2022-03-21T13:45:20.703 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-21T13:45:20.705 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906 --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} --verbose --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906/orig.config.yaml 2022-03-21T13:45:20.764 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-21T13:45:20.832 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906 branch: wip-vshankar-testing-20220319-163539 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} email: vshankar@redhat.com first_in_suite: false job_id: '6749906' kernel: client: branch: testing flavor: default kdb: true sha1: 8c5f4f463f1899102a5e979f88c37daab5f66d1e last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-vshankar-testing-20220319-163539 ceph: cephfs: session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.dir_split install: ceph: flavor: default sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 kclient: mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20220319-163539 sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 owner: scheduled_vshankar@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20220319-163539 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDB6jwNn81O2gOja8NsikJbRtrj6iuRQtsKttwcIa4ofietoHkFGsx8TpeeVXCPGLpHMzR9gzuo24laVFvnskCAzPLQwW/YlQveIJkSn3IoVjoZUK2WLNK88XmoULW+82Hn/KoT9iQ8/o+uIY2ttlAlEmDFyKCQFk2Ec2dzCcttXehFJQmc6a7sc4dQzb/KS68w2bTAUdBZ0pIGdGeWqe0oC7u7ELjurgjjUPp+3+GmtWfj+IhzymZQay4WmEzprJohadOKWRwxjhNLVZuxchk65gtoe+Gyk2LMNayvRx72WINeeOgbE+DY+eRjT4pULhqBfNELN6hCXX8W6EhYcBCWZMNpA4uybDyg/9tpFnPmhkz4mYRI+E5NQFzWtqBTIVCsunRk4GGr4VrgTQM5sB46L2eBz5+1bSannolxdCDV6XWyUN1Ije+1iT1rlx2sq8/jsH4cSd6V149y84o4naBnkax9IcmGwDcnRZUZYaCnCovQPslkSRgxAYrIKxpurZ8= smithi016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDffuWiFUYqMsfIb1z+QiJrQiP7nuJHEjBmgcEaxk6MR6R27VhMgKliWhvOKZNaqIctDnC6GNh2D6v+WkmVKtcKuONVz+yNDXcBr/kwyFMk/k+Fuk5u7P3TmIk43ZPxp/xFuL01OEV1PP7EYqd+xg/GRusXmTIQ9i3J6G9AIBZuMDhKRUIHlMG88lAH833D874vTHQShfjWJxj+c8nraMpa9c6hE/MgveMOYIvkHJotqayBTc1dMy6ratNHRgEHWyiZL2zR6wPYrx4BmcbT2TXZjkrKce3s2doSIJhP/Ja0ZHgSo+l78G24UURrEMWw1LGSAAJwddVO4V7D9zaRK9kUCn5dVu9OYpLjjubWhNk4XE+KoMdZDogykYo1uuxiejx1eKE4dOehsE4R8rUu1K3osVvIdP96B+panaYPeOKu+2PX3FChhjcuo+PDHLMdFeCwW3WiXngNsDeV59xlHogncFgf1ztqU1z3irborfgXmtBk8Izu31R5IqqNLl63Q2U= smithi079.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/92WItg5Rt7gv5pwPT4SDZkb/c8zeRfSdkwmC0S9Wa0my2mJ50kFL4Mc8luylNFr1P4geBOW40PdlXQQTNuNV6K8mxP667H5v21gtfuuKTj2D9jVBkbARL4+rylkWQB+eq8cZGOkXcheCg5rbPod8CZu7mpV1Ez8e5DQ1NPNkJ77xcVE8pKBDJdzoz9X7pSk0TttBPexysh1O7XbbSiOx9NXGKCUkU57aWcTeUZtL9LVZVNahtDfLEhIdty2IjF1igpuhWvtNfXZiFFaUa/+d3NKTYkcfoBFSi5vGA4eDMmFpDStYdgLuM8ZxUA89yX8MkiIW6ZaE6PgH6+64xB8Mi4KsXGC4aN+Jkh6SIflSoQq24gJm7Ou8esybbT2zXo9Crku/CylRwYEEgN9UpvDg+ZTmudgagV0LI3vf8gQ3VA7g1eNKurahlJEgPg0y1AHvGZzZ/SI91tuV6aY+7AbRKWOKM811yuPqdlDCdYoGG4+LFKD2tL17s2vV7T04lUk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - workunit: clients: all: - fs/misc teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-21_06:07:09 tube: smithi use_shaman: true user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-21T13:45:20.833 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa; will attempt to use it 2022-03-21T13:45:20.834 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks 2022-03-21T13:45:20.834 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-21T13:45:20.836 INFO:teuthology.task.internal:Checking packages... 2022-03-21T13:45:20.856 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'd3799b8cfa5046465710eadeea2bed7bb4681409' 2022-03-21T13:45:20.856 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-21T13:45:20.857 INFO:teuthology.packaging:ref: None 2022-03-21T13:45:20.857 INFO:teuthology.packaging:tag: None 2022-03-21T13:45:20.858 INFO:teuthology.packaging:branch: wip-vshankar-testing-20220319-163539 2022-03-21T13:45:20.858 INFO:teuthology.packaging:sha1: d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:45:20.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20220319-163539 2022-03-21T13:45:21.009 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11208.gd3799b8c 2022-03-21T13:45:21.010 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-21T13:45:21.021 INFO:teuthology.task.internal:no buildpackages task found 2022-03-21T13:45:21.021 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-21T13:45:21.052 INFO:teuthology.task.internal:Saving configuration 2022-03-21T13:45:21.071 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-21T13:45:21.117 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-21T13:45:21.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 13:40:23.333763', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYe5ixXX2OGZIe5dGqM2KfohbI1BWB6zW6Z6l4xsXXj9TDhNWmooDMiFN4E/igEzKPjb2sNsyVaq7kN/7omlnw='} 2022-03-21T13:45:21.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 13:40:23.332427', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGI2S5yfLmsLr8ROg7M6Pfp1fx6NcNIlMx3N5BQpIuZaEOh6+73JwwH+bD+xDs4cAfVg7DvocF2q/jaM2T/e/k='} 2022-03-21T13:45:21.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 13:40:23.334793', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKeghyXIlRB6WJF9QIg8opvWcDr93mxwVar/9Gc7XHEFeWX2dPoC1n04l69wcVQJJiTY1EKdn06ncPv+uCBcipM='} 2022-03-21T13:45:21.180 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-21T13:45:21.192 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-03-21T13:45:21.192 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-03-21T13:45:21.192 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-03-21T13:45:21.193 INFO:teuthology.run_tasks:Running task console_log... 2022-03-21T13:45:21.313 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29a2427ea0>, signals=[15]) 2022-03-21T13:45:21.315 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-21T13:45:21.326 INFO:teuthology.task.internal:Opening connections... 2022-03-21T13:45:21.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2022-03-21T13:45:21.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:45:21.400 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2022-03-21T13:45:21.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:45:21.473 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2022-03-21T13:45:21.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:45:21.541 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-21T13:45:21.552 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2022-03-21T13:45:21.570 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2022-03-21T13:45:21.570 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2022-03-21T13:45:21.626 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Red Hat Enterprise Linux" 2022-03-21T13:45:21.626 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8.4 (Ootpa)" 2022-03-21T13:45:21.626 INFO:teuthology.orchestra.run.smithi002.stdout:ID="rhel" 2022-03-21T13:45:21.627 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="fedora" 2022-03-21T13:45:21.627 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8.4" 2022-03-21T13:45:21.627 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2022-03-21T13:45:21.627 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-21T13:45:21.628 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2022-03-21T13:45:21.628 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-21T13:45:21.628 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.redhat.com/" 2022-03-21T13:45:21.628 INFO:teuthology.orchestra.run.smithi002.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-21T13:45:21.628 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-21T13:45:21.629 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:45:21.629 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-21T13:45:21.629 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-21T13:45:21.629 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-21T13:45:21.629 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-21T13:45:21.630 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2022-03-21T13:45:21.648 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2022-03-21T13:45:21.665 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2022-03-21T13:45:21.665 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2022-03-21T13:45:21.721 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Red Hat Enterprise Linux" 2022-03-21T13:45:21.721 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8.4 (Ootpa)" 2022-03-21T13:45:21.721 INFO:teuthology.orchestra.run.smithi016.stdout:ID="rhel" 2022-03-21T13:45:21.722 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="fedora" 2022-03-21T13:45:21.722 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8.4" 2022-03-21T13:45:21.722 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2022-03-21T13:45:21.722 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-21T13:45:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2022-03-21T13:45:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-21T13:45:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.redhat.com/" 2022-03-21T13:45:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-21T13:45:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-21T13:45:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:45:21.724 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-21T13:45:21.724 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-21T13:45:21.724 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-21T13:45:21.725 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-21T13:45:21.725 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2022-03-21T13:45:21.743 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2022-03-21T13:45:21.759 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2022-03-21T13:45:21.760 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2022-03-21T13:45:21.815 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="Red Hat Enterprise Linux" 2022-03-21T13:45:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="8.4 (Ootpa)" 2022-03-21T13:45:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:ID="rhel" 2022-03-21T13:45:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="fedora" 2022-03-21T13:45:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="8.4" 2022-03-21T13:45:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el8" 2022-03-21T13:45:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-21T13:45:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2022-03-21T13:45:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-21T13:45:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.redhat.com/" 2022-03-21T13:45:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-21T13:45:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-21T13:45:21.818 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:45:21.819 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-21T13:45:21.819 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-21T13:45:21.819 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-21T13:45:21.819 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-21T13:45:21.820 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2022-03-21T13:45:21.837 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-21T13:45:21.850 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-21T13:45:21.872 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-21T13:45:21.873 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2022-03-21T13:45:21.876 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2022-03-21T13:45:21.878 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2022-03-21T13:45:21.894 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-21T13:45:21.906 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-21T13:45:21.907 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2022-03-21T13:45:21.937 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2022-03-21T13:45:21.939 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2022-03-21T13:45:21.963 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-21T13:45:22.034 INFO:teuthology.run_tasks:Running task kernel... 2022-03-21T13:45:22.054 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '8c5f4f463f1899102a5e979f88c37daab5f66d1e'} 2022-03-21T13:45:22.055 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-03-21T13:45:22.055 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-03-21T13:45:22.056 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T13:45:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:45:22.073 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-03-21T13:45:22.182 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-03-21T13:45:22.184 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:45:22.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-03-21T13:45:22.392 DEBUG:teuthology.task.kernel:sha1 for client.0 is e35497518a9dfff614fd9941052fc942017c3589 2022-03-21T13:45:22.392 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "e35497518a9dfff614fd9941052fc942017c3589"... 2022-03-21T13:45:22.393 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2022-03-21T13:45:22.410 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-348.20.1.el8_5.x86_64 2022-03-21T13:45:22.411 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.20.1.el8_5.x86_64 vs e35497518a9dfff614fd9941052fc942017c3589 2022-03-21T13:45:22.411 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-03-21T13:45:22.412 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2022-03-21T13:45:23.795 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-03-21T13:45:23.795 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2022-03-21T13:45:23.902 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:45:23.903 INFO:teuthology.orchestra.run.smithi002.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2022-03-21T13:45:23.903 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:45:25.684 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 9.7 kB/s | 3.5 kB 00:00 2022-03-21T13:45:27.830 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 11 MB 00:02 2022-03-21T13:45:30.278 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 1.2 MB/s | 24 kB 00:00 2022-03-21T13:45:31.086 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-03-21T13:45:31.087 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2022-03-21T13:45:31.087 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-03-21T13:45:31.226 INFO:teuthology.task.kernel:Downloading kernel e35497518a9dfff614fd9941052fc942017c3589 on client.0... 2022-03-21T13:45:31.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-03-21T13:45:31.266 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-03-21T13:45:31.267 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:45:31.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e35497518a9dfff614fd9941052fc942017c3589 2022-03-21T13:45:31.420 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/e35497518a9dfff614fd9941052fc942017c3589/centos/8/flavors/default/x86_64/ 2022-03-21T13:45:31.420 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.17.0_rc6_ceph_ge35497518a9d-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/e35497518a9dfff614fd9941052fc942017c3589/centos/8/flavors/default/x86_64/ --input-file=- 2022-03-21T13:45:31.423 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2022-03-21T13:45:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21 13:45:48 URL:https://2.chacra.ceph.com/r/kernel/testing/e35497518a9dfff614fd9941052fc942017c3589/centos/8/flavors/default/x86_64/kernel-5.17.0_rc6_ceph_ge35497518a9d-1.x86_64.rpm [509935468/509935468] -> "/tmp/kernel.x86_64.rpm" [1] 2022-03-21T13:45:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2022-03-21 13:45:48-- 2022-03-21T13:45:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 17s 2022-03-21T13:45:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 486M in 17s (28.9 MB/s) 2022-03-21T13:45:48.541 INFO:teuthology.task.kernel:Installing kernel e35497518a9dfff614fd9941052fc942017c3589 on client.0... 2022-03-21T13:45:48.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-03-21T13:45:50.826 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2022-03-21T13:45:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2022-03-21T13:45:51.084 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2022-03-21T13:46:31.678 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.17.0_rc6_ceph_ge35497518a9d-1######################################## 2022-03-21T13:46:55.430 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-03-21T13:46:55.431 DEBUG:teuthology.orchestra.run.smithi002:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-03-21T13:46:55.490 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/System.map-5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.490 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/config-5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.490 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinux-5.17.0-rc6-ceph-ge35497518a9d.bz2 2022-03-21T13:46:55.491 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinuz-5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel 2022-03-21T13:46:55.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch 2022-03-21T13:46:55.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86 2022-03-21T13:46:55.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto 2022-03-21T13:46:55.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-03-21T13:46:55.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-03-21T13:46:55.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-03-21T13:46:55.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-03-21T13:46:55.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-03-21T13:46:55.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-03-21T13:46:55.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-03-21T13:46:55.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/crc32c-intel.ko 2022-03-21T13:46:55.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-03-21T13:46:55.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-03-21T13:46:55.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-03-21T13:46:55.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-03-21T13:46:55.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/serpent-avx2.ko 2022-03-21T13:46:55.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-03-21T13:46:55.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-03-21T13:46:55.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-03-21T13:46:55.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-03-21T13:46:55.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-03-21T13:46:55.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-03-21T13:46:55.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-03-21T13:46:55.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kernel 2022-03-21T13:46:55.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kernel/cpu 2022-03-21T13:46:55.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kernel/cpu/mce 2022-03-21T13:46:55.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-03-21T13:46:55.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kvm 2022-03-21T13:46:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kvm/kvm-amd.ko 2022-03-21T13:46:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kvm/kvm-intel.ko 2022-03-21T13:46:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/arch/x86/kvm/kvm.ko 2022-03-21T13:46:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto 2022-03-21T13:46:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/ansi_cprng.ko 2022-03-21T13:46:55.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/anubis.ko 2022-03-21T13:46:55.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/arc4.ko 2022-03-21T13:46:55.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx 2022-03-21T13:46:55.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/async_memcpy.ko 2022-03-21T13:46:55.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/async_pq.ko 2022-03-21T13:46:55.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/async_raid6_recov.ko 2022-03-21T13:46:55.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/async_tx.ko 2022-03-21T13:46:55.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/async_xor.ko 2022-03-21T13:46:55.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/async_tx/raid6test.ko 2022-03-21T13:46:55.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/authenc.ko 2022-03-21T13:46:55.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/authencesn.ko 2022-03-21T13:46:55.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/blake2b_generic.ko 2022-03-21T13:46:55.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/blowfish_common.ko 2022-03-21T13:46:55.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/blowfish_generic.ko 2022-03-21T13:46:55.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/camellia_generic.ko 2022-03-21T13:46:55.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/cast5_generic.ko 2022-03-21T13:46:55.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/cast6_generic.ko 2022-03-21T13:46:55.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/cast_common.ko 2022-03-21T13:46:55.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/ccm.ko 2022-03-21T13:46:55.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/cmac.ko 2022-03-21T13:46:55.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/crc32_generic.ko 2022-03-21T13:46:55.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/crypto_engine.ko 2022-03-21T13:46:55.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/crypto_user.ko 2022-03-21T13:46:55.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/cts.ko 2022-03-21T13:46:55.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/des_generic.ko 2022-03-21T13:46:55.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/dh_generic.ko 2022-03-21T13:46:55.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/echainiv.ko 2022-03-21T13:46:55.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/essiv.ko 2022-03-21T13:46:55.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/fcrypt.ko 2022-03-21T13:46:55.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/khazad.ko 2022-03-21T13:46:55.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/lz4.ko 2022-03-21T13:46:55.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/lz4hc.ko 2022-03-21T13:46:55.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/md4.ko 2022-03-21T13:46:55.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/michael_mic.ko 2022-03-21T13:46:55.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/pcbc.ko 2022-03-21T13:46:55.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/pcrypt.ko 2022-03-21T13:46:55.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/rmd160.ko 2022-03-21T13:46:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/seed.ko 2022-03-21T13:46:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/serpent_generic.ko 2022-03-21T13:46:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/tcrypt.ko 2022-03-21T13:46:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/tea.ko 2022-03-21T13:46:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/twofish_common.ko 2022-03-21T13:46:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/twofish_generic.ko 2022-03-21T13:46:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/vmac.ko 2022-03-21T13:46:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/wp512.ko 2022-03-21T13:46:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/xcbc.ko 2022-03-21T13:46:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/xor.ko 2022-03-21T13:46:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/crypto/xxhash_generic.ko 2022-03-21T13:46:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers 2022-03-21T13:46:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi 2022-03-21T13:46:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/acpi_ipmi.ko 2022-03-21T13:46:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/acpi_pad.ko 2022-03-21T13:46:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/custom_method.ko 2022-03-21T13:46:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/ec_sys.ko 2022-03-21T13:46:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/platform_profile.ko 2022-03-21T13:46:55.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/sbs.ko 2022-03-21T13:46:55.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/sbshc.ko 2022-03-21T13:46:55.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/acpi/video.ko 2022-03-21T13:46:55.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata 2022-03-21T13:46:55.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/acard-ahci.ko 2022-03-21T13:46:55.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/ahci_platform.ko 2022-03-21T13:46:55.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/ata_generic.ko 2022-03-21T13:46:55.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/libahci_platform.ko 2022-03-21T13:46:55.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_acpi.ko 2022-03-21T13:46:55.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_ali.ko 2022-03-21T13:46:55.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_amd.ko 2022-03-21T13:46:55.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_artop.ko 2022-03-21T13:46:55.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_atiixp.ko 2022-03-21T13:46:55.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_atp867x.ko 2022-03-21T13:46:55.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_cmd640.ko 2022-03-21T13:46:55.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_cmd64x.ko 2022-03-21T13:46:55.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_cypress.ko 2022-03-21T13:46:55.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_efar.ko 2022-03-21T13:46:55.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_hpt366.ko 2022-03-21T13:46:55.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_hpt37x.ko 2022-03-21T13:46:55.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_hpt3x2n.ko 2022-03-21T13:46:55.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_hpt3x3.ko 2022-03-21T13:46:55.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_it8213.ko 2022-03-21T13:46:55.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_it821x.ko 2022-03-21T13:46:55.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_jmicron.ko 2022-03-21T13:46:55.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_marvell.ko 2022-03-21T13:46:55.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_mpiix.ko 2022-03-21T13:46:55.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_netcell.ko 2022-03-21T13:46:55.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_ninja32.ko 2022-03-21T13:46:55.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_ns87410.ko 2022-03-21T13:46:55.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_ns87415.ko 2022-03-21T13:46:55.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_oldpiix.ko 2022-03-21T13:46:55.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_opti.ko 2022-03-21T13:46:55.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_optidma.ko 2022-03-21T13:46:55.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_pcmcia.ko 2022-03-21T13:46:55.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_pdc2027x.ko 2022-03-21T13:46:55.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-03-21T13:46:55.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_piccolo.ko 2022-03-21T13:46:55.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_rdc.ko 2022-03-21T13:46:55.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_sch.ko 2022-03-21T13:46:55.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_serverworks.ko 2022-03-21T13:46:55.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_sil680.ko 2022-03-21T13:46:55.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_sis.ko 2022-03-21T13:46:55.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_sl82c105.ko 2022-03-21T13:46:55.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_triflex.ko 2022-03-21T13:46:55.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pata_via.ko 2022-03-21T13:46:55.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/pdc_adma.ko 2022-03-21T13:46:55.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_inic162x.ko 2022-03-21T13:46:55.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_mv.ko 2022-03-21T13:46:55.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_nv.ko 2022-03-21T13:46:55.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_promise.ko 2022-03-21T13:46:55.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_qstor.ko 2022-03-21T13:46:55.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_sil.ko 2022-03-21T13:46:55.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_sil24.ko 2022-03-21T13:46:55.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_sis.ko 2022-03-21T13:46:55.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_svw.ko 2022-03-21T13:46:55.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_sx4.ko 2022-03-21T13:46:55.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_uli.ko 2022-03-21T13:46:55.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_via.ko 2022-03-21T13:46:55.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ata/sata_vsc.ko 2022-03-21T13:46:55.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/auxdisplay 2022-03-21T13:46:55.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/auxdisplay/ks0108.ko 2022-03-21T13:46:55.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/base 2022-03-21T13:46:55.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/base/regmap 2022-03-21T13:46:55.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/base/regmap/regmap-i2c.ko 2022-03-21T13:46:55.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/bcma 2022-03-21T13:46:55.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/bcma/bcma.ko 2022-03-21T13:46:55.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block 2022-03-21T13:46:55.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/aoe 2022-03-21T13:46:55.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/aoe/aoe.ko 2022-03-21T13:46:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/brd.ko 2022-03-21T13:46:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/drbd 2022-03-21T13:46:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/drbd/drbd.ko 2022-03-21T13:46:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/floppy.ko 2022-03-21T13:46:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/loop.ko 2022-03-21T13:46:55.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/mtip32xx 2022-03-21T13:46:55.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-03-21T13:46:55.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/nbd.ko 2022-03-21T13:46:55.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/null_blk 2022-03-21T13:46:55.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/null_blk/null_blk.ko 2022-03-21T13:46:55.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/pktcdvd.ko 2022-03-21T13:46:55.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/rbd.ko 2022-03-21T13:46:55.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/sx8.ko 2022-03-21T13:46:55.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/virtio_blk.ko 2022-03-21T13:46:55.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/xen-blkback 2022-03-21T13:46:55.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-03-21T13:46:55.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/xen-blkfront.ko 2022-03-21T13:46:55.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/zram 2022-03-21T13:46:55.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/block/zram/zram.ko 2022-03-21T13:46:55.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char 2022-03-21T13:46:55.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hangcheck-timer.ko 2022-03-21T13:46:55.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random 2022-03-21T13:46:55.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random/amd-rng.ko 2022-03-21T13:46:55.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random/intel-rng.ko 2022-03-21T13:46:55.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-03-21T13:46:55.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random/via-rng.ko 2022-03-21T13:46:55.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/hw_random/virtio-rng.ko 2022-03-21T13:46:55.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi 2022-03-21T13:46:55.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-03-21T13:46:55.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-03-21T13:46:55.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-03-21T13:46:55.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_si.ko 2022-03-21T13:46:55.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-03-21T13:46:55.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-03-21T13:46:55.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/lp.ko 2022-03-21T13:46:55.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/mwave 2022-03-21T13:46:55.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/mwave/mwave.ko 2022-03-21T13:46:55.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/pcmcia 2022-03-21T13:46:55.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-03-21T13:46:55.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-03-21T13:46:55.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/ppdev.ko 2022-03-21T13:46:55.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tlclk.ko 2022-03-21T13:46:55.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm 2022-03-21T13:46:55.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm.ko 2022-03-21T13:46:55.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm_atmel.ko 2022-03-21T13:46:55.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm_infineon.ko 2022-03-21T13:46:55.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm_nsc.ko 2022-03-21T13:46:55.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm_tis.ko 2022-03-21T13:46:55.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-03-21T13:46:55.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/uv_mmtimer.ko 2022-03-21T13:46:55.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/char/virtio_console.ko 2022-03-21T13:46:55.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq 2022-03-21T13:46:55.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-03-21T13:46:55.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-03-21T13:46:55.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/p4-clockmod.ko 2022-03-21T13:46:55.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-03-21T13:46:55.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/powernow-k8.ko 2022-03-21T13:46:55.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/cpufreq/speedstep-lib.ko 2022-03-21T13:46:55.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto 2022-03-21T13:46:55.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/ccp 2022-03-21T13:46:55.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-03-21T13:46:55.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/ccp/ccp.ko 2022-03-21T13:46:55.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/padlock-aes.ko 2022-03-21T13:46:55.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/padlock-sha.ko 2022-03-21T13:46:55.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/qat 2022-03-21T13:46:55.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/qat/qat_common 2022-03-21T13:46:55.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-03-21T13:46:55.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/qat/qat_dh895xcc 2022-03-21T13:46:55.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-03-21T13:46:55.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/virtio 2022-03-21T13:46:55.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-03-21T13:46:55.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dca 2022-03-21T13:46:55.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dca/dca.ko 2022-03-21T13:46:55.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/devfreq 2022-03-21T13:46:55.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-03-21T13:46:55.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dma 2022-03-21T13:46:55.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dma/dw 2022-03-21T13:46:55.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dma/dw/dw_dmac.ko 2022-03-21T13:46:55.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dma/ioat 2022-03-21T13:46:55.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/dma/ioat/ioatdma.ko 2022-03-21T13:46:55.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac 2022-03-21T13:46:55.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/amd64_edac.ko 2022-03-21T13:46:55.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/e752x_edac.ko 2022-03-21T13:46:55.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/edac_mce_amd.ko 2022-03-21T13:46:55.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i3000_edac.ko 2022-03-21T13:46:55.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i3200_edac.ko 2022-03-21T13:46:55.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i5000_edac.ko 2022-03-21T13:46:55.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i5100_edac.ko 2022-03-21T13:46:55.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i5400_edac.ko 2022-03-21T13:46:55.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i7300_edac.ko 2022-03-21T13:46:55.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i7core_edac.ko 2022-03-21T13:46:55.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/i82975x_edac.ko 2022-03-21T13:46:55.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/ie31200_edac.ko 2022-03-21T13:46:55.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/sb_edac.ko 2022-03-21T13:46:55.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/edac/x38_edac.ko 2022-03-21T13:46:55.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/firmware 2022-03-21T13:46:55.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/firmware/edd.ko 2022-03-21T13:46:55.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/firmware/iscsi_ibft.ko 2022-03-21T13:46:55.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/gpio 2022-03-21T13:46:55.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/gpio/gpio-ich.ko 2022-03-21T13:46:55.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/gpio/gpio-viperboard.ko 2022-03-21T13:46:55.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid 2022-03-21T13:46:55.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-appleir.ko 2022-03-21T13:46:55.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-aureal.ko 2022-03-21T13:46:55.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-axff.ko 2022-03-21T13:46:55.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-dr.ko 2022-03-21T13:46:55.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-elecom.ko 2022-03-21T13:46:55.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-elo.ko 2022-03-21T13:46:55.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-emsff.ko 2022-03-21T13:46:55.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-gaff.ko 2022-03-21T13:46:55.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-gt683r.ko 2022-03-21T13:46:55.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-gyration.ko 2022-03-21T13:46:55.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-holtek-kbd.ko 2022-03-21T13:46:55.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-holtek-mouse.ko 2022-03-21T13:46:55.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-holtekff.ko 2022-03-21T13:46:55.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-hyperv.ko 2022-03-21T13:46:55.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-icade.ko 2022-03-21T13:46:55.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-keytouch.ko 2022-03-21T13:46:55.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-kye.ko 2022-03-21T13:46:55.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-lcpower.ko 2022-03-21T13:46:55.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-led.ko 2022-03-21T13:46:55.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-lenovo.ko 2022-03-21T13:46:55.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-logitech-dj.ko 2022-03-21T13:46:55.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-03-21T13:46:55.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-multitouch.ko 2022-03-21T13:46:55.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-ortek.ko 2022-03-21T13:46:55.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-penmount.ko 2022-03-21T13:46:55.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-petalynx.ko 2022-03-21T13:46:55.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-picolcd.ko 2022-03-21T13:46:55.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-pl.ko 2022-03-21T13:46:55.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-plantronics.ko 2022-03-21T13:46:55.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-primax.ko 2022-03-21T13:46:55.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-rmi.ko 2022-03-21T13:46:55.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-arvo.ko 2022-03-21T13:46:55.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-common.ko 2022-03-21T13:46:55.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-isku.ko 2022-03-21T13:46:55.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-kone.ko 2022-03-21T13:46:55.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-03-21T13:46:55.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-konepure.ko 2022-03-21T13:46:55.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-03-21T13:46:55.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-lua.ko 2022-03-21T13:46:55.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-pyra.ko 2022-03-21T13:46:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-ryos.ko 2022-03-21T13:46:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat-savu.ko 2022-03-21T13:46:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-roccat.ko 2022-03-21T13:46:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-saitek.ko 2022-03-21T13:46:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-samsung.ko 2022-03-21T13:46:55.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-sensor-hub.ko 2022-03-21T13:46:55.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-sjoy.ko 2022-03-21T13:46:55.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-sony.ko 2022-03-21T13:46:55.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-speedlink.ko 2022-03-21T13:46:55.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-steelseries.ko 2022-03-21T13:46:55.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-sunplus.ko 2022-03-21T13:46:55.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-thrustmaster.ko 2022-03-21T13:46:55.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-tivo.ko 2022-03-21T13:46:55.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-tmff.ko 2022-03-21T13:46:55.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-topseed.ko 2022-03-21T13:46:55.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-twinhan.ko 2022-03-21T13:46:55.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-uclogic.ko 2022-03-21T13:46:55.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-waltop.ko 2022-03-21T13:46:55.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-wiimote.ko 2022-03-21T13:46:55.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-xinmo.ko 2022-03-21T13:46:55.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-zpff.ko 2022-03-21T13:46:55.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/hid-zydacron.ko 2022-03-21T13:46:55.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/uhid.ko 2022-03-21T13:46:55.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hid/wacom.ko 2022-03-21T13:46:55.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hv 2022-03-21T13:46:55.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hv/hv_balloon.ko 2022-03-21T13:46:55.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hv/hv_utils.ko 2022-03-21T13:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hv/hv_vmbus.ko 2022-03-21T13:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon 2022-03-21T13:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/abituguru.ko 2022-03-21T13:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/abituguru3.ko 2022-03-21T13:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/acpi_power_meter.ko 2022-03-21T13:46:55.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ad7414.ko 2022-03-21T13:46:55.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ad7418.ko 2022-03-21T13:46:55.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adc128d818.ko 2022-03-21T13:46:55.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm1021.ko 2022-03-21T13:46:55.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm1025.ko 2022-03-21T13:46:55.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm1026.ko 2022-03-21T13:46:55.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm1029.ko 2022-03-21T13:46:55.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm1031.ko 2022-03-21T13:46:55.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adm9240.ko 2022-03-21T13:46:55.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ads7828.ko 2022-03-21T13:46:55.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7410.ko 2022-03-21T13:46:55.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7411.ko 2022-03-21T13:46:55.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7462.ko 2022-03-21T13:46:55.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7470.ko 2022-03-21T13:46:55.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7475.ko 2022-03-21T13:46:55.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/adt7x10.ko 2022-03-21T13:46:55.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/amc6821.ko 2022-03-21T13:46:55.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/applesmc.ko 2022-03-21T13:46:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/asb100.ko 2022-03-21T13:46:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/asc7621.ko 2022-03-21T13:46:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/asus_atk0110.ko 2022-03-21T13:46:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/atxp1.ko 2022-03-21T13:46:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/coretemp.ko 2022-03-21T13:46:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-03-21T13:46:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/dme1737.ko 2022-03-21T13:46:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ds1621.ko 2022-03-21T13:46:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ds620.ko 2022-03-21T13:46:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/emc1403.ko 2022-03-21T13:46:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/emc6w201.ko 2022-03-21T13:46:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/f71805f.ko 2022-03-21T13:46:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/f71882fg.ko 2022-03-21T13:46:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/f75375s.ko 2022-03-21T13:46:55.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/fam15h_power.ko 2022-03-21T13:46:55.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/fschmd.ko 2022-03-21T13:46:55.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/g760a.ko 2022-03-21T13:46:55.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/g762.ko 2022-03-21T13:46:55.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/gl518sm.ko 2022-03-21T13:46:55.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/gl520sm.ko 2022-03-21T13:46:55.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/hwmon-vid.ko 2022-03-21T13:46:55.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/i5500_temp.ko 2022-03-21T13:46:55.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/i5k_amb.ko 2022-03-21T13:46:55.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ibmaem.ko 2022-03-21T13:46:55.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ibmpex.ko 2022-03-21T13:46:55.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ina209.ko 2022-03-21T13:46:55.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ina2xx.ko 2022-03-21T13:46:55.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/it87.ko 2022-03-21T13:46:55.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/k10temp.ko 2022-03-21T13:46:55.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/k8temp.ko 2022-03-21T13:46:55.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lineage-pem.ko 2022-03-21T13:46:55.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm63.ko 2022-03-21T13:46:55.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm73.ko 2022-03-21T13:46:55.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm75.ko 2022-03-21T13:46:55.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm77.ko 2022-03-21T13:46:55.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm78.ko 2022-03-21T13:46:55.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm80.ko 2022-03-21T13:46:55.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm83.ko 2022-03-21T13:46:55.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm85.ko 2022-03-21T13:46:55.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm87.ko 2022-03-21T13:46:55.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm90.ko 2022-03-21T13:46:55.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm92.ko 2022-03-21T13:46:55.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm93.ko 2022-03-21T13:46:55.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm95234.ko 2022-03-21T13:46:55.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm95241.ko 2022-03-21T13:46:55.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/lm95245.ko 2022-03-21T13:46:55.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc2945.ko 2022-03-21T13:46:55.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4151.ko 2022-03-21T13:46:55.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4215.ko 2022-03-21T13:46:55.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4222.ko 2022-03-21T13:46:55.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4245.ko 2022-03-21T13:46:55.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4260.ko 2022-03-21T13:46:55.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/ltc4261.ko 2022-03-21T13:46:55.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max16065.ko 2022-03-21T13:46:55.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max1619.ko 2022-03-21T13:46:55.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max1668.ko 2022-03-21T13:46:55.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max197.ko 2022-03-21T13:46:55.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max6639.ko 2022-03-21T13:46:55.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max6642.ko 2022-03-21T13:46:55.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max6650.ko 2022-03-21T13:46:55.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/max6697.ko 2022-03-21T13:46:55.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/mcp3021.ko 2022-03-21T13:46:55.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/nct6683.ko 2022-03-21T13:46:55.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/nct6775.ko 2022-03-21T13:46:55.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/nct7802.ko 2022-03-21T13:46:55.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pc87360.ko 2022-03-21T13:46:55.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pc87427.ko 2022-03-21T13:46:55.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pcf8591.ko 2022-03-21T13:46:55.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus 2022-03-21T13:46:55.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-03-21T13:46:55.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-03-21T13:46:55.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-03-21T13:46:55.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/max16064.ko 2022-03-21T13:46:55.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/max34440.ko 2022-03-21T13:46:55.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/max8688.ko 2022-03-21T13:46:55.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-03-21T13:46:55.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-03-21T13:46:55.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-03-21T13:46:55.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-03-21T13:46:55.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-03-21T13:46:55.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-03-21T13:46:55.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/powr1220.ko 2022-03-21T13:46:55.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sch5627.ko 2022-03-21T13:46:55.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sch5636.ko 2022-03-21T13:46:55.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sch56xx-common.ko 2022-03-21T13:46:55.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sht15.ko 2022-03-21T13:46:55.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sht21.ko 2022-03-21T13:46:55.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/shtc1.ko 2022-03-21T13:46:55.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/sis5595.ko 2022-03-21T13:46:55.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/smsc47b397.ko 2022-03-21T13:46:55.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/smsc47m1.ko 2022-03-21T13:46:55.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/smsc47m192.ko 2022-03-21T13:46:55.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/thmc50.ko 2022-03-21T13:46:55.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/tmp102.ko 2022-03-21T13:46:55.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/tmp103.ko 2022-03-21T13:46:55.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/tmp401.ko 2022-03-21T13:46:55.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/tmp421.ko 2022-03-21T13:46:55.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/via-cputemp.ko 2022-03-21T13:46:55.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/via686a.ko 2022-03-21T13:46:55.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/vt1211.ko 2022-03-21T13:46:55.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/vt8231.ko 2022-03-21T13:46:55.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83627ehf.ko 2022-03-21T13:46:55.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83627hf.ko 2022-03-21T13:46:55.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83781d.ko 2022-03-21T13:46:55.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83791d.ko 2022-03-21T13:46:55.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83792d.ko 2022-03-21T13:46:55.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83793.ko 2022-03-21T13:46:55.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83795.ko 2022-03-21T13:46:55.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83l785ts.ko 2022-03-21T13:46:55.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/hwmon/w83l786ng.ko 2022-03-21T13:46:55.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c 2022-03-21T13:46:55.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/algos 2022-03-21T13:46:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-03-21T13:46:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-03-21T13:46:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses 2022-03-21T13:46:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-03-21T13:46:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-03-21T13:46:55.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-03-21T13:46:55.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-03-21T13:46:55.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-03-21T13:46:55.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-03-21T13:46:55.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-03-21T13:46:55.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-i801.ko 2022-03-21T13:46:55.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-isch.ko 2022-03-21T13:46:55.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-03-21T13:46:55.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-03-21T13:46:55.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-03-21T13:46:55.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-parport.ko 2022-03-21T13:46:55.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-03-21T13:46:55.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-03-21T13:46:55.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-03-21T13:46:55.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-03-21T13:46:55.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-03-21T13:46:55.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-03-21T13:46:55.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-via.ko 2022-03-21T13:46:55.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-03-21T13:46:55.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-03-21T13:46:55.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/i2c-dev.ko 2022-03-21T13:46:55.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/i2c-mux.ko 2022-03-21T13:46:55.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-03-21T13:46:55.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/i2c-smbus.ko 2022-03-21T13:46:55.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/i2c/i2c-stub.ko 2022-03-21T13:46:55.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband 2022-03-21T13:46:55.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core 2022-03-21T13:46:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/ib_cm.ko 2022-03-21T13:46:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/ib_core.ko 2022-03-21T13:46:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/ib_umad.ko 2022-03-21T13:46:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-03-21T13:46:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/iw_cm.ko 2022-03-21T13:46:55.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/rdma_cm.ko 2022-03-21T13:46:55.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-03-21T13:46:55.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw 2022-03-21T13:46:55.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/cxgb4 2022-03-21T13:46:55.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-03-21T13:46:55.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mlx4 2022-03-21T13:46:55.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-03-21T13:46:55.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mlx5 2022-03-21T13:46:55.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-03-21T13:46:55.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mthca 2022-03-21T13:46:55.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-03-21T13:46:55.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp 2022-03-21T13:46:55.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/ipoib 2022-03-21T13:46:55.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-03-21T13:46:55.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/iser 2022-03-21T13:46:55.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-03-21T13:46:55.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/isert 2022-03-21T13:46:55.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-03-21T13:46:55.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/srp 2022-03-21T13:46:55.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-03-21T13:46:55.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/srpt 2022-03-21T13:46:55.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-03-21T13:46:55.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input 2022-03-21T13:46:55.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport 2022-03-21T13:46:55.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-03-21T13:46:55.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport/fm801-gp.ko 2022-03-21T13:46:55.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport/gameport.ko 2022-03-21T13:46:55.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport/lightning.ko 2022-03-21T13:46:55.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/gameport/ns558.ko 2022-03-21T13:46:55.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/joydev.ko 2022-03-21T13:46:55.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/keyboard 2022-03-21T13:46:55.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/keyboard/gpio_keys.ko 2022-03-21T13:46:55.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse 2022-03-21T13:46:55.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/appletouch.ko 2022-03-21T13:46:55.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/bcm5974.ko 2022-03-21T13:46:55.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/cyapatp.ko 2022-03-21T13:46:55.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/elan_i2c.ko 2022-03-21T13:46:55.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/sermouse.ko 2022-03-21T13:46:55.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-03-21T13:46:55.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/synaptics_usb.ko 2022-03-21T13:46:55.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/mouse/vsxxxaa.ko 2022-03-21T13:46:55.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/rmi4 2022-03-21T13:46:55.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/rmi4/rmi_core.ko 2022-03-21T13:46:55.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/serio 2022-03-21T13:46:55.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/serio/altera_ps2.ko 2022-03-21T13:46:55.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/serio/arc_ps2.ko 2022-03-21T13:46:55.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-03-21T13:46:55.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/serio/serio_raw.ko 2022-03-21T13:46:55.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/input/sparse-keymap.ko 2022-03-21T13:46:55.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/iommu 2022-03-21T13:46:55.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/iommu/amd 2022-03-21T13:46:55.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/iommu/amd/iommu_v2.ko 2022-03-21T13:46:55.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds 2022-03-21T13:46:55.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/leds-blinkm.ko 2022-03-21T13:46:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/leds-clevo-mail.ko 2022-03-21T13:46:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/leds-lm3530.ko 2022-03-21T13:46:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/leds-lp3944.ko 2022-03-21T13:46:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/leds-ss4200.ko 2022-03-21T13:46:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger 2022-03-21T13:46:55.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-03-21T13:46:55.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-03-21T13:46:55.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-03-21T13:46:55.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-03-21T13:46:55.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-03-21T13:46:55.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-03-21T13:46:55.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-03-21T13:46:55.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-03-21T13:46:55.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-03-21T13:46:55.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md 2022-03-21T13:46:55.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/bcache 2022-03-21T13:46:55.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/bcache/bcache.ko 2022-03-21T13:46:55.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-bio-prison.ko 2022-03-21T13:46:55.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-cache-smq.ko 2022-03-21T13:46:55.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-cache.ko 2022-03-21T13:46:55.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-crypt.ko 2022-03-21T13:46:55.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-delay.ko 2022-03-21T13:46:55.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-flakey.ko 2022-03-21T13:46:55.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-log-userspace.ko 2022-03-21T13:46:55.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-multipath.ko 2022-03-21T13:46:55.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-queue-length.ko 2022-03-21T13:46:55.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-raid.ko 2022-03-21T13:46:55.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-round-robin.ko 2022-03-21T13:46:55.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-service-time.ko 2022-03-21T13:46:55.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-switch.ko 2022-03-21T13:46:55.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-thin-pool.ko 2022-03-21T13:46:55.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/dm-verity.ko 2022-03-21T13:46:55.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/faulty.ko 2022-03-21T13:46:55.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/linear.ko 2022-03-21T13:46:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/multipath.ko 2022-03-21T13:46:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/persistent-data 2022-03-21T13:46:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-03-21T13:46:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/raid0.ko 2022-03-21T13:46:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/raid1.ko 2022-03-21T13:46:55.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/raid10.ko 2022-03-21T13:46:55.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/md/raid456.ko 2022-03-21T13:46:55.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message 2022-03-21T13:46:55.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion 2022-03-21T13:46:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptbase.ko 2022-03-21T13:46:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptctl.ko 2022-03-21T13:46:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptfc.ko 2022-03-21T13:46:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptlan.ko 2022-03-21T13:46:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptsas.ko 2022-03-21T13:46:55.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptscsih.ko 2022-03-21T13:46:55.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/message/fusion/mptspi.ko 2022-03-21T13:46:55.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd 2022-03-21T13:46:55.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/lpc_ich.ko 2022-03-21T13:46:55.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/lpc_sch.ko 2022-03-21T13:46:55.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/mfd-core.ko 2022-03-21T13:46:55.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/sm501.ko 2022-03-21T13:46:55.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/viperboard.ko 2022-03-21T13:46:55.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/vx855.ko 2022-03-21T13:46:55.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/mfd/wl1273-core.ko 2022-03-21T13:46:55.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc 2022-03-21T13:46:55.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/altera-stapl 2022-03-21T13:46:55.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-03-21T13:46:55.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/apds9802als.ko 2022-03-21T13:46:55.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/apds990x.ko 2022-03-21T13:46:55.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/bh1770glc.ko 2022-03-21T13:46:55.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/cb710 2022-03-21T13:46:55.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/cb710/cb710.ko 2022-03-21T13:46:55.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/eeprom 2022-03-21T13:46:55.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/eeprom/at24.ko 2022-03-21T13:46:55.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/eeprom/eeprom.ko 2022-03-21T13:46:55.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-03-21T13:46:55.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/eeprom/max6875.ko 2022-03-21T13:46:55.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/enclosure.ko 2022-03-21T13:46:55.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/hpilo.ko 2022-03-21T13:46:55.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/isl29003.ko 2022-03-21T13:46:55.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/isl29020.ko 2022-03-21T13:46:55.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/lis3lv02d 2022-03-21T13:46:55.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-03-21T13:46:55.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-03-21T13:46:55.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/mei 2022-03-21T13:46:55.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/mei/mei-me.ko 2022-03-21T13:46:55.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/mei/mei-txe.ko 2022-03-21T13:46:55.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/mei/mei.ko 2022-03-21T13:46:55.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-gru 2022-03-21T13:46:55.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-gru/gru.ko 2022-03-21T13:46:55.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-xp 2022-03-21T13:46:55.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-xp/xp.ko 2022-03-21T13:46:55.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-xp/xpc.ko 2022-03-21T13:46:55.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-03-21T13:46:55.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/tifm_7xx1.ko 2022-03-21T13:46:55.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/tifm_core.ko 2022-03-21T13:46:55.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/tsl2550.ko 2022-03-21T13:46:55.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/vmw_balloon.ko 2022-03-21T13:46:55.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/vmw_vmci 2022-03-21T13:46:55.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-03-21T13:46:55.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net 2022-03-21T13:46:55.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/bonding 2022-03-21T13:46:55.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/bonding/bonding.ko 2022-03-21T13:46:55.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa 2022-03-21T13:46:55.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/b53 2022-03-21T13:46:55.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/b53/b53_common.ko 2022-03-21T13:46:55.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/bcm-sf2.ko 2022-03-21T13:46:55.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/mv88e6060.ko 2022-03-21T13:46:55.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/mv88e6xxx 2022-03-21T13:46:55.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-03-21T13:46:55.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/dummy.ko 2022-03-21T13:46:55.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/eql.ko 2022-03-21T13:46:55.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet 2022-03-21T13:46:55.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/3com 2022-03-21T13:46:55.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-03-21T13:46:55.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-03-21T13:46:55.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-03-21T13:46:55.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-03-21T13:46:55.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/8390 2022-03-21T13:46:55.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/8390/8390.ko 2022-03-21T13:46:55.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-03-21T13:46:55.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-03-21T13:46:55.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-03-21T13:46:55.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/adaptec 2022-03-21T13:46:55.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-03-21T13:46:55.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/agere 2022-03-21T13:46:55.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/agere/et131x.ko 2022-03-21T13:46:55.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/alteon 2022-03-21T13:46:55.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-03-21T13:46:55.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/altera 2022-03-21T13:46:55.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-03-21T13:46:55.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/amd 2022-03-21T13:46:55.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-03-21T13:46:55.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-03-21T13:46:55.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-03-21T13:46:55.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros 2022-03-21T13:46:55.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/alx 2022-03-21T13:46:55.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-03-21T13:46:55.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atl1c 2022-03-21T13:46:55.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-03-21T13:46:55.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atl1e 2022-03-21T13:46:55.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-03-21T13:46:55.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atlx 2022-03-21T13:46:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-03-21T13:46:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-03-21T13:46:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom 2022-03-21T13:46:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-03-21T13:46:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-03-21T13:46:55.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-03-21T13:46:55.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-03-21T13:46:55.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-03-21T13:46:55.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/genet 2022-03-21T13:46:55.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-03-21T13:46:55.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-03-21T13:46:55.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/brocade 2022-03-21T13:46:55.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/brocade/bna 2022-03-21T13:46:55.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-03-21T13:46:55.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio 2022-03-21T13:46:55.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb 2022-03-21T13:46:55.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-03-21T13:46:55.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-03-21T13:46:55.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-03-21T13:46:55.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-03-21T13:46:55.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-03-21T13:46:55.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-03-21T13:46:55.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-03-21T13:46:55.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-03-21T13:46:55.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-03-21T13:46:55.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/cisco 2022-03-21T13:46:55.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/cisco/enic 2022-03-21T13:46:55.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-03-21T13:46:55.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec 2022-03-21T13:46:55.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip 2022-03-21T13:46:55.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-03-21T13:46:55.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2022-03-21T13:46:55.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-03-21T13:46:55.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-03-21T13:46:55.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-03-21T13:46:55.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-03-21T13:46:55.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-03-21T13:46:55.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dlink 2022-03-21T13:46:55.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-03-21T13:46:55.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-03-21T13:46:55.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/dnet.ko 2022-03-21T13:46:55.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/emulex 2022-03-21T13:46:55.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/emulex/benet 2022-03-21T13:46:55.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-03-21T13:46:55.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/ethoc.ko 2022-03-21T13:46:55.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/fealnx.ko 2022-03-21T13:46:55.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel 2022-03-21T13:46:55.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/e100.ko 2022-03-21T13:46:55.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/e1000 2022-03-21T13:46:55.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-03-21T13:46:55.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/e1000e 2022-03-21T13:46:55.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-03-21T13:46:55.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/fm10k 2022-03-21T13:46:55.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-03-21T13:46:55.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/i40e 2022-03-21T13:46:55.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-03-21T13:46:55.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/iavf 2022-03-21T13:46:55.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-03-21T13:46:55.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/igb 2022-03-21T13:46:55.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-03-21T13:46:55.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/igbvf 2022-03-21T13:46:55.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-03-21T13:46:55.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgb 2022-03-21T13:46:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-03-21T13:46:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgbe 2022-03-21T13:46:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-03-21T13:46:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgbevf 2022-03-21T13:46:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-03-21T13:46:55.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/jme.ko 2022-03-21T13:46:55.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/marvell 2022-03-21T13:46:55.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-03-21T13:46:55.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/marvell/skge.ko 2022-03-21T13:46:55.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-03-21T13:46:55.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox 2022-03-21T13:46:55.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx4 2022-03-21T13:46:55.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-03-21T13:46:55.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-03-21T13:46:55.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx5 2022-03-21T13:46:55.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-03-21T13:46:55.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-03-21T13:46:55.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/micrel 2022-03-21T13:46:55.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-03-21T13:46:55.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/myricom 2022-03-21T13:46:55.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/myricom/myri10ge 2022-03-21T13:46:55.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-03-21T13:46:55.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/natsemi 2022-03-21T13:46:55.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-03-21T13:46:55.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-03-21T13:46:55.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/neterion 2022-03-21T13:46:55.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-03-21T13:46:55.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/neterion/vxge 2022-03-21T13:46:55.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2022-03-21T13:46:55.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/nvidia 2022-03-21T13:46:55.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-03-21T13:46:55.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/packetengines 2022-03-21T13:46:55.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-03-21T13:46:55.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-03-21T13:46:55.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic 2022-03-21T13:46:55.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic/netxen 2022-03-21T13:46:55.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-03-21T13:46:55.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-03-21T13:46:55.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-03-21T13:46:55.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-03-21T13:46:55.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/rdc 2022-03-21T13:46:55.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-03-21T13:46:55.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/realtek 2022-03-21T13:46:55.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-03-21T13:46:55.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-03-21T13:46:55.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/realtek/atp.ko 2022-03-21T13:46:55.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-03-21T13:46:55.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/rocker 2022-03-21T13:46:55.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-03-21T13:46:55.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sfc 2022-03-21T13:46:55.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-03-21T13:46:55.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/silan 2022-03-21T13:46:55.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-03-21T13:46:55.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sis 2022-03-21T13:46:55.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sis/sis190.ko 2022-03-21T13:46:55.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sis/sis900.ko 2022-03-21T13:46:55.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/smsc 2022-03-21T13:46:55.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-03-21T13:46:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-03-21T13:46:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-03-21T13:46:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-03-21T13:46:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/stmicro 2022-03-21T13:46:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/stmicro/stmmac 2022-03-21T13:46:55.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-03-21T13:46:55.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-03-21T13:46:55.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sun 2022-03-21T13:46:55.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sun/cassini.ko 2022-03-21T13:46:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sun/niu.ko 2022-03-21T13:46:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sun/sungem.ko 2022-03-21T13:46:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-03-21T13:46:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/tehuti 2022-03-21T13:46:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-03-21T13:46:55.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/ti 2022-03-21T13:46:55.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/ti/tlan.ko 2022-03-21T13:46:55.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/via 2022-03-21T13:46:55.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-03-21T13:46:55.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-03-21T13:46:55.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/wiznet 2022-03-21T13:46:55.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-03-21T13:46:55.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-03-21T13:46:55.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/xircom 2022-03-21T13:46:55.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-03-21T13:46:55.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/geneve.ko 2022-03-21T13:46:55.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/hyperv 2022-03-21T13:46:55.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-03-21T13:46:55.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ieee802154 2022-03-21T13:46:55.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ieee802154/fakelb.ko 2022-03-21T13:46:55.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ifb.ko 2022-03-21T13:46:55.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ipvlan 2022-03-21T13:46:55.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ipvlan/ipvlan.ko 2022-03-21T13:46:55.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/macvlan.ko 2022-03-21T13:46:55.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/macvtap.ko 2022-03-21T13:46:55.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/mdio 2022-03-21T13:46:55.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/mdio.ko 2022-03-21T13:46:55.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-03-21T13:46:55.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-03-21T13:46:55.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/mii.ko 2022-03-21T13:46:55.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/net_failover.ko 2022-03-21T13:46:55.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/netconsole.ko 2022-03-21T13:46:55.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/nlmon.ko 2022-03-21T13:46:55.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/pcs 2022-03-21T13:46:55.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-03-21T13:46:55.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy 2022-03-21T13:46:55.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/amd.ko 2022-03-21T13:46:55.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-03-21T13:46:55.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/bcm7xxx.ko 2022-03-21T13:46:55.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/bcm87xx.ko 2022-03-21T13:46:55.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/broadcom.ko 2022-03-21T13:46:55.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/cicada.ko 2022-03-21T13:46:55.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/davicom.ko 2022-03-21T13:46:55.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/dp83640.ko 2022-03-21T13:46:55.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/et1011c.ko 2022-03-21T13:46:55.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/icplus.ko 2022-03-21T13:46:55.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/lxt.ko 2022-03-21T13:46:55.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/marvell.ko 2022-03-21T13:46:55.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/micrel.ko 2022-03-21T13:46:55.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/national.ko 2022-03-21T13:46:55.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/phylink.ko 2022-03-21T13:46:55.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/qsemi.ko 2022-03-21T13:46:55.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/realtek.ko 2022-03-21T13:46:55.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/smsc.ko 2022-03-21T13:46:55.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/ste10Xp.ko 2022-03-21T13:46:55.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/phy/vitesse.ko 2022-03-21T13:46:55.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp 2022-03-21T13:46:55.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/bsd_comp.ko 2022-03-21T13:46:55.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/ppp_async.ko 2022-03-21T13:46:55.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/ppp_deflate.ko 2022-03-21T13:46:55.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/ppp_generic.ko 2022-03-21T13:46:55.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/ppp_mppe.ko 2022-03-21T13:46:55.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/ppp_synctty.ko 2022-03-21T13:46:55.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/pppoe.ko 2022-03-21T13:46:55.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/pppox.ko 2022-03-21T13:46:55.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/ppp/pptp.ko 2022-03-21T13:46:55.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/slip 2022-03-21T13:46:55.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/slip/slhc.ko 2022-03-21T13:46:55.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/slip/slip.ko 2022-03-21T13:46:55.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/sungem_phy.ko 2022-03-21T13:46:55.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/tap.ko 2022-03-21T13:46:55.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team 2022-03-21T13:46:55.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team.ko 2022-03-21T13:46:55.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team_mode_activebackup.ko 2022-03-21T13:46:55.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team_mode_broadcast.ko 2022-03-21T13:46:55.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-03-21T13:46:55.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team_mode_random.ko 2022-03-21T13:46:55.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-03-21T13:46:55.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/tun.ko 2022-03-21T13:46:55.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/veth.ko 2022-03-21T13:46:55.696 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/virtio_net.ko 2022-03-21T13:46:55.696 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/vmxnet3 2022-03-21T13:46:55.696 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-03-21T13:46:55.696 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/vxlan.ko 2022-03-21T13:46:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/xen-netback 2022-03-21T13:46:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/xen-netback/xen-netback.ko 2022-03-21T13:46:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/net/xen-netfront.ko 2022-03-21T13:46:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ntb 2022-03-21T13:46:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ntb/ntb.ko 2022-03-21T13:46:55.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/nvme 2022-03-21T13:46:55.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/nvme/host 2022-03-21T13:46:55.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/nvme/host/nvme-core.ko 2022-03-21T13:46:55.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/nvme/host/nvme.ko 2022-03-21T13:46:55.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/parport 2022-03-21T13:46:55.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/parport/parport.ko 2022-03-21T13:46:55.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/parport/parport_cs.ko 2022-03-21T13:46:55.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/parport/parport_pc.ko 2022-03-21T13:46:55.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/parport/parport_serial.ko 2022-03-21T13:46:55.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci 2022-03-21T13:46:55.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci/hotplug 2022-03-21T13:46:55.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-03-21T13:46:55.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci/pcie 2022-03-21T13:46:55.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci/pcie/aer_inject.ko 2022-03-21T13:46:55.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pci/xen-pcifront.ko 2022-03-21T13:46:55.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pcmcia 2022-03-21T13:46:55.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pcmcia/i82092.ko 2022-03-21T13:46:55.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pcmcia/pd6729.ko 2022-03-21T13:46:55.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pcmcia/yenta_socket.ko 2022-03-21T13:46:55.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pinctrl 2022-03-21T13:46:55.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pinctrl/intel 2022-03-21T13:46:55.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-03-21T13:46:55.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform 2022-03-21T13:46:55.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86 2022-03-21T13:46:55.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/acer-wmi.ko 2022-03-21T13:46:55.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/acerhdf.ko 2022-03-21T13:46:55.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/apple-gmux.ko 2022-03-21T13:46:55.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/asus-laptop.ko 2022-03-21T13:46:55.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-03-21T13:46:55.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/asus-wmi.ko 2022-03-21T13:46:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/classmate-laptop.ko 2022-03-21T13:46:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-03-21T13:46:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-03-21T13:46:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-03-21T13:46:55.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-03-21T13:46:55.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/hdaps.ko 2022-03-21T13:46:55.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/hp-wmi.ko 2022-03-21T13:46:55.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/hp_accel.ko 2022-03-21T13:46:55.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/intel 2022-03-21T13:46:55.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-03-21T13:46:55.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/intel_ips.ko 2022-03-21T13:46:55.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/msi-wmi.ko 2022-03-21T13:46:55.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/mxm-wmi.ko 2022-03-21T13:46:55.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-03-21T13:46:55.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/samsung-laptop.ko 2022-03-21T13:46:55.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/samsung-q10.ko 2022-03-21T13:46:55.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/topstar-laptop.ko 2022-03-21T13:46:55.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-03-21T13:46:55.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/toshiba_haps.ko 2022-03-21T13:46:55.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/wmi-bmof.ko 2022-03-21T13:46:55.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/platform/x86/wmi.ko 2022-03-21T13:46:55.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/powercap 2022-03-21T13:46:55.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/powercap/intel_rapl_common.ko 2022-03-21T13:46:55.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/powercap/intel_rapl_msr.ko 2022-03-21T13:46:55.713 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps 2022-03-21T13:46:55.714 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps/clients 2022-03-21T13:46:55.714 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps/clients/pps-gpio.ko 2022-03-21T13:46:55.714 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps/clients/pps-ldisc.ko 2022-03-21T13:46:55.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps/clients/pps_parport.ko 2022-03-21T13:46:55.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/pps/pps_core.ko 2022-03-21T13:46:55.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ptp 2022-03-21T13:46:55.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ptp/ptp.ko 2022-03-21T13:46:55.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ptp/ptp_kvm.ko 2022-03-21T13:46:55.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc 2022-03-21T13:46:55.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-bq32k.ko 2022-03-21T13:46:55.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-bq4802.ko 2022-03-21T13:46:55.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1286.ko 2022-03-21T13:46:55.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1307.ko 2022-03-21T13:46:55.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1374.ko 2022-03-21T13:46:55.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1511.ko 2022-03-21T13:46:55.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1553.ko 2022-03-21T13:46:55.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1672.ko 2022-03-21T13:46:55.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds1742.ko 2022-03-21T13:46:55.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds2404.ko 2022-03-21T13:46:55.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-ds3232.ko 2022-03-21T13:46:55.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-em3027.ko 2022-03-21T13:46:55.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-fm3130.ko 2022-03-21T13:46:55.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-isl12022.ko 2022-03-21T13:46:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-isl1208.ko 2022-03-21T13:46:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-m41t80.ko 2022-03-21T13:46:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-m48t35.ko 2022-03-21T13:46:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-m48t59.ko 2022-03-21T13:46:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-max6900.ko 2022-03-21T13:46:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-msm6242.ko 2022-03-21T13:46:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-pcf2127.ko 2022-03-21T13:46:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-pcf85063.ko 2022-03-21T13:46:55.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-pcf8523.ko 2022-03-21T13:46:55.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-pcf8563.ko 2022-03-21T13:46:55.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-pcf8583.ko 2022-03-21T13:46:55.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-rp5c01.ko 2022-03-21T13:46:55.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-rs5c372.ko 2022-03-21T13:46:55.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-03-21T13:46:55.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-rx8025.ko 2022-03-21T13:46:55.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-rx8581.ko 2022-03-21T13:46:55.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-03-21T13:46:55.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-v3020.ko 2022-03-21T13:46:55.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/rtc/rtc-x1205.ko 2022-03-21T13:46:55.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi 2022-03-21T13:46:55.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/3w-9xxx.ko 2022-03-21T13:46:55.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/3w-sas.ko 2022-03-21T13:46:55.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/3w-xxxx.ko 2022-03-21T13:46:55.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/BusLogic.ko 2022-03-21T13:46:55.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/a100u2w.ko 2022-03-21T13:46:55.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/aacraid 2022-03-21T13:46:55.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/aacraid/aacraid.ko 2022-03-21T13:46:55.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/advansys.ko 2022-03-21T13:46:55.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/aic7xxx 2022-03-21T13:46:55.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-03-21T13:46:55.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-03-21T13:46:55.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/am53c974.ko 2022-03-21T13:46:55.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/arcmsr 2022-03-21T13:46:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-03-21T13:46:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/atp870u.ko 2022-03-21T13:46:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/be2iscsi 2022-03-21T13:46:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-03-21T13:46:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bfa 2022-03-21T13:46:55.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bfa/bfa.ko 2022-03-21T13:46:55.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bnx2fc 2022-03-21T13:46:55.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-03-21T13:46:55.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bnx2i 2022-03-21T13:46:55.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-03-21T13:46:55.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ch.ko 2022-03-21T13:46:55.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/csiostor 2022-03-21T13:46:55.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/csiostor/csiostor.ko 2022-03-21T13:46:55.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi 2022-03-21T13:46:55.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi/cxgb3i 2022-03-21T13:46:55.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-03-21T13:46:55.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi/cxgb4i 2022-03-21T13:46:55.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-03-21T13:46:55.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-03-21T13:46:55.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/dc395x.ko 2022-03-21T13:46:55.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/device_handler 2022-03-21T13:46:55.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-03-21T13:46:55.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-03-21T13:46:55.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-03-21T13:46:55.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-03-21T13:46:55.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/dmx3191d.ko 2022-03-21T13:46:55.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/esas2r 2022-03-21T13:46:55.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/esas2r/esas2r.ko 2022-03-21T13:46:55.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/esp_scsi.ko 2022-03-21T13:46:55.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/fcoe 2022-03-21T13:46:55.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/fcoe/fcoe.ko 2022-03-21T13:46:55.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-03-21T13:46:55.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/fnic 2022-03-21T13:46:55.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/fnic/fnic.ko 2022-03-21T13:46:55.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/hpsa.ko 2022-03-21T13:46:55.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/hptiop.ko 2022-03-21T13:46:55.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/hv_storvsc.ko 2022-03-21T13:46:55.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/initio.ko 2022-03-21T13:46:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ipr.ko 2022-03-21T13:46:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ips.ko 2022-03-21T13:46:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/isci 2022-03-21T13:46:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/isci/isci.ko 2022-03-21T13:46:55.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-03-21T13:46:55.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/iscsi_tcp.ko 2022-03-21T13:46:55.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libfc 2022-03-21T13:46:55.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libfc/libfc.ko 2022-03-21T13:46:55.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libiscsi.ko 2022-03-21T13:46:55.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libiscsi_tcp.ko 2022-03-21T13:46:55.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libsas 2022-03-21T13:46:55.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/libsas/libsas.ko 2022-03-21T13:46:55.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/lpfc 2022-03-21T13:46:55.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/lpfc/lpfc.ko 2022-03-21T13:46:55.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/megaraid 2022-03-21T13:46:55.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/megaraid.ko 2022-03-21T13:46:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-03-21T13:46:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-03-21T13:46:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-03-21T13:46:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/mpt3sas 2022-03-21T13:46:55.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-03-21T13:46:55.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/mvsas 2022-03-21T13:46:55.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/mvsas/mvsas.ko 2022-03-21T13:46:55.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/mvumi.ko 2022-03-21T13:46:55.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/pm8001 2022-03-21T13:46:55.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-03-21T13:46:55.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/pmcraid.ko 2022-03-21T13:46:55.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla1280.ko 2022-03-21T13:46:55.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla2xxx 2022-03-21T13:46:55.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-03-21T13:46:55.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-03-21T13:46:55.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla4xxx 2022-03-21T13:46:55.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-03-21T13:46:55.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/raid_class.ko 2022-03-21T13:46:55.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_debug.ko 2022-03-21T13:46:55.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_transport_fc.ko 2022-03-21T13:46:55.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-03-21T13:46:55.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_transport_sas.ko 2022-03-21T13:46:55.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_transport_spi.ko 2022-03-21T13:46:55.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/scsi_transport_srp.ko 2022-03-21T13:46:55.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ses.ko 2022-03-21T13:46:55.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/st.ko 2022-03-21T13:46:55.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/stex.ko 2022-03-21T13:46:55.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/sym53c8xx_2 2022-03-21T13:46:55.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-03-21T13:46:55.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ufs 2022-03-21T13:46:55.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ufs/ufshcd-core.ko 2022-03-21T13:46:55.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2022-03-21T13:46:55.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/virtio_scsi.ko 2022-03-21T13:46:55.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/vmw_pvscsi.ko 2022-03-21T13:46:55.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/scsi/wd719x.ko 2022-03-21T13:46:55.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ssb 2022-03-21T13:46:55.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/ssb/ssb.ko 2022-03-21T13:46:55.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target 2022-03-21T13:46:55.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/iscsi 2022-03-21T13:46:55.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-03-21T13:46:55.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/loopback 2022-03-21T13:46:55.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/loopback/tcm_loop.ko 2022-03-21T13:46:55.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/target_core_file.ko 2022-03-21T13:46:55.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/target_core_iblock.ko 2022-03-21T13:46:55.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/target_core_mod.ko 2022-03-21T13:46:55.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/target_core_pscsi.ko 2022-03-21T13:46:55.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/target_core_user.ko 2022-03-21T13:46:55.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/tcm_fc 2022-03-21T13:46:55.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-03-21T13:46:55.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal 2022-03-21T13:46:55.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel 2022-03-21T13:46:55.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal 2022-03-21T13:46:55.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-03-21T13:46:55.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-03-21T13:46:55.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-03-21T13:46:55.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-03-21T13:46:55.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-03-21T13:46:55.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-03-21T13:46:55.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-03-21T13:46:55.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-03-21T13:46:55.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-03-21T13:46:55.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-03-21T13:46:55.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-03-21T13:46:55.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-03-21T13:46:55.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-03-21T13:46:55.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-03-21T13:46:55.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-03-21T13:46:55.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty 2022-03-21T13:46:55.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/ipwireless 2022-03-21T13:46:55.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-03-21T13:46:55.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/n_gsm.ko 2022-03-21T13:46:55.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/n_hdlc.ko 2022-03-21T13:46:55.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/nozomi.ko 2022-03-21T13:46:55.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial 2022-03-21T13:46:55.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial/8250 2022-03-21T13:46:55.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-03-21T13:46:55.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial/arc_uart.ko 2022-03-21T13:46:55.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial/jsm 2022-03-21T13:46:55.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/serial/jsm/jsm.ko 2022-03-21T13:46:55.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/tty/synclink_gt.ko 2022-03-21T13:46:55.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio 2022-03-21T13:46:55.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio/uio.ko 2022-03-21T13:46:55.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio/uio_aec.ko 2022-03-21T13:46:55.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio/uio_cif.ko 2022-03-21T13:46:55.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio/uio_pci_generic.ko 2022-03-21T13:46:55.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/uio/uio_sercos3.ko 2022-03-21T13:46:55.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb 2022-03-21T13:46:55.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/class 2022-03-21T13:46:55.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/class/cdc-acm.ko 2022-03-21T13:46:55.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/class/cdc-wdm.ko 2022-03-21T13:46:55.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/class/usblp.ko 2022-03-21T13:46:55.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/class/usbtmc.ko 2022-03-21T13:46:55.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/host 2022-03-21T13:46:55.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/host/sl811-hcd.ko 2022-03-21T13:46:55.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/host/u132-hcd.ko 2022-03-21T13:46:55.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/image 2022-03-21T13:46:55.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/image/mdc800.ko 2022-03-21T13:46:55.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/image/microtek.ko 2022-03-21T13:46:55.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc 2022-03-21T13:46:55.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/adutux.ko 2022-03-21T13:46:55.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/appledisplay.ko 2022-03-21T13:46:55.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/emi26.ko 2022-03-21T13:46:55.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/emi62.ko 2022-03-21T13:46:55.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/ezusb.ko 2022-03-21T13:46:55.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/ftdi-elan.ko 2022-03-21T13:46:55.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/idmouse.ko 2022-03-21T13:46:55.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/iowarrior.ko 2022-03-21T13:46:55.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/isight_firmware.ko 2022-03-21T13:46:55.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/ldusb.ko 2022-03-21T13:46:55.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/legousbtower.ko 2022-03-21T13:46:55.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/sisusbvga 2022-03-21T13:46:55.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-03-21T13:46:55.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/trancevibrator.ko 2022-03-21T13:46:55.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/usb3503.ko 2022-03-21T13:46:55.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/usblcd.ko 2022-03-21T13:46:55.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/usbsevseg.ko 2022-03-21T13:46:55.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/uss720.ko 2022-03-21T13:46:55.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/misc/yurex.ko 2022-03-21T13:46:55.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/phy 2022-03-21T13:46:55.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/phy/phy-generic.ko 2022-03-21T13:46:55.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial 2022-03-21T13:46:55.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/aircable.ko 2022-03-21T13:46:55.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ark3116.ko 2022-03-21T13:46:55.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/belkin_sa.ko 2022-03-21T13:46:55.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ch341.ko 2022-03-21T13:46:55.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/cp210x.ko 2022-03-21T13:46:55.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/cyberjack.ko 2022-03-21T13:46:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/cypress_m8.ko 2022-03-21T13:46:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/digi_acceleport.ko 2022-03-21T13:46:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/empeg.ko 2022-03-21T13:46:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ftdi_sio.ko 2022-03-21T13:46:55.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/garmin_gps.ko 2022-03-21T13:46:55.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/io_edgeport.ko 2022-03-21T13:46:55.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/io_ti.ko 2022-03-21T13:46:55.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ipaq.ko 2022-03-21T13:46:55.774 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ipw.ko 2022-03-21T13:46:55.774 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ir-usb.ko 2022-03-21T13:46:55.774 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-03-21T13:46:55.774 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/keyspan.ko 2022-03-21T13:46:55.775 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/keyspan_pda.ko 2022-03-21T13:46:55.775 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/kl5kusb105.ko 2022-03-21T13:46:55.775 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/kobil_sct.ko 2022-03-21T13:46:55.775 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/mct_u232.ko 2022-03-21T13:46:55.776 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/mos7720.ko 2022-03-21T13:46:55.776 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/mos7840.ko 2022-03-21T13:46:55.776 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/navman.ko 2022-03-21T13:46:55.776 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/omninet.ko 2022-03-21T13:46:55.776 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/opticon.ko 2022-03-21T13:46:55.777 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/option.ko 2022-03-21T13:46:55.777 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/oti6858.ko 2022-03-21T13:46:55.777 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/pl2303.ko 2022-03-21T13:46:55.777 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/qcaux.ko 2022-03-21T13:46:55.778 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/qcserial.ko 2022-03-21T13:46:55.778 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/quatech2.ko 2022-03-21T13:46:55.778 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/safe_serial.ko 2022-03-21T13:46:55.778 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/sierra.ko 2022-03-21T13:46:55.779 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/spcp8x5.ko 2022-03-21T13:46:55.779 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ssu100.ko 2022-03-21T13:46:55.779 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/symbolserial.ko 2022-03-21T13:46:55.779 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-03-21T13:46:55.779 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-03-21T13:46:55.780 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/usb_debug.ko 2022-03-21T13:46:55.780 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/usb_wwan.ko 2022-03-21T13:46:55.780 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/visor.ko 2022-03-21T13:46:55.780 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/whiteheat.ko 2022-03-21T13:46:55.781 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/serial/xsens_mt.ko 2022-03-21T13:46:55.781 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage 2022-03-21T13:46:55.781 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/uas.ko 2022-03-21T13:46:55.781 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-alauda.ko 2022-03-21T13:46:55.782 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-cypress.ko 2022-03-21T13:46:55.782 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-datafab.ko 2022-03-21T13:46:55.782 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-03-21T13:46:55.782 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-freecom.ko 2022-03-21T13:46:55.782 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-isd200.ko 2022-03-21T13:46:55.783 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-03-21T13:46:55.783 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-karma.ko 2022-03-21T13:46:55.783 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-onetouch.ko 2022-03-21T13:46:55.783 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-realtek.ko 2022-03-21T13:46:55.784 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-sddr09.ko 2022-03-21T13:46:55.784 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-sddr55.ko 2022-03-21T13:46:55.784 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/ums-usbat.ko 2022-03-21T13:46:55.784 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/storage/usb-storage.ko 2022-03-21T13:46:55.785 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/usbip 2022-03-21T13:46:55.785 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/usbip/usbip-core.ko 2022-03-21T13:46:55.785 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/usbip/usbip-host.ko 2022-03-21T13:46:55.785 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-03-21T13:46:55.785 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio 2022-03-21T13:46:55.786 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/pci 2022-03-21T13:46:55.786 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-03-21T13:46:55.786 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/pci/vfio-pci.ko 2022-03-21T13:46:55.786 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/vfio.ko 2022-03-21T13:46:55.787 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-03-21T13:46:55.787 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vfio/vfio_virqfd.ko 2022-03-21T13:46:55.787 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vhost 2022-03-21T13:46:55.787 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vhost/vhost.ko 2022-03-21T13:46:55.788 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vhost/vhost_iotlb.ko 2022-03-21T13:46:55.788 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vhost/vhost_net.ko 2022-03-21T13:46:55.788 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/vhost/vhost_scsi.ko 2022-03-21T13:46:55.788 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/video 2022-03-21T13:46:55.788 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/video/backlight 2022-03-21T13:46:55.789 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/video/backlight/apple_bl.ko 2022-03-21T13:46:55.789 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/video/backlight/lcd.ko 2022-03-21T13:46:55.789 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/video/backlight/platform_lcd.ko 2022-03-21T13:46:55.789 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio 2022-03-21T13:46:55.790 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio.ko 2022-03-21T13:46:55.790 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_balloon.ko 2022-03-21T13:46:55.790 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_mmio.ko 2022-03-21T13:46:55.790 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_pci.ko 2022-03-21T13:46:55.790 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-03-21T13:46:55.791 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-03-21T13:46:55.791 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/virtio/virtio_ring.ko 2022-03-21T13:46:55.791 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1 2022-03-21T13:46:55.791 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/masters 2022-03-21T13:46:55.792 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/masters/ds1wm.ko 2022-03-21T13:46:55.792 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/masters/ds2482.ko 2022-03-21T13:46:55.792 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/masters/ds2490.ko 2022-03-21T13:46:55.792 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves 2022-03-21T13:46:55.793 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-03-21T13:46:55.793 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-03-21T13:46:55.793 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-03-21T13:46:55.793 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-03-21T13:46:55.794 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-03-21T13:46:55.794 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-03-21T13:46:55.794 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-03-21T13:46:55.794 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-03-21T13:46:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-03-21T13:46:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_smem.ko 2022-03-21T13:46:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/slaves/w1_therm.ko 2022-03-21T13:46:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/w1/wire.ko 2022-03-21T13:46:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog 2022-03-21T13:46:55.796 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/alim1535_wdt.ko 2022-03-21T13:46:55.796 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/alim7101_wdt.ko 2022-03-21T13:46:55.796 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/f71808e_wdt.ko 2022-03-21T13:46:55.796 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/hpwdt.ko 2022-03-21T13:46:55.797 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/i6300esb.ko 2022-03-21T13:46:55.797 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-03-21T13:46:55.797 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/iTCO_wdt.ko 2022-03-21T13:46:55.797 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/ib700wdt.ko 2022-03-21T13:46:55.798 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/ibmasr.ko 2022-03-21T13:46:55.798 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-03-21T13:46:55.798 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/it8712f_wdt.ko 2022-03-21T13:46:55.798 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/it87_wdt.ko 2022-03-21T13:46:55.799 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/machzwd.ko 2022-03-21T13:46:55.799 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/nv_tco.ko 2022-03-21T13:46:55.799 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/pcwd_pci.ko 2022-03-21T13:46:55.799 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/pcwd_usb.ko 2022-03-21T13:46:55.799 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-03-21T13:46:55.800 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/sch311x_wdt.ko 2022-03-21T13:46:55.800 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/softdog.ko 2022-03-21T13:46:55.800 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/sp5100_tco.ko 2022-03-21T13:46:55.800 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/via_wdt.ko 2022-03-21T13:46:55.801 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-03-21T13:46:55.801 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/w83877f_wdt.ko 2022-03-21T13:46:55.801 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/w83977f_wdt.ko 2022-03-21T13:46:55.801 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/wdt_pci.ko 2022-03-21T13:46:55.801 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/watchdog/xen_wdt.ko 2022-03-21T13:46:55.802 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen 2022-03-21T13:46:55.802 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-acpi-processor.ko 2022-03-21T13:46:55.802 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-evtchn.ko 2022-03-21T13:46:55.802 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-gntalloc.ko 2022-03-21T13:46:55.803 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-gntdev.ko 2022-03-21T13:46:55.803 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-pciback 2022-03-21T13:46:55.803 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-03-21T13:46:55.803 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xen-privcmd.ko 2022-03-21T13:46:55.804 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xenfs 2022-03-21T13:46:55.804 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/drivers/xen/xenfs/xenfs.ko 2022-03-21T13:46:55.804 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs 2022-03-21T13:46:55.804 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/9p 2022-03-21T13:46:55.805 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/9p/9p.ko 2022-03-21T13:46:55.805 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/affs 2022-03-21T13:46:55.805 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/affs/affs.ko 2022-03-21T13:46:55.805 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/befs 2022-03-21T13:46:55.806 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/befs/befs.ko 2022-03-21T13:46:55.806 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/binfmt_misc.ko 2022-03-21T13:46:55.806 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/btrfs 2022-03-21T13:46:55.806 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/btrfs/btrfs.ko 2022-03-21T13:46:55.807 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cachefiles 2022-03-21T13:46:55.807 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cachefiles/cachefiles.ko 2022-03-21T13:46:55.807 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ceph 2022-03-21T13:46:55.807 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ceph/ceph.ko 2022-03-21T13:46:55.808 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cifs 2022-03-21T13:46:55.808 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cifs/cifs.ko 2022-03-21T13:46:55.808 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/coda 2022-03-21T13:46:55.808 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/coda/coda.ko 2022-03-21T13:46:55.809 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cramfs 2022-03-21T13:46:55.809 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/cramfs/cramfs.ko 2022-03-21T13:46:55.809 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/dlm 2022-03-21T13:46:55.809 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/dlm/dlm.ko 2022-03-21T13:46:55.810 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ecryptfs 2022-03-21T13:46:55.810 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ecryptfs/ecryptfs.ko 2022-03-21T13:46:55.810 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/f2fs 2022-03-21T13:46:55.810 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/f2fs/f2fs.ko 2022-03-21T13:46:55.811 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fat 2022-03-21T13:46:55.811 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fat/fat.ko 2022-03-21T13:46:55.811 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fat/msdos.ko 2022-03-21T13:46:55.811 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fat/vfat.ko 2022-03-21T13:46:55.812 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fscache 2022-03-21T13:46:55.812 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fscache/fscache.ko 2022-03-21T13:46:55.812 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fuse 2022-03-21T13:46:55.812 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fuse/cuse.ko 2022-03-21T13:46:55.812 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/fuse/fuse.ko 2022-03-21T13:46:55.813 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/gfs2 2022-03-21T13:46:55.813 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/gfs2/gfs2.ko 2022-03-21T13:46:55.813 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/hfs 2022-03-21T13:46:55.813 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/hfs/hfs.ko 2022-03-21T13:46:55.814 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/hfsplus 2022-03-21T13:46:55.814 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/hfsplus/hfsplus.ko 2022-03-21T13:46:55.814 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/isofs 2022-03-21T13:46:55.814 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/isofs/isofs.ko 2022-03-21T13:46:55.815 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/jfs 2022-03-21T13:46:55.815 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/jfs/jfs.ko 2022-03-21T13:46:55.815 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/lockd 2022-03-21T13:46:55.815 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/lockd/lockd.ko 2022-03-21T13:46:55.816 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/minix 2022-03-21T13:46:55.816 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/minix/minix.ko 2022-03-21T13:46:55.816 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/netfs 2022-03-21T13:46:55.816 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/netfs/netfs.ko 2022-03-21T13:46:55.816 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs 2022-03-21T13:46:55.817 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/blocklayout 2022-03-21T13:46:55.817 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-03-21T13:46:55.817 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/filelayout 2022-03-21T13:46:55.817 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-03-21T13:46:55.818 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/flexfilelayout 2022-03-21T13:46:55.818 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-03-21T13:46:55.818 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/nfs.ko 2022-03-21T13:46:55.818 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/nfsv3.ko 2022-03-21T13:46:55.819 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs/nfsv4.ko 2022-03-21T13:46:55.819 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs_common 2022-03-21T13:46:55.819 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs_common/grace.ko 2022-03-21T13:46:55.819 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfs_common/nfs_acl.ko 2022-03-21T13:46:55.820 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfsd 2022-03-21T13:46:55.820 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nfsd/nfsd.ko 2022-03-21T13:46:55.820 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nilfs2 2022-03-21T13:46:55.820 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nilfs2/nilfs2.ko 2022-03-21T13:46:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls 2022-03-21T13:46:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-celtic.ko 2022-03-21T13:46:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-centeuro.ko 2022-03-21T13:46:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-croatian.ko 2022-03-21T13:46:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-cyrillic.ko 2022-03-21T13:46:55.822 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-gaelic.ko 2022-03-21T13:46:55.822 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-greek.ko 2022-03-21T13:46:55.822 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-iceland.ko 2022-03-21T13:46:55.822 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-inuit.ko 2022-03-21T13:46:55.823 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-roman.ko 2022-03-21T13:46:55.823 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-romanian.ko 2022-03-21T13:46:55.823 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/mac-turkish.ko 2022-03-21T13:46:55.823 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp1250.ko 2022-03-21T13:46:55.824 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp1251.ko 2022-03-21T13:46:55.824 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp1255.ko 2022-03-21T13:46:55.824 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp737.ko 2022-03-21T13:46:55.824 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp775.ko 2022-03-21T13:46:55.825 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp850.ko 2022-03-21T13:46:55.825 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp852.ko 2022-03-21T13:46:55.825 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp855.ko 2022-03-21T13:46:55.825 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp857.ko 2022-03-21T13:46:55.826 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp860.ko 2022-03-21T13:46:55.826 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp861.ko 2022-03-21T13:46:55.826 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp862.ko 2022-03-21T13:46:55.826 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp863.ko 2022-03-21T13:46:55.827 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp864.ko 2022-03-21T13:46:55.827 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp865.ko 2022-03-21T13:46:55.827 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp866.ko 2022-03-21T13:46:55.827 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp869.ko 2022-03-21T13:46:55.828 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp874.ko 2022-03-21T13:46:55.828 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp932.ko 2022-03-21T13:46:55.828 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp936.ko 2022-03-21T13:46:55.828 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp949.ko 2022-03-21T13:46:55.828 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_cp950.ko 2022-03-21T13:46:55.829 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_euc-jp.ko 2022-03-21T13:46:55.829 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-1.ko 2022-03-21T13:46:55.829 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-13.ko 2022-03-21T13:46:55.829 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-14.ko 2022-03-21T13:46:55.830 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-15.ko 2022-03-21T13:46:55.830 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-2.ko 2022-03-21T13:46:55.830 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-3.ko 2022-03-21T13:46:55.830 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-4.ko 2022-03-21T13:46:55.831 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-5.ko 2022-03-21T13:46:55.831 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-6.ko 2022-03-21T13:46:55.831 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-7.ko 2022-03-21T13:46:55.831 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_iso8859-9.ko 2022-03-21T13:46:55.832 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_koi8-r.ko 2022-03-21T13:46:55.832 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_koi8-ru.ko 2022-03-21T13:46:55.832 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_koi8-u.ko 2022-03-21T13:46:55.832 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/nls/nls_utf8.ko 2022-03-21T13:46:55.832 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2 2022-03-21T13:46:55.833 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/cluster 2022-03-21T13:46:55.833 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-03-21T13:46:55.833 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/dlm 2022-03-21T13:46:55.833 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-03-21T13:46:55.833 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/dlmfs 2022-03-21T13:46:55.834 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-03-21T13:46:55.834 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/ocfs2.ko 2022-03-21T13:46:55.834 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-03-21T13:46:55.835 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-03-21T13:46:55.835 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-03-21T13:46:55.835 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/overlayfs 2022-03-21T13:46:55.835 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/overlayfs/overlay.ko 2022-03-21T13:46:55.835 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/pstore 2022-03-21T13:46:55.836 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/pstore/ramoops.ko 2022-03-21T13:46:55.836 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/reiserfs 2022-03-21T13:46:55.836 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/reiserfs/reiserfs.ko 2022-03-21T13:46:55.836 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/romfs 2022-03-21T13:46:55.837 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/romfs/romfs.ko 2022-03-21T13:46:55.837 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/smbfs_common 2022-03-21T13:46:55.837 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/smbfs_common/cifs_arc4.ko 2022-03-21T13:46:55.837 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/smbfs_common/cifs_md4.ko 2022-03-21T13:46:55.837 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/squashfs 2022-03-21T13:46:55.838 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/squashfs/squashfs.ko 2022-03-21T13:46:55.838 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/sysv 2022-03-21T13:46:55.838 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/sysv/sysv.ko 2022-03-21T13:46:55.838 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/udf 2022-03-21T13:46:55.839 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/udf/udf.ko 2022-03-21T13:46:55.839 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ufs 2022-03-21T13:46:55.839 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/ufs/ufs.ko 2022-03-21T13:46:55.839 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/xfs 2022-03-21T13:46:55.839 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/fs/xfs/xfs.ko 2022-03-21T13:46:55.840 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel 2022-03-21T13:46:55.840 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel/rcu 2022-03-21T13:46:55.840 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel/rcu/rcutorture.ko 2022-03-21T13:46:55.840 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel/torture.ko 2022-03-21T13:46:55.841 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel/trace 2022-03-21T13:46:55.841 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-03-21T13:46:55.841 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib 2022-03-21T13:46:55.841 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/asn1_encoder.ko 2022-03-21T13:46:55.842 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crc-itu-t.ko 2022-03-21T13:46:55.842 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crc64.ko 2022-03-21T13:46:55.842 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crc8.ko 2022-03-21T13:46:55.842 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crypto 2022-03-21T13:46:55.843 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crypto/libarc4.ko 2022-03-21T13:46:55.843 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/crypto/libdes.ko 2022-03-21T13:46:55.843 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/libcrc32c.ko 2022-03-21T13:46:55.843 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/lru_cache.ko 2022-03-21T13:46:55.843 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/lz4 2022-03-21T13:46:55.844 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/lz4/lz4_compress.ko 2022-03-21T13:46:55.844 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/lz4/lz4hc_compress.ko 2022-03-21T13:46:55.844 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/math 2022-03-21T13:46:55.844 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/math/cordic.ko 2022-03-21T13:46:55.845 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/raid6 2022-03-21T13:46:55.845 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/raid6/raid6_pq.ko 2022-03-21T13:46:55.845 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/reed_solomon 2022-03-21T13:46:55.845 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/reed_solomon/reed_solomon.ko 2022-03-21T13:46:55.845 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/ts_bm.ko 2022-03-21T13:46:55.846 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/ts_fsm.ko 2022-03-21T13:46:55.846 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/ts_kmp.ko 2022-03-21T13:46:55.846 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/zstd 2022-03-21T13:46:55.846 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/lib/zstd/zstd_compress.ko 2022-03-21T13:46:55.847 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/mm 2022-03-21T13:46:55.847 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/mm/hwpoison-inject.ko 2022-03-21T13:46:55.847 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net 2022-03-21T13:46:55.847 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan 2022-03-21T13:46:55.847 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/6lowpan.ko 2022-03-21T13:46:55.848 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_dest.ko 2022-03-21T13:46:55.848 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_fragment.ko 2022-03-21T13:46:55.848 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_hop.ko 2022-03-21T13:46:55.848 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_ipv6.ko 2022-03-21T13:46:55.849 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_mobility.ko 2022-03-21T13:46:55.849 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_routing.ko 2022-03-21T13:46:55.849 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/6lowpan/nhc_udp.ko 2022-03-21T13:46:55.849 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802 2022-03-21T13:46:55.849 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802/garp.ko 2022-03-21T13:46:55.850 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802/mrp.ko 2022-03-21T13:46:55.850 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802/p8022.ko 2022-03-21T13:46:55.850 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802/psnap.ko 2022-03-21T13:46:55.850 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/802/stp.ko 2022-03-21T13:46:55.851 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/8021q 2022-03-21T13:46:55.851 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/8021q/8021q.ko 2022-03-21T13:46:55.851 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/9p 2022-03-21T13:46:55.851 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/9p/9pnet.ko 2022-03-21T13:46:55.852 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/9p/9pnet_fd.ko 2022-03-21T13:46:55.852 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/9p/9pnet_rdma.ko 2022-03-21T13:46:55.852 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/9p/9pnet_virtio.ko 2022-03-21T13:46:55.852 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/batman-adv 2022-03-21T13:46:55.853 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/batman-adv/batman-adv.ko 2022-03-21T13:46:55.853 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge 2022-03-21T13:46:55.853 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/br_netfilter.ko 2022-03-21T13:46:55.853 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/bridge.ko 2022-03-21T13:46:55.853 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter 2022-03-21T13:46:55.854 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-03-21T13:46:55.854 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_among.ko 2022-03-21T13:46:55.854 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_arp.ko 2022-03-21T13:46:55.854 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-03-21T13:46:55.855 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-03-21T13:46:55.855 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_ip.ko 2022-03-21T13:46:55.855 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-03-21T13:46:55.855 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_limit.ko 2022-03-21T13:46:55.856 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_log.ko 2022-03-21T13:46:55.856 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_mark.ko 2022-03-21T13:46:55.856 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-03-21T13:46:55.856 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-03-21T13:46:55.856 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-03-21T13:46:55.857 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-03-21T13:46:55.857 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_snat.ko 2022-03-21T13:46:55.857 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_stp.ko 2022-03-21T13:46:55.857 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-03-21T13:46:55.858 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-03-21T13:46:55.858 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-03-21T13:46:55.858 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-03-21T13:46:55.858 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/ebtables.ko 2022-03-21T13:46:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-03-21T13:46:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-03-21T13:46:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-03-21T13:46:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ceph 2022-03-21T13:46:55.859 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ceph/libceph.ko 2022-03-21T13:46:55.860 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/core 2022-03-21T13:46:55.860 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/core/failover.ko 2022-03-21T13:46:55.860 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/core/pktgen.ko 2022-03-21T13:46:55.860 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dccp 2022-03-21T13:46:55.861 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dccp/dccp.ko 2022-03-21T13:46:55.861 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dccp/dccp_diag.ko 2022-03-21T13:46:55.861 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dccp/dccp_ipv4.ko 2022-03-21T13:46:55.861 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dccp/dccp_ipv6.ko 2022-03-21T13:46:55.862 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dns_resolver 2022-03-21T13:46:55.862 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dns_resolver/dns_resolver.ko 2022-03-21T13:46:55.862 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dsa 2022-03-21T13:46:55.862 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dsa/dsa_core.ko 2022-03-21T13:46:55.863 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dsa/tag_brcm.ko 2022-03-21T13:46:55.863 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dsa/tag_dsa.ko 2022-03-21T13:46:55.863 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/dsa/tag_trailer.ko 2022-03-21T13:46:55.863 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ieee802154 2022-03-21T13:46:55.864 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ieee802154/6lowpan 2022-03-21T13:46:55.864 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-03-21T13:46:55.864 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ieee802154/ieee802154.ko 2022-03-21T13:46:55.865 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ieee802154/ieee802154_socket.ko 2022-03-21T13:46:55.865 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4 2022-03-21T13:46:55.865 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ah4.ko 2022-03-21T13:46:55.865 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/esp4.ko 2022-03-21T13:46:55.865 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/fou.ko 2022-03-21T13:46:55.866 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/gre.ko 2022-03-21T13:46:55.866 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/inet_diag.ko 2022-03-21T13:46:55.866 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ip_gre.ko 2022-03-21T13:46:55.867 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ip_tunnel.ko 2022-03-21T13:46:55.867 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ip_vti.ko 2022-03-21T13:46:55.867 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ipcomp.ko 2022-03-21T13:46:55.867 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/ipip.ko 2022-03-21T13:46:55.868 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter 2022-03-21T13:46:55.868 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/arp_tables.ko 2022-03-21T13:46:55.868 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-03-21T13:46:55.868 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-03-21T13:46:55.868 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-03-21T13:46:55.869 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-03-21T13:46:55.869 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-03-21T13:46:55.869 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-03-21T13:46:55.869 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-03-21T13:46:55.870 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-03-21T13:46:55.870 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-03-21T13:46:55.870 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-03-21T13:46:55.870 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/iptable_security.ko 2022-03-21T13:46:55.871 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-03-21T13:46:55.871 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-03-21T13:46:55.871 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-03-21T13:46:55.871 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-03-21T13:46:55.872 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-03-21T13:46:55.872 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-03-21T13:46:55.872 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-03-21T13:46:55.872 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-03-21T13:46:55.873 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-03-21T13:46:55.873 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-03-21T13:46:55.873 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_bic.ko 2022-03-21T13:46:55.873 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_dctcp.ko 2022-03-21T13:46:55.874 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_diag.ko 2022-03-21T13:46:55.874 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_highspeed.ko 2022-03-21T13:46:55.874 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_htcp.ko 2022-03-21T13:46:55.874 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_hybla.ko 2022-03-21T13:46:55.875 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_illinois.ko 2022-03-21T13:46:55.875 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_lp.ko 2022-03-21T13:46:55.875 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_scalable.ko 2022-03-21T13:46:55.875 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_vegas.ko 2022-03-21T13:46:55.876 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_veno.ko 2022-03-21T13:46:55.876 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_westwood.ko 2022-03-21T13:46:55.876 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tcp_yeah.ko 2022-03-21T13:46:55.876 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/tunnel4.ko 2022-03-21T13:46:55.877 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/udp_diag.ko 2022-03-21T13:46:55.877 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/udp_tunnel.ko 2022-03-21T13:46:55.877 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv4/xfrm4_tunnel.ko 2022-03-21T13:46:55.877 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6 2022-03-21T13:46:55.877 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/ah6.ko 2022-03-21T13:46:55.878 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/esp6.ko 2022-03-21T13:46:55.878 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/fou6.ko 2022-03-21T13:46:55.878 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/ip6_tunnel.ko 2022-03-21T13:46:55.878 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-03-21T13:46:55.879 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/ip6_vti.ko 2022-03-21T13:46:55.879 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/ipcomp6.ko 2022-03-21T13:46:55.879 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter 2022-03-21T13:46:55.879 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-03-21T13:46:55.880 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-03-21T13:46:55.880 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-03-21T13:46:55.880 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-03-21T13:46:55.880 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-03-21T13:46:55.881 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-03-21T13:46:55.881 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-03-21T13:46:55.881 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-03-21T13:46:55.881 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-03-21T13:46:55.881 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-03-21T13:46:55.882 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-03-21T13:46:55.882 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-03-21T13:46:55.882 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-03-21T13:46:55.882 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-03-21T13:46:55.883 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-03-21T13:46:55.883 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-03-21T13:46:55.883 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-03-21T13:46:55.883 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-03-21T13:46:55.883 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-03-21T13:46:55.884 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-03-21T13:46:55.884 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-03-21T13:46:55.884 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-03-21T13:46:55.884 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-03-21T13:46:55.885 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-03-21T13:46:55.885 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/sit.ko 2022-03-21T13:46:55.885 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/tunnel6.ko 2022-03-21T13:46:55.885 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/ipv6/xfrm6_tunnel.ko 2022-03-21T13:46:55.886 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/key 2022-03-21T13:46:55.886 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/key/af_key.ko 2022-03-21T13:46:55.886 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp 2022-03-21T13:46:55.886 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_core.ko 2022-03-21T13:46:55.886 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_debugfs.ko 2022-03-21T13:46:55.887 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_eth.ko 2022-03-21T13:46:55.887 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_ip.ko 2022-03-21T13:46:55.887 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_ip6.ko 2022-03-21T13:46:55.887 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_netlink.ko 2022-03-21T13:46:55.887 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/l2tp/l2tp_ppp.ko 2022-03-21T13:46:55.888 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/llc 2022-03-21T13:46:55.888 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/llc/llc.ko 2022-03-21T13:46:55.888 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/mac802154 2022-03-21T13:46:55.888 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/mac802154/mac802154.ko 2022-03-21T13:46:55.889 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/mpls 2022-03-21T13:46:55.889 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/mpls/mpls_gso.ko 2022-03-21T13:46:55.889 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter 2022-03-21T13:46:55.889 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset 2022-03-21T13:46:55.890 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set.ko 2022-03-21T13:46:55.890 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-03-21T13:46:55.890 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-03-21T13:46:55.890 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-03-21T13:46:55.891 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-03-21T13:46:55.891 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-03-21T13:46:55.891 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-03-21T13:46:55.891 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-03-21T13:46:55.892 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-03-21T13:46:55.892 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-03-21T13:46:55.892 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-03-21T13:46:55.892 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-03-21T13:46:55.893 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-03-21T13:46:55.893 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-03-21T13:46:55.893 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-03-21T13:46:55.893 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-03-21T13:46:55.893 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs 2022-03-21T13:46:55.894 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs.ko 2022-03-21T13:46:55.894 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-03-21T13:46:55.894 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-03-21T13:46:55.894 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-03-21T13:46:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-03-21T13:46:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-03-21T13:46:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-03-21T13:46:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-03-21T13:46:55.896 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-03-21T13:46:55.896 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-03-21T13:46:55.896 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-03-21T13:46:55.896 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-03-21T13:46:55.896 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-03-21T13:46:55.897 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-03-21T13:46:55.897 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conncount.ko 2022-03-21T13:46:55.897 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack.ko 2022-03-21T13:46:55.897 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-03-21T13:46:55.898 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-03-21T13:46:55.898 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-03-21T13:46:55.898 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_h323.ko 2022-03-21T13:46:55.898 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_irc.ko 2022-03-21T13:46:55.899 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-03-21T13:46:55.899 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-03-21T13:46:55.899 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-03-21T13:46:55.899 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_sane.ko 2022-03-21T13:46:55.900 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_sip.ko 2022-03-21T13:46:55.900 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-03-21T13:46:55.900 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-03-21T13:46:55.900 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_dup_netdev.ko 2022-03-21T13:46:55.901 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_flow_table.ko 2022-03-21T13:46:55.901 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_flow_table_inet.ko 2022-03-21T13:46:55.901 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_log_syslog.ko 2022-03-21T13:46:55.901 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat.ko 2022-03-21T13:46:55.902 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat_amanda.ko 2022-03-21T13:46:55.902 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat_ftp.ko 2022-03-21T13:46:55.902 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat_irc.ko 2022-03-21T13:46:55.902 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat_sip.ko 2022-03-21T13:46:55.902 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_nat_tftp.ko 2022-03-21T13:46:55.903 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_synproxy_core.ko 2022-03-21T13:46:55.903 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nf_tables.ko 2022-03-21T13:46:55.903 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nfnetlink.ko 2022-03-21T13:46:55.903 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nfnetlink_acct.ko 2022-03-21T13:46:55.904 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nfnetlink_log.ko 2022-03-21T13:46:55.904 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nfnetlink_osf.ko 2022-03-21T13:46:55.904 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nfnetlink_queue.ko 2022-03-21T13:46:55.904 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_chain_nat.ko 2022-03-21T13:46:55.905 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_compat.ko 2022-03-21T13:46:55.905 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_ct.ko 2022-03-21T13:46:55.905 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_dup_netdev.ko 2022-03-21T13:46:55.905 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_fib.ko 2022-03-21T13:46:55.906 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_fwd_netdev.ko 2022-03-21T13:46:55.906 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_hash.ko 2022-03-21T13:46:55.906 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_limit.ko 2022-03-21T13:46:55.906 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_log.ko 2022-03-21T13:46:55.907 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_masq.ko 2022-03-21T13:46:55.907 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_nat.ko 2022-03-21T13:46:55.907 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_numgen.ko 2022-03-21T13:46:55.907 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_objref.ko 2022-03-21T13:46:55.908 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_queue.ko 2022-03-21T13:46:55.908 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_quota.ko 2022-03-21T13:46:55.908 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_redir.ko 2022-03-21T13:46:55.908 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_reject.ko 2022-03-21T13:46:55.908 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_reject_inet.ko 2022-03-21T13:46:55.909 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_socket.ko 2022-03-21T13:46:55.909 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/nft_tunnel.ko 2022-03-21T13:46:55.909 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_AUDIT.ko 2022-03-21T13:46:55.909 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_CHECKSUM.ko 2022-03-21T13:46:55.910 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_CLASSIFY.ko 2022-03-21T13:46:55.910 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-03-21T13:46:55.910 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_CT.ko 2022-03-21T13:46:55.910 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_DSCP.ko 2022-03-21T13:46:55.911 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_HL.ko 2022-03-21T13:46:55.911 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_HMARK.ko 2022-03-21T13:46:55.911 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_IDLETIMER.ko 2022-03-21T13:46:55.911 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_LED.ko 2022-03-21T13:46:55.912 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_LOG.ko 2022-03-21T13:46:55.912 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_MASQUERADE.ko 2022-03-21T13:46:55.912 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_NETMAP.ko 2022-03-21T13:46:55.912 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_NFLOG.ko 2022-03-21T13:46:55.912 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_NFQUEUE.ko 2022-03-21T13:46:55.913 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_RATEEST.ko 2022-03-21T13:46:55.913 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_REDIRECT.ko 2022-03-21T13:46:55.913 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_SECMARK.ko 2022-03-21T13:46:55.913 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_TCPMSS.ko 2022-03-21T13:46:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-03-21T13:46:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_TEE.ko 2022-03-21T13:46:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_TPROXY.ko 2022-03-21T13:46:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_TRACE.ko 2022-03-21T13:46:55.915 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_addrtype.ko 2022-03-21T13:46:55.915 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_bpf.ko 2022-03-21T13:46:55.915 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_cgroup.ko 2022-03-21T13:46:55.915 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_cluster.ko 2022-03-21T13:46:55.915 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_comment.ko 2022-03-21T13:46:55.916 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_connbytes.ko 2022-03-21T13:46:55.916 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_connlabel.ko 2022-03-21T13:46:55.916 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_connlimit.ko 2022-03-21T13:46:55.916 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_connmark.ko 2022-03-21T13:46:55.916 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_conntrack.ko 2022-03-21T13:46:55.917 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_cpu.ko 2022-03-21T13:46:55.917 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_dccp.ko 2022-03-21T13:46:55.917 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_devgroup.ko 2022-03-21T13:46:55.917 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_dscp.ko 2022-03-21T13:46:55.918 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_ecn.ko 2022-03-21T13:46:55.918 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_esp.ko 2022-03-21T13:46:55.918 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_hashlimit.ko 2022-03-21T13:46:55.918 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_helper.ko 2022-03-21T13:46:55.919 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_hl.ko 2022-03-21T13:46:55.919 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_ipcomp.ko 2022-03-21T13:46:55.919 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_iprange.ko 2022-03-21T13:46:55.919 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_ipvs.ko 2022-03-21T13:46:55.920 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_l2tp.ko 2022-03-21T13:46:55.920 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_length.ko 2022-03-21T13:46:55.920 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_limit.ko 2022-03-21T13:46:55.920 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_mac.ko 2022-03-21T13:46:55.921 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_mark.ko 2022-03-21T13:46:55.921 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_multiport.ko 2022-03-21T13:46:55.921 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_nat.ko 2022-03-21T13:46:55.921 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_nfacct.ko 2022-03-21T13:46:55.922 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_osf.ko 2022-03-21T13:46:55.922 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_owner.ko 2022-03-21T13:46:55.922 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_physdev.ko 2022-03-21T13:46:55.922 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_pkttype.ko 2022-03-21T13:46:55.923 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_policy.ko 2022-03-21T13:46:55.923 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_quota.ko 2022-03-21T13:46:55.923 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_rateest.ko 2022-03-21T13:46:55.923 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_realm.ko 2022-03-21T13:46:55.924 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_recent.ko 2022-03-21T13:46:55.924 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_sctp.ko 2022-03-21T13:46:55.924 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_set.ko 2022-03-21T13:46:55.924 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_socket.ko 2022-03-21T13:46:55.924 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_state.ko 2022-03-21T13:46:55.925 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_statistic.ko 2022-03-21T13:46:55.925 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_string.ko 2022-03-21T13:46:55.925 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_tcpmss.ko 2022-03-21T13:46:55.925 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_time.ko 2022-03-21T13:46:55.926 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netfilter/xt_u32.ko 2022-03-21T13:46:55.926 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netlink 2022-03-21T13:46:55.926 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/netlink/netlink_diag.ko 2022-03-21T13:46:55.926 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/nsh 2022-03-21T13:46:55.927 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/nsh/nsh.ko 2022-03-21T13:46:55.927 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/openvswitch 2022-03-21T13:46:55.927 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/openvswitch/openvswitch.ko 2022-03-21T13:46:55.927 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/openvswitch/vport-geneve.ko 2022-03-21T13:46:55.927 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/openvswitch/vport-gre.ko 2022-03-21T13:46:55.928 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/openvswitch/vport-vxlan.ko 2022-03-21T13:46:55.928 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/packet 2022-03-21T13:46:55.928 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/packet/af_packet_diag.ko 2022-03-21T13:46:55.928 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/rds 2022-03-21T13:46:55.929 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/rds/rds.ko 2022-03-21T13:46:55.929 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/rds/rds_rdma.ko 2022-03-21T13:46:55.929 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/rds/rds_tcp.ko 2022-03-21T13:46:55.929 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched 2022-03-21T13:46:55.929 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_csum.ko 2022-03-21T13:46:55.930 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_gact.ko 2022-03-21T13:46:55.930 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_ipt.ko 2022-03-21T13:46:55.930 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_mirred.ko 2022-03-21T13:46:55.930 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_nat.ko 2022-03-21T13:46:55.931 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_pedit.ko 2022-03-21T13:46:55.931 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_police.ko 2022-03-21T13:46:55.931 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_simple.ko 2022-03-21T13:46:55.931 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_skbedit.ko 2022-03-21T13:46:55.932 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/act_vlan.ko 2022-03-21T13:46:55.932 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_basic.ko 2022-03-21T13:46:55.932 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_bpf.ko 2022-03-21T13:46:55.932 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_flow.ko 2022-03-21T13:46:55.932 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_fw.ko 2022-03-21T13:46:55.933 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_route.ko 2022-03-21T13:46:55.933 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_rsvp.ko 2022-03-21T13:46:55.933 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_rsvp6.ko 2022-03-21T13:46:55.933 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_tcindex.ko 2022-03-21T13:46:55.934 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/cls_u32.ko 2022-03-21T13:46:55.934 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_cmp.ko 2022-03-21T13:46:55.934 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_ipset.ko 2022-03-21T13:46:55.934 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_meta.ko 2022-03-21T13:46:55.935 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_nbyte.ko 2022-03-21T13:46:55.935 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_text.ko 2022-03-21T13:46:55.935 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/em_u32.ko 2022-03-21T13:46:55.935 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_cbq.ko 2022-03-21T13:46:55.936 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_choke.ko 2022-03-21T13:46:55.936 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_codel.ko 2022-03-21T13:46:55.936 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_drr.ko 2022-03-21T13:46:55.936 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_dsmark.ko 2022-03-21T13:46:55.936 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_fq.ko 2022-03-21T13:46:55.937 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_gred.ko 2022-03-21T13:46:55.937 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_hfsc.ko 2022-03-21T13:46:55.937 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_hhf.ko 2022-03-21T13:46:55.937 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_htb.ko 2022-03-21T13:46:55.938 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_ingress.ko 2022-03-21T13:46:55.938 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_mqprio.ko 2022-03-21T13:46:55.938 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_multiq.ko 2022-03-21T13:46:55.938 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_netem.ko 2022-03-21T13:46:55.939 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_pie.ko 2022-03-21T13:46:55.939 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_plug.ko 2022-03-21T13:46:55.939 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_prio.ko 2022-03-21T13:46:55.939 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_qfq.ko 2022-03-21T13:46:55.940 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_red.ko 2022-03-21T13:46:55.940 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_sfb.ko 2022-03-21T13:46:55.940 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_sfq.ko 2022-03-21T13:46:55.940 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_tbf.ko 2022-03-21T13:46:55.941 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sched/sch_teql.ko 2022-03-21T13:46:55.941 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sctp 2022-03-21T13:46:55.941 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sctp/sctp.ko 2022-03-21T13:46:55.941 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sctp/sctp_diag.ko 2022-03-21T13:46:55.942 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc 2022-03-21T13:46:55.942 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/auth_gss 2022-03-21T13:46:55.942 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-03-21T13:46:55.942 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-03-21T13:46:55.943 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/sunrpc.ko 2022-03-21T13:46:55.943 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/xprtrdma 2022-03-21T13:46:55.943 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-03-21T13:46:55.943 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/tipc 2022-03-21T13:46:55.944 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/tipc/diag.ko 2022-03-21T13:46:55.944 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/tipc/tipc.ko 2022-03-21T13:46:55.944 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/unix 2022-03-21T13:46:55.944 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/unix/unix_diag.ko 2022-03-21T13:46:55.944 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock 2022-03-21T13:46:55.945 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-03-21T13:46:55.945 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-03-21T13:46:55.945 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock/vsock.ko 2022-03-21T13:46:55.945 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock/vsock_diag.ko 2022-03-21T13:46:55.946 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/vmw_vsock/vsock_loopback.ko 2022-03-21T13:46:55.946 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/xfrm 2022-03-21T13:46:55.946 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/net/xfrm/xfrm_ipcomp.ko 2022-03-21T13:46:55.946 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security 2022-03-21T13:46:55.947 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security/keys 2022-03-21T13:46:55.947 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security/keys/encrypted-keys 2022-03-21T13:46:55.947 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-03-21T13:46:55.947 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security/keys/trusted-keys 2022-03-21T13:46:55.948 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/security/keys/trusted-keys/trusted.ko 2022-03-21T13:46:55.948 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/virt 2022-03-21T13:46:55.948 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/virt/lib 2022-03-21T13:46:55.948 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/kernel/virt/lib/irqbypass.ko 2022-03-21T13:46:55.949 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.alias 2022-03-21T13:46:55.949 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.alias.bin 2022-03-21T13:46:55.949 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.builtin 2022-03-21T13:46:55.949 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.builtin.bin 2022-03-21T13:46:55.950 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.builtin.modinfo 2022-03-21T13:46:55.950 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.dep 2022-03-21T13:46:55.950 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.dep.bin 2022-03-21T13:46:55.950 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.devname 2022-03-21T13:46:55.951 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.order 2022-03-21T13:46:55.951 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.softdep 2022-03-21T13:46:55.951 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.symbols 2022-03-21T13:46:55.952 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.17.0-rc6-ceph-ge35497518a9d/modules.symbols.bin 2022-03-21T13:46:55.952 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id 2022-03-21T13:46:55.952 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id/70 2022-03-21T13:46:55.952 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id/70/7f301f9f0d9c25a16c733cdacf56b5caf9445d 2022-03-21T13:46:55.953 DEBUG:teuthology.task.kernel:get_image_version: 5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.954 DEBUG:teuthology.orchestra.run.smithi002:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-03-21T13:46:55.979 INFO:teuthology.orchestra.run.smithi002.stdout:postinstall scriptlet (using /bin/sh): 2022-03-21T13:46:55.979 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.17.0-rc6-ceph-ge35497518a9d -a -r /boot/System.map-5.17.0-rc6-ceph-ge35497518a9d ]; then 2022-03-21T13:46:55.980 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/vmlinuz-5.17.0-rc6-ceph-ge35497518a9d /boot/.vmlinuz-5.17.0-rc6-ceph-ge35497518a9d-rpm 2022-03-21T13:46:55.980 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/System.map-5.17.0-rc6-ceph-ge35497518a9d /boot/.System.map-5.17.0-rc6-ceph-ge35497518a9d-rpm 2022-03-21T13:46:55.980 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/vmlinuz-5.17.0-rc6-ceph-ge35497518a9d /boot/System.map-5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.980 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/installkernel 5.17.0-rc6-ceph-ge35497518a9d /boot/.vmlinuz-5.17.0-rc6-ceph-ge35497518a9d-rpm /boot/.System.map-5.17.0-rc6-ceph-ge35497518a9d-rpm 2022-03-21T13:46:55.981 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/.vmlinuz-5.17.0-rc6-ceph-ge35497518a9d-rpm /boot/.System.map-5.17.0-rc6-ceph-ge35497518a9d-rpm 2022-03-21T13:46:55.981 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-03-21T13:46:55.981 INFO:teuthology.orchestra.run.smithi002.stdout:preuninstall scriptlet (using /bin/sh): 2022-03-21T13:46:55.981 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-03-21T13:46:55.982 INFO:teuthology.orchestra.run.smithi002.stdout:new-kernel-pkg --remove 5.17.0-rc6-ceph-ge35497518a9d --rminitrd --initrdfile=/boot/initramfs-5.17.0-rc6-ceph-ge35497518a9d.img 2022-03-21T13:46:55.982 INFO:teuthology.orchestra.run.smithi002.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-03-21T13:46:55.982 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-install remove 5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.983 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-03-21T13:46:55.983 INFO:teuthology.orchestra.run.smithi002.stdout:postuninstall scriptlet (using /bin/sh): 2022-03-21T13:46:55.983 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/update-bootloader ]; then 2022-03-21T13:46:55.983 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/update-bootloader --remove 5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:55.984 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-03-21T13:46:55.985 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2022-03-21T13:46:56.061 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2022-03-21T13:46:56.061 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2022-03-21T13:46:56.061 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.02 2022-03-21T13:46:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 99.el8 2022-03-21T13:46:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2022-03-21T13:46:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 18 Mar 2022 06:26:52 PM UTC 2022-03-21T13:46:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:Group : System Environment/Base 2022-03-21T13:46:56.063 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 9264713 2022-03-21T13:46:56.063 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2022-03-21T13:46:56.063 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 08:11:27 PM UTC, Key ID 199e2f91fd431d51 2022-03-21T13:46:56.063 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2022-03-21T13:46:56.064 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Thu 25 Feb 2021 04:59:31 PM UTC 2022-03-21T13:46:56.064 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2022-03-21T13:46:56.064 INFO:teuthology.orchestra.run.smithi002.stdout:Relocations : (not relocatable) 2022-03-21T13:46:56.064 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : Red Hat, Inc. 2022-03-21T13:46:56.064 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : Red Hat, Inc. 2022-03-21T13:46:56.065 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2022-03-21T13:46:56.065 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2022-03-21T13:46:56.065 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2022-03-21T13:46:56.065 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-03-21T13:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2022-03-21T13:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2022-03-21T13:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2022-03-21T13:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:46:56.067 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2022-03-21T13:46:56.068 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-03-21T13:46:56.068 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:46:56.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-03-21T13:46:59.507 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2022-03-21T13:47:01.926 INFO:teuthology.orchestra.run.smithi002.stderr:done 2022-03-21T13:47:01.928 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2022-03-21T13:47:01.943 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.1uUiHlZUy2 2022-03-21T13:47:01.943 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1uUiHlZUy2 2022-03-21T13:47:02.012 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.1uUiHlZUy2 2022-03-21T13:47:02.164 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.1uUiHlZUy2 is 6KB 2022-03-21T13:47:02.214 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.1uUiHlZUy2 2022-03-21T13:47:02.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2022-03-21T13:47:02.291 INFO:teuthology.orchestra.run.smithi002.stdout:2e9f1b340eca4ccf93c8af366fb23726-0-rescue.conf 2022-03-21T13:47:02.291 INFO:teuthology.orchestra.run.smithi002.stdout:2e9f1b340eca4ccf93c8af366fb23726-4.18.0-305.el8.x86_64.conf 2022-03-21T13:47:02.291 INFO:teuthology.orchestra.run.smithi002.stdout:2e9f1b340eca4ccf93c8af366fb23726-4.18.0-348.20.1.el8_5.x86_64.conf 2022-03-21T13:47:02.292 INFO:teuthology.orchestra.run.smithi002.stdout:2e9f1b340eca4ccf93c8af366fb23726-5.17.0-rc6-ceph-ge35497518a9d.conf 2022-03-21T13:47:02.293 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 2e9f1b340eca4ccf93c8af366fb23726-5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:47:02.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2022-03-21T13:47:32.445 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-03-21T13:47:32.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-03-21T13:47:32.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:47:51.115 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2022-03-21T13:48:21.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-03-21T13:48:21.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:48:37.978 DEBUG:teuthology.orchestra.run.smithi002:> true 2022-03-21T13:48:38.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-03-21T13:48:38.342 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.17.0-rc6-ceph-ge35497518a9d"... 2022-03-21T13:48:38.343 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2022-03-21T13:48:38.387 INFO:teuthology.orchestra.run.smithi002.stdout:5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:48:38.387 DEBUG:teuthology.task.kernel:current kernel version is 5.17.0-rc6-ceph-ge35497518a9d vs 5.17.0-rc6-ceph-ge35497518a9d 2022-03-21T13:48:38.387 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-21T13:48:38.388 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-03-21T13:48:39.389 DEBUG:teuthology.parallel:result is None 2022-03-21T13:48:39.390 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-21T13:48:39.398 INFO:teuthology.task.internal:Creating test directory... 2022-03-21T13:48:39.399 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T13:48:39.404 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T13:48:39.407 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T13:48:39.448 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-21T13:48:39.458 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-21T13:48:39.480 INFO:teuthology.task.internal:Creating archive directory... 2022-03-21T13:48:39.480 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T13:48:39.485 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T13:48:39.487 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T13:48:39.662 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-21T13:48:39.674 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-21T13:48:39.674 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T13:48:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:48:39.691 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T13:48:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:48:39.708 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T13:48:39.712 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T13:48:39.734 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T13:48:39.796 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.818 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.830 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.865 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.888 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.909 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T13:48:39.911 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-21T13:48:39.922 INFO:teuthology.task.internal:Configuring sudo... 2022-03-21T13:48:39.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T13:48:39.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T13:48:39.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T13:48:39.996 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-21T13:48:40.020 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-21T13:48:40.020 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T13:48:40.025 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T13:48:40.028 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T13:48:40.065 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.260 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.329 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.399 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:48:40.400 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T13:48:40.471 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.569 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.693 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T13:48:40.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T13:48:40.760 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.802 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T13:48:40.857 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.935 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T13:48:40.981 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T13:48:40.981 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T13:48:41.047 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2022-03-21T13:48:41.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2022-03-21T13:48:41.055 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2022-03-21T13:48:41.104 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-03-21T13:48:41.132 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-03-21T13:48:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-03-21T13:48:41.484 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-21T13:48:41.493 INFO:teuthology.task.internal:Starting timer... 2022-03-21T13:48:41.493 INFO:teuthology.run_tasks:Running task pcp... 2022-03-21T13:48:41.541 INFO:teuthology.run_tasks:Running task selinux... 2022-03-21T13:48:41.571 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-03-21T13:48:41.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2022-03-21T13:48:41.720 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: [ OK ] 2022-03-21T13:48:41.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2022-03-21T13:48:41.781 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: [ OK ] 2022-03-21T13:48:41.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2022-03-21T13:48:41.836 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: [ OK ] 2022-03-21T13:48:41.837 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-21T13:48:41.838 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2022-03-21T13:48:41.884 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2022-03-21T13:48:41.886 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2022-03-21T13:48:41.908 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2022-03-21T13:48:41.909 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2022-03-21T13:48:41.931 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2022-03-21T13:48:41.931 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi079.front.sepia.ceph.com': 'permissive'} 2022-03-21T13:48:41.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-03-21T13:48:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:48:42.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-03-21T13:48:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:48:42.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-03-21T13:48:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T13:48:42.059 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-21T13:48:42.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2022-03-21T13:48:42.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2022-03-21T13:48:42.160 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2022-03-21T13:48:42.191 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-21T13:48:42.202 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-21T13:48:42.209 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-21T13:48:42.263 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-21T13:48:42.284 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-21T13:48:42.285 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi016.front.sepia.ceph.com,smithi079.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-21T13:54:22.309 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi079.front.sepia.ceph.com')] 2022-03-21T13:54:22.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-03-21T13:54:22.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:54:22.420 DEBUG:teuthology.orchestra.run.smithi002:> true 2022-03-21T13:54:22.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2022-03-21T13:54:22.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:54:22.599 DEBUG:teuthology.orchestra.run.smithi016:> true 2022-03-21T13:54:22.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2022-03-21T13:54:22.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T13:54:22.739 DEBUG:teuthology.orchestra.run.smithi079:> true 2022-03-21T13:54:22.806 INFO:teuthology.run_tasks:Running task clock... 2022-03-21T13:54:22.818 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-21T13:54:22.818 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T13:54:22.821 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T13:54:22.826 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T13:54:22.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T13:54:22.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T13:54:22.829 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T13:54:22.863 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-03-21T13:54:22.875 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-03-21T13:54:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-03-21T13:54:22.890 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-03-21T13:54:22.898 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-03-21T13:54:22.920 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2022-03-21T13:54:22.921 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2022-03-21T13:54:22.931 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2022-03-21T13:54:22.932 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2022-03-21T13:54:22.936 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-03-21T13:54:22.946 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-03-21T13:54:22.947 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-03-21T13:54:22.959 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-03-21T13:54:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-03-21T13:54:23.004 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2022-03-21T13:54:23.015 INFO:teuthology.orchestra.run.smithi079.stderr:bash: ntpq: command not found 2022-03-21T13:54:23.018 INFO:teuthology.orchestra.run.smithi079.stdout:210 Number of sources = 4 2022-03-21T13:54:23.018 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-03-21T13:54:23.019 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2022-03-21T13:54:23.019 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.019 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.019 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.020 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.031 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2022-03-21T13:54:23.032 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2022-03-21T13:54:23.034 INFO:teuthology.orchestra.run.smithi016.stdout:210 Number of sources = 4 2022-03-21T13:54:23.035 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-03-21T13:54:23.035 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2022-03-21T13:54:23.035 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.035 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.036 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.036 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.070 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-03-21T13:54:23.108 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-03-21T13:54:23.192 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2022-03-21T13:54:23.197 INFO:teuthology.orchestra.run.smithi002.stdout:210 Number of sources = 4 2022-03-21T13:54:23.197 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-03-21T13:54:23.198 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2022-03-21T13:54:23.198 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.198 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.198 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.199 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-21T13:54:23.200 INFO:teuthology.run_tasks:Running task pexec... 2022-03-21T13:54:23.210 INFO:teuthology.task.pexec:Executing custom commands... 2022-03-21T13:54:23.210 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi002.front.sepia.ceph.com 2022-03-21T13:54:23.211 DEBUG:teuthology.orchestra.run.smithi002:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-03-21T13:54:23.211 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi016.front.sepia.ceph.com 2022-03-21T13:54:23.212 DEBUG:teuthology.orchestra.run.smithi016:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-03-21T13:54:23.212 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi079.front.sepia.ceph.com 2022-03-21T13:54:23.213 DEBUG:teuthology.orchestra.run.smithi079:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-03-21T13:54:23.571 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:23.596 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:23.725 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:24.123 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2022-03-21T13:54:24.232 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2022-03-21T13:54:24.306 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 75 kB/s | 2.4 kB 00:00 2022-03-21T13:54:24.383 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 71 kB/s | 2.8 kB 00:00 2022-03-21T13:54:24.403 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2022-03-21T13:54:24.637 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2022-03-21T13:54:26.386 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2022-03-21T13:54:26.387 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2022-03-21T13:54:26.388 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2022-03-21T13:54:26.388 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2022-03-21T13:54:26.388 INFO:teuthology.orchestra.run.smithi079.stdout:Resetting modules: 2022-03-21T13:54:26.389 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools 2022-03-21T13:54:26.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:54:26.389 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2022-03-21T13:54:26.389 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2022-03-21T13:54:26.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:54:26.666 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2022-03-21T13:54:26.667 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2022-03-21T13:54:26.667 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2022-03-21T13:54:26.667 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2022-03-21T13:54:26.667 INFO:teuthology.orchestra.run.smithi016.stdout:Resetting modules: 2022-03-21T13:54:26.668 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools 2022-03-21T13:54:26.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:54:26.668 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2022-03-21T13:54:26.668 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2022-03-21T13:54:26.669 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:54:27.130 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-03-21T13:54:27.131 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-03-21T13:54:27.131 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2022-03-21T13:54:27.132 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-03-21T13:54:27.132 INFO:teuthology.orchestra.run.smithi002.stdout:Resetting modules: 2022-03-21T13:54:27.132 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools 2022-03-21T13:54:27.132 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:54:27.133 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-03-21T13:54:27.133 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-03-21T13:54:27.133 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:54:39.015 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:52:19 PM UTC. 2022-03-21T13:54:40.226 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:52:25 PM UTC. 2022-03-21T13:54:41.647 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:07 ago on Mon 21 Mar 2022 01:52:54 PM UTC. 2022-03-21T13:54:41.820 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2022-03-21T13:54:42.229 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:42.733 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2022-03-21T13:54:43.239 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2022-03-21T13:54:43.669 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:44.262 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2022-03-21T13:54:44.887 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-03-21T13:54:45.555 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-03-21T13:54:46.259 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2022-03-21T13:54:47.777 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2022-03-21T13:54:49.932 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2022-03-21T13:54:52.009 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2022-03-21T13:54:53.526 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:05 ago on Mon 21 Mar 2022 01:51:06 PM UTC. 2022-03-21T13:54:55.327 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:05 ago on Mon 21 Mar 2022 01:51:08 PM UTC. 2022-03-21T13:54:56.358 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2022-03-21T13:54:56.359 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================================= 2022-03-21T13:54:56.360 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2022-03-21T13:54:56.360 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================================= 2022-03-21T13:54:56.360 INFO:teuthology.orchestra.run.smithi079.stdout:Installing group/module packages: 2022-03-21T13:54:56.361 INFO:teuthology.orchestra.run.smithi079.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-03-21T13:54:56.361 INFO:teuthology.orchestra.run.smithi079.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-03-21T13:54:56.361 INFO:teuthology.orchestra.run.smithi079.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-03-21T13:54:56.361 INFO:teuthology.orchestra.run.smithi079.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-03-21T13:54:56.361 INFO:teuthology.orchestra.run.smithi079.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-03-21T13:54:56.362 INFO:teuthology.orchestra.run.smithi079.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-03-21T13:54:56.362 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2022-03-21T13:54:56.362 INFO:teuthology.orchestra.run.smithi079.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-03-21T13:54:56.362 INFO:teuthology.orchestra.run.smithi079.stdout:Installing module profiles: 2022-03-21T13:54:56.363 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools/common 2022-03-21T13:54:56.363 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:54:56.363 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling module streams: 2022-03-21T13:54:56.364 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools rhel8 2022-03-21T13:54:56.364 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:54:56.364 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2022-03-21T13:54:56.364 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================================= 2022-03-21T13:54:56.365 INFO:teuthology.orchestra.run.smithi079.stdout:Install 7 Packages 2022-03-21T13:54:56.365 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:54:56.366 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 16 M 2022-03-21T13:54:56.366 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 57 M 2022-03-21T13:54:56.366 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2022-03-21T13:54:56.412 INFO:teuthology.orchestra.run.smithi079.stdout:(1/7): yajl-2.1.0-10.el8.x86_64.rpm 814 kB/s | 41 kB 00:00 2022-03-21T13:54:56.419 INFO:teuthology.orchestra.run.smithi079.stdout:(2/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 277 kB/s | 16 kB 00:00 2022-03-21T13:54:56.430 INFO:teuthology.orchestra.run.smithi079.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 2.7 MB/s | 192 kB 00:00 2022-03-21T13:54:56.449 INFO:teuthology.orchestra.run.smithi079.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.7 MB/s | 51 kB 00:00 2022-03-21T13:54:56.464 INFO:teuthology.orchestra.run.smithi079.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 12 MB/s | 410 kB 00:00 2022-03-21T13:54:56.509 INFO:teuthology.orchestra.run.smithi079.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 73 MB/s | 7.0 MB 00:00 2022-03-21T13:54:56.539 INFO:teuthology.orchestra.run.smithi079.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 91 MB/s | 8.2 MB 00:00 2022-03-21T13:54:56.540 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:54:56.540 INFO:teuthology.orchestra.run.smithi079.stdout:Total 89 MB/s | 16 MB 00:00 2022-03-21T13:54:56.545 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2022-03-21T13:54:56.561 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2022-03-21T13:54:56.562 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2022-03-21T13:54:56.726 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2022-03-21T13:54:56.731 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2022-03-21T13:54:57.141 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2022-03-21T13:54:57.340 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:54:58.167 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:05 ago on Mon 21 Mar 2022 01:51:29 PM UTC. 2022-03-21T13:54:58.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-03-21T13:54:58.330 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2022-03-21T13:54:58.331 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================= 2022-03-21T13:54:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2022-03-21T13:54:58.331 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================= 2022-03-21T13:54:58.332 INFO:teuthology.orchestra.run.smithi016.stdout:Installing group/module packages: 2022-03-21T13:54:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-03-21T13:54:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-03-21T13:54:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-03-21T13:54:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-03-21T13:54:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-03-21T13:54:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-03-21T13:54:58.333 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2022-03-21T13:54:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-03-21T13:54:58.334 INFO:teuthology.orchestra.run.smithi016.stdout:Installing module profiles: 2022-03-21T13:54:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools/common 2022-03-21T13:54:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:54:58.334 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling module streams: 2022-03-21T13:54:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools rhel8 2022-03-21T13:54:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:54:58.335 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2022-03-21T13:54:58.335 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================= 2022-03-21T13:54:58.335 INFO:teuthology.orchestra.run.smithi016.stdout:Install 7 Packages 2022-03-21T13:54:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:54:58.336 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 16 M 2022-03-21T13:54:58.337 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 57 M 2022-03-21T13:54:58.337 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2022-03-21T13:54:58.404 INFO:teuthology.orchestra.run.smithi016.stdout:(1/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 223 kB/s | 16 kB 00:00 2022-03-21T13:54:58.405 INFO:teuthology.orchestra.run.smithi016.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 570 kB/s | 41 kB 00:00 2022-03-21T13:54:58.421 INFO:teuthology.orchestra.run.smithi016.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 2.1 MB/s | 192 kB 00:00 2022-03-21T13:54:58.445 INFO:teuthology.orchestra.run.smithi016.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.2 MB/s | 51 kB 00:00 2022-03-21T13:54:58.472 INFO:teuthology.orchestra.run.smithi016.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 8.0 MB/s | 410 kB 00:00 2022-03-21T13:54:58.480 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-03-21T13:54:58.521 INFO:teuthology.orchestra.run.smithi016.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 60 MB/s | 7.0 MB 00:00 2022-03-21T13:54:58.555 INFO:teuthology.orchestra.run.smithi016.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 75 MB/s | 8.2 MB 00:00 2022-03-21T13:54:58.556 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:54:58.556 INFO:teuthology.orchestra.run.smithi016.stdout:Total 71 MB/s | 16 MB 00:00 2022-03-21T13:54:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2022-03-21T13:54:58.581 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2022-03-21T13:54:58.581 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2022-03-21T13:54:58.709 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-03-21T13:54:58.760 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2022-03-21T13:54:58.764 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2022-03-21T13:54:59.212 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2022-03-21T13:54:59.445 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:54:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:54:59.710 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-03-21T13:54:59.914 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:00.147 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:00.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:55:00.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-03-21T13:55:00.148 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-03-21T13:55:00.148 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-03-21T13:55:00.148 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:55:00.148 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-03-21T13:55:00.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-03-21T13:55:00.605 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-03-21T13:55:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-03-21T13:55:01.328 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-03-21T13:55:01.330 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-03-21T13:55:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2022-03-21T13:55:01.330 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-03-21T13:55:01.331 INFO:teuthology.orchestra.run.smithi002.stdout:Installing group/module packages: 2022-03-21T13:55:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-03-21T13:55:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-03-21T13:55:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-03-21T13:55:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-03-21T13:55:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-03-21T13:55:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-03-21T13:55:01.332 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2022-03-21T13:55:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-03-21T13:55:01.333 INFO:teuthology.orchestra.run.smithi002.stdout:Installing module profiles: 2022-03-21T13:55:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools/common 2022-03-21T13:55:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:01.333 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2022-03-21T13:55:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools rhel8 2022-03-21T13:55:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:01.334 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-03-21T13:55:01.334 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-03-21T13:55:01.334 INFO:teuthology.orchestra.run.smithi002.stdout:Install 7 Packages 2022-03-21T13:55:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:01.336 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 16 M 2022-03-21T13:55:01.337 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 57 M 2022-03-21T13:55:01.337 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2022-03-21T13:55:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:(1/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 215 kB/s | 16 kB 00:00 2022-03-21T13:55:01.407 INFO:teuthology.orchestra.run.smithi002.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2022-03-21T13:55:01.440 INFO:teuthology.orchestra.run.smithi002.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 1.7 MB/s | 192 kB 00:00 2022-03-21T13:55:01.452 INFO:teuthology.orchestra.run.smithi002.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.1 MB/s | 51 kB 00:00 2022-03-21T13:55:01.483 INFO:teuthology.orchestra.run.smithi002.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 9.3 MB/s | 410 kB 00:00 2022-03-21T13:55:01.604 INFO:teuthology.orchestra.run.smithi002.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 36 MB/s | 7.0 MB 00:00 2022-03-21T13:55:01.652 INFO:teuthology.orchestra.run.smithi002.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 41 MB/s | 8.2 MB 00:00 2022-03-21T13:55:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:55:01.654 INFO:teuthology.orchestra.run.smithi002.stdout:Total 49 MB/s | 16 MB 00:00 2022-03-21T13:55:01.666 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2022-03-21T13:55:01.689 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2022-03-21T13:55:01.690 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2022-03-21T13:55:01.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:55:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-03-21T13:55:02.098 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:02.244 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2022-03-21T13:55:02.249 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2022-03-21T13:55:02.340 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:02.340 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:55:02.340 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-03-21T13:55:02.340 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-03-21T13:55:02.341 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-03-21T13:55:02.341 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:55:02.341 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-03-21T13:55:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2022-03-21T13:55:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:55:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-03-21T13:55:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-03-21T13:55:04.469 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-03-21T13:55:05.437 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:55:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-03-21T13:55:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:06.271 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-03-21T13:55:06.272 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-03-21T13:55:06.272 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-03-21T13:55:06.272 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-03-21T13:55:06.273 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-03-21T13:55:06.273 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-03-21T13:55:06.273 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-03-21T13:55:13.374 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-03-21T13:55:13.374 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:54:32 PM UTC. 2022-03-21T13:55:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-03-21T13:55:16.070 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:07 ago on Mon 21 Mar 2022 01:54:33 PM UTC. 2022-03-21T13:55:16.459 INFO:teuthology.orchestra.run.smithi079.stdout:Installed products updated. 2022-03-21T13:55:16.574 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:55:16.574 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2022-03-21T13:55:16.574 INFO:teuthology.orchestra.run.smithi079.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-03-21T13:55:16.574 INFO:teuthology.orchestra.run.smithi079.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-03-21T13:55:16.575 INFO:teuthology.orchestra.run.smithi079.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-03-21T13:55:16.575 INFO:teuthology.orchestra.run.smithi079.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-03-21T13:55:16.575 INFO:teuthology.orchestra.run.smithi079.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:16.575 INFO:teuthology.orchestra.run.smithi079.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:16.575 INFO:teuthology.orchestra.run.smithi079.stdout: yajl-2.1.0-10.el8.x86_64 2022-03-21T13:55:16.576 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:55:16.576 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2022-03-21T13:55:16.722 DEBUG:teuthology.parallel:result is None 2022-03-21T13:55:19.228 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2022-03-21T13:55:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:55:19.345 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2022-03-21T13:55:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-03-21T13:55:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-03-21T13:55:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-03-21T13:55:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-03-21T13:55:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: yajl-2.1.0-10.el8.x86_64 2022-03-21T13:55:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:55:19.347 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2022-03-21T13:55:19.514 DEBUG:teuthology.parallel:result is None 2022-03-21T13:55:21.099 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-03-21T13:55:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:54:34 PM UTC. 2022-03-21T13:55:24.657 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2022-03-21T13:55:24.769 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:24.770 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2022-03-21T13:55:24.770 INFO:teuthology.orchestra.run.smithi002.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-03-21T13:55:24.770 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-03-21T13:55:24.770 INFO:teuthology.orchestra.run.smithi002.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-03-21T13:55:24.770 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-03-21T13:55:24.771 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:24.771 INFO:teuthology.orchestra.run.smithi002.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-03-21T13:55:24.771 INFO:teuthology.orchestra.run.smithi002.stdout: yajl-2.1.0-10.el8.x86_64 2022-03-21T13:55:24.771 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:24.772 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-03-21T13:55:25.050 DEBUG:teuthology.parallel:result is None 2022-03-21T13:55:25.050 INFO:teuthology.run_tasks:Running task install... 2022-03-21T13:55:25.059 DEBUG:teuthology.task.install:project ceph 2022-03-21T13:55:25.060 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409'}} 2022-03-21T13:55:25.060 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409'} 2022-03-21T13:55:25.060 INFO:teuthology.task.install:Using flavor: default 2022-03-21T13:55:25.066 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-21T13:55:25.067 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-03-21T13:55:25.067 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-03-21T13:55:25.190 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-03-21T13:55:25.194 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2022-03-21T13:55:25.195 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409', 'tag': None, 'wait_for_package': False} 2022-03-21T13:55:25.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-03-21T13:55:25.245 INFO:teuthology.orchestra.run.smithi016.stdout:RedHatEnterprise 2022-03-21T13:55:25.246 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.247 DEBUG:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2022-03-21T13:55:25.247 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409', 'tag': None, 'wait_for_package': False} 2022-03-21T13:55:25.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2022-03-21T13:55:25.289 INFO:teuthology.orchestra.run.smithi016.stdout:RedHatEnterprise 2022-03-21T13:55:25.291 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:55:25.299 INFO:teuthology.orchestra.run.smithi079.stdout:RedHatEnterprise 2022-03-21T13:55:25.301 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.301 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409', 'tag': None, 'wait_for_package': False} 2022-03-21T13:55:25.302 DEBUG:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2022-03-21T13:55:25.326 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-03-21T13:55:25.327 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:55:25.342 INFO:teuthology.orchestra.run.smithi079.stdout:RedHatEnterprise 2022-03-21T13:55:25.343 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:55:25.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:55:25.450 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/ 2022-03-21T13:55:25.451 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11208.gd3799b8c 2022-03-21T13:55:25.473 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/ 2022-03-21T13:55:25.473 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11208.gd3799b8c 2022-03-21T13:55:25.508 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/ 2022-03-21T13:55:25.509 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11208.gd3799b8c 2022-03-21T13:55:25.582 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-03-21T13:55:25.583 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T13:55:25.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-03-21T13:55:25.588 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-03-21T13:55:25.588 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:55:25.589 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-03-21T13:55:25.610 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-03-21T13:55:25.610 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d3799b8cfa5046465710eadeea2bed7bb4681409/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-03-21T13:55:25.625 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220319-163539/d3799b8cfa5046465710eadeea2bed7bb4681409/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-03-21T13:55:25.625 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T13:55:25.626 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-03-21T13:55:25.652 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-03-21T13:55:25.652 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d3799b8cfa5046465710eadeea2bed7bb4681409/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-03-21T13:55:25.659 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-03-21T13:55:25.660 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d3799b8cfa5046465710eadeea2bed7bb4681409/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-03-21T13:55:25.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-03-21T13:55:25.719 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-03-21T13:55:25.757 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-03-21T13:55:25.772 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-03-21T13:55:25.796 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-03-21T13:55:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2022-03-21T13:55:25.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2022-03-21T13:55:25.865 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2022-03-21T13:55:25.868 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2022-03-21T13:55:25.874 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-03-21T13:55:25.946 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2022-03-21T13:55:25.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2022-03-21T13:55:26.207 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:26.225 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:26.420 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:26.528 INFO:teuthology.orchestra.run.smithi079.stdout:54 files removed 2022-03-21T13:55:26.570 INFO:teuthology.orchestra.run.smithi016.stdout:54 files removed 2022-03-21T13:55:26.573 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-03-21T13:55:26.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-03-21T13:55:26.845 INFO:teuthology.orchestra.run.smithi002.stdout:54 files removed 2022-03-21T13:55:26.885 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:26.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-03-21T13:55:26.948 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:27.377 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-03-21T13:55:27.600 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 246 kB/s | 75 kB 00:00 2022-03-21T13:55:27.670 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 273 kB/s | 75 kB 00:00 2022-03-21T13:55:27.834 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 46 kB/s | 9.1 kB 00:00 2022-03-21T13:55:27.906 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 46 kB/s | 9.1 kB 00:00 2022-03-21T13:55:28.057 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2022-03-21T13:55:28.139 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2022-03-21T13:55:28.157 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 309 kB/s | 75 kB 00:00 2022-03-21T13:55:28.161 INFO:teuthology.orchestra.run.smithi079.stdout:Copr repo for python3-asyncssh owned by ceph 42 kB/s | 3.5 kB 00:00 2022-03-21T13:55:28.243 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 43 kB/s | 3.5 kB 00:00 2022-03-21T13:55:28.399 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 45 kB/s | 9.1 kB 00:00 2022-03-21T13:55:28.631 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2022-03-21T13:55:28.733 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 45 kB/s | 3.5 kB 00:00 2022-03-21T13:55:29.052 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 13 MB/s | 11 MB 00:00 2022-03-21T13:55:29.097 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-03-21T13:55:29.758 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:00 2022-03-21T13:55:31.345 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2022-03-21T13:55:31.571 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2022-03-21T13:55:31.698 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 33 MB 00:00 2022-03-21T13:55:31.965 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 117 MB/s | 38 MB 00:00 2022-03-21T13:55:32.415 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 3.7 MB/s | 24 kB 00:00 2022-03-21T13:55:33.009 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 63 MB/s | 33 MB 00:00 2022-03-21T13:55:36.817 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 141 MB/s | 38 MB 00:00 2022-03-21T13:55:37.782 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 MB/s | 33 MB 00:00 2022-03-21T13:55:39.268 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 68 MB/s | 38 MB 00:00 2022-03-21T13:55:45.077 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:45.077 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:45.088 INFO:teuthology.orchestra.run.smithi079.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-03-21T13:55:45.088 INFO:teuthology.orchestra.run.smithi079.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-03-21T13:55:45.089 INFO:teuthology.orchestra.run.smithi079.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-03-21T13:55:45.090 INFO:teuthology.orchestra.run.smithi079.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-03-21T13:55:45.092 INFO:teuthology.orchestra.run.smithi079.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-03-21T13:55:45.094 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-03-21T13:55:45.095 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-03-21T13:55:45.560 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2022-03-21T13:55:45.572 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================================================== 2022-03-21T13:55:45.572 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2022-03-21T13:55:45.572 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================================================== 2022-03-21T13:55:45.573 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2022-03-21T13:55:45.573 INFO:teuthology.orchestra.run.smithi079.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-03-21T13:55:45.573 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 6.5 k 2022-03-21T13:55:45.573 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 5.3 M 2022-03-21T13:55:45.574 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 881 k 2022-03-21T13:55:45.574 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 169 k 2022-03-21T13:55:45.574 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 1.5 M 2022-03-21T13:55:45.574 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 104 k 2022-03-21T13:55:45.575 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 1.6 M 2022-03-21T13:55:45.575 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 7.4 M 2022-03-21T13:55:45.575 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 47 k 2022-03-21T13:55:45.575 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 9.3 M 2022-03-21T13:55:45.575 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 M 2022-03-21T13:55:45.576 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 257 k 2022-03-21T13:55:45.576 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 71 k 2022-03-21T13:55:45.576 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 273 k 2022-03-21T13:55:45.576 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 14 k 2022-03-21T13:55:45.577 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-03-21T13:55:45.577 INFO:teuthology.orchestra.run.smithi079.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-03-21T13:55:45.577 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-03-21T13:55:45.577 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 25 k 2022-03-21T13:55:45.578 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 795 k 2022-03-21T13:55:45.578 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 125 k 2022-03-21T13:55:45.578 INFO:teuthology.orchestra.run.smithi079.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-03-21T13:55:45.578 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-03-21T13:55:45.579 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-03-21T13:55:45.579 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 179 k 2022-03-21T13:55:45.579 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 366 k 2022-03-21T13:55:45.579 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 345 k 2022-03-21T13:55:45.580 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 109 k 2022-03-21T13:55:45.580 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 94 k 2022-03-21T13:55:45.580 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.8 M 2022-03-21T13:55:45.580 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 174 k 2022-03-21T13:55:45.580 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-03-21T13:55:45.581 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2022-03-21T13:55:45.581 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-03-21T13:55:45.581 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-03-21T13:55:45.581 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-03-21T13:55:45.582 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-03-21T13:55:45.582 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.4 M 2022-03-21T13:55:45.582 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:45.582 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2022-03-21T13:55:45.583 INFO:teuthology.orchestra.run.smithi079.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-03-21T13:55:45.583 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 20 M 2022-03-21T13:55:45.583 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 23 k 2022-03-21T13:55:45.583 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 2.3 M 2022-03-21T13:55:45.584 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 226 k 2022-03-21T13:55:45.584 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:45.584 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 17 M 2022-03-21T13:55:45.584 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 15 k 2022-03-21T13:55:45.584 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 24 k 2022-03-21T13:55:45.585 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-03-21T13:55:45.585 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-03-21T13:55:45.585 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-03-21T13:55:45.585 INFO:teuthology.orchestra.run.smithi079.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-03-21T13:55:45.586 INFO:teuthology.orchestra.run.smithi079.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:45.586 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 170 k 2022-03-21T13:55:45.586 INFO:teuthology.orchestra.run.smithi079.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-03-21T13:55:45.586 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-03-21T13:55:45.587 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-03-21T13:55:45.587 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-03-21T13:55:45.587 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-03-21T13:55:45.587 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 483 k 2022-03-21T13:55:45.588 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-03-21T13:55:45.588 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 4.0 M 2022-03-21T13:55:45.588 INFO:teuthology.orchestra.run.smithi079.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-03-21T13:55:45.588 INFO:teuthology.orchestra.run.smithi079.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-03-21T13:55:45.589 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-03-21T13:55:45.589 INFO:teuthology.orchestra.run.smithi079.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-03-21T13:55:45.589 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-03-21T13:55:45.589 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-03-21T13:55:45.590 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-03-21T13:55:45.590 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-03-21T13:55:45.590 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-03-21T13:55:45.590 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-03-21T13:55:45.591 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-03-21T13:55:45.591 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-03-21T13:55:45.591 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-03-21T13:55:45.591 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-03-21T13:55:45.591 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-03-21T13:55:45.592 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-03-21T13:55:45.592 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 k 2022-03-21T13:55:45.592 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 105 k 2022-03-21T13:55:45.592 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-03-21T13:55:45.593 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-03-21T13:55:45.593 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-03-21T13:55:45.593 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-03-21T13:55:45.593 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:45.594 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-03-21T13:55:45.594 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-03-21T13:55:45.594 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-03-21T13:55:45.594 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-03-21T13:55:45.595 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-03-21T13:55:45.595 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-03-21T13:55:45.595 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:45.595 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-03-21T13:55:45.596 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-03-21T13:55:45.596 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-03-21T13:55:45.596 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-03-21T13:55:45.596 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-03-21T13:55:45.596 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-03-21T13:55:45.597 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-03-21T13:55:45.597 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-03-21T13:55:45.597 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-03-21T13:55:45.597 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-03-21T13:55:45.598 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-03-21T13:55:45.598 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-03-21T13:55:45.598 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-03-21T13:55:45.598 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-03-21T13:55:45.599 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-03-21T13:55:45.599 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-03-21T13:55:45.599 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-03-21T13:55:45.599 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-03-21T13:55:45.600 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:45.600 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-03-21T13:55:45.600 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-03-21T13:55:45.600 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-03-21T13:55:45.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-03-21T13:55:45.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-03-21T13:55:45.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-03-21T13:55:45.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-03-21T13:55:45.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-03-21T13:55:45.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-03-21T13:55:45.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-03-21T13:55:45.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-03-21T13:55:45.603 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-03-21T13:55:45.603 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-03-21T13:55:45.603 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-03-21T13:55:45.603 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-03-21T13:55:45.603 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-03-21T13:55:45.604 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:55:45.604 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2022-03-21T13:55:45.604 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================================================== 2022-03-21T13:55:45.604 INFO:teuthology.orchestra.run.smithi079.stdout:Install 121 Packages 2022-03-21T13:55:45.605 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 6 Packages 2022-03-21T13:55:45.605 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:55:45.606 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 187 M 2022-03-21T13:55:45.606 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2022-03-21T13:55:45.688 INFO:teuthology.orchestra.run.smithi079.stdout:(1/127): ceph-17.0.0-11208.gd3799b8c.el8.x86_64 64 kB/s | 6.5 kB 00:00 2022-03-21T13:55:45.864 INFO:teuthology.orchestra.run.smithi079.stdout:(2/127): ceph-fuse-17.0.0-11208.gd3799b8c.el8.x 4.9 MB/s | 881 kB 00:00 2022-03-21T13:55:45.892 INFO:teuthology.orchestra.run.smithi079.stdout:(3/127): ceph-immutable-object-cache-17.0.0-112 5.9 MB/s | 169 kB 00:00 2022-03-21T13:55:46.178 INFO:teuthology.orchestra.run.smithi079.stdout:(4/127): ceph-mds-17.0.0-11208.gd3799b8c.el8.x8 8.0 MB/s | 2.3 MB 00:00 2022-03-21T13:55:46.319 INFO:teuthology.orchestra.run.smithi079.stdout:(5/127): ceph-base-17.0.0-11208.gd3799b8c.el8.x 7.2 MB/s | 5.3 MB 00:00 2022-03-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi079.stdout:(6/127): ceph-mgr-17.0.0-11208.gd3799b8c.el8.x8 6.5 MB/s | 1.5 MB 00:00 2022-03-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-03-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-03-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-03-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-03-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-03-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-03-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-03-21T13:55:46.742 INFO:teuthology.orchestra.run.smithi079.stdout:(7/127): ceph-mon-17.0.0-11208.gd3799b8c.el8.x8 9.2 MB/s | 3.9 MB 00:00 2022-03-21T13:55:46.874 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2022-03-21T13:55:46.887 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2022-03-21T13:55:46.887 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2022-03-21T13:55:46.888 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2022-03-21T13:55:46.888 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2022-03-21T13:55:46.888 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-03-21T13:55:46.888 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 6.5 k 2022-03-21T13:55:46.889 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 5.3 M 2022-03-21T13:55:46.889 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 881 k 2022-03-21T13:55:46.889 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 169 k 2022-03-21T13:55:46.889 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 1.5 M 2022-03-21T13:55:46.889 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 104 k 2022-03-21T13:55:46.890 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 1.6 M 2022-03-21T13:55:46.890 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 7.4 M 2022-03-21T13:55:46.890 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 47 k 2022-03-21T13:55:46.890 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 9.3 M 2022-03-21T13:55:46.891 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 M 2022-03-21T13:55:46.891 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 257 k 2022-03-21T13:55:46.896 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 71 k 2022-03-21T13:55:46.897 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 273 k 2022-03-21T13:55:46.897 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 14 k 2022-03-21T13:55:46.897 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-03-21T13:55:46.897 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-03-21T13:55:46.898 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-03-21T13:55:46.898 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 25 k 2022-03-21T13:55:46.898 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 795 k 2022-03-21T13:55:46.898 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 125 k 2022-03-21T13:55:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-03-21T13:55:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-03-21T13:55:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-03-21T13:55:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 179 k 2022-03-21T13:55:46.899 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 366 k 2022-03-21T13:55:46.900 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 345 k 2022-03-21T13:55:46.900 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 109 k 2022-03-21T13:55:46.900 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 94 k 2022-03-21T13:55:46.900 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.8 M 2022-03-21T13:55:46.900 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 174 k 2022-03-21T13:55:46.901 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-03-21T13:55:46.901 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2022-03-21T13:55:46.901 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-03-21T13:55:46.901 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-03-21T13:55:46.902 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-03-21T13:55:46.902 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-03-21T13:55:46.902 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.4 M 2022-03-21T13:55:46.902 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:46.902 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2022-03-21T13:55:46.903 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-03-21T13:55:46.903 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 20 M 2022-03-21T13:55:46.903 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 23 k 2022-03-21T13:55:46.903 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 2.3 M 2022-03-21T13:55:46.904 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 226 k 2022-03-21T13:55:46.904 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:46.904 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 17 M 2022-03-21T13:55:46.904 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 15 k 2022-03-21T13:55:46.904 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 24 k 2022-03-21T13:55:46.905 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-03-21T13:55:46.905 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-03-21T13:55:46.905 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-03-21T13:55:46.905 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-03-21T13:55:46.906 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:46.906 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 170 k 2022-03-21T13:55:46.906 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-03-21T13:55:46.906 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-03-21T13:55:46.907 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-03-21T13:55:46.907 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-03-21T13:55:46.907 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-03-21T13:55:46.907 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 483 k 2022-03-21T13:55:46.907 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-03-21T13:55:46.908 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 4.0 M 2022-03-21T13:55:46.908 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-03-21T13:55:46.908 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-03-21T13:55:46.908 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-03-21T13:55:46.909 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-03-21T13:55:46.909 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-03-21T13:55:46.909 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-03-21T13:55:46.909 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-03-21T13:55:46.910 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-03-21T13:55:46.910 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-03-21T13:55:46.910 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-03-21T13:55:46.910 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-03-21T13:55:46.910 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-03-21T13:55:46.911 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-03-21T13:55:46.911 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-03-21T13:55:46.911 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-03-21T13:55:46.911 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-03-21T13:55:46.911 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 k 2022-03-21T13:55:46.912 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 105 k 2022-03-21T13:55:46.912 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-03-21T13:55:46.912 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-03-21T13:55:46.912 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-03-21T13:55:46.913 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-03-21T13:55:46.913 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:46.913 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-03-21T13:55:46.913 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-03-21T13:55:46.914 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-03-21T13:55:46.914 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-03-21T13:55:46.914 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-03-21T13:55:46.914 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-03-21T13:55:46.914 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:46.915 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-03-21T13:55:46.915 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-03-21T13:55:46.915 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-03-21T13:55:46.915 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-03-21T13:55:46.915 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-03-21T13:55:46.916 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-03-21T13:55:46.916 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-03-21T13:55:46.916 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-03-21T13:55:46.916 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-03-21T13:55:46.917 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-03-21T13:55:46.917 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-03-21T13:55:46.917 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-03-21T13:55:46.917 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-03-21T13:55:46.918 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-03-21T13:55:46.918 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-03-21T13:55:46.918 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-03-21T13:55:46.918 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-03-21T13:55:46.918 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-03-21T13:55:46.919 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:46.919 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-03-21T13:55:46.919 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-03-21T13:55:46.919 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-03-21T13:55:46.920 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-03-21T13:55:46.920 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-03-21T13:55:46.920 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-03-21T13:55:46.920 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-03-21T13:55:46.921 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-03-21T13:55:46.921 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-03-21T13:55:46.921 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-03-21T13:55:46.921 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-03-21T13:55:46.921 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-03-21T13:55:46.922 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-03-21T13:55:46.922 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-03-21T13:55:46.922 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-03-21T13:55:46.922 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-03-21T13:55:46.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:55:46.923 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2022-03-21T13:55:46.923 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2022-03-21T13:55:46.923 INFO:teuthology.orchestra.run.smithi016.stdout:Install 121 Packages 2022-03-21T13:55:46.923 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 6 Packages 2022-03-21T13:55:46.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:55:46.925 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 187 M 2022-03-21T13:55:46.925 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2022-03-21T13:55:47.220 INFO:teuthology.orchestra.run.smithi016.stdout:(1/127): ceph-17.0.0-11208.gd3799b8c.el8.x86_64 20 kB/s | 6.5 kB 00:00 2022-03-21T13:55:47.797 INFO:teuthology.orchestra.run.smithi016.stdout:(2/127): ceph-fuse-17.0.0-11208.gd3799b8c.el8.x 1.5 MB/s | 881 kB 00:00 2022-03-21T13:55:47.865 INFO:teuthology.orchestra.run.smithi016.stdout:(3/127): ceph-immutable-object-cache-17.0.0-112 2.4 MB/s | 169 kB 00:00 2022-03-21T13:55:47.919 INFO:teuthology.orchestra.run.smithi079.stdout:(8/127): ceph-common-17.0.0-11208.gd3799b8c.el8 8.7 MB/s | 20 MB 00:02 2022-03-21T13:55:48.034 INFO:teuthology.orchestra.run.smithi079.stdout:(9/127): ceph-radosgw-17.0.0-11208.gd3799b8c.el 7.2 MB/s | 9.3 MB 00:01 2022-03-21T13:55:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:(10/127): ceph-selinux-17.0.0-11208.gd3799b8c.e 209 kB/s | 24 kB 00:00 2022-03-21T13:55:48.103 INFO:teuthology.orchestra.run.smithi079.stdout:(11/127): cephfs-mirror-17.0.0-11208.gd3799b8c. 4.0 MB/s | 273 kB 00:00 2022-03-21T13:55:48.141 INFO:teuthology.orchestra.run.smithi079.stdout:(12/127): libcephfs-devel-17.0.0-11208.gd3799b8 649 kB/s | 25 kB 00:00 2022-03-21T13:55:48.265 INFO:teuthology.orchestra.run.smithi079.stdout:(13/127): libcephfs2-17.0.0-11208.gd3799b8c.el8 6.3 MB/s | 795 kB 00:00 2022-03-21T13:55:48.318 INFO:teuthology.orchestra.run.smithi079.stdout:(14/127): libcephsqlite-17.0.0-11208.gd3799b8c. 3.2 MB/s | 170 kB 00:00 2022-03-21T13:55:48.377 INFO:teuthology.orchestra.run.smithi079.stdout:(15/127): librados-devel-17.0.0-11208.gd3799b8c 2.1 MB/s | 125 kB 00:00 2022-03-21T13:55:48.417 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:48.417 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-03-21T13:55:48.429 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-03-21T13:55:48.430 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-03-21T13:55:48.431 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-03-21T13:55:48.432 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-03-21T13:55:48.433 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-03-21T13:55:48.436 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-03-21T13:55:48.437 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-03-21T13:55:48.439 INFO:teuthology.orchestra.run.smithi079.stdout:(16/127): libradosstriper1-17.0.0-11208.gd3799b 7.6 MB/s | 483 kB 00:00 2022-03-21T13:55:48.491 INFO:teuthology.orchestra.run.smithi016.stdout:(4/127): ceph-mds-17.0.0-11208.gd3799b8c.el8.x8 3.7 MB/s | 2.3 MB 00:00 2022-03-21T13:55:48.723 INFO:teuthology.orchestra.run.smithi079.stdout:(17/127): ceph-osd-17.0.0-11208.gd3799b8c.el8.x 7.3 MB/s | 17 MB 00:02 2022-03-21T13:55:48.778 INFO:teuthology.orchestra.run.smithi079.stdout:(18/127): python3-ceph-argparse-17.0.0-11208.gd 820 kB/s | 45 kB 00:00 2022-03-21T13:55:48.836 INFO:teuthology.orchestra.run.smithi079.stdout:(19/127): python3-ceph-common-17.0.0-11208.gd37 1.8 MB/s | 105 kB 00:00 2022-03-21T13:55:48.884 INFO:teuthology.orchestra.run.smithi079.stdout:(20/127): python3-cephfs-17.0.0-11208.gd3799b8c 3.7 MB/s | 179 kB 00:00 2022-03-21T13:55:48.925 INFO:teuthology.orchestra.run.smithi079.stdout:(21/127): librgw2-17.0.0-11208.gd3799b8c.el8.x8 8.2 MB/s | 4.0 MB 00:00 2022-03-21T13:55:48.942 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-03-21T13:55:48.946 INFO:teuthology.orchestra.run.smithi079.stdout:(22/127): python3-rados-17.0.0-11208.gd3799b8c. 5.8 MB/s | 366 kB 00:00 2022-03-21T13:55:48.956 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-03-21T13:55:48.956 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2022-03-21T13:55:48.956 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-03-21T13:55:48.957 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2022-03-21T13:55:48.957 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-03-21T13:55:48.957 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 6.5 k 2022-03-21T13:55:48.957 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 5.3 M 2022-03-21T13:55:48.958 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 881 k 2022-03-21T13:55:48.958 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 169 k 2022-03-21T13:55:48.958 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 1.5 M 2022-03-21T13:55:48.958 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 104 k 2022-03-21T13:55:48.958 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 1.6 M 2022-03-21T13:55:48.959 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 7.4 M 2022-03-21T13:55:48.959 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 47 k 2022-03-21T13:55:48.959 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 9.3 M 2022-03-21T13:55:48.959 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 M 2022-03-21T13:55:48.960 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 257 k 2022-03-21T13:55:48.960 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 71 k 2022-03-21T13:55:48.960 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 273 k 2022-03-21T13:55:48.960 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 14 k 2022-03-21T13:55:48.960 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-03-21T13:55:48.961 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-03-21T13:55:48.961 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-03-21T13:55:48.961 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 25 k 2022-03-21T13:55:48.961 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 795 k 2022-03-21T13:55:48.962 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 125 k 2022-03-21T13:55:48.962 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-03-21T13:55:48.962 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-03-21T13:55:48.962 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-03-21T13:55:48.962 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 179 k 2022-03-21T13:55:48.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 366 k 2022-03-21T13:55:48.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 345 k 2022-03-21T13:55:48.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 109 k 2022-03-21T13:55:48.963 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 94 k 2022-03-21T13:55:48.964 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.8 M 2022-03-21T13:55:48.964 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 174 k 2022-03-21T13:55:48.964 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-03-21T13:55:48.964 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2022-03-21T13:55:48.964 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-03-21T13:55:48.965 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-03-21T13:55:48.965 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-03-21T13:55:48.965 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-03-21T13:55:48.965 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.4 M 2022-03-21T13:55:48.966 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:48.966 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2022-03-21T13:55:48.966 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-03-21T13:55:48.966 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 20 M 2022-03-21T13:55:48.967 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 23 k 2022-03-21T13:55:48.967 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 2.3 M 2022-03-21T13:55:48.967 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 226 k 2022-03-21T13:55:48.967 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 3.9 M 2022-03-21T13:55:48.968 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 17 M 2022-03-21T13:55:48.968 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11208.gd3799b8c.el8 ceph-noarch 15 k 2022-03-21T13:55:48.968 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 24 k 2022-03-21T13:55:48.968 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-03-21T13:55:48.968 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-03-21T13:55:48.969 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-03-21T13:55:48.969 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-03-21T13:55:48.969 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:48.969 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 170 k 2022-03-21T13:55:48.970 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-03-21T13:55:48.970 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-03-21T13:55:48.970 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-03-21T13:55:48.970 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-03-21T13:55:48.970 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-03-21T13:55:48.971 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 483 k 2022-03-21T13:55:48.971 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-03-21T13:55:48.971 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 4.0 M 2022-03-21T13:55:48.971 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-03-21T13:55:48.971 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-03-21T13:55:48.972 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-03-21T13:55:48.972 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-03-21T13:55:48.972 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-03-21T13:55:48.972 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-03-21T13:55:48.973 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-03-21T13:55:48.973 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-03-21T13:55:48.973 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-03-21T13:55:48.973 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-03-21T13:55:48.974 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-03-21T13:55:48.974 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-03-21T13:55:48.974 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-03-21T13:55:48.974 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-03-21T13:55:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-03-21T13:55:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-03-21T13:55:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 45 k 2022-03-21T13:55:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:17.0.0-11208.gd3799b8c.el8 ceph 105 k 2022-03-21T13:55:48.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-03-21T13:55:48.976 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-03-21T13:55:48.976 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-03-21T13:55:48.976 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-03-21T13:55:48.976 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-03-21T13:55:48.977 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-03-21T13:55:48.977 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-03-21T13:55:48.977 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-03-21T13:55:48.977 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-03-21T13:55:48.977 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-03-21T13:55:48.978 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-03-21T13:55:48.978 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:48.978 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-03-21T13:55:48.978 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-03-21T13:55:48.979 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-03-21T13:55:48.979 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-03-21T13:55:48.979 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-03-21T13:55:48.979 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-03-21T13:55:48.979 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-03-21T13:55:48.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-03-21T13:55:48.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-03-21T13:55:48.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-03-21T13:55:48.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-03-21T13:55:48.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-03-21T13:55:48.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-03-21T13:55:48.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-03-21T13:55:48.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-03-21T13:55:48.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-03-21T13:55:48.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-03-21T13:55:48.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-03-21T13:55:48.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-03-21T13:55:48.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-03-21T13:55:48.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-03-21T13:55:48.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-03-21T13:55:48.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-03-21T13:55:48.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-03-21T13:55:48.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-03-21T13:55:48.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-03-21T13:55:48.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-03-21T13:55:48.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-03-21T13:55:48.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-03-21T13:55:48.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-03-21T13:55:48.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-03-21T13:55:48.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-03-21T13:55:48.985 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-03-21T13:55:48.985 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-03-21T13:55:48.986 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-03-21T13:55:48.986 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:48.986 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-03-21T13:55:48.986 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-03-21T13:55:48.987 INFO:teuthology.orchestra.run.smithi002.stdout:Install 121 Packages 2022-03-21T13:55:48.987 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 6 Packages 2022-03-21T13:55:48.987 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:55:48.988 INFO:teuthology.orchestra.run.smithi079.stdout:(23/127): python3-rbd-17.0.0-11208.gd3799b8c.el 6.1 MB/s | 345 kB 00:00 2022-03-21T13:55:48.988 INFO:teuthology.orchestra.run.smithi079.stdout:(24/127): python3-rgw-17.0.0-11208.gd3799b8c.el 2.6 MB/s | 109 kB 00:00 2022-03-21T13:55:48.989 INFO:teuthology.orchestra.run.smithi016.stdout:(5/127): ceph-base-17.0.0-11208.gd3799b8c.el8.x 2.6 MB/s | 5.3 MB 00:02 2022-03-21T13:55:48.989 INFO:teuthology.orchestra.run.smithi016.stdout:(6/127): ceph-mgr-17.0.0-11208.gd3799b8c.el8.x8 3.2 MB/s | 1.5 MB 00:00 2022-03-21T13:55:48.989 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 187 M 2022-03-21T13:55:48.989 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2022-03-21T13:55:49.022 INFO:teuthology.orchestra.run.smithi079.stdout:(25/127): rbd-fuse-17.0.0-11208.gd3799b8c.el8.x 2.3 MB/s | 94 kB 00:00 2022-03-21T13:55:49.077 INFO:teuthology.orchestra.run.smithi079.stdout:(26/127): rbd-nbd-17.0.0-11208.gd3799b8c.el8.x8 3.1 MB/s | 174 kB 00:00 2022-03-21T13:55:49.126 INFO:teuthology.orchestra.run.smithi079.stdout:(27/127): ceph-grafana-dashboards-17.0.0-11208. 462 kB/s | 23 kB 00:00 2022-03-21T13:55:49.180 INFO:teuthology.orchestra.run.smithi079.stdout:(28/127): ceph-mgr-cephadm-17.0.0-11208.gd3799b 1.9 MB/s | 104 kB 00:00 2022-03-21T13:55:49.194 INFO:teuthology.orchestra.run.smithi002.stdout:(1/127): ceph-17.0.0-11208.gd3799b8c.el8.x86_64 30 kB/s | 6.5 kB 00:00 2022-03-21T13:55:49.454 INFO:teuthology.orchestra.run.smithi079.stdout:(29/127): ceph-mgr-dashboard-17.0.0-11208.gd379 5.9 MB/s | 1.6 MB 00:00 2022-03-21T13:55:49.546 INFO:teuthology.orchestra.run.smithi079.stdout:(30/127): rbd-mirror-17.0.0-11208.gd3799b8c.el8 6.7 MB/s | 3.8 MB 00:00 2022-03-21T13:55:49.626 INFO:teuthology.orchestra.run.smithi079.stdout:(31/127): ceph-mgr-modules-core-17.0.0-11208.gd 2.8 MB/s | 226 kB 00:00 2022-03-21T13:55:49.707 INFO:teuthology.orchestra.run.smithi079.stdout:(32/127): ceph-mgr-rook-17.0.0-11208.gd3799b8c. 583 kB/s | 47 kB 00:00 2022-03-21T13:55:49.774 INFO:teuthology.orchestra.run.smithi079.stdout:(33/127): ceph-prometheus-alerts-17.0.0-11208.g 219 kB/s | 15 kB 00:00 2022-03-21T13:55:49.844 INFO:teuthology.orchestra.run.smithi079.stdout:(34/127): ceph-volume-17.0.0-11208.gd3799b8c.el 3.6 MB/s | 257 kB 00:00 2022-03-21T13:55:49.913 INFO:teuthology.orchestra.run.smithi079.stdout:(35/127): cephadm-17.0.0-11208.gd3799b8c.el8.no 1.0 MB/s | 71 kB 00:00 2022-03-21T13:55:49.972 INFO:teuthology.orchestra.run.smithi079.stdout:(36/127): cephfs-top-17.0.0-11208.gd3799b8c.el8 242 kB/s | 14 kB 00:00 2022-03-21T13:55:50.080 INFO:teuthology.orchestra.run.smithi002.stdout:(2/127): ceph-fuse-17.0.0-11208.gd3799b8c.el8.x 995 kB/s | 881 kB 00:00 2022-03-21T13:55:50.207 INFO:teuthology.orchestra.run.smithi002.stdout:(3/127): ceph-immutable-object-cache-17.0.0-112 1.3 MB/s | 169 kB 00:00 2022-03-21T13:55:50.288 INFO:teuthology.orchestra.run.smithi016.stdout:(7/127): ceph-mon-17.0.0-11208.gd3799b8c.el8.x8 2.9 MB/s | 3.9 MB 00:01 2022-03-21T13:55:50.294 INFO:teuthology.orchestra.run.smithi079.stdout:(37/127): gperftools-libs-2.7-9.el8.x86_64.rpm 952 kB/s | 306 kB 00:00 2022-03-21T13:55:50.347 INFO:teuthology.orchestra.run.smithi079.stdout:(38/127): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2022-03-21T13:55:50.403 INFO:teuthology.orchestra.run.smithi079.stdout:(39/127): ceph-mgr-diskprediction-local-17.0.0- 7.8 MB/s | 7.4 MB 00:00 2022-03-21T13:55:50.413 INFO:teuthology.orchestra.run.smithi079.stdout:(40/127): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2022-03-21T13:55:50.480 INFO:teuthology.orchestra.run.smithi079.stdout:(41/127): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 655 kB/s | 44 kB 00:00 2022-03-21T13:55:50.532 INFO:teuthology.orchestra.run.smithi079.stdout:(42/127): python3-beautifulsoup4-4.6.3-2.el8.1. 3.5 MB/s | 185 kB 00:00 2022-03-21T13:55:50.592 INFO:teuthology.orchestra.run.smithi079.stdout:(43/127): python3-cachetools-3.1.1-4.el8.noarch 556 kB/s | 33 kB 00:00 2022-03-21T13:55:50.652 INFO:teuthology.orchestra.run.smithi079.stdout:(44/127): python3-certifi-2018.10.15-7.el8.noar 272 kB/s | 16 kB 00:00 2022-03-21T13:55:50.712 INFO:teuthology.orchestra.run.smithi079.stdout:(45/127): python3-cheroot-8.5.2-1.el8.noarch.rp 2.8 MB/s | 173 kB 00:00 2022-03-21T13:55:50.821 INFO:teuthology.orchestra.run.smithi079.stdout:(46/127): python3-asyncssh-2.7.0-2.el8.noarch.r 1.1 MB/s | 476 kB 00:00 2022-03-21T13:55:50.841 INFO:teuthology.orchestra.run.smithi079.stdout:(47/127): python3-cherrypy-18.4.0-1.el8.noarch. 2.9 MB/s | 384 kB 00:00 2022-03-21T13:55:50.890 INFO:teuthology.orchestra.run.smithi079.stdout:(48/127): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2022-03-21T13:55:50.907 INFO:teuthology.orchestra.run.smithi079.stdout:(49/127): python3-jaraco-6.2-6.el8.noarch.rpm 168 kB/s | 11 kB 00:00 2022-03-21T13:55:50.956 INFO:teuthology.orchestra.run.smithi079.stdout:(50/127): python3-jaraco-functools-2.0-4.el8.no 267 kB/s | 18 kB 00:00 2022-03-21T13:55:51.022 INFO:teuthology.orchestra.run.smithi079.stdout:(51/127): python3-logutils-0.3.5-11.el8.noarch. 734 kB/s | 49 kB 00:00 2022-03-21T13:55:51.089 INFO:teuthology.orchestra.run.smithi079.stdout:(52/127): python3-more-itertools-7.2.0-3.el8.no 887 kB/s | 59 kB 00:00 2022-03-21T13:55:51.093 INFO:teuthology.orchestra.run.smithi002.stdout:(4/127): ceph-mds-17.0.0-11208.gd3799b8c.el8.x8 2.6 MB/s | 2.3 MB 00:00 2022-03-21T13:55:51.146 INFO:teuthology.orchestra.run.smithi079.stdout:(53/127): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2022-03-21T13:55:51.237 INFO:teuthology.orchestra.run.smithi079.stdout:(54/127): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2022-03-21T13:55:51.248 INFO:teuthology.orchestra.run.smithi079.stdout:(55/127): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2022-03-21T13:55:51.283 INFO:teuthology.orchestra.run.smithi079.stdout:(56/127): python3-portend-2.6-1.el8.noarch.rpm 341 kB/s | 16 kB 00:00 2022-03-21T13:55:51.292 INFO:teuthology.orchestra.run.smithi079.stdout:(57/127): python3-repoze-lru-0.7-6.el8.noarch.r 773 kB/s | 34 kB 00:00 2022-03-21T13:55:51.327 INFO:teuthology.orchestra.run.smithi079.stdout:(58/127): python3-routes-2.4.1-12.el8.noarch.rp 4.4 MB/s | 196 kB 00:00 2022-03-21T13:55:51.335 INFO:teuthology.orchestra.run.smithi079.stdout:(59/127): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2022-03-21T13:55:51.374 INFO:teuthology.orchestra.run.smithi079.stdout:(60/127): python3-simplegeneric-0.8.1-17.el8.no 399 kB/s | 19 kB 00:00 2022-03-21T13:55:51.380 INFO:teuthology.orchestra.run.smithi079.stdout:(61/127): python3-singledispatch-3.4.0.3-18.el8 526 kB/s | 24 kB 00:00 2022-03-21T13:55:51.420 INFO:teuthology.orchestra.run.smithi079.stdout:(62/127): python3-tempora-1.14.1-5.el8.noarch.r 646 kB/s | 29 kB 00:00 2022-03-21T13:55:51.423 INFO:teuthology.orchestra.run.smithi079.stdout:(63/127): python3-trustme-0.6.0-4.el8.noarch.rp 642 kB/s | 27 kB 00:00 2022-03-21T13:55:51.431 INFO:teuthology.orchestra.run.smithi002.stdout:(5/127): ceph-mgr-17.0.0-11208.gd3799b8c.el8.x8 4.6 MB/s | 1.5 MB 00:00 2022-03-21T13:55:51.461 INFO:teuthology.orchestra.run.smithi079.stdout:(64/127): python3-waitress-1.2.1-2.el8.1.noarch 5.7 MB/s | 241 kB 00:00 2022-03-21T13:55:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:(65/127): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2022-03-21T13:55:51.507 INFO:teuthology.orchestra.run.smithi079.stdout:(66/127): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2022-03-21T13:55:51.543 INFO:teuthology.orchestra.run.smithi079.stdout:(67/127): python3-webtest-2.0.33-1.el8.noarch.r 2.1 MB/s | 85 kB 00:00 2022-03-21T13:55:51.546 INFO:teuthology.orchestra.run.smithi079.stdout:(68/127): python3-zc-lockfile-2.0-2.el8.noarch. 584 kB/s | 23 kB 00:00 2022-03-21T13:55:51.580 INFO:teuthology.orchestra.run.smithi079.stdout:(69/127): python3-mako-1.0.6-13.el8.noarch.rpm 4.6 MB/s | 157 kB 00:00 2022-03-21T13:55:51.582 INFO:teuthology.orchestra.run.smithi079.stdout:(70/127): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.7 MB/s | 69 kB 00:00 2022-03-21T13:55:51.612 INFO:teuthology.orchestra.run.smithi079.stdout:(71/127): flex-2.6.1-9.el8.x86_64.rpm 9.7 MB/s | 320 kB 00:00 2022-03-21T13:55:51.620 INFO:teuthology.orchestra.run.smithi079.stdout:(72/127): jq-1.5-12.el8.x86_64.rpm 4.1 MB/s | 161 kB 00:00 2022-03-21T13:55:51.638 INFO:teuthology.orchestra.run.smithi079.stdout:(73/127): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2022-03-21T13:55:51.647 INFO:teuthology.orchestra.run.smithi079.stdout:(74/127): librdkafka-0.11.4-1.el8.x86_64.rpm 13 MB/s | 353 kB 00:00 2022-03-21T13:55:51.666 INFO:teuthology.orchestra.run.smithi079.stdout:(75/127): python3-jinja2-2.10.1-2.el8_0.noarch. 19 MB/s | 538 kB 00:00 2022-03-21T13:55:51.682 INFO:teuthology.orchestra.run.smithi079.stdout:(76/127): automake-1.16.1-7.el8.noarch.rpm 23 MB/s | 713 kB 00:00 2022-03-21T13:55:51.695 INFO:teuthology.orchestra.run.smithi079.stdout:(77/127): libtool-2.4.6-25.el8.x86_64.rpm 24 MB/s | 709 kB 00:00 2022-03-21T13:55:51.721 INFO:teuthology.orchestra.run.smithi079.stdout:(78/127): python3-pyasn1-modules-0.3.7-6.el8.no 5.0 MB/s | 110 kB 00:00 2022-03-21T13:55:51.738 INFO:teuthology.orchestra.run.smithi079.stdout:(79/127): python3-numpy-1.14.3-9.el8.x86_64.rpm 66 MB/s | 3.7 MB 00:00 2022-03-21T13:55:51.750 INFO:teuthology.orchestra.run.smithi079.stdout:(80/127): python3-jsonpointer-1.10-11.el8.noarc 692 kB/s | 20 kB 00:00 2022-03-21T13:55:51.767 INFO:teuthology.orchestra.run.smithi079.stdout:(81/127): python3-pytz-2017.2-9.el8.noarch.rpm 2.2 MB/s | 54 kB 00:00 2022-03-21T13:55:51.776 INFO:teuthology.orchestra.run.smithi079.stdout:(82/127): oniguruma-6.8.2-2.el8.x86_64.rpm 7.1 MB/s | 187 kB 00:00 2022-03-21T13:55:51.788 INFO:teuthology.orchestra.run.smithi079.stdout:(83/127): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.0 MB/s | 126 kB 00:00 2022-03-21T13:55:51.813 INFO:teuthology.orchestra.run.smithi079.stdout:(84/127): python3-pyOpenSSL-19.0.0-1.el8.noarch 4.0 MB/s | 103 kB 00:00 2022-03-21T13:55:51.838 INFO:teuthology.orchestra.run.smithi079.stdout:(85/127): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2022-03-21T13:55:51.864 INFO:teuthology.orchestra.run.smithi002.stdout:(6/127): ceph-base-17.0.0-11208.gd3799b8c.el8.x 1.8 MB/s | 5.3 MB 00:02 2022-03-21T13:55:51.896 INFO:teuthology.orchestra.run.smithi079.stdout:(86/127): openblas-threads-0.3.12-1.el8.x86_64. 83 MB/s | 4.7 MB 00:00 2022-03-21T13:55:52.021 INFO:teuthology.orchestra.run.smithi079.stdout:(87/127): python3-scipy-1.0.0-20.module+el8.1.0 60 MB/s | 15 MB 00:00 2022-03-21T13:55:52.045 INFO:teuthology.orchestra.run.smithi079.stdout:(88/127): python3-werkzeug-0.12.2-4.el8.noarch. 19 MB/s | 457 kB 00:00 2022-03-21T13:55:52.072 INFO:teuthology.orchestra.run.smithi079.stdout:(89/127): python3-babel-2.5.1-5.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2022-03-21T13:55:52.073 INFO:teuthology.orchestra.run.smithi079.stdout:(90/127): python3-markupsafe-0.23-19.el8.x86_64 1.4 MB/s | 39 kB 00:00 2022-03-21T13:55:52.080 INFO:teuthology.orchestra.run.smithi002.stdout:(7/127): ceph-mon-17.0.0-11208.gd3799b8c.el8.x8 6.0 MB/s | 3.9 MB 00:00 2022-03-21T13:55:52.096 INFO:teuthology.orchestra.run.smithi079.stdout:(91/127): python3-prettytable-0.7.2-14.el8.noar 1.8 MB/s | 44 kB 00:00 2022-03-21T13:55:52.105 INFO:teuthology.orchestra.run.smithi079.stdout:(92/127): socat-1.7.3.3-2.el8.x86_64.rpm 9.1 MB/s | 302 kB 00:00 2022-03-21T13:55:52.135 INFO:teuthology.orchestra.run.smithi079.stdout:(93/127): bison-3.0.4-10.el8.x86_64.rpm 23 MB/s | 688 kB 00:00 2022-03-21T13:55:52.161 INFO:teuthology.orchestra.run.smithi079.stdout:(94/127): openblas-0.3.12-1.el8.x86_64.rpm 71 MB/s | 4.6 MB 00:00 2022-03-21T13:55:52.163 INFO:teuthology.orchestra.run.smithi079.stdout:(95/127): python3-numpy-f2py-1.14.3-9.el8.x86_6 7.6 MB/s | 225 kB 00:00 2022-03-21T13:55:52.187 INFO:teuthology.orchestra.run.smithi079.stdout:(96/127): libaio-devel-0.3.112-1.el8.x86_64.rpm 812 kB/s | 19 kB 00:00 2022-03-21T13:55:52.193 INFO:teuthology.orchestra.run.smithi079.stdout:(97/127): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2022-03-21T13:55:52.212 INFO:teuthology.orchestra.run.smithi079.stdout:(98/127): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2022-03-21T13:55:52.220 INFO:teuthology.orchestra.run.smithi079.stdout:(99/127): libquadmath-8.4.1-1.el8.x86_64.rpm 6.4 MB/s | 169 kB 00:00 2022-03-21T13:55:52.238 INFO:teuthology.orchestra.run.smithi079.stdout:(100/127): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.8 MB/s | 238 kB 00:00 2022-03-21T13:55:52.253 INFO:teuthology.orchestra.run.smithi079.stdout:(101/127): libgfortran-8.4.1-1.el8.x86_64.rpm 19 MB/s | 639 kB 00:00 2022-03-21T13:55:52.260 INFO:teuthology.orchestra.run.smithi079.stdout:(102/127): elfutils-libelf-devel-0.182-3.el8.x8 2.6 MB/s | 59 kB 00:00 2022-03-21T13:55:52.275 INFO:teuthology.orchestra.run.smithi079.stdout:(103/127): libselinux-devel-2.9-5.el8.x86_64.rp 9.1 MB/s | 200 kB 00:00 2022-03-21T13:55:52.286 INFO:teuthology.orchestra.run.smithi079.stdout:(104/127): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.2 MB/s | 58 kB 00:00 2022-03-21T13:55:52.305 INFO:teuthology.orchestra.run.smithi079.stdout:(105/127): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.5 MB/s | 229 kB 00:00 2022-03-21T13:55:52.312 INFO:teuthology.orchestra.run.smithi079.stdout:(106/127): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-03-21T13:55:52.329 INFO:teuthology.orchestra.run.smithi079.stdout:(107/127): libverto-devel-0.3.0-5.el8.x86_64.rp 752 kB/s | 18 kB 00:00 2022-03-21T13:55:52.336 INFO:teuthology.orchestra.run.smithi079.stdout:(108/127): libsepol-devel-2.9-2.el8.x86_64.rpm 3.5 MB/s | 86 kB 00:00 2022-03-21T13:55:52.349 INFO:teuthology.orchestra.run.smithi079.stdout:(109/127): libuuid-devel-2.32.1-27.el8.x86_64.r 4.9 MB/s | 97 kB 00:00 2022-03-21T13:55:52.360 INFO:teuthology.orchestra.run.smithi079.stdout:(110/127): python3-oauthlib-2.1.0-1.el8.noarch. 6.3 MB/s | 155 kB 00:00 2022-03-21T13:55:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:(111/127): fuse-2.9.7-12.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2022-03-21T13:55:52.393 INFO:teuthology.orchestra.run.smithi079.stdout:(112/127): openssl-devel-1.1.1g-15.el8_3.x86_64 58 MB/s | 2.3 MB 00:00 2022-03-21T13:55:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:(113/127): python3-pycparser-2.14-14.el8.noarch 3.6 MB/s | 109 kB 00:00 2022-03-21T13:55:52.426 INFO:teuthology.orchestra.run.smithi079.stdout:(114/127): pcre2-devel-10.32-2.el8.x86_64.rpm 20 MB/s | 605 kB 00:00 2022-03-21T13:55:52.440 INFO:teuthology.orchestra.run.smithi079.stdout:(115/127): python3-cryptography-3.2.1-4.el8.x86 22 MB/s | 559 kB 00:00 2022-03-21T13:55:52.451 INFO:teuthology.orchestra.run.smithi079.stdout:(116/127): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 3.2 MB/s | 70 kB 00:00 2022-03-21T13:55:52.469 INFO:teuthology.orchestra.run.smithi079.stdout:(117/127): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.6 MB/s | 220 kB 00:00 2022-03-21T13:55:52.477 INFO:teuthology.orchestra.run.smithi079.stdout:(118/127): keyutils-libs-devel-1.5.10-6.el8.x86 1.8 MB/s | 48 kB 00:00 2022-03-21T13:55:52.495 INFO:teuthology.orchestra.run.smithi079.stdout:(119/127): libcom_err-devel-1.45.6-1.el8.x86_64 1.5 MB/s | 38 kB 00:00 2022-03-21T13:55:52.504 INFO:teuthology.orchestra.run.smithi079.stdout:(120/127): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 21 MB/s | 559 kB 00:00 2022-03-21T13:55:53.507 INFO:teuthology.orchestra.run.smithi002.stdout:(8/127): ceph-radosgw-17.0.0-11208.gd3799b8c.el 6.5 MB/s | 9.3 MB 00:01 2022-03-21T13:55:53.569 INFO:teuthology.orchestra.run.smithi002.stdout:(9/127): ceph-selinux-17.0.0-11208.gd3799b8c.el 395 kB/s | 24 kB 00:00 2022-03-21T13:55:53.816 INFO:teuthology.orchestra.run.smithi079.stdout:(121/127): librados2-17.0.0-11208.gd3799b8c.el8 2.6 MB/s | 3.4 MB 00:01 2022-03-21T13:55:53.838 INFO:teuthology.orchestra.run.smithi079.stdout:(122/127): NetworkManager-team-1.30.0-13.el8_4. 6.5 MB/s | 146 kB 00:00 2022-03-21T13:55:53.871 INFO:teuthology.orchestra.run.smithi079.stdout:(123/127): NetworkManager-libnm-1.30.0-13.el8_4 55 MB/s | 1.8 MB 00:00 2022-03-21T13:55:53.895 INFO:teuthology.orchestra.run.smithi079.stdout:(124/127): NetworkManager-tui-1.30.0-13.el8_4.x 14 MB/s | 329 kB 00:00 2022-03-21T13:55:53.930 INFO:teuthology.orchestra.run.smithi079.stdout:(125/127): NetworkManager-1.30.0-13.el8_4.x86_6 73 MB/s | 2.6 MB 00:00 2022-03-21T13:55:54.119 INFO:teuthology.orchestra.run.smithi016.stdout:(8/127): ceph-common-17.0.0-11208.gd3799b8c.el8 2.8 MB/s | 20 MB 00:07 2022-03-21T13:55:54.177 INFO:teuthology.orchestra.run.smithi016.stdout:(9/127): ceph-selinux-17.0.0-11208.gd3799b8c.el 416 kB/s | 24 kB 00:00 2022-03-21T13:55:54.177 INFO:teuthology.orchestra.run.smithi079.stdout:(126/127): librbd1-17.0.0-11208.gd3799b8c.el8.x 2.3 MB/s | 3.9 MB 00:01 2022-03-21T13:55:54.933 INFO:teuthology.orchestra.run.smithi002.stdout:(10/127): ceph-common-17.0.0-11208.gd3799b8c.el 3.4 MB/s | 20 MB 00:05 2022-03-21T13:55:54.989 INFO:teuthology.orchestra.run.smithi002.stdout:(11/127): cephfs-mirror-17.0.0-11208.gd3799b8c. 4.8 MB/s | 273 kB 00:00 2022-03-21T13:55:55.025 INFO:teuthology.orchestra.run.smithi002.stdout:(12/127): libcephfs-devel-17.0.0-11208.gd3799b8 693 kB/s | 25 kB 00:00 2022-03-21T13:55:55.172 INFO:teuthology.orchestra.run.smithi002.stdout:(13/127): libcephfs2-17.0.0-11208.gd3799b8c.el8 5.3 MB/s | 795 kB 00:00 2022-03-21T13:55:55.178 INFO:teuthology.orchestra.run.smithi016.stdout:(10/127): ceph-radosgw-17.0.0-11208.gd3799b8c.e 1.9 MB/s | 9.3 MB 00:04 2022-03-21T13:55:55.211 INFO:teuthology.orchestra.run.smithi002.stdout:(14/127): libcephsqlite-17.0.0-11208.gd3799b8c. 4.2 MB/s | 170 kB 00:00 2022-03-21T13:55:55.263 INFO:teuthology.orchestra.run.smithi002.stdout:(15/127): librados-devel-17.0.0-11208.gd3799b8c 2.4 MB/s | 125 kB 00:00 2022-03-21T13:55:55.334 INFO:teuthology.orchestra.run.smithi016.stdout:(11/127): cephfs-mirror-17.0.0-11208.gd3799b8c. 1.7 MB/s | 273 kB 00:00 2022-03-21T13:55:55.360 INFO:teuthology.orchestra.run.smithi079.stdout:(127/127): ceph-test-17.0.0-11208.gd3799b8c.el8 6.2 MB/s | 45 MB 00:07 2022-03-21T13:55:55.361 INFO:teuthology.orchestra.run.smithi002.stdout:(16/127): libradosstriper1-17.0.0-11208.gd3799b 4.8 MB/s | 483 kB 00:00 2022-03-21T13:55:55.365 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:55:55.366 INFO:teuthology.orchestra.run.smithi079.stdout:Total 19 MB/s | 187 MB 00:09 2022-03-21T13:55:55.373 INFO:teuthology.orchestra.run.smithi016.stdout:(12/127): libcephfs-devel-17.0.0-11208.gd3799b8 639 kB/s | 25 kB 00:00 2022-03-21T13:55:55.427 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2022-03-21T13:55:55.498 INFO:teuthology.orchestra.run.smithi002.stdout:(17/127): ceph-osd-17.0.0-11208.gd3799b8c.el8.x 4.6 MB/s | 17 MB 00:03 2022-03-21T13:55:55.537 INFO:teuthology.orchestra.run.smithi002.stdout:(18/127): python3-ceph-argparse-17.0.0-11208.gd 1.2 MB/s | 45 kB 00:00 2022-03-21T13:55:55.543 INFO:teuthology.orchestra.run.smithi016.stdout:(13/127): ceph-osd-17.0.0-11208.gd3799b8c.el8.x 2.6 MB/s | 17 MB 00:06 2022-03-21T13:55:55.553 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2022-03-21T13:55:55.554 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2022-03-21T13:55:55.569 INFO:teuthology.orchestra.run.smithi002.stdout:(19/127): python3-ceph-common-17.0.0-11208.gd37 3.2 MB/s | 105 kB 00:00 2022-03-21T13:55:55.623 INFO:teuthology.orchestra.run.smithi002.stdout:(20/127): python3-cephfs-17.0.0-11208.gd3799b8c 3.2 MB/s | 179 kB 00:00 2022-03-21T13:55:55.644 INFO:teuthology.orchestra.run.smithi016.stdout:(14/127): libcephsqlite-17.0.0-11208.gd3799b8c. 1.7 MB/s | 170 kB 00:00 2022-03-21T13:55:55.690 INFO:teuthology.orchestra.run.smithi016.stdout:(15/127): librados-devel-17.0.0-11208.gd3799b8c 2.7 MB/s | 125 kB 00:00 2022-03-21T13:55:55.711 INFO:teuthology.orchestra.run.smithi002.stdout:(21/127): python3-rados-17.0.0-11208.gd3799b8c. 4.1 MB/s | 366 kB 00:00 2022-03-21T13:55:55.739 INFO:teuthology.orchestra.run.smithi016.stdout:(16/127): libcephfs2-17.0.0-11208.gd3799b8c.el8 2.1 MB/s | 795 kB 00:00 2022-03-21T13:55:55.805 INFO:teuthology.orchestra.run.smithi002.stdout:(22/127): python3-rbd-17.0.0-11208.gd3799b8c.el 3.6 MB/s | 345 kB 00:00 2022-03-21T13:55:55.852 INFO:teuthology.orchestra.run.smithi002.stdout:(23/127): python3-rgw-17.0.0-11208.gd3799b8c.el 2.3 MB/s | 109 kB 00:00 2022-03-21T13:55:55.870 INFO:teuthology.orchestra.run.smithi016.stdout:(17/127): libradosstriper1-17.0.0-11208.gd3799b 2.6 MB/s | 483 kB 00:00 2022-03-21T13:55:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:(24/127): rbd-fuse-17.0.0-11208.gd3799b8c.el8.x 2.1 MB/s | 94 kB 00:00 2022-03-21T13:55:55.912 INFO:teuthology.orchestra.run.smithi016.stdout:(18/127): python3-ceph-argparse-17.0.0-11208.gd 1.0 MB/s | 45 kB 00:00 2022-03-21T13:55:55.950 INFO:teuthology.orchestra.run.smithi002.stdout:(25/127): librgw2-17.0.0-11208.gd3799b8c.el8.x8 6.8 MB/s | 4.0 MB 00:00 2022-03-21T13:55:55.956 INFO:teuthology.orchestra.run.smithi016.stdout:(19/127): python3-ceph-common-17.0.0-11208.gd37 2.4 MB/s | 105 kB 00:00 2022-03-21T13:55:55.997 INFO:teuthology.orchestra.run.smithi002.stdout:(26/127): rbd-nbd-17.0.0-11208.gd3799b8c.el8.x8 3.7 MB/s | 174 kB 00:00 2022-03-21T13:55:56.041 INFO:teuthology.orchestra.run.smithi002.stdout:(27/127): ceph-grafana-dashboards-17.0.0-11208. 512 kB/s | 23 kB 00:00 2022-03-21T13:55:56.042 INFO:teuthology.orchestra.run.smithi016.stdout:(20/127): python3-cephfs-17.0.0-11208.gd3799b8c 2.0 MB/s | 179 kB 00:00 2022-03-21T13:55:56.083 INFO:teuthology.orchestra.run.smithi002.stdout:(28/127): ceph-mgr-cephadm-17.0.0-11208.gd3799b 2.5 MB/s | 104 kB 00:00 2022-03-21T13:55:56.192 INFO:teuthology.orchestra.run.smithi016.stdout:(21/127): python3-rados-17.0.0-11208.gd3799b8c. 2.4 MB/s | 366 kB 00:00 2022-03-21T13:55:56.349 INFO:teuthology.orchestra.run.smithi016.stdout:(22/127): python3-rbd-17.0.0-11208.gd3799b8c.el 2.2 MB/s | 345 kB 00:00 2022-03-21T13:55:56.351 INFO:teuthology.orchestra.run.smithi002.stdout:(29/127): ceph-mgr-dashboard-17.0.0-11208.gd379 6.0 MB/s | 1.6 MB 00:00 2022-03-21T13:55:56.389 INFO:teuthology.orchestra.run.smithi016.stdout:(23/127): python3-rgw-17.0.0-11208.gd3799b8c.el 2.7 MB/s | 109 kB 00:00 2022-03-21T13:55:56.442 INFO:teuthology.orchestra.run.smithi016.stdout:(24/127): rbd-fuse-17.0.0-11208.gd3799b8c.el8.x 1.8 MB/s | 94 kB 00:00 2022-03-21T13:55:56.555 INFO:teuthology.orchestra.run.smithi002.stdout:(30/127): rbd-mirror-17.0.0-11208.gd3799b8c.el8 5.7 MB/s | 3.8 MB 00:00 2022-03-21T13:55:56.563 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2022-03-21T13:55:56.568 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2022-03-21T13:55:56.610 INFO:teuthology.orchestra.run.smithi002.stdout:(31/127): ceph-mgr-modules-core-17.0.0-11208.gd 4.0 MB/s | 226 kB 00:00 2022-03-21T13:55:56.657 INFO:teuthology.orchestra.run.smithi002.stdout:(32/127): ceph-mgr-rook-17.0.0-11208.gd3799b8c. 1.0 MB/s | 47 kB 00:00 2022-03-21T13:55:56.708 INFO:teuthology.orchestra.run.smithi002.stdout:(33/127): ceph-prometheus-alerts-17.0.0-11208.g 290 kB/s | 15 kB 00:00 2022-03-21T13:55:56.759 INFO:teuthology.orchestra.run.smithi002.stdout:(34/127): ceph-volume-17.0.0-11208.gd3799b8c.el 4.9 MB/s | 257 kB 00:00 2022-03-21T13:55:56.802 INFO:teuthology.orchestra.run.smithi002.stdout:(35/127): cephadm-17.0.0-11208.gd3799b8c.el8.no 1.6 MB/s | 71 kB 00:00 2022-03-21T13:55:56.846 INFO:teuthology.orchestra.run.smithi002.stdout:(36/127): cephfs-top-17.0.0-11208.gd3799b8c.el8 331 kB/s | 14 kB 00:00 2022-03-21T13:55:57.030 INFO:teuthology.orchestra.run.smithi002.stdout:(37/127): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2022-03-21T13:55:57.058 INFO:teuthology.orchestra.run.smithi002.stdout:(38/127): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2022-03-21T13:55:57.086 INFO:teuthology.orchestra.run.smithi002.stdout:(39/127): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2022-03-21T13:55:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:(40/127): python3-asyncssh-2.7.0-2.el8.noarch.r 7.9 MB/s | 476 kB 00:00 2022-03-21T13:55:57.173 INFO:teuthology.orchestra.run.smithi002.stdout:(41/127): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.6 MB/s | 44 kB 00:00 2022-03-21T13:55:57.203 INFO:teuthology.orchestra.run.smithi002.stdout:(42/127): python3-beautifulsoup4-4.6.3-2.el8.1. 6.1 MB/s | 185 kB 00:00 2022-03-21T13:55:57.230 INFO:teuthology.orchestra.run.smithi016.stdout:(25/127): librgw2-17.0.0-11208.gd3799b8c.el8.x8 2.7 MB/s | 4.0 MB 00:01 2022-03-21T13:55:57.231 INFO:teuthology.orchestra.run.smithi002.stdout:(43/127): python3-cachetools-3.1.1-4.el8.noarch 1.2 MB/s | 33 kB 00:00 2022-03-21T13:55:57.259 INFO:teuthology.orchestra.run.smithi002.stdout:(44/127): python3-certifi-2018.10.15-7.el8.noar 597 kB/s | 16 kB 00:00 2022-03-21T13:55:57.288 INFO:teuthology.orchestra.run.smithi002.stdout:(45/127): python3-cheroot-8.5.2-1.el8.noarch.rp 5.8 MB/s | 173 kB 00:00 2022-03-21T13:55:57.318 INFO:teuthology.orchestra.run.smithi016.stdout:(26/127): rbd-nbd-17.0.0-11208.gd3799b8c.el8.x8 1.9 MB/s | 174 kB 00:00 2022-03-21T13:55:57.346 INFO:teuthology.orchestra.run.smithi002.stdout:(46/127): python3-cherrypy-18.4.0-1.el8.noarch. 6.6 MB/s | 384 kB 00:00 2022-03-21T13:55:57.359 INFO:teuthology.orchestra.run.smithi016.stdout:(27/127): ceph-grafana-dashboards-17.0.0-11208. 561 kB/s | 23 kB 00:00 2022-03-21T13:55:57.374 INFO:teuthology.orchestra.run.smithi002.stdout:(47/127): python3-google-auth-1.1.1-10.el8.noar 2.8 MB/s | 81 kB 00:00 2022-03-21T13:55:57.399 INFO:teuthology.orchestra.run.smithi016.stdout:(28/127): ceph-mgr-cephadm-17.0.0-11208.gd3799b 2.5 MB/s | 104 kB 00:00 2022-03-21T13:55:57.419 INFO:teuthology.orchestra.run.smithi002.stdout:(48/127): ceph-mgr-diskprediction-local-17.0.0- 6.9 MB/s | 7.4 MB 00:01 2022-03-21T13:55:57.420 INFO:teuthology.orchestra.run.smithi002.stdout:(49/127): python3-jaraco-6.2-6.el8.noarch.rpm 247 kB/s | 11 kB 00:00 2022-03-21T13:55:57.447 INFO:teuthology.orchestra.run.smithi002.stdout:(50/127): python3-jaraco-functools-2.0-4.el8.no 640 kB/s | 18 kB 00:00 2022-03-21T13:55:57.475 INFO:teuthology.orchestra.run.smithi002.stdout:(51/127): python3-logutils-0.3.5-11.el8.noarch. 1.7 MB/s | 49 kB 00:00 2022-03-21T13:55:57.500 INFO:teuthology.orchestra.run.smithi016.stdout:(29/127): rbd-mirror-17.0.0-11208.gd3799b8c.el8 3.6 MB/s | 3.8 MB 00:01 2022-03-21T13:55:57.504 INFO:teuthology.orchestra.run.smithi002.stdout:(52/127): python3-more-itertools-7.2.0-3.el8.no 2.0 MB/s | 59 kB 00:00 2022-03-21T13:55:57.532 INFO:teuthology.orchestra.run.smithi002.stdout:(53/127): python3-natsort-7.1.1-2.el8.noarch.rp 2.1 MB/s | 60 kB 00:00 2022-03-21T13:55:57.588 INFO:teuthology.orchestra.run.smithi002.stdout:(54/127): python3-pecan-1.3.2-9.el8.noarch.rpm 5.0 MB/s | 283 kB 00:00 2022-03-21T13:55:57.616 INFO:teuthology.orchestra.run.smithi002.stdout:(55/127): python3-portend-2.6-1.el8.noarch.rpm 563 kB/s | 16 kB 00:00 2022-03-21T13:55:57.644 INFO:teuthology.orchestra.run.smithi002.stdout:(56/127): python3-repoze-lru-0.7-6.el8.noarch.r 1.2 MB/s | 34 kB 00:00 2022-03-21T13:55:57.674 INFO:teuthology.orchestra.run.smithi002.stdout:(57/127): python3-routes-2.4.1-12.el8.noarch.rp 6.4 MB/s | 196 kB 00:00 2022-03-21T13:55:57.702 INFO:teuthology.orchestra.run.smithi002.stdout:(58/127): python3-rsa-4.8-1.el8.noarch.rpm 2.2 MB/s | 61 kB 00:00 2022-03-21T13:55:57.730 INFO:teuthology.orchestra.run.smithi002.stdout:(59/127): python3-simplegeneric-0.8.1-17.el8.no 686 kB/s | 19 kB 00:00 2022-03-21T13:55:57.736 INFO:teuthology.orchestra.run.smithi002.stdout:(60/127): python3-kubernetes-11.0.0-6.el8.noarc 3.7 MB/s | 1.2 MB 00:00 2022-03-21T13:55:57.757 INFO:teuthology.orchestra.run.smithi002.stdout:(61/127): python3-singledispatch-3.4.0.3-18.el8 876 kB/s | 24 kB 00:00 2022-03-21T13:55:57.772 INFO:teuthology.orchestra.run.smithi002.stdout:(62/127): python3-tempora-1.14.1-5.el8.noarch.r 808 kB/s | 29 kB 00:00 2022-03-21T13:55:57.786 INFO:teuthology.orchestra.run.smithi002.stdout:(63/127): python3-trustme-0.6.0-4.el8.noarch.rp 993 kB/s | 27 kB 00:00 2022-03-21T13:55:57.797 INFO:teuthology.orchestra.run.smithi016.stdout:(30/127): ceph-mgr-dashboard-17.0.0-11208.gd379 4.1 MB/s | 1.6 MB 00:00 2022-03-21T13:55:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:(64/127): python3-waitress-1.2.1-2.el8.1.noarch 7.1 MB/s | 241 kB 00:00 2022-03-21T13:55:57.836 INFO:teuthology.orchestra.run.smithi002.stdout:(65/127): python3-websocket-client-0.56.0-5.el8 2.0 MB/s | 61 kB 00:00 2022-03-21T13:55:57.840 INFO:teuthology.orchestra.run.smithi002.stdout:(66/127): python3-webob-1.8.5-1.el8.1.noarch.rp 4.5 MB/s | 251 kB 00:00 2022-03-21T13:55:57.866 INFO:teuthology.orchestra.run.smithi016.stdout:(31/127): ceph-mgr-modules-core-17.0.0-11208.gd 3.2 MB/s | 226 kB 00:00 2022-03-21T13:55:57.866 INFO:teuthology.orchestra.run.smithi002.stdout:(67/127): python3-webtest-2.0.33-1.el8.noarch.r 2.8 MB/s | 85 kB 00:00 2022-03-21T13:55:57.868 INFO:teuthology.orchestra.run.smithi002.stdout:(68/127): python3-zc-lockfile-2.0-2.el8.noarch. 838 kB/s | 23 kB 00:00 2022-03-21T13:55:57.898 INFO:teuthology.orchestra.run.smithi016.stdout:(32/127): ceph-mgr-rook-17.0.0-11208.gd3799b8c. 1.5 MB/s | 47 kB 00:00 2022-03-21T13:55:57.903 INFO:teuthology.orchestra.run.smithi002.stdout:(69/127): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.9 MB/s | 69 kB 00:00 2022-03-21T13:55:57.918 INFO:teuthology.orchestra.run.smithi002.stdout:(70/127): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-03-21T13:55:57.928 INFO:teuthology.orchestra.run.smithi016.stdout:(33/127): ceph-prometheus-alerts-17.0.0-11208.g 479 kB/s | 15 kB 00:00 2022-03-21T13:55:57.954 INFO:teuthology.orchestra.run.smithi002.stdout:(71/127): flex-2.6.1-9.el8.x86_64.rpm 6.1 MB/s | 320 kB 00:00 2022-03-21T13:55:57.966 INFO:teuthology.orchestra.run.smithi002.stdout:(72/127): jq-1.5-12.el8.x86_64.rpm 3.3 MB/s | 161 kB 00:00 2022-03-21T13:55:57.983 INFO:teuthology.orchestra.run.smithi002.stdout:(73/127): python3-jsonpatch-1.21-2.el8.noarch.r 929 kB/s | 27 kB 00:00 2022-03-21T13:55:58.001 INFO:teuthology.orchestra.run.smithi016.stdout:(34/127): ceph-volume-17.0.0-11208.gd3799b8c.el 3.5 MB/s | 257 kB 00:00 2022-03-21T13:55:58.014 INFO:teuthology.orchestra.run.smithi002.stdout:(74/127): librdkafka-0.11.4-1.el8.x86_64.rpm 8.3 MB/s | 353 kB 00:00 2022-03-21T13:55:58.030 INFO:teuthology.orchestra.run.smithi016.stdout:(35/127): cephadm-17.0.0-11208.gd3799b8c.el8.no 2.4 MB/s | 71 kB 00:00 2022-03-21T13:55:58.038 INFO:teuthology.orchestra.run.smithi002.stdout:(75/127): python3-jinja2-2.10.1-2.el8_0.noarch. 9.7 MB/s | 538 kB 00:00 2022-03-21T13:55:58.056 INFO:teuthology.orchestra.run.smithi016.stdout:(36/127): cephfs-top-17.0.0-11208.gd3799b8c.el8 561 kB/s | 14 kB 00:00 2022-03-21T13:55:58.066 INFO:teuthology.orchestra.run.smithi002.stdout:(76/127): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2022-03-21T13:55:58.091 INFO:teuthology.orchestra.run.smithi002.stdout:(77/127): libtool-2.4.6-25.el8.x86_64.rpm 13 MB/s | 709 kB 00:00 2022-03-21T13:55:58.120 INFO:teuthology.orchestra.run.smithi002.stdout:(78/127): python3-pyasn1-modules-0.3.7-6.el8.no 3.7 MB/s | 110 kB 00:00 2022-03-21T13:55:58.148 INFO:teuthology.orchestra.run.smithi002.stdout:(79/127): python3-jsonpointer-1.10-11.el8.noarc 746 kB/s | 20 kB 00:00 2022-03-21T13:55:58.177 INFO:teuthology.orchestra.run.smithi002.stdout:(80/127): python3-pytz-2017.2-9.el8.noarch.rpm 1.9 MB/s | 54 kB 00:00 2022-03-21T13:55:58.213 INFO:teuthology.orchestra.run.smithi002.stdout:(81/127): python3-numpy-1.14.3-9.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2022-03-21T13:55:58.224 INFO:teuthology.orchestra.run.smithi002.stdout:(82/127): oniguruma-6.8.2-2.el8.x86_64.rpm 3.9 MB/s | 187 kB 00:00 2022-03-21T13:55:58.242 INFO:teuthology.orchestra.run.smithi002.stdout:(83/127): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.3 MB/s | 126 kB 00:00 2022-03-21T13:55:58.279 INFO:teuthology.orchestra.run.smithi002.stdout:(84/127): python3-pyOpenSSL-19.0.0-1.el8.noarch 3.4 MB/s | 103 kB 00:00 2022-03-21T13:55:58.309 INFO:teuthology.orchestra.run.smithi002.stdout:(85/127): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.4 MB/s | 259 kB 00:00 2022-03-21T13:55:58.352 INFO:teuthology.orchestra.run.smithi016.stdout:(37/127): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2022-03-21T13:55:58.398 INFO:teuthology.orchestra.run.smithi016.stdout:(38/127): liboath-2.6.2-3.el8.x86_64.rpm 1.3 MB/s | 59 kB 00:00 2022-03-21T13:55:58.443 INFO:teuthology.orchestra.run.smithi016.stdout:(39/127): libunwind-1.3.1-3.el8.x86_64.rpm 1.6 MB/s | 75 kB 00:00 2022-03-21T13:55:58.511 INFO:teuthology.orchestra.run.smithi002.stdout:(86/127): ceph-test-17.0.0-11208.gd3799b8c.el8. 9.1 MB/s | 45 MB 00:04 2022-03-21T13:55:58.659 INFO:teuthology.orchestra.run.smithi002.stdout:(87/127): openblas-threads-0.3.12-1.el8.x86_64. 14 MB/s | 4.7 MB 00:00 2022-03-21T13:55:58.676 INFO:teuthology.orchestra.run.smithi016.stdout:(40/127): python3-asyncssh-2.7.0-2.el8.noarch.r 2.0 MB/s | 476 kB 00:00 2022-03-21T13:55:58.710 INFO:teuthology.orchestra.run.smithi002.stdout:(88/127): python3-babel-2.5.1-5.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2022-03-21T13:55:58.722 INFO:teuthology.orchestra.run.smithi002.stdout:(89/127): python3-werkzeug-0.12.2-4.el8.noarch. 7.2 MB/s | 457 kB 00:00 2022-03-21T13:55:58.724 INFO:teuthology.orchestra.run.smithi016.stdout:(41/127): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 906 kB/s | 44 kB 00:00 2022-03-21T13:55:58.740 INFO:teuthology.orchestra.run.smithi002.stdout:(90/127): python3-markupsafe-0.23-19.el8.x86_64 1.3 MB/s | 39 kB 00:00 2022-03-21T13:55:58.758 INFO:teuthology.orchestra.run.smithi002.stdout:(91/127): python3-prettytable-0.7.2-14.el8.noar 1.2 MB/s | 44 kB 00:00 2022-03-21T13:55:58.787 INFO:teuthology.orchestra.run.smithi002.stdout:(92/127): socat-1.7.3.3-2.el8.x86_64.rpm 6.4 MB/s | 302 kB 00:00 2022-03-21T13:55:58.809 INFO:teuthology.orchestra.run.smithi016.stdout:(42/127): python3-beautifulsoup4-4.6.3-2.el8.1. 2.2 MB/s | 185 kB 00:00 2022-03-21T13:55:58.831 INFO:teuthology.orchestra.run.smithi002.stdout:(93/127): bison-3.0.4-10.el8.x86_64.rpm 16 MB/s | 688 kB 00:00 2022-03-21T13:55:58.843 INFO:teuthology.orchestra.run.smithi016.stdout:(43/127): python3-cachetools-3.1.1-4.el8.noarch 984 kB/s | 33 kB 00:00 2022-03-21T13:55:58.863 INFO:teuthology.orchestra.run.smithi002.stdout:(94/127): python3-numpy-f2py-1.14.3-9.el8.x86_6 7.0 MB/s | 225 kB 00:00 2022-03-21T13:55:58.872 INFO:teuthology.orchestra.run.smithi016.stdout:(44/127): python3-certifi-2018.10.15-7.el8.noar 563 kB/s | 16 kB 00:00 2022-03-21T13:55:58.890 INFO:teuthology.orchestra.run.smithi002.stdout:(95/127): python3-jwt-1.6.1-2.el8.noarch.rpm 1.6 MB/s | 43 kB 00:00 2022-03-21T13:55:58.903 INFO:teuthology.orchestra.run.smithi016.stdout:(45/127): ceph-mgr-diskprediction-local-17.0.0- 5.3 MB/s | 7.4 MB 00:01 2022-03-21T13:55:58.916 INFO:teuthology.orchestra.run.smithi002.stdout:(96/127): libaio-devel-0.3.112-1.el8.x86_64.rpm 727 kB/s | 19 kB 00:00 2022-03-21T13:55:58.939 INFO:teuthology.orchestra.run.smithi016.stdout:(46/127): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2022-03-21T13:55:58.977 INFO:teuthology.orchestra.run.smithi016.stdout:(47/127): python3-google-auth-1.1.1-10.el8.noar 2.1 MB/s | 81 kB 00:00 2022-03-21T13:55:58.994 INFO:teuthology.orchestra.run.smithi002.stdout:(97/127): python3-scipy-1.0.0-20.module+el8.1.0 19 MB/s | 15 MB 00:00 2022-03-21T13:55:58.996 INFO:teuthology.orchestra.run.smithi002.stdout:(98/127): python3-requests-oauthlib-1.0.0-1.el8 540 kB/s | 43 kB 00:00 2022-03-21T13:55:59.024 INFO:teuthology.orchestra.run.smithi016.stdout:(48/127): python3-jaraco-6.2-6.el8.noarch.rpm 236 kB/s | 11 kB 00:00 2022-03-21T13:55:59.046 INFO:teuthology.orchestra.run.smithi002.stdout:(99/127): openblas-0.3.12-1.el8.x86_64.rpm 16 MB/s | 4.6 MB 00:00 2022-03-21T13:55:59.049 INFO:teuthology.orchestra.run.smithi002.stdout:(100/127): libquadmath-8.4.1-1.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2022-03-21T13:55:59.058 INFO:teuthology.orchestra.run.smithi002.stdout:(101/127): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.7 MB/s | 238 kB 00:00 2022-03-21T13:55:59.070 INFO:teuthology.orchestra.run.smithi016.stdout:(49/127): python3-jaraco-functools-2.0-4.el8.no 391 kB/s | 18 kB 00:00 2022-03-21T13:55:59.098 INFO:teuthology.orchestra.run.smithi002.stdout:(102/127): elfutils-libelf-devel-0.182-3.el8.x8 1.4 MB/s | 59 kB 00:00 2022-03-21T13:55:59.103 INFO:teuthology.orchestra.run.smithi002.stdout:(103/127): libgfortran-8.4.1-1.el8.x86_64.rpm 11 MB/s | 639 kB 00:00 2022-03-21T13:55:59.119 INFO:teuthology.orchestra.run.smithi002.stdout:(104/127): libselinux-devel-2.9-5.el8.x86_64.rp 4.1 MB/s | 200 kB 00:00 2022-03-21T13:55:59.143 INFO:teuthology.orchestra.run.smithi002.stdout:(105/127): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2022-03-21T13:55:59.153 INFO:teuthology.orchestra.run.smithi002.stdout:(106/127): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.2 MB/s | 229 kB 00:00 2022-03-21T13:55:59.161 INFO:teuthology.orchestra.run.smithi002.stdout:(107/127): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.3 MB/s | 47 kB 00:00 2022-03-21T13:55:59.184 INFO:teuthology.orchestra.run.smithi002.stdout:(108/127): libverto-devel-0.3.0-5.el8.x86_64.rp 447 kB/s | 18 kB 00:00 2022-03-21T13:55:59.199 INFO:teuthology.orchestra.run.smithi002.stdout:(109/127): libsepol-devel-2.9-2.el8.x86_64.rpm 2.1 MB/s | 86 kB 00:00 2022-03-21T13:55:59.204 INFO:teuthology.orchestra.run.smithi016.stdout:(50/127): python3-cherrypy-18.4.0-1.el8.noarch. 1.3 MB/s | 384 kB 00:00 2022-03-21T13:55:59.208 INFO:teuthology.orchestra.run.smithi002.stdout:(110/127): libuuid-devel-2.32.1-27.el8.x86_64.r 2.0 MB/s | 97 kB 00:00 2022-03-21T13:55:59.234 INFO:teuthology.orchestra.run.smithi016.stdout:(51/127): python3-logutils-0.3.5-11.el8.noarch. 1.6 MB/s | 49 kB 00:00 2022-03-21T13:55:59.249 INFO:teuthology.orchestra.run.smithi002.stdout:(111/127): python3-oauthlib-2.1.0-1.el8.noarch. 2.3 MB/s | 155 kB 00:00 2022-03-21T13:55:59.261 INFO:teuthology.orchestra.run.smithi002.stdout:(112/127): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2022-03-21T13:55:59.263 INFO:teuthology.orchestra.run.smithi016.stdout:(52/127): python3-more-itertools-7.2.0-3.el8.no 1.9 MB/s | 59 kB 00:00 2022-03-21T13:55:59.280 INFO:teuthology.orchestra.run.smithi002.stdout:(113/127): python3-pycparser-2.14-14.el8.noarch 3.6 MB/s | 109 kB 00:00 2022-03-21T13:55:59.292 INFO:teuthology.orchestra.run.smithi016.stdout:(53/127): python3-natsort-7.1.1-2.el8.noarch.rp 2.1 MB/s | 60 kB 00:00 2022-03-21T13:55:59.317 INFO:teuthology.orchestra.run.smithi002.stdout:(114/127): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2022-03-21T13:55:59.340 INFO:teuthology.orchestra.run.smithi002.stdout:(115/127): python3-cryptography-3.2.1-4.el8.x86 10 MB/s | 559 kB 00:00 2022-03-21T13:55:59.352 INFO:teuthology.orchestra.run.smithi002.stdout:(116/127): openssl-devel-1.1.1g-15.el8_3.x86_64 16 MB/s | 2.3 MB 00:00 2022-03-21T13:55:59.359 INFO:teuthology.orchestra.run.smithi002.stdout:(117/127): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 1.6 MB/s | 70 kB 00:00 2022-03-21T13:55:59.374 INFO:teuthology.orchestra.run.smithi016.stdout:(54/127): python3-kubernetes-11.0.0-6.el8.noarc 3.8 MB/s | 1.2 MB 00:00 2022-03-21T13:55:59.399 INFO:teuthology.orchestra.run.smithi002.stdout:(118/127): keyutils-libs-devel-1.5.10-6.el8.x86 1.2 MB/s | 48 kB 00:00 2022-03-21T13:55:59.402 INFO:teuthology.orchestra.run.smithi002.stdout:(119/127): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.5 MB/s | 220 kB 00:00 2022-03-21T13:55:59.405 INFO:teuthology.orchestra.run.smithi016.stdout:(55/127): python3-portend-2.6-1.el8.noarch.rpm 523 kB/s | 16 kB 00:00 2022-03-21T13:55:59.412 INFO:teuthology.orchestra.run.smithi002.stdout:(120/127): libcom_err-devel-1.45.6-1.el8.x86_64 738 kB/s | 38 kB 00:00 2022-03-21T13:55:59.415 INFO:teuthology.orchestra.run.smithi016.stdout:(56/127): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2022-03-21T13:55:59.433 INFO:teuthology.orchestra.run.smithi002.stdout:(121/127): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 17 MB/s | 559 kB 00:00 2022-03-21T13:55:59.439 INFO:teuthology.orchestra.run.smithi016.stdout:(57/127): python3-repoze-lru-0.7-6.el8.noarch.r 1.0 MB/s | 34 kB 00:00 2022-03-21T13:55:59.459 INFO:teuthology.orchestra.run.smithi002.stdout:(122/127): NetworkManager-team-1.30.0-13.el8_4. 5.4 MB/s | 146 kB 00:00 2022-03-21T13:55:59.477 INFO:teuthology.orchestra.run.smithi016.stdout:(58/127): python3-rsa-4.8-1.el8.noarch.rpm 1.5 MB/s | 61 kB 00:00 2022-03-21T13:55:59.507 INFO:teuthology.orchestra.run.smithi016.stdout:(59/127): python3-simplegeneric-0.8.1-17.el8.no 649 kB/s | 19 kB 00:00 2022-03-21T13:55:59.509 INFO:teuthology.orchestra.run.smithi002.stdout:(123/127): NetworkManager-libnm-1.30.0-13.el8_4 36 MB/s | 1.8 MB 00:00 2022-03-21T13:55:59.512 INFO:teuthology.orchestra.run.smithi016.stdout:(60/127): python3-routes-2.4.1-12.el8.noarch.rp 2.0 MB/s | 196 kB 00:00 2022-03-21T13:55:59.537 INFO:teuthology.orchestra.run.smithi016.stdout:(61/127): python3-singledispatch-3.4.0.3-18.el8 792 kB/s | 24 kB 00:00 2022-03-21T13:55:59.540 INFO:teuthology.orchestra.run.smithi016.stdout:(62/127): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2022-03-21T13:55:59.541 INFO:teuthology.orchestra.run.smithi002.stdout:(124/127): NetworkManager-tui-1.30.0-13.el8_4.x 10 MB/s | 329 kB 00:00 2022-03-21T13:55:59.567 INFO:teuthology.orchestra.run.smithi016.stdout:(63/127): python3-trustme-0.6.0-4.el8.noarch.rp 918 kB/s | 27 kB 00:00 2022-03-21T13:55:59.607 INFO:teuthology.orchestra.run.smithi002.stdout:(125/127): NetworkManager-1.30.0-13.el8_4.x86_6 39 MB/s | 2.6 MB 00:00 2022-03-21T13:55:59.628 INFO:teuthology.orchestra.run.smithi016.stdout:(64/127): python3-waitress-1.2.1-2.el8.1.noarch 2.7 MB/s | 241 kB 00:00 2022-03-21T13:55:59.630 INFO:teuthology.orchestra.run.smithi016.stdout:(65/127): python3-webob-1.8.5-1.el8.1.noarch.rp 4.0 MB/s | 251 kB 00:00 2022-03-21T13:55:59.656 INFO:teuthology.orchestra.run.smithi016.stdout:(66/127): python3-websocket-client-0.56.0-5.el8 2.1 MB/s | 61 kB 00:00 2022-03-21T13:55:59.660 INFO:teuthology.orchestra.run.smithi016.stdout:(67/127): python3-webtest-2.0.33-1.el8.noarch.r 2.8 MB/s | 85 kB 00:00 2022-03-21T13:55:59.690 INFO:teuthology.orchestra.run.smithi016.stdout:(68/127): python3-zc-lockfile-2.0-2.el8.noarch. 679 kB/s | 23 kB 00:00 2022-03-21T13:55:59.704 INFO:teuthology.orchestra.run.smithi016.stdout:(69/127): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.5 MB/s | 69 kB 00:00 2022-03-21T13:55:59.739 INFO:teuthology.orchestra.run.smithi016.stdout:(70/127): python3-jwt-1.6.1-2.el8.noarch.rpm 879 kB/s | 43 kB 00:00 2022-03-21T13:55:59.752 INFO:teuthology.orchestra.run.smithi002.stdout:(126/127): librados2-17.0.0-11208.gd3799b8c.el8 9.8 MB/s | 3.4 MB 00:00 2022-03-21T13:55:59.755 INFO:teuthology.orchestra.run.smithi016.stdout:(71/127): libaio-devel-0.3.112-1.el8.x86_64.rpm 367 kB/s | 19 kB 00:00 2022-03-21T13:55:59.771 INFO:teuthology.orchestra.run.smithi016.stdout:(72/127): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2022-03-21T13:55:59.788 INFO:teuthology.orchestra.run.smithi016.stdout:(73/127): libquadmath-8.4.1-1.el8.x86_64.rpm 5.0 MB/s | 169 kB 00:00 2022-03-21T13:55:59.806 INFO:teuthology.orchestra.run.smithi016.stdout:(74/127): python3-cffi-1.11.5-5.el8.x86_64.rpm 6.7 MB/s | 238 kB 00:00 2022-03-21T13:55:59.833 INFO:teuthology.orchestra.run.smithi016.stdout:(75/127): libgfortran-8.4.1-1.el8.x86_64.rpm 14 MB/s | 639 kB 00:00 2022-03-21T13:55:59.843 INFO:teuthology.orchestra.run.smithi016.stdout:(76/127): elfutils-libelf-devel-0.182-3.el8.x86 1.6 MB/s | 59 kB 00:00 2022-03-21T13:55:59.860 INFO:teuthology.orchestra.run.smithi016.stdout:(77/127): libselinux-devel-2.9-5.el8.x86_64.rpm 7.4 MB/s | 200 kB 00:00 2022-03-21T13:55:59.876 INFO:teuthology.orchestra.run.smithi016.stdout:(78/127): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2022-03-21T13:55:59.893 INFO:teuthology.orchestra.run.smithi016.stdout:(79/127): pcre2-utf16-10.32-2.el8.x86_64.rpm 6.6 MB/s | 229 kB 00:00 2022-03-21T13:55:59.905 INFO:teuthology.orchestra.run.smithi002.stdout:(127/127): librbd1-17.0.0-11208.gd3799b8c.el8.x 7.8 MB/s | 3.9 MB 00:00 2022-03-21T13:55:59.909 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:55:59.909 INFO:teuthology.orchestra.run.smithi002.stdout:Total 17 MB/s | 187 MB 00:10 2022-03-21T13:55:59.910 INFO:teuthology.orchestra.run.smithi016.stdout:(80/127): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2022-03-21T13:55:59.934 INFO:teuthology.orchestra.run.smithi016.stdout:(81/127): libverto-devel-0.3.0-5.el8.x86_64.rpm 532 kB/s | 18 kB 00:00 2022-03-21T13:55:59.944 INFO:teuthology.orchestra.run.smithi016.stdout:(82/127): libsepol-devel-2.9-2.el8.x86_64.rpm 2.4 MB/s | 86 kB 00:00 2022-03-21T13:55:59.960 INFO:teuthology.orchestra.run.smithi016.stdout:(83/127): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.6 MB/s | 97 kB 00:00 2022-03-21T13:55:59.978 INFO:teuthology.orchestra.run.smithi016.stdout:(84/127): python3-oauthlib-2.1.0-1.el8.noarch.r 4.6 MB/s | 155 kB 00:00 2022-03-21T13:56:00.016 INFO:teuthology.orchestra.run.smithi016.stdout:(85/127): openssl-devel-1.1.1g-15.el8_3.x86_64. 42 MB/s | 2.3 MB 00:00 2022-03-21T13:56:00.017 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2022-03-21T13:56:00.017 INFO:teuthology.orchestra.run.smithi016.stdout:(86/127): fuse-2.9.7-12.el8.x86_64.rpm 2.1 MB/s | 83 kB 00:00 2022-03-21T13:56:00.048 INFO:teuthology.orchestra.run.smithi016.stdout:(87/127): python3-pycparser-2.14-14.el8.noarch. 3.3 MB/s | 109 kB 00:00 2022-03-21T13:56:00.068 INFO:teuthology.orchestra.run.smithi016.stdout:(88/127): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2022-03-21T13:56:00.088 INFO:teuthology.orchestra.run.smithi016.stdout:(89/127): python3-cryptography-3.2.1-4.el8.x86_ 14 MB/s | 559 kB 00:00 2022-03-21T13:56:00.098 INFO:teuthology.orchestra.run.smithi016.stdout:(90/127): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 2.3 MB/s | 70 kB 00:00 2022-03-21T13:56:00.115 INFO:teuthology.orchestra.run.smithi016.stdout:(91/127): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.0 MB/s | 220 kB 00:00 2022-03-21T13:56:00.130 INFO:teuthology.orchestra.run.smithi016.stdout:(92/127): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.4 MB/s | 48 kB 00:00 2022-03-21T13:56:00.147 INFO:teuthology.orchestra.run.smithi016.stdout:(93/127): libcom_err-devel-1.45.6-1.el8.x86_64. 1.2 MB/s | 38 kB 00:00 2022-03-21T13:56:00.166 INFO:teuthology.orchestra.run.smithi016.stdout:(94/127): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 15 MB/s | 559 kB 00:00 2022-03-21T13:56:00.173 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2022-03-21T13:56:00.173 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2022-03-21T13:56:00.183 INFO:teuthology.orchestra.run.smithi016.stdout:(95/127): python3-mako-1.0.6-13.el8.noarch.rpm 4.2 MB/s | 157 kB 00:00 2022-03-21T13:56:00.201 INFO:teuthology.orchestra.run.smithi016.stdout:(96/127): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2022-03-21T13:56:00.219 INFO:teuthology.orchestra.run.smithi016.stdout:(97/127): jq-1.5-12.el8.x86_64.rpm 4.4 MB/s | 161 kB 00:00 2022-03-21T13:56:00.234 INFO:teuthology.orchestra.run.smithi016.stdout:(98/127): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2022-03-21T13:56:00.252 INFO:teuthology.orchestra.run.smithi016.stdout:(99/127): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2022-03-21T13:56:00.273 INFO:teuthology.orchestra.run.smithi016.stdout:(100/127): python3-jinja2-2.10.1-2.el8_0.noarch 14 MB/s | 538 kB 00:00 2022-03-21T13:56:00.303 INFO:teuthology.orchestra.run.smithi016.stdout:(101/127): automake-1.16.1-7.el8.noarch.rpm 17 MB/s | 713 kB 00:00 2022-03-21T13:56:00.324 INFO:teuthology.orchestra.run.smithi016.stdout:(102/127): libtool-2.4.6-25.el8.x86_64.rpm 16 MB/s | 709 kB 00:00 2022-03-21T13:56:00.351 INFO:teuthology.orchestra.run.smithi016.stdout:(103/127): python3-pyasn1-modules-0.3.7-6.el8.n 4.0 MB/s | 110 kB 00:00 2022-03-21T13:56:00.381 INFO:teuthology.orchestra.run.smithi016.stdout:(104/127): python3-numpy-1.14.3-9.el8.x86_64.rp 47 MB/s | 3.7 MB 00:00 2022-03-21T13:56:00.382 INFO:teuthology.orchestra.run.smithi016.stdout:(105/127): python3-jsonpointer-1.10-11.el8.noar 653 kB/s | 20 kB 00:00 2022-03-21T13:56:00.416 INFO:teuthology.orchestra.run.smithi016.stdout:(106/127): oniguruma-6.8.2-2.el8.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2022-03-21T13:56:00.425 INFO:teuthology.orchestra.run.smithi016.stdout:(107/127): python3-pytz-2017.2-9.el8.noarch.rpm 1.2 MB/s | 54 kB 00:00 2022-03-21T13:56:00.442 INFO:teuthology.orchestra.run.smithi016.stdout:(108/127): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.7 MB/s | 126 kB 00:00 2022-03-21T13:56:00.474 INFO:teuthology.orchestra.run.smithi016.stdout:(109/127): python3-pyOpenSSL-19.0.0-1.el8.noarc 3.2 MB/s | 103 kB 00:00 2022-03-21T13:56:00.501 INFO:teuthology.orchestra.run.smithi016.stdout:(110/127): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.6 MB/s | 259 kB 00:00 2022-03-21T13:56:00.567 INFO:teuthology.orchestra.run.smithi016.stdout:(111/127): openblas-threads-0.3.12-1.el8.x86_64 72 MB/s | 4.7 MB 00:00 2022-03-21T13:56:00.700 INFO:teuthology.orchestra.run.smithi016.stdout:(112/127): python3-scipy-1.0.0-20.module+el8.1. 53 MB/s | 15 MB 00:00 2022-03-21T13:56:00.730 INFO:teuthology.orchestra.run.smithi016.stdout:(113/127): python3-werkzeug-0.12.2-4.el8.noarch 15 MB/s | 457 kB 00:00 2022-03-21T13:56:00.765 INFO:teuthology.orchestra.run.smithi016.stdout:(114/127): python3-babel-2.5.1-5.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2022-03-21T13:56:00.765 INFO:teuthology.orchestra.run.smithi016.stdout:(115/127): python3-markupsafe-0.23-19.el8.x86_6 1.1 MB/s | 39 kB 00:00 2022-03-21T13:56:00.796 INFO:teuthology.orchestra.run.smithi016.stdout:(116/127): python3-prettytable-0.7.2-14.el8.noa 1.4 MB/s | 44 kB 00:00 2022-03-21T13:56:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:(117/127): socat-1.7.3.3-2.el8.x86_64.rpm 7.0 MB/s | 302 kB 00:00 2022-03-21T13:56:00.846 INFO:teuthology.orchestra.run.smithi016.stdout:(118/127): bison-3.0.4-10.el8.x86_64.rpm 22 MB/s | 688 kB 00:00 2022-03-21T13:56:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:(119/127): openblas-0.3.12-1.el8.x86_64.rpm 54 MB/s | 4.6 MB 00:00 2022-03-21T13:56:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:(120/127): python3-numpy-f2py-1.14.3-9.el8.x86_ 6.0 MB/s | 225 kB 00:00 2022-03-21T13:56:01.681 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2022-03-21T13:56:01.707 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-03-21T13:56:01.826 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2022-03-21T13:56:01.830 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2022-03-21T13:56:01.857 INFO:teuthology.orchestra.run.smithi016.stdout:(121/127): ceph-test-17.0.0-11208.gd3799b8c.el8 5.9 MB/s | 45 MB 00:07 2022-03-21T13:56:01.873 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/133 2022-03-21T13:56:01.877 INFO:teuthology.orchestra.run.smithi016.stdout:(122/127): librados2-17.0.0-11208.gd3799b8c.el8 3.5 MB/s | 3.4 MB 00:00 2022-03-21T13:56:01.894 INFO:teuthology.orchestra.run.smithi016.stdout:(123/127): NetworkManager-team-1.30.0-13.el8_4. 3.9 MB/s | 146 kB 00:00 2022-03-21T13:56:01.930 INFO:teuthology.orchestra.run.smithi016.stdout:(124/127): NetworkManager-libnm-1.30.0-13.el8_4 34 MB/s | 1.8 MB 00:00 2022-03-21T13:56:01.933 INFO:teuthology.orchestra.run.smithi016.stdout:(125/127): NetworkManager-tui-1.30.0-13.el8_4.x 8.3 MB/s | 329 kB 00:00 2022-03-21T13:56:01.972 INFO:teuthology.orchestra.run.smithi016.stdout:(126/127): NetworkManager-1.30.0-13.el8_4.x86_6 62 MB/s | 2.6 MB 00:00 2022-03-21T13:56:02.025 INFO:teuthology.orchestra.run.smithi016.stdout:(127/127): librbd1-17.0.0-11208.gd3799b8c.el8.x 3.4 MB/s | 3.9 MB 00:01 2022-03-21T13:56:02.029 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2022-03-21T13:56:02.030 INFO:teuthology.orchestra.run.smithi016.stdout:Total 12 MB/s | 187 MB 00:15 2022-03-21T13:56:02.093 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2022-03-21T13:56:02.129 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/133 2022-03-21T13:56:02.230 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2022-03-21T13:56:02.230 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2022-03-21T13:56:02.294 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/133 2022-03-21T13:56:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/133 2022-03-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2022-03-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2022-03-21T13:56:03.619 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:04.260 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:04.374 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:04.577 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:56:04.661 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:56:04.924 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 7/133 2022-03-21T13:56:05.085 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/133 2022-03-21T13:56:05.511 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/133 2022-03-21T13:56:05.718 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:05.830 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:06.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 10/133 2022-03-21T13:56:06.258 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 11/133 2022-03-21T13:56:06.418 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:06.526 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:06.659 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:06.822 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2022-03-21T13:56:06.907 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-03-21T13:56:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/133 2022-03-21T13:56:07.284 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:07.337 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:07.486 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/133 2022-03-21T13:56:07.577 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:07.669 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/133 2022-03-21T13:56:07.852 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:08.181 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:08.197 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/133 2022-03-21T13:56:08.356 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 16/133 2022-03-21T13:56:08.435 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:09.007 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:09.062 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2022-03-21T13:56:09.084 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 17/133 2022-03-21T13:56:09.091 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-03-21T13:56:09.201 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:09.232 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 17/133 2022-03-21T13:56:09.406 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:09.417 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 18/133 2022-03-21T13:56:09.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/133 2022-03-21T13:56:09.735 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:56:09.899 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/133 2022-03-21T13:56:09.912 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:56:09.953 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 18/133 2022-03-21T13:56:10.151 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 7/133 2022-03-21T13:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 19/133 2022-03-21T13:56:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/133 2022-03-21T13:56:10.318 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/133 2022-03-21T13:56:10.574 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 3/133 2022-03-21T13:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 19/133 2022-03-21T13:56:10.897 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/133 2022-03-21T13:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.12-1.el8.x86_64 20/133 2022-03-21T13:56:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 3/133 2022-03-21T13:56:11.134 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:56:11.638 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 20/133 2022-03-21T13:56:11.640 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:11.648 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 5/133 2022-03-21T13:56:11.904 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 10/133 2022-03-21T13:56:11.906 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 21/133 2022-03-21T13:56:12.059 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 6/133 2022-03-21T13:56:12.095 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 11/133 2022-03-21T13:56:12.192 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/133 2022-03-21T13:56:12.259 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:12.266 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/133 2022-03-21T13:56:12.317 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/133 2022-03-21T13:56:12.422 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:12.432 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/133 2022-03-21T13:56:12.559 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:12.695 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/133 2022-03-21T13:56:12.956 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/133 2022-03-21T13:56:13.004 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/133 2022-03-21T13:56:13.044 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:13.236 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:56:13.240 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 26/133 2022-03-21T13:56:13.351 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:13.460 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 27/133 2022-03-21T13:56:13.488 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 10/133 2022-03-21T13:56:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 11/133 2022-03-21T13:56:13.795 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 12/133 2022-03-21T13:56:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:14.444 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 28/133 2022-03-21T13:56:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 13/133 2022-03-21T13:56:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:14.808 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 29/133 2022-03-21T13:56:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 16/133 2022-03-21T13:56:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 14/133 2022-03-21T13:56:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 17/133 2022-03-21T13:56:15.149 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 30/133 2022-03-21T13:56:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 17/133 2022-03-21T13:56:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:15.459 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/133 2022-03-21T13:56:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 18/133 2022-03-21T13:56:15.683 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/133 2022-03-21T13:56:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 15/133 2022-03-21T13:56:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 18/133 2022-03-21T13:56:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/133 2022-03-21T13:56:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 19/133 2022-03-21T13:56:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/133 2022-03-21T13:56:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/133 2022-03-21T13:56:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 19/133 2022-03-21T13:56:17.315 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 19/133 2022-03-21T13:56:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 20/133 2022-03-21T13:56:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 20/133 2022-03-21T13:56:17.500 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.12-1.el8.x86_64 20/133 2022-03-21T13:56:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 21/133 2022-03-21T13:56:17.712 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/133 2022-03-21T13:56:18.072 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 33/133 2022-03-21T13:56:18.106 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 21/133 2022-03-21T13:56:18.265 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 22/133 2022-03-21T13:56:18.340 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/133 2022-03-21T13:56:18.532 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 20/133 2022-03-21T13:56:18.552 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/133 2022-03-21T13:56:18.766 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 36/133 2022-03-21T13:56:18.828 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 21/133 2022-03-21T13:56:18.853 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 22/133 2022-03-21T13:56:18.978 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 37/133 2022-03-21T13:56:19.081 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.12-1.el8.x86_64 23/133 2022-03-21T13:56:19.128 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 38/133 2022-03-21T13:56:19.154 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/133 2022-03-21T13:56:19.333 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 39/133 2022-03-21T13:56:19.422 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/133 2022-03-21T13:56:19.520 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 40/133 2022-03-21T13:56:19.686 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/133 2022-03-21T13:56:19.737 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 41/133 2022-03-21T13:56:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 23/133 2022-03-21T13:56:19.993 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 42/133 2022-03-21T13:56:19.997 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/133 2022-03-21T13:56:20.173 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 24/133 2022-03-21T13:56:20.208 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 26/133 2022-03-21T13:56:20.339 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 27/133 2022-03-21T13:56:20.402 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 43/133 2022-03-21T13:56:20.695 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 44/133 2022-03-21T13:56:20.834 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/133 2022-03-21T13:56:20.935 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 45/133 2022-03-21T13:56:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 26/133 2022-03-21T13:56:21.227 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 46/133 2022-03-21T13:56:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 27/133 2022-03-21T13:56:21.498 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 47/133 2022-03-21T13:56:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 28/133 2022-03-21T13:56:21.563 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 28/133 2022-03-21T13:56:21.781 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/133 2022-03-21T13:56:21.859 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 48/133 2022-03-21T13:56:21.972 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 29/133 2022-03-21T13:56:22.090 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 49/133 2022-03-21T13:56:22.267 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 50/133 2022-03-21T13:56:22.274 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 30/133 2022-03-21T13:56:22.526 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 51/133 2022-03-21T13:56:22.583 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/133 2022-03-21T13:56:22.718 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.7-12.el8.x86_64 52/133 2022-03-21T13:56:22.833 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/133 2022-03-21T13:56:22.919 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 53/133 2022-03-21T13:56:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 54/133 2022-03-21T13:56:23.319 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 55/133 2022-03-21T13:56:23.663 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 56/133 2022-03-21T13:56:23.967 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/133 2022-03-21T13:56:24.027 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 57/133 2022-03-21T13:56:24.315 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/133 2022-03-21T13:56:24.676 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 59/133 2022-03-21T13:56:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 30/133 2022-03-21T13:56:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 60/133 2022-03-21T13:56:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 31/133 2022-03-21T13:56:25.583 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/133 2022-03-21T13:56:25.602 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 61/133 2022-03-21T13:56:25.805 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/133 2022-03-21T13:56:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 32/133 2022-03-21T13:56:25.919 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/133 2022-03-21T13:56:25.926 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 33/133 2022-03-21T13:56:26.068 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/133 2022-03-21T13:56:26.140 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.5-12.el8.x86_64 63/133 2022-03-21T13:56:26.290 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/133 2022-03-21T13:56:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/133 2022-03-21T13:56:26.347 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/133 2022-03-21T13:56:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/133 2022-03-21T13:56:26.604 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/133 2022-03-21T13:56:26.783 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 36/133 2022-03-21T13:56:26.864 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/133 2022-03-21T13:56:26.875 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : automake-1.16.1-7.el8.noarch 66/133 2022-03-21T13:56:27.011 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 37/133 2022-03-21T13:56:27.092 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 36/133 2022-03-21T13:56:27.107 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 67/133 2022-03-21T13:56:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 38/133 2022-03-21T13:56:27.281 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/133 2022-03-21T13:56:27.283 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 37/133 2022-03-21T13:56:27.431 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 38/133 2022-03-21T13:56:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 39/133 2022-03-21T13:56:27.614 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 39/133 2022-03-21T13:56:27.636 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/133 2022-03-21T13:56:27.670 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 40/133 2022-03-21T13:56:27.783 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 40/133 2022-03-21T13:56:27.864 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 41/133 2022-03-21T13:56:27.897 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/133 2022-03-21T13:56:27.953 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 41/133 2022-03-21T13:56:28.025 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 41/133 2022-03-21T13:56:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/133 2022-03-21T13:56:28.132 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.5-12.el8.x86_64 42/133 2022-03-21T13:56:28.233 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/133 2022-03-21T13:56:28.313 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 42/133 2022-03-21T13:56:28.324 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 43/133 2022-03-21T13:56:28.471 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/133 2022-03-21T13:56:28.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 44/133 2022-03-21T13:56:28.706 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/133 2022-03-21T13:56:28.818 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 43/133 2022-03-21T13:56:28.853 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.1-7.el8.noarch 45/133 2022-03-21T13:56:28.921 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/133 2022-03-21T13:56:29.031 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 46/133 2022-03-21T13:56:29.158 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/133 2022-03-21T13:56:29.233 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 44/133 2022-03-21T13:56:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/133 2022-03-21T13:56:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/133 2022-03-21T13:56:29.453 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 45/133 2022-03-21T13:56:29.518 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/133 2022-03-21T13:56:29.628 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/133 2022-03-21T13:56:29.739 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 46/133 2022-03-21T13:56:29.789 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 49/133 2022-03-21T13:56:29.860 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/133 2022-03-21T13:56:29.965 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 47/133 2022-03-21T13:56:30.086 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/133 2022-03-21T13:56:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 50/133 2022-03-21T13:56:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 48/133 2022-03-21T13:56:30.371 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/133 2022-03-21T13:56:30.502 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 51/133 2022-03-21T13:56:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 49/133 2022-03-21T13:56:30.656 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/133 2022-03-21T13:56:30.728 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 52/133 2022-03-21T13:56:30.779 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 50/133 2022-03-21T13:56:30.874 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/133 2022-03-21T13:56:31.021 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 53/133 2022-03-21T13:56:31.028 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 51/133 2022-03-21T13:56:31.071 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/133 2022-03-21T13:56:31.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 54/133 2022-03-21T13:56:31.352 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.7-12.el8.x86_64 52/133 2022-03-21T13:56:31.589 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 53/133 2022-03-21T13:56:31.702 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 55/133 2022-03-21T13:56:31.810 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 54/133 2022-03-21T13:56:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 56/133 2022-03-21T13:56:32.008 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 55/133 2022-03-21T13:56:32.167 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 57/133 2022-03-21T13:56:32.243 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/133 2022-03-21T13:56:32.420 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 56/133 2022-03-21T13:56:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 58/133 2022-03-21T13:56:32.585 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/133 2022-03-21T13:56:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.7-12.el8.x86_64 59/133 2022-03-21T13:56:32.777 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 57/133 2022-03-21T13:56:32.830 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/133 2022-03-21T13:56:32.838 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 60/133 2022-03-21T13:56:33.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 61/133 2022-03-21T13:56:33.086 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/133 2022-03-21T13:56:33.181 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/133 2022-03-21T13:56:33.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 62/133 2022-03-21T13:56:33.379 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 59/133 2022-03-21T13:56:33.479 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/133 2022-03-21T13:56:33.568 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 60/133 2022-03-21T13:56:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 63/133 2022-03-21T13:56:33.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/133 2022-03-21T13:56:33.781 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 61/133 2022-03-21T13:56:33.912 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/133 2022-03-21T13:56:33.964 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/133 2022-03-21T13:56:33.976 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/133 2022-03-21T13:56:34.089 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/133 2022-03-21T13:56:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/133 2022-03-21T13:56:34.325 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.5-12.el8.x86_64 63/133 2022-03-21T13:56:34.534 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/133 2022-03-21T13:56:34.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 66/133 2022-03-21T13:56:34.773 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 67/133 2022-03-21T13:56:34.854 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/133 2022-03-21T13:56:34.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/133 2022-03-21T13:56:35.088 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.1-7.el8.noarch 66/133 2022-03-21T13:56:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/133 2022-03-21T13:56:35.301 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 67/133 2022-03-21T13:56:35.446 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/133 2022-03-21T13:56:35.512 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/133 2022-03-21T13:56:35.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/133 2022-03-21T13:56:35.814 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/133 2022-03-21T13:56:35.847 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/133 2022-03-21T13:56:36.030 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/133 2022-03-21T13:56:36.091 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/133 2022-03-21T13:56:36.237 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/133 2022-03-21T13:56:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/133 2022-03-21T13:56:36.422 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/133 2022-03-21T13:56:36.548 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/133 2022-03-21T13:56:36.627 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/133 2022-03-21T13:56:36.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/133 2022-03-21T13:56:36.869 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/133 2022-03-21T13:56:36.945 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/133 2022-03-21T13:56:37.109 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/133 2022-03-21T13:56:37.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/133 2022-03-21T13:56:37.352 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/133 2022-03-21T13:56:37.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/133 2022-03-21T13:56:37.573 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/133 2022-03-21T13:56:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/133 2022-03-21T13:56:37.821 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/133 2022-03-21T13:56:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/133 2022-03-21T13:56:38.048 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/133 2022-03-21T13:56:38.215 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/133 2022-03-21T13:56:38.255 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/133 2022-03-21T13:56:38.400 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/133 2022-03-21T13:56:38.571 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/133 2022-03-21T13:56:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/133 2022-03-21T13:56:38.825 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/133 2022-03-21T13:56:38.884 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:39.046 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/133 2022-03-21T13:56:39.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/133 2022-03-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/133 2022-03-21T13:56:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:40.010 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:40.161 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:56:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/133 2022-03-21T13:56:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/133 2022-03-21T13:56:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/133 2022-03-21T13:56:40.938 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/133 2022-03-21T13:56:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/133 2022-03-21T13:56:41.294 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/133 2022-03-21T13:56:41.594 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/133 2022-03-21T13:56:41.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/133 2022-03-21T13:56:41.993 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/133 2022-03-21T13:56:42.037 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/133 2022-03-21T13:56:42.298 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/133 2022-03-21T13:56:43.044 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/133 2022-03-21T13:56:43.280 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/133 2022-03-21T13:56:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:47.337 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:47.951 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:56:50.529 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:50.763 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:51.588 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 92/133 2022-03-21T13:56:51.858 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:52.104 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 93/133 2022-03-21T13:56:52.228 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:56:57.997 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:56:57.999 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2022-03-21T13:56:58.000 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2022-03-21T13:56:58.004 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2022-03-21T13:56:58.004 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /run 2022-03-21T13:56:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2022-03-21T13:56:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2022-03-21T13:56:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2022-03-21T13:56:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2022-03-21T13:56:58.006 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2022-03-21T13:56:58.006 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:56:58.365 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:57:00.040 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:00.265 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:00.464 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:00.854 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:01.073 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:01.091 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:03.436 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:03.646 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:03.828 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 100/133 2022-03-21T13:57:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 101/133 2022-03-21T13:57:04.222 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:04.427 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:04.580 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:05.545 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:05.722 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:05.819 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:06.234 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 105/133 2022-03-21T13:57:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:06.629 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:06.641 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:07.404 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:57:07.404 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2022-03-21T13:57:07.405 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2022-03-21T13:57:07.405 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2022-03-21T13:57:07.405 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /run 2022-03-21T13:57:07.405 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2022-03-21T13:57:07.406 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2022-03-21T13:57:07.406 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2022-03-21T13:57:07.406 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2022-03-21T13:57:07.406 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2022-03-21T13:57:07.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:57:07.690 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 108/133 2022-03-21T13:57:07.817 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:57:07.954 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:08.030 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:08.238 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:08.283 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:08.480 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:08.866 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:09.071 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:09.413 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:09.878 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:10.462 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:10.667 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:10.686 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:12.850 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 94/133 2022-03-21T13:57:12.850 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2022-03-21T13:57:12.850 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2022-03-21T13:57:12.850 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2022-03-21T13:57:12.851 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /run 2022-03-21T13:57:12.851 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2022-03-21T13:57:12.851 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2022-03-21T13:57:12.851 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2022-03-21T13:57:12.852 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2022-03-21T13:57:12.852 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2022-03-21T13:57:12.852 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:57:13.282 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:13.292 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:57:13.423 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:13.508 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 100/133 2022-03-21T13:57:13.756 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 113/133 2022-03-21T13:57:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 101/133 2022-03-21T13:57:13.928 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libtool-2.4.6-25.el8.x86_64 114/133 2022-03-21T13:57:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:14.641 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 114/133 2022-03-21T13:57:14.956 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 115/133 2022-03-21T13:57:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 95/133 2022-03-21T13:57:15.255 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 116/133 2022-03-21T13:57:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:15.474 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 117/133 2022-03-21T13:57:15.632 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 118/133 2022-03-21T13:57:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 96/133 2022-03-21T13:57:15.797 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 119/133 2022-03-21T13:57:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:15.998 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 120/133 2022-03-21T13:57:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:16.191 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 121/133 2022-03-21T13:57:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 97/133 2022-03-21T13:57:16.395 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 122/133 2022-03-21T13:57:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 105/133 2022-03-21T13:57:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 98/133 2022-03-21T13:57:16.611 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 123/133 2022-03-21T13:57:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:16.812 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:17.012 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 125/133 2022-03-21T13:57:17.252 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : bison-3.0.4-10.el8.x86_64 126/133 2022-03-21T13:57:17.306 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 126/133 2022-03-21T13:57:17.527 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flex-2.6.1-9.el8.x86_64 127/133 2022-03-21T13:57:17.537 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 127/133 2022-03-21T13:57:17.662 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 128/133 2022-03-21T13:57:17.663 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:17.871 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:17.991 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 108/133 2022-03-21T13:57:18.046 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 130/133 2022-03-21T13:57:18.056 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:18.056 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:18.221 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:18.304 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:18.507 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:18.539 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:18.540 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:18.556 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:18.869 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:18.870 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:19.163 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:19.766 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:19.767 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 1/133 2022-03-21T13:57:19.767 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2/133 2022-03-21T13:57:19.767 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 3/133 2022-03-21T13:57:19.767 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:57:19.768 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11208.gd379 5/133 2022-03-21T13:57:19.768 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:57:19.768 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 7/133 2022-03-21T13:57:19.768 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 8/133 2022-03-21T13:57:19.769 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:57:19.769 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 10/133 2022-03-21T13:57:19.769 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 11/133 2022-03-21T13:57:19.769 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 12/133 2022-03-21T13:57:19.769 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 13/133 2022-03-21T13:57:19.770 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 14/133 2022-03-21T13:57:19.770 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 15/133 2022-03-21T13:57:19.770 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 16/133 2022-03-21T13:57:19.770 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 17/133 2022-03-21T13:57:19.771 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 18/133 2022-03-21T13:57:19.771 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 19/133 2022-03-21T13:57:19.771 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 20/133 2022-03-21T13:57:19.771 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 21/133 2022-03-21T13:57:19.772 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 22/133 2022-03-21T13:57:19.772 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 23/133 2022-03-21T13:57:19.772 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 24/133 2022-03-21T13:57:19.772 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 25/133 2022-03-21T13:57:19.772 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 26/133 2022-03-21T13:57:19.773 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 27/133 2022-03-21T13:57:19.773 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 28/133 2022-03-21T13:57:19.773 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 29/133 2022-03-21T13:57:19.773 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 30/133 2022-03-21T13:57:19.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 31/133 2022-03-21T13:57:19.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 32/133 2022-03-21T13:57:19.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 33/133 2022-03-21T13:57:19.774 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 34/133 2022-03-21T13:57:19.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 35/133 2022-03-21T13:57:19.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 36/133 2022-03-21T13:57:19.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 37/133 2022-03-21T13:57:19.775 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 38/133 2022-03-21T13:57:19.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/133 2022-03-21T13:57:19.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/133 2022-03-21T13:57:19.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/133 2022-03-21T13:57:19.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/133 2022-03-21T13:57:19.776 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/133 2022-03-21T13:57:19.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/133 2022-03-21T13:57:19.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/133 2022-03-21T13:57:19.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/133 2022-03-21T13:57:19.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/133 2022-03-21T13:57:19.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/133 2022-03-21T13:57:19.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/133 2022-03-21T13:57:19.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/133 2022-03-21T13:57:19.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/133 2022-03-21T13:57:19.778 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/133 2022-03-21T13:57:19.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/133 2022-03-21T13:57:19.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/133 2022-03-21T13:57:19.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/133 2022-03-21T13:57:19.779 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/133 2022-03-21T13:57:19.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/133 2022-03-21T13:57:19.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/133 2022-03-21T13:57:19.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/133 2022-03-21T13:57:19.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/133 2022-03-21T13:57:19.780 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/133 2022-03-21T13:57:19.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/133 2022-03-21T13:57:19.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/133 2022-03-21T13:57:19.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/133 2022-03-21T13:57:19.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/133 2022-03-21T13:57:19.782 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/133 2022-03-21T13:57:19.782 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/133 2022-03-21T13:57:19.782 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/133 2022-03-21T13:57:19.782 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/133 2022-03-21T13:57:19.782 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 70/133 2022-03-21T13:57:19.783 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 71/133 2022-03-21T13:57:19.783 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flex-2.6.1-9.el8.x86_64 72/133 2022-03-21T13:57:19.783 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.5-12.el8.x86_64 73/133 2022-03-21T13:57:19.783 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/133 2022-03-21T13:57:19.784 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 75/133 2022-03-21T13:57:19.784 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 76/133 2022-03-21T13:57:19.784 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : automake-1.16.1-7.el8.noarch 77/133 2022-03-21T13:57:19.784 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 78/133 2022-03-21T13:57:19.784 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 79/133 2022-03-21T13:57:19.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 80/133 2022-03-21T13:57:19.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/133 2022-03-21T13:57:19.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 82/133 2022-03-21T13:57:19.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/133 2022-03-21T13:57:19.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/133 2022-03-21T13:57:19.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 85/133 2022-03-21T13:57:19.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 86/133 2022-03-21T13:57:19.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 87/133 2022-03-21T13:57:19.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 88/133 2022-03-21T13:57:19.787 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 89/133 2022-03-21T13:57:19.787 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 90/133 2022-03-21T13:57:19.787 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 91/133 2022-03-21T13:57:19.787 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 92/133 2022-03-21T13:57:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 93/133 2022-03-21T13:57:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 94/133 2022-03-21T13:57:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : bison-3.0.4-10.el8.x86_64 95/133 2022-03-21T13:57:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 96/133 2022-03-21T13:57:19.788 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 97/133 2022-03-21T13:57:19.789 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 98/133 2022-03-21T13:57:19.789 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 99/133 2022-03-21T13:57:19.789 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 100/133 2022-03-21T13:57:19.789 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 101/133 2022-03-21T13:57:19.790 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 102/133 2022-03-21T13:57:19.790 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 103/133 2022-03-21T13:57:19.790 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/133 2022-03-21T13:57:19.790 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 105/133 2022-03-21T13:57:19.790 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 106/133 2022-03-21T13:57:19.791 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 107/133 2022-03-21T13:57:19.791 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/133 2022-03-21T13:57:19.791 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 109/133 2022-03-21T13:57:19.791 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 110/133 2022-03-21T13:57:19.792 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/133 2022-03-21T13:57:19.792 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 112/133 2022-03-21T13:57:19.792 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 113/133 2022-03-21T13:57:19.792 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 114/133 2022-03-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 115/133 2022-03-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 116/133 2022-03-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 117/133 2022-03-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 118/133 2022-03-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 119/133 2022-03-21T13:57:19.794 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 120/133 2022-03-21T13:57:19.794 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 121/133 2022-03-21T13:57:19.794 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 122/133 2022-03-21T13:57:19.794 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 123/133 2022-03-21T13:57:19.795 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:19.795 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 125/133 2022-03-21T13:57:19.795 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 126/133 2022-03-21T13:57:19.795 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 127/133 2022-03-21T13:57:19.795 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 128/133 2022-03-21T13:57:19.796 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 129/133 2022-03-21T13:57:19.796 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 130/133 2022-03-21T13:57:19.796 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:19.796 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 132/133 2022-03-21T13:57:20.947 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:21.150 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 99/133 2022-03-21T13:57:21.383 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 100/133 2022-03-21T13:57:21.722 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 101/133 2022-03-21T13:57:21.898 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:22.201 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 102/133 2022-03-21T13:57:22.411 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:23.705 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 103/133 2022-03-21T13:57:23.906 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:23.990 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:24.082 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 104/133 2022-03-21T13:57:24.513 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 105/133 2022-03-21T13:57:24.720 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:24.817 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 106/133 2022-03-21T13:57:25.006 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 113/133 2022-03-21T13:57:25.056 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 107/133 2022-03-21T13:57:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 114/133 2022-03-21T13:57:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 115/133 2022-03-21T13:57:26.106 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 116/133 2022-03-21T13:57:26.324 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 108/133 2022-03-21T13:57:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-25.el8.x86_64 117/133 2022-03-21T13:57:26.345 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 117/133 2022-03-21T13:57:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 118/133 2022-03-21T13:57:26.553 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:26.719 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 109/133 2022-03-21T13:57:26.726 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 119/133 2022-03-21T13:57:26.898 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 120/133 2022-03-21T13:57:27.002 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11208.gd379 110/133 2022-03-21T13:57:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 121/133 2022-03-21T13:57:27.181 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 122/133 2022-03-21T13:57:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 123/133 2022-03-21T13:57:27.712 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 111/133 2022-03-21T13:57:27.825 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:27.865 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:28.039 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.0.4-10.el8.x86_64 125/133 2022-03-21T13:57:28.096 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 125/133 2022-03-21T13:57:28.322 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.1-9.el8.x86_64 126/133 2022-03-21T13:57:28.333 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 126/133 2022-03-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 127/133 2022-03-21T13:57:28.683 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 128/133 2022-03-21T13:57:28.683 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:28.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:29.100 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 130/133 2022-03-21T13:57:29.110 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:29.111 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:30.839 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:30.839 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 1/133 2022-03-21T13:57:30.839 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2/133 2022-03-21T13:57:30.840 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 3/133 2022-03-21T13:57:30.840 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:57:30.840 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11208.gd379 5/133 2022-03-21T13:57:30.840 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:57:30.841 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 7/133 2022-03-21T13:57:30.841 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 8/133 2022-03-21T13:57:30.841 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:57:30.841 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 10/133 2022-03-21T13:57:30.841 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 11/133 2022-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 12/133 2022-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 13/133 2022-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 14/133 2022-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 15/133 2022-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 16/133 2022-03-21T13:57:30.843 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 17/133 2022-03-21T13:57:30.843 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 18/133 2022-03-21T13:57:30.843 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 19/133 2022-03-21T13:57:30.843 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 20/133 2022-03-21T13:57:30.843 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 21/133 2022-03-21T13:57:30.844 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 22/133 2022-03-21T13:57:30.844 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 23/133 2022-03-21T13:57:30.844 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 24/133 2022-03-21T13:57:30.844 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 25/133 2022-03-21T13:57:30.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 26/133 2022-03-21T13:57:30.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 27/133 2022-03-21T13:57:30.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 28/133 2022-03-21T13:57:30.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 29/133 2022-03-21T13:57:30.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 30/133 2022-03-21T13:57:30.855 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 31/133 2022-03-21T13:57:30.855 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 32/133 2022-03-21T13:57:30.855 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 33/133 2022-03-21T13:57:30.855 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 34/133 2022-03-21T13:57:30.856 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 35/133 2022-03-21T13:57:30.856 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 36/133 2022-03-21T13:57:30.856 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 37/133 2022-03-21T13:57:30.856 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 38/133 2022-03-21T13:57:30.856 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/133 2022-03-21T13:57:30.857 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/133 2022-03-21T13:57:30.857 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/133 2022-03-21T13:57:30.857 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/133 2022-03-21T13:57:30.857 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/133 2022-03-21T13:57:30.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/133 2022-03-21T13:57:30.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/133 2022-03-21T13:57:30.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/133 2022-03-21T13:57:30.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/133 2022-03-21T13:57:30.858 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/133 2022-03-21T13:57:30.859 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/133 2022-03-21T13:57:30.859 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/133 2022-03-21T13:57:30.859 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/133 2022-03-21T13:57:30.859 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/133 2022-03-21T13:57:30.859 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/133 2022-03-21T13:57:30.860 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/133 2022-03-21T13:57:30.860 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/133 2022-03-21T13:57:30.860 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/133 2022-03-21T13:57:30.860 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/133 2022-03-21T13:57:30.861 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/133 2022-03-21T13:57:30.861 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/133 2022-03-21T13:57:30.861 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/133 2022-03-21T13:57:30.861 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/133 2022-03-21T13:57:30.861 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/133 2022-03-21T13:57:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/133 2022-03-21T13:57:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/133 2022-03-21T13:57:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/133 2022-03-21T13:57:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/133 2022-03-21T13:57:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/133 2022-03-21T13:57:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/133 2022-03-21T13:57:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/133 2022-03-21T13:57:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 70/133 2022-03-21T13:57:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 71/133 2022-03-21T13:57:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/133 2022-03-21T13:57:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/133 2022-03-21T13:57:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 74/133 2022-03-21T13:57:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 75/133 2022-03-21T13:57:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 76/133 2022-03-21T13:57:30.865 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 77/133 2022-03-21T13:57:30.865 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 78/133 2022-03-21T13:57:30.865 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 79/133 2022-03-21T13:57:30.865 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/133 2022-03-21T13:57:30.865 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 81/133 2022-03-21T13:57:30.866 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 82/133 2022-03-21T13:57:30.866 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 83/133 2022-03-21T13:57:30.866 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 84/133 2022-03-21T13:57:30.866 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 85/133 2022-03-21T13:57:30.866 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 86/133 2022-03-21T13:57:30.867 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 87/133 2022-03-21T13:57:30.867 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/133 2022-03-21T13:57:30.867 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 89/133 2022-03-21T13:57:30.867 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 90/133 2022-03-21T13:57:30.867 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/133 2022-03-21T13:57:30.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 92/133 2022-03-21T13:57:30.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 93/133 2022-03-21T13:57:30.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 94/133 2022-03-21T13:57:30.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 95/133 2022-03-21T13:57:30.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 96/133 2022-03-21T13:57:30.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.1-9.el8.x86_64 97/133 2022-03-21T13:57:30.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.5-12.el8.x86_64 98/133 2022-03-21T13:57:30.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 99/133 2022-03-21T13:57:30.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 100/133 2022-03-21T13:57:30.870 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 101/133 2022-03-21T13:57:30.870 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.1-7.el8.noarch 102/133 2022-03-21T13:57:30.870 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 103/133 2022-03-21T13:57:30.870 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 104/133 2022-03-21T13:57:30.870 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 105/133 2022-03-21T13:57:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 106/133 2022-03-21T13:57:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 107/133 2022-03-21T13:57:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 108/133 2022-03-21T13:57:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 109/133 2022-03-21T13:57:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 110/133 2022-03-21T13:57:30.872 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/133 2022-03-21T13:57:30.872 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 112/133 2022-03-21T13:57:30.872 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 113/133 2022-03-21T13:57:30.872 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 114/133 2022-03-21T13:57:30.872 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 115/133 2022-03-21T13:57:30.873 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 116/133 2022-03-21T13:57:30.873 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 117/133 2022-03-21T13:57:30.873 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 118/133 2022-03-21T13:57:30.873 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 119/133 2022-03-21T13:57:30.874 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.0.4-10.el8.x86_64 120/133 2022-03-21T13:57:30.874 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 121/133 2022-03-21T13:57:30.875 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 122/133 2022-03-21T13:57:30.875 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 123/133 2022-03-21T13:57:30.875 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:30.875 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 125/133 2022-03-21T13:57:30.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 126/133 2022-03-21T13:57:30.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 127/133 2022-03-21T13:57:30.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 128/133 2022-03-21T13:57:30.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 129/133 2022-03-21T13:57:30.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 130/133 2022-03-21T13:57:30.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:30.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 132/133 2022-03-21T13:57:33.347 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 112/133 2022-03-21T13:57:33.750 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 113/133 2022-03-21T13:57:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-25.el8.x86_64 114/133 2022-03-21T13:57:34.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 133/133 2022-03-21T13:57:34.549 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:57:28 PM UTC. 2022-03-21T13:57:35.268 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 114/133 2022-03-21T13:57:35.547 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 115/133 2022-03-21T13:57:35.839 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 116/133 2022-03-21T13:57:36.035 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 117/133 2022-03-21T13:57:36.186 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 118/133 2022-03-21T13:57:36.367 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 119/133 2022-03-21T13:57:36.547 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 120/133 2022-03-21T13:57:36.903 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 121/133 2022-03-21T13:57:37.056 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 122/133 2022-03-21T13:57:37.259 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 123/133 2022-03-21T13:57:37.423 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:37.705 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 125/133 2022-03-21T13:57:37.921 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.0.4-10.el8.x86_64 126/133 2022-03-21T13:57:38.029 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 126/133 2022-03-21T13:57:38.254 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.1-9.el8.x86_64 127/133 2022-03-21T13:57:38.297 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 127/133 2022-03-21T13:57:38.299 INFO:teuthology.orchestra.run.smithi079.stdout:Installed products updated. 2022-03-21T13:57:38.419 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:57:38.420 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2022-03-21T13:57:38.420 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:38.420 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:38.420 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:38.421 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:38.421 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.421 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.421 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2022-03-21T13:57:38.422 INFO:teuthology.orchestra.run.smithi079.stdout: automake-1.16.1-7.el8.noarch 2022-03-21T13:57:38.422 INFO:teuthology.orchestra.run.smithi079.stdout: bison-3.0.4-10.el8.x86_64 2022-03-21T13:57:38.422 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.422 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.423 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.423 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.423 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.423 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.424 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.424 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.424 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.424 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.425 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.425 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.425 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.425 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.425 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.426 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.426 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.426 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.426 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.427 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.427 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.427 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.427 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:38.428 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-03-21T13:57:38.428 INFO:teuthology.orchestra.run.smithi079.stdout: flex-2.6.1-9.el8.x86_64 2022-03-21T13:57:38.428 INFO:teuthology.orchestra.run.smithi079.stdout: fuse-2.9.7-12.el8.x86_64 2022-03-21T13:57:38.428 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-03-21T13:57:38.428 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.5-12.el8.x86_64 2022-03-21T13:57:38.429 INFO:teuthology.orchestra.run.smithi079.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-03-21T13:57:38.429 INFO:teuthology.orchestra.run.smithi079.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-03-21T13:57:38.429 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-03-21T13:57:38.429 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.430 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.430 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.430 INFO:teuthology.orchestra.run.smithi079.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-03-21T13:57:38.430 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-03-21T13:57:38.431 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.2-3.el8.x86_64 2022-03-21T13:57:38.431 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-03-21T13:57:38.431 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-03-21T13:57:38.431 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.431 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.432 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-03-21T13:57:38.432 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.432 INFO:teuthology.orchestra.run.smithi079.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-03-21T13:57:38.432 INFO:teuthology.orchestra.run.smithi079.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-03-21T13:57:38.433 INFO:teuthology.orchestra.run.smithi079.stdout: libtool-2.4.6-25.el8.x86_64 2022-03-21T13:57:38.433 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.3.1-3.el8.x86_64 2022-03-21T13:57:38.433 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-03-21T13:57:38.433 INFO:teuthology.orchestra.run.smithi079.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-03-21T13:57:38.434 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-03-21T13:57:38.434 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-03-21T13:57:38.434 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.12-1.el8.x86_64 2022-03-21T13:57:38.434 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-03-21T13:57:38.435 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-03-21T13:57:38.435 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-03-21T13:57:38.435 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-03-21T13:57:38.435 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-03-21T13:57:38.435 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-03-21T13:57:38.436 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.5.1-5.el8.noarch 2022-03-21T13:57:38.436 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-03-21T13:57:38.436 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-03-21T13:57:38.436 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-03-21T13:57:38.437 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.437 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.437 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.437 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-03-21T13:57:38.438 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-03-21T13:57:38.438 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-03-21T13:57:38.438 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-03-21T13:57:38.438 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-03-21T13:57:38.439 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-03-21T13:57:38.439 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-6.2-6.el8.noarch 2022-03-21T13:57:38.439 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-03-21T13:57:38.439 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-03-21T13:57:38.439 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-03-21T13:57:38.440 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-03-21T13:57:38.440 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-03-21T13:57:38.440 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-03-21T13:57:38.440 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-03-21T13:57:38.441 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.0.6-13.el8.noarch 2022-03-21T13:57:38.441 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-03-21T13:57:38.441 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-03-21T13:57:38.441 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-03-21T13:57:38.442 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-03-21T13:57:38.442 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-03-21T13:57:38.442 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-03-21T13:57:38.442 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-03-21T13:57:38.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-2.6-1.el8.noarch 2022-03-21T13:57:38.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-03-21T13:57:38.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-03-21T13:57:38.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-03-21T13:57:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-03-21T13:57:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.14-14.el8.noarch 2022-03-21T13:57:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2017.2-9.el8.noarch 2022-03-21T13:57:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-03-21T13:57:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-03-21T13:57:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.4.1-12.el8.noarch 2022-03-21T13:57:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.8-1.el8.noarch 2022-03-21T13:57:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-03-21T13:57:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-03-21T13:57:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-03-21T13:57:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-03-21T13:57:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-03-21T13:57:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-03-21T13:57:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-03-21T13:57:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-03-21T13:57:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-03-21T13:57:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-03-21T13:57:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-03-21T13:57:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.3.3-2.el8.x86_64 2022-03-21T13:57:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-03-21T13:57:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-03-21T13:57:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-03-21T13:57:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T13:57:38.451 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2022-03-21T13:57:38.458 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 128/133 2022-03-21T13:57:38.458 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:38.571 DEBUG:teuthology.parallel:result is None 2022-03-21T13:57:38.732 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 129/133 2022-03-21T13:57:38.865 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 130/133 2022-03-21T13:57:38.906 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:38.907 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 132/133 2022-03-21T13:57:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:40.692 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/133 2022-03-21T13:57:40.692 INFO:teuthology.orchestra.run.smithi002.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-03-21T13:57:40.693 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:57:41.624 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 1/133 2022-03-21T13:57:41.624 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2/133 2022-03-21T13:57:41.624 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 3/133 2022-03-21T13:57:41.624 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 4/133 2022-03-21T13:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11208.gd379 5/133 2022-03-21T13:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 6/133 2022-03-21T13:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 7/133 2022-03-21T13:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 8/133 2022-03-21T13:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 9/133 2022-03-21T13:57:41.626 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 10/133 2022-03-21T13:57:41.626 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 11/133 2022-03-21T13:57:41.626 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 12/133 2022-03-21T13:57:41.626 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_6 13/133 2022-03-21T13:57:41.627 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86 14/133 2022-03-21T13:57:41.627 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 15/133 2022-03-21T13:57:41.627 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_6 16/133 2022-03-21T13:57:41.627 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_ 17/133 2022-03-21T13:57:41.627 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x8 18/133 2022-03-21T13:57:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 19/133 2022-03-21T13:57:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.e 20/133 2022-03-21T13:57:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8 21/133 2022-03-21T13:57:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_ 22/133 2022-03-21T13:57:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_6 23/133 2022-03-21T13:57:41.629 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 24/133 2022-03-21T13:57:41.629 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 25/133 2022-03-21T13:57:41.629 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 26/133 2022-03-21T13:57:41.629 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 27/133 2022-03-21T13:57:41.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 28/133 2022-03-21T13:57:41.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c 29/133 2022-03-21T13:57:41.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.no 30/133 2022-03-21T13:57:41.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8. 31/133 2022-03-21T13:57:41.630 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3 32/133 2022-03-21T13:57:41.631 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.e 33/133 2022-03-21T13:57:41.631 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarc 34/133 2022-03-21T13:57:41.631 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c. 35/133 2022-03-21T13:57:41.631 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 36/133 2022-03-21T13:57:41.632 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 37/133 2022-03-21T13:57:41.632 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 38/133 2022-03-21T13:57:41.632 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/133 2022-03-21T13:57:41.632 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/133 2022-03-21T13:57:41.632 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/133 2022-03-21T13:57:41.633 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/133 2022-03-21T13:57:41.633 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/133 2022-03-21T13:57:41.633 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/133 2022-03-21T13:57:41.633 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/133 2022-03-21T13:57:41.633 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/133 2022-03-21T13:57:41.634 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/133 2022-03-21T13:57:41.634 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/133 2022-03-21T13:57:41.634 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/133 2022-03-21T13:57:41.634 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/133 2022-03-21T13:57:41.635 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/133 2022-03-21T13:57:41.635 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/133 2022-03-21T13:57:41.635 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/133 2022-03-21T13:57:41.635 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/133 2022-03-21T13:57:41.635 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/133 2022-03-21T13:57:41.636 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/133 2022-03-21T13:57:41.636 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/133 2022-03-21T13:57:41.636 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/133 2022-03-21T13:57:41.636 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/133 2022-03-21T13:57:41.636 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/133 2022-03-21T13:57:41.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/133 2022-03-21T13:57:41.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/133 2022-03-21T13:57:41.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/133 2022-03-21T13:57:41.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/133 2022-03-21T13:57:41.637 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/133 2022-03-21T13:57:41.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/133 2022-03-21T13:57:41.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/133 2022-03-21T13:57:41.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/133 2022-03-21T13:57:41.638 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/133 2022-03-21T13:57:41.639 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 70/133 2022-03-21T13:57:41.639 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 71/133 2022-03-21T13:57:41.639 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.1-9.el8.x86_64 72/133 2022-03-21T13:57:41.639 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.5-12.el8.x86_64 73/133 2022-03-21T13:57:41.640 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/133 2022-03-21T13:57:41.640 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 75/133 2022-03-21T13:57:41.640 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 76/133 2022-03-21T13:57:41.640 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.1-7.el8.noarch 77/133 2022-03-21T13:57:41.641 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 78/133 2022-03-21T13:57:41.641 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 79/133 2022-03-21T13:57:41.641 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 80/133 2022-03-21T13:57:41.641 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/133 2022-03-21T13:57:41.641 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 82/133 2022-03-21T13:57:41.642 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/133 2022-03-21T13:57:41.642 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/133 2022-03-21T13:57:41.642 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 85/133 2022-03-21T13:57:41.642 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 86/133 2022-03-21T13:57:41.643 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 87/133 2022-03-21T13:57:41.643 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 88/133 2022-03-21T13:57:41.643 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 89/133 2022-03-21T13:57:41.643 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 90/133 2022-03-21T13:57:41.643 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 91/133 2022-03-21T13:57:41.644 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 92/133 2022-03-21T13:57:41.644 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 93/133 2022-03-21T13:57:41.644 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 94/133 2022-03-21T13:57:41.644 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.0.4-10.el8.x86_64 95/133 2022-03-21T13:57:41.644 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 96/133 2022-03-21T13:57:41.645 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 97/133 2022-03-21T13:57:41.645 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 98/133 2022-03-21T13:57:41.645 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 99/133 2022-03-21T13:57:41.645 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 100/133 2022-03-21T13:57:41.646 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 101/133 2022-03-21T13:57:41.646 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 102/133 2022-03-21T13:57:41.646 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 103/133 2022-03-21T13:57:41.646 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/133 2022-03-21T13:57:41.646 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 105/133 2022-03-21T13:57:41.647 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 106/133 2022-03-21T13:57:41.647 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 107/133 2022-03-21T13:57:41.647 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/133 2022-03-21T13:57:41.647 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 109/133 2022-03-21T13:57:41.647 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 110/133 2022-03-21T13:57:41.648 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/133 2022-03-21T13:57:41.648 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 112/133 2022-03-21T13:57:41.648 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 113/133 2022-03-21T13:57:41.648 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 114/133 2022-03-21T13:57:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 115/133 2022-03-21T13:57:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 116/133 2022-03-21T13:57:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 117/133 2022-03-21T13:57:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 118/133 2022-03-21T13:57:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 119/133 2022-03-21T13:57:41.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 120/133 2022-03-21T13:57:41.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 121/133 2022-03-21T13:57:41.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 122/133 2022-03-21T13:57:41.650 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 123/133 2022-03-21T13:57:41.651 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 124/133 2022-03-21T13:57:41.651 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 125/133 2022-03-21T13:57:41.651 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 126/133 2022-03-21T13:57:41.651 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 127/133 2022-03-21T13:57:41.651 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 128/133 2022-03-21T13:57:41.652 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 129/133 2022-03-21T13:57:41.652 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 130/133 2022-03-21T13:57:41.652 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 131/133 2022-03-21T13:57:41.652 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 132/133 2022-03-21T13:57:46.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 133/133 2022-03-21T13:57:46.365 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:06 ago on Mon 21 Mar 2022 01:57:39 PM UTC. 2022-03-21T13:57:50.549 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2022-03-21T13:57:50.682 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:57:50.683 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2022-03-21T13:57:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:50.683 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:57:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.684 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.685 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2022-03-21T13:57:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.1-7.el8.noarch 2022-03-21T13:57:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.0.4-10.el8.x86_64 2022-03-21T13:57:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.685 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.686 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.687 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.688 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.689 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.690 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:57:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-03-21T13:57:50.691 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.1-9.el8.x86_64 2022-03-21T13:57:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.7-12.el8.x86_64 2022-03-21T13:57:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-03-21T13:57:50.692 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.5-12.el8.x86_64 2022-03-21T13:57:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-03-21T13:57:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-03-21T13:57:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-03-21T13:57:50.693 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-03-21T13:57:50.694 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-03-21T13:57:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.2-3.el8.x86_64 2022-03-21T13:57:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-03-21T13:57:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-03-21T13:57:50.695 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-03-21T13:57:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.696 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-03-21T13:57:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-03-21T13:57:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-25.el8.x86_64 2022-03-21T13:57:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.3.1-3.el8.x86_64 2022-03-21T13:57:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-03-21T13:57:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-03-21T13:57:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-03-21T13:57:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-03-21T13:57:50.698 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.12-1.el8.x86_64 2022-03-21T13:57:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-03-21T13:57:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-03-21T13:57:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-03-21T13:57:50.699 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-03-21T13:57:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-03-21T13:57:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-03-21T13:57:50.700 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.5.1-5.el8.noarch 2022-03-21T13:57:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-03-21T13:57:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-03-21T13:57:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-03-21T13:57:50.701 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-03-21T13:57:50.702 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-03-21T13:57:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-03-21T13:57:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-03-21T13:57:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-03-21T13:57:50.703 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-03-21T13:57:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-6.2-6.el8.noarch 2022-03-21T13:57:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-03-21T13:57:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-03-21T13:57:50.704 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-03-21T13:57:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-03-21T13:57:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-03-21T13:57:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-03-21T13:57:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-03-21T13:57:50.705 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.0.6-13.el8.noarch 2022-03-21T13:57:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-03-21T13:57:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-03-21T13:57:50.706 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-03-21T13:57:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-03-21T13:57:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-03-21T13:57:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-03-21T13:57:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-03-21T13:57:50.707 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-2.6-1.el8.noarch 2022-03-21T13:57:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-03-21T13:57:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-03-21T13:57:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-03-21T13:57:50.708 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-03-21T13:57:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.14-14.el8.noarch 2022-03-21T13:57:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2017.2-9.el8.noarch 2022-03-21T13:57:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.709 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-03-21T13:57:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-03-21T13:57:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.710 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.4.1-12.el8.noarch 2022-03-21T13:57:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.8-1.el8.noarch 2022-03-21T13:57:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-03-21T13:57:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-03-21T13:57:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-03-21T13:57:50.711 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-03-21T13:57:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-03-21T13:57:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-03-21T13:57:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-03-21T13:57:50.712 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-03-21T13:57:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-03-21T13:57:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-03-21T13:57:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-03-21T13:57:50.713 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:57:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.3.3-2.el8.x86_64 2022-03-21T13:57:50.714 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-03-21T13:57:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-03-21T13:57:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-03-21T13:57:50.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2022-03-21T13:57:50.716 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2022-03-21T13:57:50.850 DEBUG:teuthology.parallel:result is None 2022-03-21T13:57:58.656 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 133/133 2022-03-21T13:57:58.657 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:07 ago on Mon 21 Mar 2022 01:57:51 PM UTC. 2022-03-21T13:58:03.023 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2022-03-21T13:58:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:58:03.167 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2022-03-21T13:58:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:58:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:58:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:58:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-03-21T13:58:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.169 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2022-03-21T13:58:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.1-7.el8.noarch 2022-03-21T13:58:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.0.4-10.el8.x86_64 2022-03-21T13:58:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:17.0.0-11208.gd3799b8c.el8.noarch 2022-03-21T13:58:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-03-21T13:58:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.1-9.el8.x86_64 2022-03-21T13:58:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.7-12.el8.x86_64 2022-03-21T13:58:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-03-21T13:58:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.5-12.el8.x86_64 2022-03-21T13:58:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-03-21T13:58:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-03-21T13:58:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-03-21T13:58:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-03-21T13:58:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-03-21T13:58:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.2-3.el8.x86_64 2022-03-21T13:58:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-03-21T13:58:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-03-21T13:58:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-03-21T13:58:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-03-21T13:58:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-03-21T13:58:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-25.el8.x86_64 2022-03-21T13:58:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.3.1-3.el8.x86_64 2022-03-21T13:58:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-03-21T13:58:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-03-21T13:58:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-03-21T13:58:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-03-21T13:58:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.12-1.el8.x86_64 2022-03-21T13:58:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-03-21T13:58:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-03-21T13:58:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-03-21T13:58:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-03-21T13:58:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-03-21T13:58:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-03-21T13:58:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.5.1-5.el8.noarch 2022-03-21T13:58:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-03-21T13:58:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-03-21T13:58:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-03-21T13:58:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-03-21T13:58:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-03-21T13:58:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-03-21T13:58:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-03-21T13:58:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-03-21T13:58:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-03-21T13:58:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-6.2-6.el8.noarch 2022-03-21T13:58:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-03-21T13:58:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-03-21T13:58:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-03-21T13:58:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-03-21T13:58:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-03-21T13:58:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-03-21T13:58:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-03-21T13:58:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.0.6-13.el8.noarch 2022-03-21T13:58:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-03-21T13:58:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-03-21T13:58:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-03-21T13:58:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-03-21T13:58:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-03-21T13:58:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-03-21T13:58:03.192 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-03-21T13:58:03.192 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-2.6-1.el8.noarch 2022-03-21T13:58:03.192 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-03-21T13:58:03.192 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-03-21T13:58:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-03-21T13:58:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-03-21T13:58:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.14-14.el8.noarch 2022-03-21T13:58:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2017.2-9.el8.noarch 2022-03-21T13:58:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-03-21T13:58:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-03-21T13:58:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.4.1-12.el8.noarch 2022-03-21T13:58:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.8-1.el8.noarch 2022-03-21T13:58:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-03-21T13:58:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-03-21T13:58:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-03-21T13:58:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-03-21T13:58:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-03-21T13:58:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-03-21T13:58:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-03-21T13:58:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-03-21T13:58:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-03-21T13:58:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-03-21T13:58:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-03-21T13:58:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:17.0.0-11208.gd3799b8c.el8.x86_64 2022-03-21T13:58:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.3.3-2.el8.x86_64 2022-03-21T13:58:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-03-21T13:58:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-03-21T13:58:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-03-21T13:58:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T13:58:03.201 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-03-21T13:58:03.551 DEBUG:teuthology.parallel:result is None 2022-03-21T13:58:03.552 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-03-21T13:58:03.696 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-03-21T13:58:03.700 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:58:03.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:58:03.867 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-03-21T13:58:03.922 INFO:teuthology.orchestra.run.smithi002.stdout:17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:03.923 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:03.923 INFO:teuthology.task.install:The correct ceph version 17.0.0-11208.gd3799b8c is installed. 2022-03-21T13:58:03.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2022-03-21T13:58:03.985 INFO:teuthology.orchestra.run.smithi016.stdout:RedHatEnterprise 2022-03-21T13:58:03.987 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:58:03.987 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:58:04.226 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-03-21T13:58:04.249 INFO:teuthology.orchestra.run.smithi016.stdout:17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:04.249 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:04.250 INFO:teuthology.task.install:The correct ceph version 17.0.0-11208.gd3799b8c is installed. 2022-03-21T13:58:04.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo lsb_release -is 2022-03-21T13:58:04.309 INFO:teuthology.orchestra.run.smithi079.stdout:RedHatEnterprise 2022-03-21T13:58:04.310 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-03-21T13:58:04.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:58:04.546 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-03-21T13:58:04.569 INFO:teuthology.orchestra.run.smithi079.stdout:17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:04.569 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11208.gd3799b8c.el8 2022-03-21T13:58:04.570 INFO:teuthology.task.install:The correct ceph version 17.0.0-11208.gd3799b8c is installed. 2022-03-21T13:58:04.570 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-21T13:58:04.571 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:58:04.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-21T13:58:04.646 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T13:58:04.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-21T13:58:04.676 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T13:58:04.676 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-21T13:58:04.705 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-21T13:58:04.705 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:58:04.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2022-03-21T13:58:04.778 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-21T13:58:04.849 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T13:58:04.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2022-03-21T13:58:04.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-21T13:58:04.948 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T13:58:04.948 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2022-03-21T13:58:04.977 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-21T13:58:05.044 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-21T13:58:05.045 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:58:05.045 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-21T13:58:05.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-21T13:58:05.227 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T13:58:05.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-21T13:58:05.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-21T13:58:05.327 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T13:58:05.327 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-21T13:58:05.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-21T13:58:05.423 INFO:teuthology.run_tasks:Running task cephadm... 2022-03-21T13:58:05.490 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd3799b8cfa5046465710eadeea2bed7bb4681409'} 2022-03-21T13:58:05.491 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T13:58:05.492 INFO:tasks.cephadm:Cluster fsid is ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:58:05.492 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-03-21T13:58:05.493 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.16', 'mon.c': '172.21.15.79'} 2022-03-21T13:58:05.493 INFO:tasks.cephadm:First mon is mon.a on smithi002 2022-03-21T13:58:05.493 INFO:tasks.cephadm:First mgr is x 2022-03-21T13:58:05.494 INFO:tasks.cephadm:Normalizing hostnames... 2022-03-21T13:58:05.494 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2022-03-21T13:58:05.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2022-03-21T13:58:05.598 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2022-03-21T13:58:05.627 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref d3799b8cfa5046465710eadeea2bed7bb4681409)... 2022-03-21T13:58:05.627 DEBUG:teuthology.orchestra.run.smithi002:> git archive --remote=git://git.ceph.com/ceph-ci.git d3799b8cfa5046465710eadeea2bed7bb4681409 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.723 DEBUG:teuthology.orchestra.run.smithi016:> git archive --remote=git://git.ceph.com/ceph-ci.git d3799b8cfa5046465710eadeea2bed7bb4681409 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.784 DEBUG:teuthology.orchestra.run.smithi079:> git archive --remote=git://git.ceph.com/ceph-ci.git d3799b8cfa5046465710eadeea2bed7bb4681409 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.847 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.895 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.915 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-03-21T13:58:05.947 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 on all hosts... 2022-03-21T13:58:05.947 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 pull 2022-03-21T13:58:05.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 pull 2022-03-21T13:58:05.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 pull 2022-03-21T13:58:07.589 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409... 2022-03-21T13:58:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409... 2022-03-21T13:58:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409... 2022-03-21T13:58:43.916 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2022-03-21T13:58:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev)", 2022-03-21T13:58:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "024bbad3f2a7e94e355e52fe31666234123f65c43c754c57d8343696898adda2", 2022-03-21T13:58:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2022-03-21T13:58:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:59bfb168b441fb691e0f63959446cd659f6d6e3eae99a4f91f23d9384714d2bf" 2022-03-21T13:58:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2022-03-21T13:58:43.917 INFO:teuthology.orchestra.run.smithi016.stdout:} 2022-03-21T13:58:58.374 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2022-03-21T13:58:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev)", 2022-03-21T13:58:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "024bbad3f2a7e94e355e52fe31666234123f65c43c754c57d8343696898adda2", 2022-03-21T13:58:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2022-03-21T13:58:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:59bfb168b441fb691e0f63959446cd659f6d6e3eae99a4f91f23d9384714d2bf" 2022-03-21T13:58:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2022-03-21T13:58:58.376 INFO:teuthology.orchestra.run.smithi079.stdout:} 2022-03-21T13:59:06.944 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T13:59:06.944 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev)", 2022-03-21T13:59:06.944 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "024bbad3f2a7e94e355e52fe31666234123f65c43c754c57d8343696898adda2", 2022-03-21T13:59:06.945 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2022-03-21T13:59:06.945 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:59bfb168b441fb691e0f63959446cd659f6d6e3eae99a4f91f23d9384714d2bf" 2022-03-21T13:59:06.945 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2022-03-21T13:59:06.945 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T13:59:06.973 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2022-03-21T13:59:07.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2022-03-21T13:59:07.089 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2022-03-21T13:59:07.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2022-03-21T13:59:07.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2022-03-21T13:59:07.225 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2022-03-21T13:59:07.254 INFO:tasks.cephadm:Writing seed config... 2022-03-21T13:59:07.255 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-03-21T13:59:07.256 INFO:tasks.cephadm: override: [client] debug client = 20 2022-03-21T13:59:07.256 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-03-21T13:59:07.256 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-03-21T13:59:07.257 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-03-21T13:59:07.257 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2022-03-21T13:59:07.257 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-03-21T13:59:07.258 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-03-21T13:59:07.258 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2022-03-21T13:59:07.258 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2022-03-21T13:59:07.258 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2022-03-21T13:59:07.259 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2022-03-21T13:59:07.259 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-03-21T13:59:07.259 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-03-21T13:59:07.260 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-03-21T13:59:07.260 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-03-21T13:59:07.260 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-03-21T13:59:07.261 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-03-21T13:59:07.261 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-03-21T13:59:07.261 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-03-21T13:59:07.262 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-03-21T13:59:07.262 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-03-21T13:59:07.262 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-03-21T13:59:07.263 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-03-21T13:59:07.263 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-03-21T13:59:07.263 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2022-03-21T13:59:07.264 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2022-03-21T13:59:07.264 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2022-03-21T13:59:07.264 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-03-21T13:59:07.265 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-03-21T13:59:07.265 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2022-03-21T13:59:07.265 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2022-03-21T13:59:07.266 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-03-21T13:59:07.266 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-03-21T13:59:07.266 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2022-03-21T13:59:07.267 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-03-21T13:59:07.267 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-03-21T13:59:07.267 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-03-21T13:59:07.268 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-03-21T13:59:07.268 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-03-21T13:59:07.268 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-03-21T13:59:07.269 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-03-21T13:59:07.269 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2022-03-21T13:59:07.270 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T13:59:07.271 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-03-21T13:59:07.315 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ee5e78f2-a91e-11ec-8c35-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-03-21T13:59:07.316 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.a.service 2022-03-21T13:59:07.360 DEBUG:teuthology.orchestra.run.smithi002:mgr.x> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.x.service 2022-03-21T13:59:07.365 INFO:tasks.cephadm:Bootstrapping... 2022-03-21T13:59:07.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 -v bootstrap --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-03-21T13:59:07.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T13:59:07.435 INFO:journalctl@ceph.mgr.x.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T13:59:07.673 INFO:teuthology.orchestra.run.smithi002.stderr:-------------------------------------------------------------------------------- 2022-03-21T13:59:07.674 INFO:teuthology.orchestra.run.smithi002.stderr:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409', '-v', 'bootstrap', '--fsid', 'ee5e78f2-a91e-11ec-8c35-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2022-03-21T13:59:07.778 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 3.2.3 2022-03-21T13:59:07.863 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying podman|docker is present... 2022-03-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 3.2.3 2022-03-21T13:59:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying lvm2 is present... 2022-03-21T13:59:08.052 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying time synchronization is in place... 2022-03-21T13:59:08.067 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-03-21T13:59:08.084 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T13:59:08.102 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2022-03-21T13:59:08.118 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2022-03-21T13:59:08.120 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2022-03-21T13:59:08.120 INFO:teuthology.orchestra.run.smithi002.stderr:Repeating the final host check... 2022-03-21T13:59:08.222 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 3.2.3 2022-03-21T13:59:08.306 INFO:teuthology.orchestra.run.smithi002.stderr:podman (/bin/podman) version 3.2.3 is present 2022-03-21T13:59:08.307 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl is present 2022-03-21T13:59:08.307 INFO:teuthology.orchestra.run.smithi002.stderr:lvcreate is present 2022-03-21T13:59:08.321 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-03-21T13:59:08.338 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T13:59:08.356 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2022-03-21T13:59:08.372 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2022-03-21T13:59:08.374 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2022-03-21T13:59:08.374 INFO:teuthology.orchestra.run.smithi002.stderr:Host looks OK 2022-03-21T13:59:08.375 INFO:teuthology.orchestra.run.smithi002.stderr:Cluster fsid: ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:08.376 INFO:teuthology.orchestra.run.smithi002.stderr:Acquiring lock 140646984489448 on /run/cephadm/ee5e78f2-a91e-11ec-8c35-001a4aab830c.lock 2022-03-21T13:59:08.376 INFO:teuthology.orchestra.run.smithi002.stderr:Lock 140646984489448 acquired on /run/cephadm/ee5e78f2-a91e-11ec-8c35-001a4aab830c.lock 2022-03-21T13:59:08.376 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 3300 ... 2022-03-21T13:59:08.377 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 6789 ... 2022-03-21T13:59:08.377 INFO:teuthology.orchestra.run.smithi002.stderr:Base mon IP is 172.21.15.2, final addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2022-03-21T13:59:08.383 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2022-03-21T13:59:08.383 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2022-03-21T13:59:08.391 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2022-03-21T13:59:08.391 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2022-03-21T13:59:08.392 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2022-03-21T13:59:08.399 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2022-03-21T13:59:08.399 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 ::1/128 scope host 2022-03-21T13:59:08.400 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2022-03-21T13:59:08.400 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2022-03-21T13:59:08.400 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2022-03-21T13:59:08.400 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2022-03-21T13:59:08.401 INFO:teuthology.orchestra.run.smithi002.stderr:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2022-03-21T13:59:08.401 INFO:teuthology.orchestra.run.smithi002.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-03-21T13:59:08.402 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409... 2022-03-21T13:59:08.540 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409... 2022-03-21T13:59:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Getting image source signatures 2022-03-21T13:59:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:be6bef52633967571f7782aed89a26dc0d0340dcc3973a250cf9064976a6e4d7 2022-03-21T13:59:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:fee18ea417d6bd9b69cbb06f4df2dcc2582b25a4742c582cf85d5c0d21abe078 2022-03-21T13:59:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:9f45cf1cd9f2cb569ccf05cdd52214665b0a635f776a8ce4e42e5dfd28f95088 2022-03-21T13:59:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:bfd1401568a8346ffcf384bd20bdda89adb0289339936df7e882803a177f374c 2022-03-21T13:59:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:f0a2109a2528cd1cf5eaa3f5867a6771e4a7c28ae7179d7cc67cbcc2b29f21fc 2022-03-21T13:59:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying config sha256:024bbad3f2a7e94e355e52fe31666234123f65c43c754c57d8343696898adda2 2022-03-21T13:59:08.731 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Writing manifest to image destination 2022-03-21T13:59:08.732 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Storing signatures 2022-03-21T13:59:08.888 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 024bbad3f2a7e94e355e52fe31666234123f65c43c754c57d8343696898adda2 2022-03-21T13:59:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:09.586 INFO:teuthology.orchestra.run.smithi002.stderr:ceph: ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev) 2022-03-21T13:59:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph version: ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev) 2022-03-21T13:59:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:Extracting ceph user uid/gid from container image... 2022-03-21T13:59:10.094 INFO:teuthology.orchestra.run.smithi002.stderr:stat: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:10.458 INFO:teuthology.orchestra.run.smithi002.stderr:stat: 167 167 2022-03-21T13:59:10.769 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial keys... 2022-03-21T13:59:10.882 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQCvhDhiGHP3DhAAIKWxkIa5QE09KT366lXcqQ== 2022-03-21T13:59:11.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:12.015 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQCwhDhi/GZbABAA6p0b+B9y+LXHeKKRwwFmTg== 2022-03-21T13:59:12.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:12.824 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQCwhDhisANEMBAAA+qEieyww/S7XgnTT+BDeA== 2022-03-21T13:59:13.635 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial monmap... 2022-03-21T13:59:13.749 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:14.098 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2022-03-21T13:59:14.099 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2022-03-21T13:59:14.099 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:14.099 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-03-21T13:59:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2022-03-21T13:59:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:setting min_mon_release = octopus 2022-03-21T13:59:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: set fsid to ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:14.428 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-03-21T13:59:14.428 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T13:59:14.428 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mon... 2022-03-21T13:59:14.544 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-mon: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:15.319 INFO:teuthology.orchestra.run.smithi002.stderr:create mon.a on 2022-03-21T13:59:15.353 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2022-03-21T13:59:15.610 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2022-03-21T13:59:15.874 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target → /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target. 2022-03-21T13:59:15.874 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target → /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target. 2022-03-21T13:59:16.472 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.a.service: Unit ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.a.service not loaded. 2022-03-21T13:59:16.491 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target.wants/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@.service. 2022-03-21T13:59:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:16 smithi002 systemd[1]: Starting Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T13:59:17.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:17 smithi002 bash[16121]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:17.831 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2022-03-21T13:59:17.851 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T13:59:17.853 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2022-03-21T13:59:17.854 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2022-03-21T13:59:17.854 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon to start... 2022-03-21T13:59:17.855 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon... 2022-03-21T13:59:17.975 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:17 smithi002 bash[16121]: af08f1158d7c128e28dcacd7789e488d868cd2f9f48db3c8d525620287866918 2022-03-21T13:59:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:17 smithi002 ceph-mon[16332]: mkfs ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:17 smithi002 ceph-mon[16332]: mon.a is new leader, mons a in quorum (ranks 0) 2022-03-21T13:59:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:17 smithi002 systemd[1]: Started Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T13:59:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.445+0000 7f0bae88c700 1 Processor -- start 2022-03-21T13:59:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.446+0000 7f0bae88c700 1 -- start start 2022-03-21T13:59:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.446+0000 7f0bae88c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.446+0000 7f0bae88c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba81040e0 con 0x7f0ba8103790 2022-03-21T13:59:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.447+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.447+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43328/0 (socket says 172.21.15.2:43328) 2022-03-21T13:59:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.447+0000 7f0ba7fff700 1 -- 172.21.15.2:0/3551499932 learned_addr learned my addr 172.21.15.2:0/3551499932 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.447+0000 7f0ba7fff700 1 -- 172.21.15.2:0/3551499932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8104220 con 0x7f0ba8103790 2022-03-21T13:59:18.454 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.448+0000 7f0ba7fff700 1 --2- 172.21.15.2:0/3551499932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009730 tx=0x7f0b980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dbcedd8c68f2da server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:18.454 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.448+0000 7f0ba6ffd700 1 -- 172.21.15.2:0/3551499932 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98004030 con 0x7f0ba8103790 2022-03-21T13:59:18.454 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.448+0000 7f0ba6ffd700 1 -- 172.21.15.2:0/3551499932 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98004190 con 0x7f0ba8103790 2022-03-21T13:59:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.448+0000 7f0ba6ffd700 1 -- 172.21.15.2:0/3551499932 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98004340 con 0x7f0ba8103790 2022-03-21T13:59:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 -- 172.21.15.2:0/3551499932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 msgr2=0x7f0ba8103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 --2- 172.21.15.2:0/3551499932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009730 tx=0x7f0b980090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 -- 172.21.15.2:0/3551499932 shutdown_connections 2022-03-21T13:59:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 --2- 172.21.15.2:0/3551499932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8103ba0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 -- 172.21.15.2:0/3551499932 >> 172.21.15.2:0/3551499932 conn(0x7f0ba8076dd0 msgr2=0x7f0ba80771d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 -- 172.21.15.2:0/3551499932 shutdown_connections 2022-03-21T13:59:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.449+0000 7f0bae88c700 1 -- 172.21.15.2:0/3551499932 wait complete. 2022-03-21T13:59:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.450+0000 7f0bae88c700 1 Processor -- start 2022-03-21T13:59:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.450+0000 7f0bae88c700 1 -- start start 2022-03-21T13:59:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.451+0000 7f0bae88c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.451+0000 7f0bae88c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8112140 con 0x7f0ba8103790 2022-03-21T13:59:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.451+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.451+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43330/0 (socket says 172.21.15.2:43330) 2022-03-21T13:59:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.451+0000 7f0ba7fff700 1 -- 172.21.15.2:0/1800547747 learned_addr learned my addr 172.21.15.2:0/1800547747 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0ba7fff700 1 -- 172.21.15.2:0/1800547747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009370 con 0x7f0ba8103790 2022-03-21T13:59:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0ba7fff700 1 --2- 172.21.15.2:0/1800547747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009130 tx=0x7f0b98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:18.459 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98004300 con 0x7f0ba8103790 2022-03-21T13:59:18.459 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98003c00 con 0x7f0ba8103790 2022-03-21T13:59:18.459 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8112340 con 0x7f0ba8103790 2022-03-21T13:59:18.459 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98026480 con 0x7f0ba8103790 2022-03-21T13:59:18.460 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.452+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba8112fe0 con 0x7f0ba8103790 2022-03-21T13:59:18.460 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.453+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f0b98017020 con 0x7f0ba8103790 2022-03-21T13:59:18.460 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.453+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0b9801c7a0 con 0x7f0ba8103790 2022-03-21T13:59:18.461 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.454+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba805e600 con 0x7f0ba8103790 2022-03-21T13:59:18.461 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.457+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f0b980269f0 con 0x7f0ba8103790 2022-03-21T13:59:18.535 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.532+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0ba8113ab0 con 0x7f0ba8103790 2022-03-21T13:59:18.536 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.533+0000 7f0ba4ff9700 1 -- 172.21.15.2:0/1800547747 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f0b98026b90 con 0x7f0ba8103790 2022-03-21T13:59:18.536 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: cluster: 2022-03-21T13:59:18.536 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: id: ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: health: HEALTH_OK 2022-03-21T13:59:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: services: 2022-03-21T13:59:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.768523s) 2022-03-21T13:59:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr: no daemons active 2022-03-21T13:59:18.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2022-03-21T13:59:18.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: data: 2022-03-21T13:59:18.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2022-03-21T13:59:18.539 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2022-03-21T13:59:18.539 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2022-03-21T13:59:18.539 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pgs: 2022-03-21T13:59:18.539 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.540 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 msgr2=0x7f0ba8111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:18.540 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 --2- 172.21.15.2:0/1800547747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009130 tx=0x7f0b98016000 comp rx=0 tx=0).stop 2022-03-21T13:59:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 shutdown_connections 2022-03-21T13:59:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 --2- 172.21.15.2:0/1800547747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ba8103790 0x7f0ba8111c00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 >> 172.21.15.2:0/1800547747 conn(0x7f0ba8076dd0 msgr2=0x7f0ba8106560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 shutdown_connections 2022-03-21T13:59:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:18.534+0000 7f0bae88c700 1 -- 172.21.15.2:0/1800547747 wait complete. 2022-03-21T13:59:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: mon.a is new leader, mons a in quorum (ranks 0) 2022-03-21T13:59:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2022-03-21T13:59:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: fsmap 2022-03-21T13:59:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: osdmap e1: 0 total, 0 up, 0 in 2022-03-21T13:59:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: mgrmap e1: no daemons active 2022-03-21T13:59:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:18 smithi002 ceph-mon[16332]: from='client.? 172.21.15.2:0/1800547747' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-03-21T13:59:18.949 INFO:teuthology.orchestra.run.smithi002.stderr:mon is available 2022-03-21T13:59:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:Assimilating anything we can from ceph.conf... 2022-03-21T13:59:19.063 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:19.582 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.579+0000 7fa853fff700 1 Processor -- start 2022-03-21T13:59:19.582 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.580+0000 7fa853fff700 1 -- start start 2022-03-21T13:59:19.582 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.580+0000 7fa853fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:19.583 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.580+0000 7fa853fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa854076f60 con 0x7fa854076b50 2022-03-21T13:59:19.583 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.581+0000 7fa852ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:19.583 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.581+0000 7fa852ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43332/0 (socket says 172.21.15.2:43332) 2022-03-21T13:59:19.583 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.581+0000 7fa852ffd700 1 -- 172.21.15.2:0/2707350684 learned_addr learned my addr 172.21.15.2:0/2707350684 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:19.584 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.581+0000 7fa852ffd700 1 -- 172.21.15.2:0/2707350684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854075630 con 0x7fa854076b50 2022-03-21T13:59:19.586 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.581+0000 7fa852ffd700 1 --2- 172.21.15.2:0/2707350684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa844009730 tx=0x7fa8440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60be4ef1d96a11c9 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:19.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.582+0000 7fa851ffb700 1 -- 172.21.15.2:0/2707350684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa844004030 con 0x7fa854076b50 2022-03-21T13:59:19.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.582+0000 7fa851ffb700 1 -- 172.21.15.2:0/2707350684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844004190 con 0x7fa854076b50 2022-03-21T13:59:19.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.582+0000 7fa851ffb700 1 -- 172.21.15.2:0/2707350684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa844004340 con 0x7fa854076b50 2022-03-21T13:59:19.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.582+0000 7fa853fff700 1 -- 172.21.15.2:0/2707350684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 msgr2=0x7fa8540750f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:19.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.582+0000 7fa853fff700 1 --2- 172.21.15.2:0/2707350684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa844009730 tx=0x7fa8440090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:19.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.583+0000 7fa853fff700 1 -- 172.21.15.2:0/2707350684 shutdown_connections 2022-03-21T13:59:19.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.583+0000 7fa853fff700 1 --2- 172.21.15.2:0/2707350684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa8540750f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:19.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.583+0000 7fa853fff700 1 -- 172.21.15.2:0/2707350684 >> 172.21.15.2:0/2707350684 conn(0x7fa8540fcbe0 msgr2=0x7fa8540feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:19.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.583+0000 7fa853fff700 1 -- 172.21.15.2:0/2707350684 shutdown_connections 2022-03-21T13:59:19.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.583+0000 7fa853fff700 1 -- 172.21.15.2:0/2707350684 wait complete. 2022-03-21T13:59:19.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.584+0000 7fa853fff700 1 Processor -- start 2022-03-21T13:59:19.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.584+0000 7fa853fff700 1 -- start start 2022-03-21T13:59:19.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.585+0000 7fa853fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:19.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.585+0000 7fa853fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa854112200 con 0x7fa854076b50 2022-03-21T13:59:19.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.585+0000 7fa852ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:19.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.585+0000 7fa852ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43334/0 (socket says 172.21.15.2:43334) 2022-03-21T13:59:19.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.585+0000 7fa852ffd700 1 -- 172.21.15.2:0/2792349539 learned_addr learned my addr 172.21.15.2:0/2792349539 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:19.591 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa852ffd700 1 -- 172.21.15.2:0/2792349539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844009370 con 0x7fa854076b50 2022-03-21T13:59:19.592 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa852ffd700 1 --2- 172.21.15.2:0/2792349539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa844000c00 tx=0x7fa844016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:19.592 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa844004300 con 0x7fa854076b50 2022-03-21T13:59:19.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844003c00 con 0x7fa854076b50 2022-03-21T13:59:19.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa844026480 con 0x7fa854076b50 2022-03-21T13:59:19.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.586+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa854112400 con 0x7fa854076b50 2022-03-21T13:59:19.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.587+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa854112f60 con 0x7fa854076b50 2022-03-21T13:59:19.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.587+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fa844017020 con 0x7fa854076b50 2022-03-21T13:59:19.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.587+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa84401c7a0 con 0x7fa854076b50 2022-03-21T13:59:19.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.588+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85405e600 con 0x7fa854076b50 2022-03-21T13:59:19.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.591+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7fa844026a40 con 0x7fa854076b50 2022-03-21T13:59:19.664 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.663+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa85404af20 con 0x7fa854076b50 2022-03-21T13:59:19.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.668+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa84401cdf0 con 0x7fa854076b50 2022-03-21T13:59:19.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.668+0000 7fa83b7fe700 1 -- 172.21.15.2:0/2792349539 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7fa84401a850 con 0x7fa854076b50 2022-03-21T13:59:19.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2022-03-21T13:59:19.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:19.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2022-03-21T13:59:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-03-21T13:59:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-03-21T13:59:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-03-21T13:59:19.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-03-21T13:59:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2022-03-21T13:59:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-03-21T13:59:19.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2022-03-21T13:59:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2022-03-21T13:59:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2022-03-21T13:59:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2022-03-21T13:59:19.675 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-03-21T13:59:19.676 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-03-21T13:59:19.676 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 msgr2=0x7fa854111cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:19.676 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 --2- 172.21.15.2:0/2792349539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa844000c00 tx=0x7fa844016000 comp rx=0 tx=0).stop 2022-03-21T13:59:19.676 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 shutdown_connections 2022-03-21T13:59:19.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 --2- 172.21.15.2:0/2792349539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa854076b50 0x7fa854111cc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:19.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 >> 172.21.15.2:0/2792349539 conn(0x7fa8540fcbe0 msgr2=0x7fa854105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:19.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 shutdown_connections 2022-03-21T13:59:19.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:19.669+0000 7fa853fff700 1 -- 172.21.15.2:0/2792349539 wait complete. 2022-03-21T13:59:20.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:19 smithi002 ceph-mon[16332]: from='client.? 172.21.15.2:0/2792349539' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-03-21T13:59:20.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:19 smithi002 ceph-mon[16332]: from='client.? 172.21.15.2:0/2792349539' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-03-21T13:59:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:Generating new minimal ceph.conf... 2022-03-21T13:59:20.351 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.818+0000 7fa04e759700 1 Processor -- start 2022-03-21T13:59:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.818+0000 7fa04e759700 1 -- start start 2022-03-21T13:59:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.819+0000 7fa04e759700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.819+0000 7fa04e759700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048101f40 con 0x7fa0481015f0 2022-03-21T13:59:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.819+0000 7fa047fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.819+0000 7fa047fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43338/0 (socket says 172.21.15.2:43338) 2022-03-21T13:59:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.819+0000 7fa047fff700 1 -- 172.21.15.2:0/2730799022 learned_addr learned my addr 172.21.15.2:0/2730799022 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.820+0000 7fa047fff700 1 -- 172.21.15.2:0/2730799022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa048102080 con 0x7fa0481015f0 2022-03-21T13:59:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.820+0000 7fa047fff700 1 --2- 172.21.15.2:0/2730799022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa038009730 tx=0x7fa0380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce995d3ab30d46d1 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.820+0000 7fa046ffd700 1 -- 172.21.15.2:0/2730799022 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa038004030 con 0x7fa0481015f0 2022-03-21T13:59:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.820+0000 7fa046ffd700 1 -- 172.21.15.2:0/2730799022 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa038004190 con 0x7fa0481015f0 2022-03-21T13:59:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.820+0000 7fa046ffd700 1 -- 172.21.15.2:0/2730799022 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa038003750 con 0x7fa0481015f0 2022-03-21T13:59:20.827 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 -- 172.21.15.2:0/2730799022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 msgr2=0x7fa048101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:20.827 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 --2- 172.21.15.2:0/2730799022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa038009730 tx=0x7fa0380090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:20.827 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 -- 172.21.15.2:0/2730799022 shutdown_connections 2022-03-21T13:59:20.827 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 --2- 172.21.15.2:0/2730799022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa048101a00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:20.828 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 -- 172.21.15.2:0/2730799022 >> 172.21.15.2:0/2730799022 conn(0x7fa0480fcdb0 msgr2=0x7fa0480ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:20.828 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 -- 172.21.15.2:0/2730799022 shutdown_connections 2022-03-21T13:59:20.828 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.821+0000 7fa04e759700 1 -- 172.21.15.2:0/2730799022 wait complete. 2022-03-21T13:59:20.828 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.822+0000 7fa04e759700 1 Processor -- start 2022-03-21T13:59:20.829 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.822+0000 7fa04e759700 1 -- start start 2022-03-21T13:59:20.829 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa04e759700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:20.829 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa04e759700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0481128e0 con 0x7fa0481015f0 2022-03-21T13:59:20.829 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa047fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:20.830 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa047fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43340/0 (socket says 172.21.15.2:43340) 2022-03-21T13:59:20.830 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa047fff700 1 -- 172.21.15.2:0/1291444409 learned_addr learned my addr 172.21.15.2:0/1291444409 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:20.830 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.823+0000 7fa047fff700 1 -- 172.21.15.2:0/1291444409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038009370 con 0x7fa0481015f0 2022-03-21T13:59:20.830 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa047fff700 1 --2- 172.21.15.2:0/1291444409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa038004530 tx=0x7fa0380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa038011a60 con 0x7fa0481015f0 2022-03-21T13:59:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa038011bc0 con 0x7fa0481015f0 2022-03-21T13:59:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa048112ae0 con 0x7fa0481015f0 2022-03-21T13:59:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa038019400 con 0x7fa0481015f0 2022-03-21T13:59:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.824+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa048113780 con 0x7fa0481015f0 2022-03-21T13:59:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.825+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fa038017020 con 0x7fa0481015f0 2022-03-21T13:59:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.825+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa03802d950 con 0x7fa0481015f0 2022-03-21T13:59:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.826+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04805e600 con 0x7fa0481015f0 2022-03-21T13:59:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.828+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7fa03801a950 con 0x7fa0481015f0 2022-03-21T13:59:20.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.905+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa048103590 con 0x7fa0481015f0 2022-03-21T13:59:20.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.905+0000 7fa044ff9700 1 -- 172.21.15.2:0/1291444409 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7fa048103590 con 0x7fa0481015f0 2022-03-21T13:59:20.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 msgr2=0x7fa0481123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:20.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 --2- 172.21.15.2:0/1291444409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa038004530 tx=0x7fa0380090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:20.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 shutdown_connections 2022-03-21T13:59:20.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 --2- 172.21.15.2:0/1291444409 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0481015f0 0x7fa0481123a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:20.909 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 >> 172.21.15.2:0/1291444409 conn(0x7fa0480fcdb0 msgr2=0x7fa0480fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:20.909 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.906+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 shutdown_connections 2022-03-21T13:59:20.909 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:20.907+0000 7fa04e759700 1 -- 172.21.15.2:0/1291444409 wait complete. 2022-03-21T13:59:21.289 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:20 smithi002 ceph-mon[16332]: from='client.? 172.21.15.2:0/1291444409' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T13:59:21.326 INFO:teuthology.orchestra.run.smithi002.stderr:Restarting the monitor... 2022-03-21T13:59:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:21 smithi002 systemd[1]: Stopping Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T13:59:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:21 smithi002 bash[16783]: Error: no container with name or ID "ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c-mon.a" found: no such container 2022-03-21T13:59:22.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:21 smithi002 conmon[16308]: 2022-03-21T13:59:21.649+0000 7fe4e2581700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2022-03-21T13:59:22.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:21 smithi002 conmon[16308]: 2022-03-21T13:59:21.650+0000 7fe4e2581700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-03-21T13:59:22.294 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 bash[16783]: ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c-mon-a 2022-03-21T13:59:22.295 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 bash[16783]: Error: no container with name or ID "ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c-mon.a" found: no such container 2022-03-21T13:59:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 systemd[1]: ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.a.service: Succeeded. 2022-03-21T13:59:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 systemd[1]: Stopped Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T13:59:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 systemd[1]: Starting Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T13:59:23.219 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:22 smithi002 bash[16955]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:23.409 INFO:teuthology.orchestra.run.smithi002.stderr:Setting mon public_network to 172.21.0.0/20 2022-03-21T13:59:23.529 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 bash[16955]: a9136a71a95be23e936fbbac731c6e4a4fc57fb309f77a3fad98e159c2494ece 2022-03-21T13:59:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: set uid:gid to 167:167 (ceph:ceph) 2022-03-21T13:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev), process ceph-mon, pid 7 2022-03-21T13:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: pidfile_write: ignore empty --pid-file 2022-03-21T13:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: load: jerasure load: lrc 2022-03-21T13:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: RocksDB version: 6.15.5 2022-03-21T13:59:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-03-21T13:59:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Compile date Mar 19 2022 2022-03-21T13:59:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: DB SUMMARY 2022-03-21T13:59:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: DB Session ID: XOLIDC4XLN0SANRSWL9Q 2022-03-21T13:59:23.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: CURRENT file: CURRENT 2022-03-21T13:59:23.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: IDENTITY file: IDENTITY 2022-03-21T13:59:23.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2022-03-21T13:59:23.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2022-03-21T13:59:23.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 96459 ; 2022-03-21T13:59:23.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.error_if_exists: 0 2022-03-21T13:59:23.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.create_if_missing: 0 2022-03-21T13:59:23.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.paranoid_checks: 1 2022-03-21T13:59:23.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-03-21T13:59:23.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.env: 0x5631170c8860 2022-03-21T13:59:23.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.fs: Posix File System 2022-03-21T13:59:23.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.info_log: 0x5631191d7e60 2022-03-21T13:59:23.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_file_opening_threads: 16 2022-03-21T13:59:23.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.statistics: (nil) 2022-03-21T13:59:23.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.use_fsync: 0 2022-03-21T13:59:23.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_log_file_size: 0 2022-03-21T13:59:23.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-03-21T13:59:23.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.log_file_time_to_roll: 0 2022-03-21T13:59:23.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.keep_log_file_num: 1000 2022-03-21T13:59:23.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.recycle_log_file_num: 0 2022-03-21T13:59:23.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_fallocate: 1 2022-03-21T13:59:23.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_mmap_reads: 0 2022-03-21T13:59:23.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_mmap_writes: 0 2022-03-21T13:59:23.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.use_direct_reads: 0 2022-03-21T13:59:23.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-03-21T13:59:23.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.create_missing_column_families: 0 2022-03-21T13:59:23.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.db_log_dir: 2022-03-21T13:59:23.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-03-21T13:59:23.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.table_cache_numshardbits: 6 2022-03-21T13:59:23.625 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.WAL_ttl_seconds: 0 2022-03-21T13:59:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.WAL_size_limit_MB: 0 2022-03-21T13:59:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-03-21T13:59:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-03-21T13:59:23.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.is_fd_close_on_exec: 1 2022-03-21T13:59:23.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.advise_random_on_open: 1 2022-03-21T13:59:23.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.db_write_buffer_size: 0 2022-03-21T13:59:23.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.write_buffer_manager: 0x5631192a4420 2022-03-21T13:59:23.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-03-21T13:59:23.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-03-21T13:59:23.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-03-21T13:59:23.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.use_adaptive_mutex: 0 2022-03-21T13:59:23.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.rate_limiter: (nil) 2022-03-21T13:59:23.628 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-03-21T13:59:23.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.wal_recovery_mode: 2 2022-03-21T13:59:23.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.enable_thread_tracking: 0 2022-03-21T13:59:23.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.enable_pipelined_write: 0 2022-03-21T13:59:23.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.unordered_write: 0 2022-03-21T13:59:23.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-03-21T13:59:23.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-03-21T13:59:23.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-03-21T13:59:23.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-03-21T13:59:23.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.row_cache: None 2022-03-21T13:59:23.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.wal_filter: None 2022-03-21T13:59:23.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-03-21T13:59:23.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_ingest_behind: 0 2022-03-21T13:59:23.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.preserve_deletes: 0 2022-03-21T13:59:23.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.two_write_queues: 0 2022-03-21T13:59:23.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.manual_wal_flush: 0 2022-03-21T13:59:23.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.atomic_flush: 0 2022-03-21T13:59:23.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-03-21T13:59:23.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.persist_stats_to_disk: 0 2022-03-21T13:59:23.633 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.write_dbid_to_manifest: 0 2022-03-21T13:59:23.633 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.log_readahead_size: 0 2022-03-21T13:59:23.633 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-03-21T13:59:23.633 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.best_efforts_recovery: 0 2022-03-21T13:59:23.634 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-03-21T13:59:23.634 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-03-21T13:59:23.634 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.allow_data_in_errors: 0 2022-03-21T13:59:23.634 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.db_host_id: __hostname__ 2022-03-21T13:59:23.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_background_jobs: 2 2022-03-21T13:59:23.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_background_compactions: -1 2022-03-21T13:59:23.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_subcompactions: 1 2022-03-21T13:59:23.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-03-21T13:59:23.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-03-21T13:59:23.636 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.delayed_write_rate : 16777216 2022-03-21T13:59:23.636 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_total_wal_size: 0 2022-03-21T13:59:23.636 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-03-21T13:59:23.636 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.stats_dump_period_sec: 600 2022-03-21T13:59:23.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.stats_persist_period_sec: 600 2022-03-21T13:59:23.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-03-21T13:59:23.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_open_files: -1 2022-03-21T13:59:23.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bytes_per_sync: 0 2022-03-21T13:59:23.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.wal_bytes_per_sync: 0 2022-03-21T13:59:23.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.strict_bytes_per_sync: 0 2022-03-21T13:59:23.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_readahead_size: 0 2022-03-21T13:59:23.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_background_flushes: -1 2022-03-21T13:59:23.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Compression algorithms supported: 2022-03-21T13:59:23.639 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-03-21T13:59:23.639 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kZSTD supported: 0 2022-03-21T13:59:23.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kXpressCompression supported: 0 2022-03-21T13:59:23.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kLZ4HCCompression supported: 1 2022-03-21T13:59:23.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kLZ4Compression supported: 1 2022-03-21T13:59:23.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kBZip2Compression supported: 0 2022-03-21T13:59:23.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kZlibCompression supported: 1 2022-03-21T13:59:23.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: kSnappyCompression supported: 1 2022-03-21T13:59:23.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Fast CRC32 supported: Supported on x86 2022-03-21T13:59:23.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-03-21T13:59:23.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-03-21T13:59:23.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-03-21T13:59:23.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.merge_operator: 2022-03-21T13:59:23.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_filter: None 2022-03-21T13:59:23.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_filter_factory: None 2022-03-21T13:59:23.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.sst_partitioner_factory: None 2022-03-21T13:59:23.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.memtable_factory: SkipListFactory 2022-03-21T13:59:23.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.table_factory: BlockBasedTable 2022-03-21T13:59:23.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5631191a3cc0) 2022-03-21T13:59:23.644 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2022-03-21T13:59:23.644 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-03-21T13:59:23.644 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-03-21T13:59:23.644 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2022-03-21T13:59:23.645 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2022-03-21T13:59:23.645 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2022-03-21T13:59:23.645 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2022-03-21T13:59:23.645 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2022-03-21T13:59:23.646 INFO:journalctl@ceph.mon.a.smithi002.stdout: hash_index_allow_collision: 1 2022-03-21T13:59:23.646 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 1 2022-03-21T13:59:23.646 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2022-03-21T13:59:23.646 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x56311920e170 2022-03-21T13:59:23.647 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2022-03-21T13:59:23.647 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2022-03-21T13:59:23.647 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2022-03-21T13:59:23.647 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2022-03-21T13:59:23.647 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2022-03-21T13:59:23.648 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2022-03-21T13:59:23.648 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2022-03-21T13:59:23.648 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2022-03-21T13:59:23.648 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2022-03-21T13:59:23.649 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2022-03-21T13:59:23.649 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2022-03-21T13:59:23.649 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2022-03-21T13:59:23.649 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2022-03-21T13:59:23.649 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2022-03-21T13:59:23.650 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2022-03-21T13:59:23.650 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-03-21T13:59:23.650 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2022-03-21T13:59:23.650 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2022-03-21T13:59:23.651 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2022-03-21T13:59:23.651 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 4 2022-03-21T13:59:23.651 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2022-03-21T13:59:23.651 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2022-03-21T13:59:23.652 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.write_buffer_size: 33554432 2022-03-21T13:59:23.652 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_write_buffer_number: 2 2022-03-21T13:59:23.652 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression: NoCompression 2022-03-21T13:59:23.652 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression: Disabled 2022-03-21T13:59:23.652 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.prefix_extractor: nullptr 2022-03-21T13:59:23.653 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-03-21T13:59:23.653 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.num_levels: 7 2022-03-21T13:59:23.653 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-03-21T13:59:23.653 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-03-21T13:59:23.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-03-21T13:59:23.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-03-21T13:59:23.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-03-21T13:59:23.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-03-21T13:59:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-03-21T13:59:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-03-21T13:59:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-03-21T13:59:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-03-21T13:59:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.window_bits: -14 2022-03-21T13:59:23.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.level: 32767 2022-03-21T13:59:23.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.strategy: 0 2022-03-21T13:59:23.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-03-21T13:59:23.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-03-21T13:59:23.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-03-21T13:59:23.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compression_opts.enabled: false 2022-03-21T13:59:23.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-03-21T13:59:23.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-03-21T13:59:23.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-03-21T13:59:23.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.target_file_size_base: 67108864 2022-03-21T13:59:23.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.target_file_size_multiplier: 1 2022-03-21T13:59:23.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-03-21T13:59:23.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-03-21T13:59:23.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-03-21T13:59:23.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-03-21T13:59:23.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-03-21T13:59:23.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-03-21T13:59:23.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-03-21T13:59:23.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-03-21T13:59:23.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-03-21T13:59:23.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-03-21T13:59:23.662 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-03-21T13:59:23.662 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-03-21T13:59:23.662 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.arena_block_size: 4194304 2022-03-21T13:59:23.662 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-03-21T13:59:23.663 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-03-21T13:59:23.663 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-03-21T13:59:23.663 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.disable_auto_compactions: 0 2022-03-21T13:59:23.663 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-03-21T13:59:23.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-03-21T13:59:23.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-03-21T13:59:23.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-03-21T13:59:23.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-03-21T13:59:23.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-03-21T13:59:23.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-03-21T13:59:23.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-03-21T13:59:23.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-03-21T13:59:23.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-03-21T13:59:23.666 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.table_properties_collectors: 2022-03-21T13:59:23.666 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.inplace_update_support: 0 2022-03-21T13:59:23.666 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.inplace_update_num_locks: 10000 2022-03-21T13:59:23.666 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-03-21T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-03-21T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.memtable_huge_page_size: 0 2022-03-21T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.bloom_locality: 0 2022-03-21T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.max_successive_merges: 0 2022-03-21T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.optimize_filters_for_hits: 0 2022-03-21T13:59:23.668 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.paranoid_file_checks: 0 2022-03-21T13:59:23.668 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.force_consistency_checks: 1 2022-03-21T13:59:23.668 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.report_bg_io_stats: 0 2022-03-21T13:59:23.668 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.ttl: 2592000 2022-03-21T13:59:23.669 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.periodic_compaction_seconds: 0 2022-03-21T13:59:23.669 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.enable_blob_files: false 2022-03-21T13:59:23.669 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.min_blob_size: 0 2022-03-21T13:59:23.669 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.blob_file_size: 268435456 2022-03-21T13:59:23.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.blob_compression_type: NoCompression 2022-03-21T13:59:23.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.enable_blob_garbage_collection: false 2022-03-21T13:59:23.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-03-21T13:59:23.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-03-21T13:59:23.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2022-03-21T13:59:23.671 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2022-03-21T13:59:23.671 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871163328835, "job": 1, "event": "recovery_started", "wal_files": [10]} 2022-03-21T13:59:23.671 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2022-03-21T13:59:23.671 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-03-21T13:59:23.672 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871163331842, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 91750, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 89843, "index_size": 267, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13376, "raw_average_key_size": 49, "raw_value_size": 82522, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1647871163, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1e7b36e4-b37b-411e-9731-0fc6ddad9c10", "db_session_id": "XOLIDC4XLN0SANRSWL9Q"}} 2022-03-21T13:59:23.672 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2022-03-21T13:59:23.672 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871163333167, "job": 1, "event": "recovery_finished"} 2022-03-21T13:59:23.672 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-03-21T13:59:23.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5631191f4a80 2022-03-21T13:59:23.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: DB pointer 0x563119204000 2022-03-21T13:59:23.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-03-21T13:59:23.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-03-21T13:59:23.674 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2022-03-21T13:59:23.674 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T13:59:23.674 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-03-21T13:59:23.674 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-03-21T13:59:23.675 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T13:59:23.675 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-03-21T13:59:23.675 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-03-21T13:59:23.675 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T13:59:23.676 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.676 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-03-21T13:59:23.676 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T13:59:23.676 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T13:59:23.676 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 91.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.677 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 91.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.677 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.677 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.678 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-03-21T13:59:23.678 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T13:59:23.678 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T13:59:23.678 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.679 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T13:59:23.679 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T13:59:23.679 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T13:59:23.679 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T13:59:23.680 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T13:59:23.680 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T13:59:23.680 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 7.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T13:59:23.680 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 7.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T13:59:23.681 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T13:59:23.681 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.681 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T13:59:23.681 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.682 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-03-21T13:59:23.682 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T13:59:23.682 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T13:59:23.683 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 91.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.683 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 91.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.684 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-03-21T13:59:23.684 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.685 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-03-21T13:59:23.685 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T13:59:23.685 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T13:59:23.686 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 42.7 0.00 0.00 1 0.002 0 0 2022-03-21T13:59:23.686 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T13:59:23.687 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T13:59:23.687 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T13:59:23.688 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T13:59:23.688 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T13:59:23.689 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T13:59:23.689 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 7.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T13:59:23.689 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T13:59:23.690 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T13:59:23.690 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.691 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T13:59:23.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:23.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???) e1 preinit fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:23.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).mds e1 new map 2022-03-21T13:59:23.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).mds e1 print_map 2022-03-21T13:59:23.693 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2022-03-21T13:59:23.694 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-03-21T13:59:23.694 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-03-21T13:59:23.695 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2022-03-21T13:59:23.695 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-03-21T13:59:23.696 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2022-03-21T13:59:23.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-03-21T13:59:23.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T13:59:23.697 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T13:59:23.697 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T13:59:23.698 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-03-21T13:59:23.698 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expand map: {default=false} 2022-03-21T13:59:23.699 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta from 'false' to 'false' 2022-03-21T13:59:23.699 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expanded map: {default=false} 2022-03-21T13:59:23.700 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expand map: {default=info} 2022-03-21T13:59:23.700 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta from 'info' to 'info' 2022-03-21T13:59:23.700 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expanded map: {default=info} 2022-03-21T13:59:23.701 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expand map: {default=daemon} 2022-03-21T13:59:23.701 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta from 'daemon' to 'daemon' 2022-03-21T13:59:23.702 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expanded map: {default=daemon} 2022-03-21T13:59:23.702 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expand map: {default=debug} 2022-03-21T13:59:23.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta from 'debug' to 'debug' 2022-03-21T13:59:23.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: expand_channel_meta expanded map: {default=debug} 2022-03-21T13:59:23.704 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 systemd[1]: Started Ceph mon.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T13:59:23.704 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mon.a is new leader, mons a in quorum (ranks 0) 2022-03-21T13:59:23.704 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2022-03-21T13:59:23.705 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: fsmap 2022-03-21T13:59:23.705 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: osdmap e1: 0 total, 0 up, 0 in 2022-03-21T13:59:23.706 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:23 smithi002 ceph-mon[17167]: mgrmap e1: no daemons active 2022-03-21T13:59:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:27 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1275036154' entity='client.admin' 2022-03-21T13:59:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.027+0000 7fe8c0365700 1 Processor -- start 2022-03-21T13:59:27.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.027+0000 7fe8c0365700 1 -- start start 2022-03-21T13:59:27.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.027+0000 7fe8c0365700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:27.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.027+0000 7fe8c0365700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b8104100 con 0x7fe8b8103790 2022-03-21T13:59:27.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.028+0000 7fe8be101700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:27.398 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.028+0000 7fe8be101700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43344/0 (socket says 172.21.15.2:43344) 2022-03-21T13:59:27.398 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.028+0000 7fe8be101700 1 -- 172.21.15.2:0/3893087834 learned_addr learned my addr 172.21.15.2:0/3893087834 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:27.398 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.028+0000 7fe8be101700 1 -- 172.21.15.2:0/3893087834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8104240 con 0x7fe8b8103790 2022-03-21T13:59:27.398 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8be101700 1 --2- 172.21.15.2:0/3893087834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009730 tx=0x7fe8b40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e47a7ae6b8524a7 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:27.399 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8bd0ff700 1 -- 172.21.15.2:0/3893087834 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4004030 con 0x7fe8b8103790 2022-03-21T13:59:27.399 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8bd0ff700 1 -- 172.21.15.2:0/3893087834 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe8b4004190 con 0x7fe8b8103790 2022-03-21T13:59:27.399 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8bd0ff700 1 -- 172.21.15.2:0/3893087834 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4003750 con 0x7fe8b8103790 2022-03-21T13:59:27.399 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8c0365700 1 -- 172.21.15.2:0/3893087834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 msgr2=0x7fe8b8103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:27.400 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.029+0000 7fe8c0365700 1 --2- 172.21.15.2:0/3893087834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009730 tx=0x7fe8b40090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:27.400 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.030+0000 7fe8c0365700 1 -- 172.21.15.2:0/3893087834 shutdown_connections 2022-03-21T13:59:27.400 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.030+0000 7fe8c0365700 1 --2- 172.21.15.2:0/3893087834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b8103bc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:27.400 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.030+0000 7fe8c0365700 1 -- 172.21.15.2:0/3893087834 >> 172.21.15.2:0/3893087834 conn(0x7fe8b8076da0 msgr2=0x7fe8b80771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:27.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.030+0000 7fe8c0365700 1 -- 172.21.15.2:0/3893087834 shutdown_connections 2022-03-21T13:59:27.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.030+0000 7fe8c0365700 1 -- 172.21.15.2:0/3893087834 wait complete. 2022-03-21T13:59:27.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8c0365700 1 Processor -- start 2022-03-21T13:59:27.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8c0365700 1 -- start start 2022-03-21T13:59:27.402 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8c0365700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:27.402 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8c0365700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b8112920 con 0x7fe8b8103790 2022-03-21T13:59:27.402 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8be101700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:27.402 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8be101700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43346/0 (socket says 172.21.15.2:43346) 2022-03-21T13:59:27.403 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.031+0000 7fe8be101700 1 -- 172.21.15.2:0/1275036154 learned_addr learned my addr 172.21.15.2:0/1275036154 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:27.403 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8be101700 1 -- 172.21.15.2:0/1275036154 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4009370 con 0x7fe8b8103790 2022-03-21T13:59:27.403 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8be101700 1 --2- 172.21.15.2:0/1275036154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4004530 tx=0x7fe8b40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:27.403 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4011a60 con 0x7fe8b8103790 2022-03-21T13:59:27.404 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe8b4011bc0 con 0x7fe8b8103790 2022-03-21T13:59:27.404 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b8112b20 con 0x7fe8b8103790 2022-03-21T13:59:27.404 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.032+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4019400 con 0x7fe8b8103790 2022-03-21T13:59:27.404 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.033+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b81137c0 con 0x7fe8b8103790 2022-03-21T13:59:27.404 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.033+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fe8b4017020 con 0x7fe8b8103790 2022-03-21T13:59:27.405 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.033+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe8b402d900 con 0x7fe8b8103790 2022-03-21T13:59:27.405 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.034+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b805e600 con 0x7fe8b8103790 2022-03-21T13:59:27.405 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.037+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7fe8b401a950 con 0x7fe8b8103790 2022-03-21T13:59:27.405 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.109+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe8b8105750 con 0x7fe8b8103790 2022-03-21T13:59:27.406 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.112+0000 7fe8a6ffd700 1 -- 172.21.15.2:0/1275036154 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe8b8105750 con 0x7fe8b8103790 2022-03-21T13:59:27.406 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 msgr2=0x7fe8b81123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:27.406 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 --2- 172.21.15.2:0/1275036154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4004530 tx=0x7fe8b40090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:27.406 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 shutdown_connections 2022-03-21T13:59:27.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 --2- 172.21.15.2:0/1275036154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8b8103790 0x7fe8b81123e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:27.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 >> 172.21.15.2:0/1275036154 conn(0x7fe8b8076da0 msgr2=0x7fe8b81016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:27.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 shutdown_connections 2022-03-21T13:59:27.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:26.113+0000 7fe8c0365700 1 -- 172.21.15.2:0/1275036154 wait complete. 2022-03-21T13:59:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote config to /etc/ceph/ceph.conf 2022-03-21T13:59:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-03-21T13:59:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mgr... 2022-03-21T13:59:29.012 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying port 9283 ... 2022-03-21T13:59:29.356 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.x.service: Unit ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.x.service not loaded. 2022-03-21T13:59:29.374 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c.target.wants/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@.service. 2022-03-21T13:59:29.624 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:29 smithi002 systemd[1]: Starting Ceph mgr.x for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T13:59:30.562 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:30 smithi002 bash[17394]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:30.704 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2022-03-21T13:59:30.724 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T13:59:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2022-03-21T13:59:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2022-03-21T13:59:30.748 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2022-03-21T13:59:30.768 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T13:59:30.770 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2022-03-21T13:59:30.771 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2022-03-21T13:59:30.771 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr to start... 2022-03-21T13:59:30.771 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr... 2022-03-21T13:59:30.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:30 smithi002 bash[17394]: 73577d22a6e50923c1e50c04b5452cb889fef46e5f59bdcb44313b633edf9242 2022-03-21T13:59:30.866 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:30 smithi002 systemd[1]: Started Ceph mgr.x for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T13:59:30.866 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:30 smithi002 conmon[17580]: 2022-03-21T13:59:30.741+0000 7fc7fc04b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-21T13:59:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:31.332 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:30 smithi002 conmon[17580]: 2022-03-21T13:59:30.960+0000 7fc7fc04b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-21T13:59:31.332 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:31 smithi002 conmon[17580]: 2022-03-21T13:59:31.096+0000 7fc7fc04b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-21T13:59:31.332 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:31 smithi002 conmon[17580]: 2022-03-21T13:59:31.330+0000 7fc7fc04b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-21T13:59:31.383 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.382+0000 7f0bc4bb3700 1 Processor -- start 2022-03-21T13:59:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.382+0000 7f0bc4bb3700 1 -- start start 2022-03-21T13:59:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.382+0000 7f0bc4bb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.382+0000 7f0bc4bb3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0076f40 con 0x7f0bc0076b30 2022-03-21T13:59:31.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.383+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:31.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.383+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43352/0 (socket says 172.21.15.2:43352) 2022-03-21T13:59:31.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.383+0000 7f0bbf7fe700 1 -- 172.21.15.2:0/4054424697 learned_addr learned my addr 172.21.15.2:0/4054424697 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.384+0000 7f0bbf7fe700 1 -- 172.21.15.2:0/4054424697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc0075610 con 0x7f0bc0076b30 2022-03-21T13:59:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.384+0000 7f0bbf7fe700 1 --2- 172.21.15.2:0/4054424697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009730 tx=0x7f0bb00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cb9958d055a872 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.384+0000 7f0bbe7fc700 1 -- 172.21.15.2:0/4054424697 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0004030 con 0x7f0bc0076b30 2022-03-21T13:59:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.384+0000 7f0bbe7fc700 1 -- 172.21.15.2:0/4054424697 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f0bb0004190 con 0x7f0bc0076b30 2022-03-21T13:59:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bbe7fc700 1 -- 172.21.15.2:0/4054424697 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0003750 con 0x7f0bc0076b30 2022-03-21T13:59:31.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4054424697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 msgr2=0x7f0bc00750d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:31.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bc4bb3700 1 --2- 172.21.15.2:0/4054424697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009730 tx=0x7f0bb00090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:31.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4054424697 shutdown_connections 2022-03-21T13:59:31.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bc4bb3700 1 --2- 172.21.15.2:0/4054424697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc00750d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:31.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.385+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4054424697 >> 172.21.15.2:0/4054424697 conn(0x7f0bc00fcbe0 msgr2=0x7f0bc00feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:31.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.386+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4054424697 shutdown_connections 2022-03-21T13:59:31.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.386+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4054424697 wait complete. 2022-03-21T13:59:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.387+0000 7f0bc4bb3700 1 Processor -- start 2022-03-21T13:59:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.387+0000 7f0bc4bb3700 1 -- start start 2022-03-21T13:59:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.388+0000 7f0bc4bb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.388+0000 7f0bc4bb3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0112700 con 0x7f0bc0076b30 2022-03-21T13:59:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.388+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.388+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43354/0 (socket says 172.21.15.2:43354) 2022-03-21T13:59:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.388+0000 7f0bbf7fe700 1 -- 172.21.15.2:0/4263382774 learned_addr learned my addr 172.21.15.2:0/4263382774 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0bbf7fe700 1 -- 172.21.15.2:0/4263382774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0009370 con 0x7f0bc0076b30 2022-03-21T13:59:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0bbf7fe700 1 --2- 172.21.15.2:0/4263382774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0004530 tx=0x7f0bb00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:31.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0011a60 con 0x7f0bc0076b30 2022-03-21T13:59:31.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc0112900 con 0x7f0bc0076b30 2022-03-21T13:59:31.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f0bb0011bc0 con 0x7f0bc0076b30 2022-03-21T13:59:31.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.389+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0019400 con 0x7f0bc0076b30 2022-03-21T13:59:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.390+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc01135a0 con 0x7f0bc0076b30 2022-03-21T13:59:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.390+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f0bb0017020 con 0x7f0bc0076b30 2022-03-21T13:59:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.390+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0bb002d740 con 0x7f0bc0076b30 2022-03-21T13:59:31.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.391+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc005e600 con 0x7f0bc0076b30 2022-03-21T13:59:31.397 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.396+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f0bb0019d60 con 0x7f0bc0076b30 2022-03-21T13:59:31.473 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.471+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0bc0107820 con 0x7f0bc0076b30 2022-03-21T13:59:31.482 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.472+0000 7f0ba7fff700 1 -- 172.21.15.2:0/4263382774 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f0bc0107820 con 0x7f0bc0076b30 2022-03-21T13:59:31.482 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.483 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T13:59:31.483 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "ee5e78f2-a91e-11ec-8c35-001a4aab830c", 2022-03-21T13:59:31.483 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2022-03-21T13:59:31.483 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-03-21T13:59:31.484 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2022-03-21T13:59:31.484 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2022-03-21T13:59:31.484 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.485 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-03-21T13:59:31.485 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2022-03-21T13:59:31.485 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2022-03-21T13:59:31.486 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:31.486 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2022-03-21T13:59:31.486 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2022-03-21T13:59:31.486 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:31.487 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 8, 2022-03-21T13:59:31.487 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2022-03-21T13:59:31.487 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:31.488 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2022-03-21T13:59:31.488 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2022-03-21T13:59:31.488 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.488 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2022-03-21T13:59:31.489 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:31.489 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2022-03-21T13:59:31.489 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-03-21T13:59:31.489 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-03-21T13:59:31.490 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-03-21T13:59:31.490 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-03-21T13:59:31.490 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-03-21T13:59:31.490 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.491 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2022-03-21T13:59:31.491 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-03-21T13:59:31.491 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-03-21T13:59:31.491 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2022-03-21T13:59:31.492 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2022-03-21T13:59:31.492 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-03-21T13:59:31.492 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-03-21T13:59:31.492 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-03-21T13:59:31.493 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2022-03-21T13:59:31.493 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.493 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2022-03-21T13:59:31.493 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:31.494 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2022-03-21T13:59:31.494 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2022-03-21T13:59:31.494 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.495 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2022-03-21T13:59:31.495 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2022-03-21T13:59:31.495 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-03-21T13:59:31.495 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2022-03-21T13:59:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2022-03-21T13:59:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2022-03-21T13:59:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2022-03-21T13:59:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:31.497 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.497 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2022-03-21T13:59:31.497 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:31.497 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2022-03-21T13:59:17.766329+0000", 2022-03-21T13:59:31.498 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:31.498 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:31.498 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2022-03-21T13:59:31.498 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T13:59:31.499 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 msgr2=0x7f0bc01121c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:31.499 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 --2- 172.21.15.2:0/4263382774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0004530 tx=0x7f0bb00090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:31.499 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 shutdown_connections 2022-03-21T13:59:31.499 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 --2- 172.21.15.2:0/4263382774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0bc0076b30 0x7f0bc01121c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 >> 172.21.15.2:0/4263382774 conn(0x7f0bc00fcbe0 msgr2=0x7f0bc00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.473+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 shutdown_connections 2022-03-21T13:59:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:31.474+0000 7f0bc4bb3700 1 -- 172.21.15.2:0/4263382774 wait complete. 2022-03-21T13:59:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:31 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4263382774' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-03-21T13:59:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (1/15)... 2022-03-21T13:59:32.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:31 smithi002 conmon[17580]: 2022-03-21T13:59:31.682+0000 7fc7fc04b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-21T13:59:32.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:31 smithi002 conmon[17580]: 2022-03-21T13:59:31.866+0000 7fc7fc04b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-21T13:59:32.615 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:32 smithi002 conmon[17580]: 2022-03-21T13:59:32.341+0000 7fc7fc04b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-21T13:59:32.616 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:32 smithi002 conmon[17580]: 2022-03-21T13:59:32.586+0000 7fc7fc04b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-21T13:59:33.219 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:32 smithi002 conmon[17580]: 2022-03-21T13:59:32.921+0000 7fc7fc04b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-21T13:59:33.616 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:33 smithi002 conmon[17580]: 2022-03-21T13:59:33.218+0000 7fc7fc04b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-21T13:59:33.616 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:33 smithi002 conmon[17580]: 2022-03-21T13:59:33.443+0000 7fc7fc04b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-21T13:59:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:34.566 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:34 smithi002 conmon[17580]: 2022-03-21T13:59:34.289+0000 7fc7fc04b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-21T13:59:34.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:34 smithi002 conmon[17580]: 2022-03-21T13:59:34.565+0000 7fc7fc04b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-21T13:59:35.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:35 smithi002 conmon[17580]: 2022-03-21T13:59:35.593+0000 7fc7fc04b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-21T13:59:37.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:36 smithi002 conmon[17580]: 2022-03-21T13:59:36.510+0000 7fc7fc04b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-21T13:59:37.116 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:36 smithi002 conmon[17580]: 2022-03-21T13:59:36.678+0000 7fc7fc04b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-21T13:59:37.571 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:37 smithi002 conmon[17580]: 2022-03-21T13:59:37.148+0000 7fc7fc04b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-21T13:59:37.571 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:37 smithi002 conmon[17580]: 2022-03-21T13:59:37.315+0000 7fc7fc04b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-21T13:59:37.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:37 smithi002 conmon[17580]: 2022-03-21T13:59:37.505+0000 7fc7fc04b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-21T13:59:38.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:38 smithi002 conmon[17580]: 2022-03-21T13:59:38.099+0000 7fc7fc04b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-21T13:59:38.366 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:38 smithi002 conmon[17580]: 2022-03-21T13:59:38.277+0000 7fc7fc04b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-21T13:59:38.427 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.426+0000 7f3c01b56700 1 Processor -- start 2022-03-21T13:59:38.428 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.426+0000 7f3c01b56700 1 -- start start 2022-03-21T13:59:38.429 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.427+0000 7f3c01b56700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:38.429 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.427+0000 7f3c01b56700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc102070 con 0x7f3bfc0700d0 2022-03-21T13:59:38.429 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.427+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:38.429 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.427+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43356/0 (socket says 172.21.15.2:43356) 2022-03-21T13:59:38.430 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.427+0000 7f3bfb7fe700 1 -- 172.21.15.2:0/3860549419 learned_addr learned my addr 172.21.15.2:0/3860549419 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:38.430 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.428+0000 7f3bfb7fe700 1 -- 172.21.15.2:0/3860549419 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc106bc0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.428+0000 7f3bfb7fe700 1 --2- 172.21.15.2:0/3860549419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009730 tx=0x7f3bec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2dafc384268cb4 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:38.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3bfa7fc700 1 -- 172.21.15.2:0/3860549419 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bec004030 con 0x7f3bfc0700d0 2022-03-21T13:59:38.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3bfa7fc700 1 -- 172.21.15.2:0/3860549419 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f3bec004190 con 0x7f3bfc0700d0 2022-03-21T13:59:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3bfa7fc700 1 -- 172.21.15.2:0/3860549419 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bec003750 con 0x7f3bfc0700d0 2022-03-21T13:59:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3c01b56700 1 -- 172.21.15.2:0/3860549419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 msgr2=0x7f3bfc106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3c01b56700 1 --2- 172.21.15.2:0/3860549419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009730 tx=0x7f3bec0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3c01b56700 1 -- 172.21.15.2:0/3860549419 shutdown_connections 2022-03-21T13:59:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3c01b56700 1 --2- 172.21.15.2:0/3860549419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc106680 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.429+0000 7f3c01b56700 1 -- 172.21.15.2:0/3860549419 >> 172.21.15.2:0/3860549419 conn(0x7f3bfc0fbc90 msgr2=0x7f3bfc0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.430+0000 7f3c01b56700 1 -- 172.21.15.2:0/3860549419 shutdown_connections 2022-03-21T13:59:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.430+0000 7f3c01b56700 1 -- 172.21.15.2:0/3860549419 wait complete. 2022-03-21T13:59:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.431+0000 7f3c01b56700 1 Processor -- start 2022-03-21T13:59:38.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.431+0000 7f3c01b56700 1 -- start start 2022-03-21T13:59:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.432+0000 7f3c01b56700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.432+0000 7f3c01b56700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc1128a0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.432+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.432+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43360/0 (socket says 172.21.15.2:43360) 2022-03-21T13:59:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.432+0000 7f3bfb7fe700 1 -- 172.21.15.2:0/556764133 learned_addr learned my addr 172.21.15.2:0/556764133 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3bfb7fe700 1 -- 172.21.15.2:0/556764133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec009370 con 0x7f3bfc0700d0 2022-03-21T13:59:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3bfb7fe700 1 --2- 172.21.15.2:0/556764133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3bec004530 tx=0x7f3bec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bec011a60 con 0x7f3bfc0700d0 2022-03-21T13:59:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f3bec011bc0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bec019400 con 0x7f3bfc0700d0 2022-03-21T13:59:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc112aa0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.433+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc113740 con 0x7f3bfc0700d0 2022-03-21T13:59:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.434+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f3bec017020 con 0x7f3bfc0700d0 2022-03-21T13:59:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.434+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3bec02d780 con 0x7f3bfc0700d0 2022-03-21T13:59:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.435+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc05e600 con 0x7f3bfc0700d0 2022-03-21T13:59:38.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.437+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f3bec01a780 con 0x7f3bfc0700d0 2022-03-21T13:59:38.515 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.514+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3bfc1079b0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.516 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.514+0000 7f3be3fff700 1 -- 172.21.15.2:0/556764133 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f3bfc1079b0 con 0x7f3bfc0700d0 2022-03-21T13:59:38.516 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T13:59:38.516 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "ee5e78f2-a91e-11ec-8c35-001a4aab830c", 2022-03-21T13:59:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2022-03-21T13:59:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-03-21T13:59:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2022-03-21T13:59:38.523 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2022-03-21T13:59:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-03-21T13:59:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2022-03-21T13:59:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2022-03-21T13:59:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2022-03-21T13:59:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2022-03-21T13:59:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 15, 2022-03-21T13:59:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2022-03-21T13:59:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2022-03-21T13:59:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2022-03-21T13:59:38.527 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.527 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2022-03-21T13:59:38.527 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:38.527 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2022-03-21T13:59:38.527 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-03-21T13:59:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-03-21T13:59:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-03-21T13:59:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-03-21T13:59:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-03-21T13:59:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2022-03-21T13:59:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-03-21T13:59:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-03-21T13:59:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2022-03-21T13:59:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2022-03-21T13:59:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-03-21T13:59:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-03-21T13:59:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2022-03-21T13:59:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2022-03-21T13:59:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2022-03-21T13:59:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2022-03-21T13:59:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-03-21T13:59:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2022-03-21T13:59:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2022-03-21T13:59:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2022-03-21T13:59:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2022-03-21T13:59:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2022-03-21T13:59:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2022-03-21T13:59:17.766329+0000", 2022-03-21T13:59:38.535 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:38.535 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:38.536 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2022-03-21T13:59:38.536 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T13:59:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.515+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 msgr2=0x7f3bfc112360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.515+0000 7f3c01b56700 1 --2- 172.21.15.2:0/556764133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3bec004530 tx=0x7f3bec0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 shutdown_connections 2022-03-21T13:59:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516+0000 7f3c01b56700 1 --2- 172.21.15.2:0/556764133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bfc0700d0 0x7f3bfc112360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 >> 172.21.15.2:0/556764133 conn(0x7f3bfc0fbc90 msgr2=0x7f3bfc0fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:38.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 shutdown_connections 2022-03-21T13:59:38.538 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:38.516+0000 7f3c01b56700 1 -- 172.21.15.2:0/556764133 wait complete. 2022-03-21T13:59:38.821 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:38 smithi002 conmon[17580]: 2022-03-21T13:59:38.450+0000 7fc7fc04b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-21T13:59:39.067 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (2/15)... 2022-03-21T13:59:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:39 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/556764133' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-03-21T13:59:39.616 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:39 smithi002 conmon[17580]: 2022-03-21T13:59:39.465+0000 7fc7fc04b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-21T13:59:40.351 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:39 smithi002 conmon[17580]: 2022-03-21T13:59:39.881+0000 7fc7fc04b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-21T13:59:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: Activating manager daemon x 2022-03-21T13:59:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: mgrmap e2: x(active, starting, since 0.00447896s) 2022-03-21T13:59:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-03-21T13:59:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-03-21T13:59:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-03-21T13:59:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T13:59:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-03-21T13:59:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: Manager daemon x is now available 2022-03-21T13:59:40.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T13:59:40.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T13:59:40.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' 2022-03-21T13:59:40.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' 2022-03-21T13:59:40.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:40 smithi002 ceph-mon[17167]: from='mgr.14100 172.21.15.2:0/2540965838' entity='mgr.x' 2022-03-21T13:59:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:41.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.674+0000 7ff856771700 1 Processor -- start 2022-03-21T13:59:41.677 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.675+0000 7ff856771700 1 -- start start 2022-03-21T13:59:41.680 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.675+0000 7ff856771700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:41.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.675+0000 7ff856771700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850101f40 con 0x7ff8501015f0 2022-03-21T13:59:41.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.675+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:41.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.675+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43376/0 (socket says 172.21.15.2:43376) 2022-03-21T13:59:41.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.676+0000 7ff84ffff700 1 -- 172.21.15.2:0/4001825900 learned_addr learned my addr 172.21.15.2:0/4001825900 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:41.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.676+0000 7ff84ffff700 1 -- 172.21.15.2:0/4001825900 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850102080 con 0x7ff8501015f0 2022-03-21T13:59:41.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.676+0000 7ff84ffff700 1 --2- 172.21.15.2:0/4001825900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff840009730 tx=0x7ff8400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b18b51559cfac58 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:41.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff84effd700 1 -- 172.21.15.2:0/4001825900 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff840004030 con 0x7ff8501015f0 2022-03-21T13:59:41.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff84effd700 1 -- 172.21.15.2:0/4001825900 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ff840004190 con 0x7ff8501015f0 2022-03-21T13:59:41.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff84effd700 1 -- 172.21.15.2:0/4001825900 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff840003750 con 0x7ff8501015f0 2022-03-21T13:59:41.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 -- 172.21.15.2:0/4001825900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 msgr2=0x7ff850101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:41.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 --2- 172.21.15.2:0/4001825900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff840009730 tx=0x7ff8400090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 -- 172.21.15.2:0/4001825900 shutdown_connections 2022-03-21T13:59:41.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 --2- 172.21.15.2:0/4001825900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff850101a00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 -- 172.21.15.2:0/4001825900 >> 172.21.15.2:0/4001825900 conn(0x7ff8500fcdb0 msgr2=0x7ff8500ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:41.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.677+0000 7ff856771700 1 -- 172.21.15.2:0/4001825900 shutdown_connections 2022-03-21T13:59:41.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.678+0000 7ff856771700 1 -- 172.21.15.2:0/4001825900 wait complete. 2022-03-21T13:59:41.685 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.678+0000 7ff856771700 1 Processor -- start 2022-03-21T13:59:41.685 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff856771700 1 -- start start 2022-03-21T13:59:41.687 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff856771700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:41.688 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff856771700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8501128e0 con 0x7ff8501015f0 2022-03-21T13:59:41.688 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:41.688 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43378/0 (socket says 172.21.15.2:43378) 2022-03-21T13:59:41.688 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.679+0000 7ff84ffff700 1 -- 172.21.15.2:0/2681544725 learned_addr learned my addr 172.21.15.2:0/2681544725 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff84ffff700 1 -- 172.21.15.2:0/2681544725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840009370 con 0x7ff8501015f0 2022-03-21T13:59:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff84ffff700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff840004530 tx=0x7ff8400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff840011a60 con 0x7ff8501015f0 2022-03-21T13:59:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ff840011bc0 con 0x7ff8501015f0 2022-03-21T13:59:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff840019400 con 0x7ff8501015f0 2022-03-21T13:59:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.680+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff850112ae0 con 0x7ff8501015f0 2022-03-21T13:59:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.681+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff850113780 con 0x7ff8501015f0 2022-03-21T13:59:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.682+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85005e600 con 0x7ff8501015f0 2022-03-21T13:59:41.691 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.682+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 3) v1 ==== 41753+0+0 (secure 0 0 0) 0x7ff840017020 con 0x7ff8501015f0 2022-03-21T13:59:41.691 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.682+0000 7ff84cff9700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 0x7ff83c036fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:41.691 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.682+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff8400478e0 con 0x7ff8501015f0 2022-03-21T13:59:41.691 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.683+0000 7ff84f7fe700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 0x7ff83c036fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:41.692 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.683+0000 7ff84f7fe700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 0x7ff83c036fb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff844009780 tx=0x7ff844006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:41.692 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.688+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7ff84001ab30 con 0x7ff8501015f0 2022-03-21T13:59:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.947+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff85004af20 con 0x7ff8501015f0 2022-03-21T13:59:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.947+0000 7ff84cff9700 1 -- 172.21.15.2:0/2681544725 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7ff84000b7e0 con 0x7ff8501015f0 2022-03-21T13:59:41.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T13:59:41.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "ee5e78f2-a91e-11ec-8c35-001a4aab830c", 2022-03-21T13:59:41.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2022-03-21T13:59:41.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-03-21T13:59:41.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2022-03-21T13:59:41.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2022-03-21T13:59:41.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-03-21T13:59:41.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2022-03-21T13:59:41.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2022-03-21T13:59:41.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:41.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2022-03-21T13:59:41.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2022-03-21T13:59:41.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:41.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 18, 2022-03-21T13:59:41.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2022-03-21T13:59:41.963 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:41.963 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2022-03-21T13:59:41.963 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2022-03-21T13:59:41.963 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2022-03-21T13:59:41.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:41.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2022-03-21T13:59:41.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-03-21T13:59:41.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-03-21T13:59:41.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-03-21T13:59:41.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-03-21T13:59:41.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-03-21T13:59:41.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2022-03-21T13:59:41.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-03-21T13:59:41.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-03-21T13:59:41.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2022-03-21T13:59:41.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2022-03-21T13:59:41.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-03-21T13:59:41.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-03-21T13:59:41.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-03-21T13:59:41.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2022-03-21T13:59:41.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2022-03-21T13:59:41.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:41.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2022-03-21T13:59:41.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2022-03-21T13:59:41.969 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.969 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2022-03-21T13:59:41.969 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2022-03-21T13:59:41.969 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-03-21T13:59:41.969 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2022-03-21T13:59:41.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2022-03-21T13:59:41.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2022-03-21T13:59:41.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2022-03-21T13:59:41.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2022-03-21T13:59:41.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:41.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2022-03-21T13:59:41.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2022-03-21T13:59:41.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2022-03-21T13:59:17.766329+0000", 2022-03-21T13:59:41.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2022-03-21T13:59:41.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2022-03-21T13:59:41.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2022-03-21T13:59:41.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T13:59:41.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 msgr2=0x7ff83c036fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:41.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 0x7ff83c036fb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff844009780 tx=0x7ff844006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 msgr2=0x7ff8501123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:41.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff840004530 tx=0x7ff8400090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 shutdown_connections 2022-03-21T13:59:41.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff83c034b00 0x7ff83c036fb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 --2- 172.21.15.2:0/2681544725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8501015f0 0x7ff8501123a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:41.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 >> 172.21.15.2:0/2681544725 conn(0x7ff8500fcdb0 msgr2=0x7ff8500fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:41.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 shutdown_connections 2022-03-21T13:59:41.975 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:41.950+0000 7ff856771700 1 -- 172.21.15.2:0/2681544725 wait complete. 2022-03-21T13:59:42.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:41 smithi002 ceph-mon[17167]: mgrmap e3: x(active, since 1.00984s) 2022-03-21T13:59:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:mgr is available 2022-03-21T13:59:42.580 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:43.069 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.065+0000 7f7141c96700 1 Processor -- start 2022-03-21T13:59:43.069 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.065+0000 7f7141c96700 1 -- start start 2022-03-21T13:59:43.069 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.066+0000 7f7141c96700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:43.069 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.066+0000 7f7141c96700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c1040e0 con 0x7f713c103790 2022-03-21T13:59:43.070 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.066+0000 7f713b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:43.070 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.066+0000 7f713b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43380/0 (socket says 172.21.15.2:43380) 2022-03-21T13:59:43.070 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.066+0000 7f713b7fe700 1 -- 172.21.15.2:0/2178755001 learned_addr learned my addr 172.21.15.2:0/2178755001 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:43.071 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.067+0000 7f713b7fe700 1 -- 172.21.15.2:0/2178755001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c104220 con 0x7f713c103790 2022-03-21T13:59:43.075 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.067+0000 7f713b7fe700 1 --2- 172.21.15.2:0/2178755001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f712c009730 tx=0x7f712c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c4ea1976b87d05 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:43.076 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.067+0000 7f713a7fc700 1 -- 172.21.15.2:0/2178755001 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f712c004030 con 0x7f713c103790 2022-03-21T13:59:43.076 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.067+0000 7f713a7fc700 1 -- 172.21.15.2:0/2178755001 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f712c004190 con 0x7f713c103790 2022-03-21T13:59:43.076 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.067+0000 7f713a7fc700 1 -- 172.21.15.2:0/2178755001 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f712c003750 con 0x7f713c103790 2022-03-21T13:59:43.077 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 -- 172.21.15.2:0/2178755001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 msgr2=0x7f713c103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:43.077 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 --2- 172.21.15.2:0/2178755001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f712c009730 tx=0x7f712c0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.077 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 -- 172.21.15.2:0/2178755001 shutdown_connections 2022-03-21T13:59:43.077 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 --2- 172.21.15.2:0/2178755001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c103ba0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.078 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 -- 172.21.15.2:0/2178755001 >> 172.21.15.2:0/2178755001 conn(0x7f713c076dd0 msgr2=0x7f713c0771d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:43.078 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 -- 172.21.15.2:0/2178755001 shutdown_connections 2022-03-21T13:59:43.078 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.068+0000 7f7141c96700 1 -- 172.21.15.2:0/2178755001 wait complete. 2022-03-21T13:59:43.078 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.069+0000 7f7141c96700 1 Processor -- start 2022-03-21T13:59:43.079 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.069+0000 7f7141c96700 1 -- start start 2022-03-21T13:59:43.079 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.069+0000 7f7141c96700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:43.079 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.070+0000 7f7141c96700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c1128a0 con 0x7f713c103790 2022-03-21T13:59:43.080 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.070+0000 7f713b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:43.080 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.070+0000 7f713b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43382/0 (socket says 172.21.15.2:43382) 2022-03-21T13:59:43.080 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.070+0000 7f713b7fe700 1 -- 172.21.15.2:0/484059324 learned_addr learned my addr 172.21.15.2:0/484059324 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:43.080 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.070+0000 7f713b7fe700 1 -- 172.21.15.2:0/484059324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c009370 con 0x7f713c103790 2022-03-21T13:59:43.081 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f713b7fe700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f712c004530 tx=0x7f712c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:43.081 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f712c011a10 con 0x7f713c103790 2022-03-21T13:59:43.081 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f712c011b70 con 0x7f713c103790 2022-03-21T13:59:43.081 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f712c024490 con 0x7f713c103790 2022-03-21T13:59:43.081 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c112aa0 con 0x7f713c103790 2022-03-21T13:59:43.082 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.071+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c113740 con 0x7f713c103790 2022-03-21T13:59:43.082 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.072+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 41833+0+0 (secure 0 0 0) 0x7f712c02b060 con 0x7f713c103790 2022-03-21T13:59:43.082 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.072+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c05e600 con 0x7f713c103790 2022-03-21T13:59:43.082 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.072+0000 7f7123fff700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 0x7f7128037030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:43.083 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.073+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f712c047c10 con 0x7f713c103790 2022-03-21T13:59:43.083 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.073+0000 7f713affd700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 0x7f7128037030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:43.083 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.073+0000 7f713affd700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 0x7f7128037030 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7130009780 tx=0x7f7130006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:43.084 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.078+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f712c02c340 con 0x7f713c103790 2022-03-21T13:59:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:42 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2681544725' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-03-21T13:59:43.266 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.264+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f713c1055f0 con 0x7f713c103790 2022-03-21T13:59:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.265+0000 7f7123fff700 1 -- 172.21.15.2:0/484059324 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f712c018b40 con 0x7f713c103790 2022-03-21T13:59:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2022-03-21T13:59:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-03-21T13:59:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-03-21T13:59:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-03-21T13:59:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-03-21T13:59:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2022-03-21T13:59:43.272 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-03-21T13:59:43.272 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.272 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2022-03-21T13:59:43.273 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2022-03-21T13:59:43.273 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2022-03-21T13:59:43.273 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2022-03-21T13:59:43.273 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-03-21T13:59:43.273 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-03-21T13:59:43.275 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.267+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 msgr2=0x7f7128037030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:43.275 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 0x7f7128037030 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7130009780 tx=0x7f7130006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.276 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 msgr2=0x7f713c112360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:43.276 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f712c004530 tx=0x7f712c0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.276 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 shutdown_connections 2022-03-21T13:59:43.276 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f7128034b80 0x7f7128037030 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.277 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 --2- 172.21.15.2:0/484059324 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c103790 0x7f713c112360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:43.277 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 >> 172.21.15.2:0/484059324 conn(0x7f713c076dd0 msgr2=0x7f713c101640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:43.277 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 shutdown_connections 2022-03-21T13:59:43.277 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:43.268+0000 7f7141c96700 1 -- 172.21.15.2:0/484059324 wait complete. 2022-03-21T13:59:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling cephadm module... 2022-03-21T13:59:43.869 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:44.052 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:43 smithi002 ceph-mon[17167]: mgrmap e4: x(active, since 3s) 2022-03-21T13:59:44.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:43 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/484059324' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-03-21T13:59:44.361 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.358+0000 7f4212dae700 1 Processor -- start 2022-03-21T13:59:44.361 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.358+0000 7f4212dae700 1 -- start start 2022-03-21T13:59:44.362 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.358+0000 7f4212dae700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:44.362 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.358+0000 7f4212dae700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c076fa0 con 0x7f420c076b90 2022-03-21T13:59:44.362 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.359+0000 7f4210b4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:44.362 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.359+0000 7f4210b4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43384/0 (socket says 172.21.15.2:43384) 2022-03-21T13:59:44.363 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.359+0000 7f4210b4a700 1 -- 172.21.15.2:0/1486088255 learned_addr learned my addr 172.21.15.2:0/1486088255 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:44.363 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.359+0000 7f4210b4a700 1 -- 172.21.15.2:0/1486088255 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c075670 con 0x7f420c076b90 2022-03-21T13:59:44.363 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.359+0000 7f4210b4a700 1 --2- 172.21.15.2:0/1486088255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4200009730 tx=0x7f42000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b12d01173ea7d93 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:44.366 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f420b7fe700 1 -- 172.21.15.2:0/1486088255 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4200004030 con 0x7f420c076b90 2022-03-21T13:59:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f420b7fe700 1 -- 172.21.15.2:0/1486088255 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f4200004190 con 0x7f420c076b90 2022-03-21T13:59:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f420b7fe700 1 -- 172.21.15.2:0/1486088255 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4200003750 con 0x7f420c076b90 2022-03-21T13:59:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f4212dae700 1 -- 172.21.15.2:0/1486088255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 msgr2=0x7f420c075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f4212dae700 1 --2- 172.21.15.2:0/1486088255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4200009730 tx=0x7f42000090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f4212dae700 1 -- 172.21.15.2:0/1486088255 shutdown_connections 2022-03-21T13:59:44.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f4212dae700 1 --2- 172.21.15.2:0/1486088255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c075130 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.360+0000 7f4212dae700 1 -- 172.21.15.2:0/1486088255 >> 172.21.15.2:0/1486088255 conn(0x7f420c0fcda0 msgr2=0x7f420c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:44.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.361+0000 7f4212dae700 1 -- 172.21.15.2:0/1486088255 shutdown_connections 2022-03-21T13:59:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.361+0000 7f4212dae700 1 -- 172.21.15.2:0/1486088255 wait complete. 2022-03-21T13:59:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4212dae700 1 Processor -- start 2022-03-21T13:59:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4212dae700 1 -- start start 2022-03-21T13:59:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4212dae700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4212dae700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c1128e0 con 0x7f420c076b90 2022-03-21T13:59:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4210b4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4210b4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43386/0 (socket says 172.21.15.2:43386) 2022-03-21T13:59:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.362+0000 7f4210b4a700 1 -- 172.21.15.2:0/324388546 learned_addr learned my addr 172.21.15.2:0/324388546 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.363+0000 7f4210b4a700 1 -- 172.21.15.2:0/324388546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200009370 con 0x7f420c076b90 2022-03-21T13:59:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.363+0000 7f4210b4a700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4200004530 tx=0x7f42000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.364+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4200011a60 con 0x7f420c076b90 2022-03-21T13:59:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.364+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f4200011bc0 con 0x7f420c076b90 2022-03-21T13:59:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.364+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c112ae0 con 0x7f420c076b90 2022-03-21T13:59:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.364+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42000193c0 con 0x7f420c076b90 2022-03-21T13:59:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.364+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c113780 con 0x7f420c076b90 2022-03-21T13:59:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.365+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 41833+0+0 (secure 0 0 0) 0x7f4200017020 con 0x7f420c076b90 2022-03-21T13:59:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.365+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c05e600 con 0x7f420c076b90 2022-03-21T13:59:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.365+0000 7f42097fa700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 0x7f41f0037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.365+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4200047950 con 0x7f420c076b90 2022-03-21T13:59:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.366+0000 7f420bfff700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 0x7f41f0037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.367+0000 7f420bfff700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 0x7f41f0037080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009780 tx=0x7f41f8006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:44.375 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.371+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f420001ab30 con 0x7f420c076b90 2022-03-21T13:59:44.610 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.608+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f420c04af20 con 0x7f420c076b90 2022-03-21T13:59:44.913 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.911+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 5) v1 ==== 41844+0+0 (secure 0 0 0) 0x7f420000f050 con 0x7f420c076b90 2022-03-21T13:59:44.914 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.912+0000 7f42097fa700 1 -- 172.21.15.2:0/324388546 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f420003f7b0 con 0x7f420c076b90 2022-03-21T13:59:44.917 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 msgr2=0x7f41f0037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 0x7f41f0037080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009780 tx=0x7f41f8006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 msgr2=0x7f420c1123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4200004530 tx=0x7f42000090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 shutdown_connections 2022-03-21T13:59:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7f41f0034bd0 0x7f41f0037080 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 --2- 172.21.15.2:0/324388546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f420c076b90 0x7f420c1123a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 >> 172.21.15.2:0/324388546 conn(0x7f420c0fcda0 msgr2=0x7f420c0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 shutdown_connections 2022-03-21T13:59:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:44.915+0000 7f4212dae700 1 -- 172.21.15.2:0/324388546 wait complete. 2022-03-21T13:59:45.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:44 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/324388546' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-03-21T13:59:45.263 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: ignoring --setuser ceph since I am not root 2022-03-21T13:59:45.263 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: ignoring --setgroup ceph since I am not root 2022-03-21T13:59:45.488 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:45.615 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: 2022-03-21T13:59:45.260+0000 7f446a9a0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-21T13:59:45.615 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: 2022-03-21T13:59:45.489+0000 7f446a9a0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-21T13:59:45.912 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: 2022-03-21T13:59:45.623+0000 7f446a9a0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-21T13:59:45.912 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:45 smithi002 conmon[17580]: 2022-03-21T13:59:45.856+0000 7f446a9a0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-21T13:59:45.939 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.936+0000 7fd48eb66700 1 Processor -- start 2022-03-21T13:59:45.939 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.936+0000 7fd48eb66700 1 -- start start 2022-03-21T13:59:45.939 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.936+0000 7fd48eb66700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:45.940 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.936+0000 7fd48eb66700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488076f80 con 0x7fd488076b70 2022-03-21T13:59:45.940 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.937+0000 7fd48db64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:45.940 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.937+0000 7fd48db64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43394/0 (socket says 172.21.15.2:43394) 2022-03-21T13:59:45.940 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.937+0000 7fd48db64700 1 -- 172.21.15.2:0/1514882537 learned_addr learned my addr 172.21.15.2:0/1514882537 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:45.941 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.937+0000 7fd48db64700 1 -- 172.21.15.2:0/1514882537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488075650 con 0x7fd488076b70 2022-03-21T13:59:45.944 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48db64700 1 --2- 172.21.15.2:0/1514882537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd484009730 tx=0x7fd4840090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d454618e8eff18 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:45.944 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48cb62700 1 -- 172.21.15.2:0/1514882537 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd484004030 con 0x7fd488076b70 2022-03-21T13:59:45.944 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48cb62700 1 -- 172.21.15.2:0/1514882537 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd484004190 con 0x7fd488076b70 2022-03-21T13:59:45.945 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48cb62700 1 -- 172.21.15.2:0/1514882537 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd484003750 con 0x7fd488076b70 2022-03-21T13:59:45.945 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48eb66700 1 -- 172.21.15.2:0/1514882537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 msgr2=0x7fd488075110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:45.945 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.938+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1514882537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd484009730 tx=0x7fd4840090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:45.945 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.939+0000 7fd48eb66700 1 -- 172.21.15.2:0/1514882537 shutdown_connections 2022-03-21T13:59:45.946 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.939+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1514882537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488075110 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:45.946 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.939+0000 7fd48eb66700 1 -- 172.21.15.2:0/1514882537 >> 172.21.15.2:0/1514882537 conn(0x7fd4880fcbe0 msgr2=0x7fd4880feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:45.946 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.939+0000 7fd48eb66700 1 -- 172.21.15.2:0/1514882537 shutdown_connections 2022-03-21T13:59:45.946 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.939+0000 7fd48eb66700 1 -- 172.21.15.2:0/1514882537 wait complete. 2022-03-21T13:59:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.940+0000 7fd48eb66700 1 Processor -- start 2022-03-21T13:59:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.940+0000 7fd48eb66700 1 -- start start 2022-03-21T13:59:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.940+0000 7fd48eb66700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:45.947 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.940+0000 7fd48eb66700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881126c0 con 0x7fd488076b70 2022-03-21T13:59:45.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.941+0000 7fd48db64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:45.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.941+0000 7fd48db64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43396/0 (socket says 172.21.15.2:43396) 2022-03-21T13:59:45.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.941+0000 7fd48db64700 1 -- 172.21.15.2:0/1984022384 learned_addr learned my addr 172.21.15.2:0/1984022384 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:45.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.941+0000 7fd48db64700 1 -- 172.21.15.2:0/1984022384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484009370 con 0x7fd488076b70 2022-03-21T13:59:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.941+0000 7fd48db64700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd484004530 tx=0x7fd4840090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.942+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd484011a60 con 0x7fd488076b70 2022-03-21T13:59:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.942+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd484011bc0 con 0x7fd488076b70 2022-03-21T13:59:45.949 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.942+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd484019400 con 0x7fd488076b70 2022-03-21T13:59:45.950 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.942+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4881128c0 con 0x7fd488076b70 2022-03-21T13:59:45.950 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.942+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488113560 con 0x7fd488076b70 2022-03-21T13:59:45.951 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.943+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fd484017020 con 0x7fd488076b70 2022-03-21T13:59:45.951 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.943+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48805e600 con 0x7fd488076b70 2022-03-21T13:59:45.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.944+0000 7fd4767fc700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 0x7fd470037030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:45.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.944+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd484047a90 con 0x7fd488076b70 2022-03-21T13:59:45.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.944+0000 7fd48d363700 1 -- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 msgr2=0x7fd470037030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:45.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.944+0000 7fd48d363700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 0x7fd470037030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T13:59:45.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:45.949+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd48401ab30 con 0x7fd488076b70 2022-03-21T13:59:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.145+0000 7fd48d363700 1 -- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 msgr2=0x7fd470037030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.145+0000 7fd48d363700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 0x7fd470037030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-03-21T13:59:46.198 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.195+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd4881076a0 con 0x7fd488076b70 2022-03-21T13:59:46.198 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.196+0000 7fd4767fc700 1 -- 172.21.15.2:0/1984022384 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd48400b7e0 con 0x7fd488076b70 2022-03-21T13:59:46.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T13:59:46.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 5, 2022-03-21T13:59:46.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2022-03-21T13:59:46.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "x", 2022-03-21T13:59:46.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2022-03-21T13:59:46.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T13:59:46.201 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:45 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/324388546' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-03-21T13:59:46.201 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:45 smithi002 ceph-mon[17167]: mgrmap e5: x(active, since 5s) 2022-03-21T13:59:46.202 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 msgr2=0x7fd470037030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T13:59:46.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 0x7fd470037030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:46.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 msgr2=0x7fd488112180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:46.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd484004530 tx=0x7fd4840090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:46.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 shutdown_connections 2022-03-21T13:59:46.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7fd470034b80 0x7fd470037030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:46.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 --2- 172.21.15.2:0/1984022384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd488076b70 0x7fd488112180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:46.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.198+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 >> 172.21.15.2:0/1984022384 conn(0x7fd4880fcbe0 msgr2=0x7fd4880fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:46.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.199+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 shutdown_connections 2022-03-21T13:59:46.205 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:46.199+0000 7fd48eb66700 1 -- 172.21.15.2:0/1984022384 wait complete. 2022-03-21T13:59:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2022-03-21T13:59:46.575 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 5... 2022-03-21T13:59:46.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:46 smithi002 conmon[17580]: 2022-03-21T13:59:46.198+0000 7f446a9a0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-21T13:59:46.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:46 smithi002 conmon[17580]: 2022-03-21T13:59:46.400+0000 7f446a9a0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-21T13:59:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:47.139 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:46 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1984022384' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-03-21T13:59:47.139 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:46 smithi002 conmon[17580]: 2022-03-21T13:59:46.890+0000 7f446a9a0000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-21T13:59:47.192 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.190+0000 7ff337292700 1 Processor -- start 2022-03-21T13:59:47.192 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.190+0000 7ff337292700 1 -- start start 2022-03-21T13:59:47.193 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.190+0000 7ff337292700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:47.194 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.190+0000 7ff337292700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff330076f40 con 0x7ff330076b30 2022-03-21T13:59:47.194 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.191+0000 7ff336290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:47.194 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.191+0000 7ff336290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43398/0 (socket says 172.21.15.2:43398) 2022-03-21T13:59:47.195 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.191+0000 7ff336290700 1 -- 172.21.15.2:0/3081630466 learned_addr learned my addr 172.21.15.2:0/3081630466 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:47.195 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.191+0000 7ff336290700 1 -- 172.21.15.2:0/3081630466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff330075610 con 0x7ff330076b30 2022-03-21T13:59:47.195 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.192+0000 7ff336290700 1 --2- 172.21.15.2:0/3081630466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009730 tx=0x7ff32c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32eac832dc50af45 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:47.198 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.192+0000 7ff33528e700 1 -- 172.21.15.2:0/3081630466 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c004030 con 0x7ff330076b30 2022-03-21T13:59:47.198 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.192+0000 7ff33528e700 1 -- 172.21.15.2:0/3081630466 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ff32c004190 con 0x7ff330076b30 2022-03-21T13:59:47.198 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.192+0000 7ff33528e700 1 -- 172.21.15.2:0/3081630466 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c003750 con 0x7ff330076b30 2022-03-21T13:59:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 -- 172.21.15.2:0/3081630466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 msgr2=0x7ff3300750d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 --2- 172.21.15.2:0/3081630466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009730 tx=0x7ff32c0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 -- 172.21.15.2:0/3081630466 shutdown_connections 2022-03-21T13:59:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 --2- 172.21.15.2:0/3081630466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3300750d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 -- 172.21.15.2:0/3081630466 >> 172.21.15.2:0/3081630466 conn(0x7ff3300fcbe0 msgr2=0x7ff3300feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 -- 172.21.15.2:0/3081630466 shutdown_connections 2022-03-21T13:59:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.193+0000 7ff337292700 1 -- 172.21.15.2:0/3081630466 wait complete. 2022-03-21T13:59:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.194+0000 7ff337292700 1 Processor -- start 2022-03-21T13:59:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.194+0000 7ff337292700 1 -- start start 2022-03-21T13:59:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.194+0000 7ff337292700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.194+0000 7ff337292700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff330112700 con 0x7ff330076b30 2022-03-21T13:59:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.195+0000 7ff336290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.195+0000 7ff336290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43400/0 (socket says 172.21.15.2:43400) 2022-03-21T13:59:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.195+0000 7ff336290700 1 -- 172.21.15.2:0/4039726888 learned_addr learned my addr 172.21.15.2:0/4039726888 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.195+0000 7ff336290700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009370 con 0x7ff330076b30 2022-03-21T13:59:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.195+0000 7ff336290700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff32c004510 tx=0x7ff32c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:47.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.196+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c011a60 con 0x7ff330076b30 2022-03-21T13:59:47.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.196+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ff32c011bc0 con 0x7ff330076b30 2022-03-21T13:59:47.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.196+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c024510 con 0x7ff330076b30 2022-03-21T13:59:47.203 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.196+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff330112900 con 0x7ff330076b30 2022-03-21T13:59:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.196+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3301135a0 con 0x7ff330076b30 2022-03-21T13:59:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.197+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 41844+0+0 (secure 0 0 0) 0x7ff32c02b060 con 0x7ff330076b30 2022-03-21T13:59:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.197+0000 7ff31effd700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.197+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff318037790 con 0x7ff318034bd0 2022-03-21T13:59:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.197+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff32c047bb0 con 0x7ff330076b30 2022-03-21T13:59:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.198+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.198+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T13:59:47.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.399+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:47.401 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.399+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-03-21T13:59:47.480 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:47 smithi002 conmon[17580]: 2022-03-21T13:59:47.136+0000 7f446a9a0000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-21T13:59:47.775 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:47 smithi002 conmon[17580]: 2022-03-21T13:59:47.478+0000 7f446a9a0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-21T13:59:47.802 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.800+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:47.802 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:47.800+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-03-21T13:59:48.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:47 smithi002 conmon[17580]: 2022-03-21T13:59:47.773+0000 7f446a9a0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-21T13:59:48.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:48 smithi002 conmon[17580]: 2022-03-21T13:59:48.000+0000 7f446a9a0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-21T13:59:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:48.601+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:48.601+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-03-21T13:59:49.076 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:48 smithi002 conmon[17580]: 2022-03-21T13:59:48.796+0000 7f446a9a0000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-21T13:59:49.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:49 smithi002 conmon[17580]: 2022-03-21T13:59:49.074+0000 7f446a9a0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-21T13:59:50.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:50.203+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:50.204 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:50.203+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-03-21T13:59:50.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:50 smithi002 conmon[17580]: 2022-03-21T13:59:50.079+0000 7f446a9a0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-21T13:59:51.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:50 smithi002 conmon[17580]: 2022-03-21T13:59:50.994+0000 7f446a9a0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-21T13:59:51.366 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:51 smithi002 conmon[17580]: 2022-03-21T13:59:51.154+0000 7f446a9a0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-21T13:59:51.953 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:51 smithi002 conmon[17580]: 2022-03-21T13:59:51.618+0000 7f446a9a0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-21T13:59:51.953 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:51 smithi002 conmon[17580]: 2022-03-21T13:59:51.783+0000 7f446a9a0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-21T13:59:52.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:51 smithi002 conmon[17580]: 2022-03-21T13:59:51.952+0000 7f446a9a0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-21T13:59:52.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:52 smithi002 conmon[17580]: 2022-03-21T13:59:52.534+0000 7f446a9a0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-21T13:59:52.866 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:52 smithi002 conmon[17580]: 2022-03-21T13:59:52.697+0000 7f446a9a0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-21T13:59:53.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:52 smithi002 conmon[17580]: 2022-03-21T13:59:52.866+0000 7f446a9a0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-21T13:59:53.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:53.406+0000 7ff335a8f700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2638362969 2022-03-21T13:59:53.407 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:53.406+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-03-21T13:59:54.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:53 smithi002 conmon[17580]: 2022-03-21T13:59:53.860+0000 7f446a9a0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-21T13:59:54.357 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:54.356+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 6) v1 ==== 41654+0+0 (secure 0 0 0) 0x7ff32c017020 con 0x7ff330076b30 2022-03-21T13:59:54.358 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:54.356+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 msgr2=0x7ff318037080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T13:59:54.358 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:54.356+0000 7ff31effd700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:54.615 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:54 smithi002 conmon[17580]: 2022-03-21T13:59:54.276+0000 7f446a9a0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-21T13:59:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:54 smithi002 ceph-mon[17167]: Active manager daemon x restarted 2022-03-21T13:59:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:54 smithi002 ceph-mon[17167]: Activating manager daemon x 2022-03-21T13:59:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:54 smithi002 ceph-mon[17167]: osdmap e2: 0 total, 0 up, 0 in 2022-03-21T13:59:55.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.364+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 41764+0+0 (secure 0 0 0) 0x7ff32c00f050 con 0x7ff330076b30 2022-03-21T13:59:55.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.365+0000 7ff31effd700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 0x7ff31803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:55.367 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.365+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff318037790 con 0x7ff318038480 2022-03-21T13:59:55.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.366+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 0x7ff31803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:55.368 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.367+0000 7ff335a8f700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 0x7ff31803a860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff320003850 tx=0x7ff320009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:55.369 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.368+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7ff318037790 con 0x7ff318038480 2022-03-21T13:59:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.374+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff330112d90 con 0x7ff318038480 2022-03-21T13:59:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.375+0000 7ff31effd700 1 -- 172.21.15.2:0/4039726888 <== mgr.14118 v2:172.21.15.2:6800/1086540904 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff330112d90 con 0x7ff318038480 2022-03-21T13:59:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T13:59:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2022-03-21T13:59:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2022-03-21T13:59:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T13:59:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.375+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 msgr2=0x7ff31803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.375+0000 7ff337292700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 0x7ff31803a860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff320003850 tx=0x7ff320009350 comp rx=0 tx=0).stop 2022-03-21T13:59:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.375+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 msgr2=0x7ff3301121c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.375+0000 7ff337292700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff32c004510 tx=0x7ff32c0090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 shutdown_connections 2022-03-21T13:59:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff330076b30 0x7ff3301121c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7ff318038480 0x7ff31803a860 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 --2- 172.21.15.2:0/4039726888 >> [v2:172.21.15.2:6800/2638362969,v1:172.21.15.2:6801/2638362969] conn(0x7ff318034bd0 0x7ff318037080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 >> 172.21.15.2:0/4039726888 conn(0x7ff3300fcbe0 msgr2=0x7ff3300fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 shutdown_connections 2022-03-21T13:59:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:55.376+0000 7ff337292700 1 -- 172.21.15.2:0/4039726888 wait complete. 2022-03-21T13:59:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: mgrmap e6: x(active, starting, since 0.0670214s) 2022-03-21T13:59:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T13:59:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-03-21T13:59:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-03-21T13:59:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-03-21T13:59:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-03-21T13:59:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: Manager daemon x is now available 2022-03-21T13:59:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T13:59:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T13:59:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T13:59:55.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T13:59:55.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:55 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T13:59:55.778 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 5 is available 2022-03-21T13:59:55.779 INFO:teuthology.orchestra.run.smithi002.stderr:Setting orchestrator backend to cephadm... 2022-03-21T13:59:55.895 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:56.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.378+0000 7feae02ca700 1 Processor -- start 2022-03-21T13:59:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.378+0000 7feae02ca700 1 -- start start 2022-03-21T13:59:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.379+0000 7feae02ca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.379+0000 7feae02ca700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead8101ee0 con 0x7fead8101590 2022-03-21T13:59:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.379+0000 7feade066700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.379+0000 7feade066700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43418/0 (socket says 172.21.15.2:43418) 2022-03-21T13:59:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.379+0000 7feade066700 1 -- 172.21.15.2:0/1013652248 learned_addr learned my addr 172.21.15.2:0/1013652248 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.380+0000 7feade066700 1 -- 172.21.15.2:0/1013652248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8102020 con 0x7fead8101590 2022-03-21T13:59:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.380+0000 7feade066700 1 --2- 172.21.15.2:0/1013652248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fead4009730 tx=0x7fead40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1536bcef285d51 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:56.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.380+0000 7feadd064700 1 -- 172.21.15.2:0/1013652248 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fead4004030 con 0x7fead8101590 2022-03-21T13:59:56.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.380+0000 7feadd064700 1 -- 172.21.15.2:0/1013652248 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fead4004190 con 0x7fead8101590 2022-03-21T13:59:56.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.380+0000 7feadd064700 1 -- 172.21.15.2:0/1013652248 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fead4003750 con 0x7fead8101590 2022-03-21T13:59:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 -- 172.21.15.2:0/1013652248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 msgr2=0x7fead81019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 --2- 172.21.15.2:0/1013652248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fead4009730 tx=0x7fead40090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 -- 172.21.15.2:0/1013652248 shutdown_connections 2022-03-21T13:59:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 --2- 172.21.15.2:0/1013652248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead81019a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 -- 172.21.15.2:0/1013652248 >> 172.21.15.2:0/1013652248 conn(0x7fead80fcda0 msgr2=0x7fead80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 -- 172.21.15.2:0/1013652248 shutdown_connections 2022-03-21T13:59:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.381+0000 7feae02ca700 1 -- 172.21.15.2:0/1013652248 wait complete. 2022-03-21T13:59:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.382+0000 7feae02ca700 1 Processor -- start 2022-03-21T13:59:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.382+0000 7feae02ca700 1 -- start start 2022-03-21T13:59:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.383+0000 7feae02ca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:56.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.383+0000 7feae02ca700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead81128d0 con 0x7fead8101590 2022-03-21T13:59:56.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.383+0000 7feade066700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:56.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.383+0000 7feade066700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43420/0 (socket says 172.21.15.2:43420) 2022-03-21T13:59:56.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.383+0000 7feade066700 1 -- 172.21.15.2:0/3480835936 learned_addr learned my addr 172.21.15.2:0/3480835936 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:56.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feade066700 1 -- 172.21.15.2:0/3480835936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4009370 con 0x7fead8101590 2022-03-21T13:59:56.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feade066700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fead4004530 tx=0x7fead40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:56.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fead4011a10 con 0x7fead8101590 2022-03-21T13:59:56.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fead4011b70 con 0x7fead8101590 2022-03-21T13:59:56.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead8112ad0 con 0x7fead8101590 2022-03-21T13:59:56.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fead4024490 con 0x7fead8101590 2022-03-21T13:59:56.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.384+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead8113770 con 0x7fead8101590 2022-03-21T13:59:56.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.386+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fead402b060 con 0x7fead8101590 2022-03-21T13:59:56.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.386+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead805e600 con 0x7fead8101590 2022-03-21T13:59:56.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.386+0000 7feac6ffd700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 0x7feac0037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:56.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.386+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fead4047c20 con 0x7fead8101590 2022-03-21T13:59:56.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.387+0000 7feadd865700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 0x7feac0037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:56.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.387+0000 7feadd865700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 0x7feac0037080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feac8009780 tx=0x7feac8006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:56.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.391+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fead402c340 con 0x7fead8101590 2022-03-21T13:59:56.591 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.590+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fead80fe120 con 0x7feac0034bd0 2022-03-21T13:59:56.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.597+0000 7feac6ffd700 1 -- 172.21.15.2:0/3480835936 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fead80fe120 con 0x7feac0034bd0 2022-03-21T13:59:56.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 msgr2=0x7feac0037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:56.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 0x7feac0037080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feac8009780 tx=0x7feac8006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 msgr2=0x7fead8112390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:56.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fead4004530 tx=0x7fead40090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 shutdown_connections 2022-03-21T13:59:56.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7feac0034bd0 0x7feac0037080 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 --2- 172.21.15.2:0/3480835936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fead8101590 0x7fead8112390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:56.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 >> 172.21.15.2:0/3480835936 conn(0x7fead80fcda0 msgr2=0x7fead80fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:56.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 shutdown_connections 2022-03-21T13:59:56.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:56.600+0000 7feae02ca700 1 -- 172.21.15.2:0/3480835936 wait complete. 2022-03-21T13:59:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:56 smithi002 ceph-mon[17167]: mgrmap e7: x(active, since 1.07679s) 2022-03-21T13:59:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:56 smithi002 ceph-mon[17167]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-03-21T13:59:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:56 smithi002 ceph-mon[17167]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-03-21T13:59:57.092 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.584+0000 7f62a6cb0700 1 Processor -- start 2022-03-21T13:59:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.584+0000 7f62a6cb0700 1 -- start start 2022-03-21T13:59:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.585+0000 7f62a6cb0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.585+0000 7f62a6cb0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0104780 con 0x7f62a0070110 2022-03-21T13:59:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.585+0000 7f62a4a4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.585+0000 7f62a4a4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43422/0 (socket says 172.21.15.2:43422) 2022-03-21T13:59:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.585+0000 7f62a4a4c700 1 -- 172.21.15.2:0/3588342688 learned_addr learned my addr 172.21.15.2:0/3588342688 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.586+0000 7f62a4a4c700 1 -- 172.21.15.2:0/3588342688 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a01048c0 con 0x7f62a0070110 2022-03-21T13:59:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.586+0000 7f62a4a4c700 1 --2- 172.21.15.2:0/3588342688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6294009730 tx=0x7f62940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb096ad08264f286 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.586+0000 7f629f7fe700 1 -- 172.21.15.2:0/3588342688 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6294004030 con 0x7f62a0070110 2022-03-21T13:59:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.586+0000 7f629f7fe700 1 -- 172.21.15.2:0/3588342688 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f6294004190 con 0x7f62a0070110 2022-03-21T13:59:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.586+0000 7f629f7fe700 1 -- 172.21.15.2:0/3588342688 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6294003750 con 0x7f62a0070110 2022-03-21T13:59:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3588342688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 msgr2=0x7f62a0104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3588342688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6294009730 tx=0x7f62940090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3588342688 shutdown_connections 2022-03-21T13:59:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3588342688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0104240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3588342688 >> 172.21.15.2:0/3588342688 conn(0x7f62a00fbc90 msgr2=0x7f62a00fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3588342688 shutdown_connections 2022-03-21T13:59:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.587+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3588342688 wait complete. 2022-03-21T13:59:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.588+0000 7f62a6cb0700 1 Processor -- start 2022-03-21T13:59:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.588+0000 7f62a6cb0700 1 -- start start 2022-03-21T13:59:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.588+0000 7f62a6cb0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.588+0000 7f62a6cb0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01128a0 con 0x7f62a0070110 2022-03-21T13:59:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.589+0000 7f62a4a4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.589+0000 7f62a4a4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43424/0 (socket says 172.21.15.2:43424) 2022-03-21T13:59:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.589+0000 7f62a4a4c700 1 -- 172.21.15.2:0/3437633955 learned_addr learned my addr 172.21.15.2:0/3437633955 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.589+0000 7f62a4a4c700 1 -- 172.21.15.2:0/3437633955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294009370 con 0x7f62a0070110 2022-03-21T13:59:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.589+0000 7f62a4a4c700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6294004530 tx=0x7f62940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.590+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6294011a60 con 0x7f62a0070110 2022-03-21T13:59:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.590+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f6294011bc0 con 0x7f62a0070110 2022-03-21T13:59:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.590+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a0112aa0 con 0x7f62a0070110 2022-03-21T13:59:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.590+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6294019400 con 0x7f62a0070110 2022-03-21T13:59:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.590+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a0113740 con 0x7f62a0070110 2022-03-21T13:59:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.591+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7f6294017020 con 0x7f62a0070110 2022-03-21T13:59:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.591+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a005e600 con 0x7f62a0070110 2022-03-21T13:59:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.591+0000 7f629d7fa700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 0x7f6284037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.591+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6294047a30 con 0x7f62a0070110 2022-03-21T13:59:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.592+0000 7f629ffff700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 0x7f6284037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.592+0000 7f629ffff700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 0x7f6284037080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f628c009780 tx=0x7f628c006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.597+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f629401ab30 con 0x7f62a0070110 2022-03-21T13:59:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:57 smithi002 ceph-mon[17167]: mgrmap e8: x(active, since 2s) 2022-03-21T13:59:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:57 smithi002 ceph-mon[17167]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T13:59:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:57 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:57 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T13:59:57.616 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:57 smithi002 conmon[17580]: [21/Mar/2022:13:59:57] ENGINE Bus STARTING 2022-03-21T13:59:57.617 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:57 smithi002 conmon[17580]: [21/Mar/2022:13:59:57] ENGINE Serving on https://172.21.15.2:7150 2022-03-21T13:59:57.617 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:57 smithi002 conmon[17580]: [21/Mar/2022:13:59:57] ENGINE Bus STARTED 2022-03-21T13:59:57.787 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.786+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f62a00fd010 con 0x7f6284034bd0 2022-03-21T13:59:57.788 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.787+0000 7f629d7fa700 1 -- 172.21.15.2:0/3437633955 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f62a00fd010 con 0x7f6284034bd0 2022-03-21T13:59:57.788 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: value unchanged 2022-03-21T13:59:57.791 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 msgr2=0x7f6284037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:57.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 0x7f6284037080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f628c009780 tx=0x7f628c006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 msgr2=0x7f62a0112360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:57.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6294004530 tx=0x7f62940090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 shutdown_connections 2022-03-21T13:59:57.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f6284034bd0 0x7f6284037080 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.793 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 --2- 172.21.15.2:0/3437633955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62a0070110 0x7f62a0112360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:57.793 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.789+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 >> 172.21.15.2:0/3437633955 conn(0x7f62a00fbc90 msgr2=0x7f62a00fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:57.793 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.790+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 shutdown_connections 2022-03-21T13:59:57.793 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:57.790+0000 7f62a6cb0700 1 -- 172.21.15.2:0/3437633955 wait complete. 2022-03-21T13:59:58.164 INFO:teuthology.orchestra.run.smithi002.stderr:Generating ssh key... 2022-03-21T13:59:58.282 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T13:59:58.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: [21/Mar/2022:13:59:57] ENGINE Bus STARTING 2022-03-21T13:59:58.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: [21/Mar/2022:13:59:57] ENGINE Serving on https://172.21.15.2:7150 2022-03-21T13:59:58.491 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: [21/Mar/2022:13:59:57] ENGINE Bus STARTED 2022-03-21T13:59:58.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:58.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T13:59:58.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:58 smithi002 ceph-mon[17167]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T13:59:58.759 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.757+0000 7f8ba58ba700 1 Processor -- start 2022-03-21T13:59:58.760 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.757+0000 7f8ba58ba700 1 -- start start 2022-03-21T13:59:58.760 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8ba58ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:58.761 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8ba58ba700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0076760 con 0x7f8ba0077dc0 2022-03-21T13:59:58.761 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8b9effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:58.761 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8b9effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43426/0 (socket says 172.21.15.2:43426) 2022-03-21T13:59:58.761 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8b9effd700 1 -- 172.21.15.2:0/3256900966 learned_addr learned my addr 172.21.15.2:0/3256900966 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:58.762 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.758+0000 7f8b9effd700 1 -- 172.21.15.2:0/3256900966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba00768a0 con 0x7f8ba0077dc0 2022-03-21T13:59:58.762 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.759+0000 7f8b9effd700 1 --2- 172.21.15.2:0/3256900966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009730 tx=0x7f8b900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccc1c7e8598cd82 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:58.765 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.759+0000 7f8b9dffb700 1 -- 172.21.15.2:0/3256900966 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b90004030 con 0x7f8ba0077dc0 2022-03-21T13:59:58.766 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.759+0000 7f8b9dffb700 1 -- 172.21.15.2:0/3256900966 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8b90004190 con 0x7f8ba0077dc0 2022-03-21T13:59:58.766 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.759+0000 7f8b9dffb700 1 -- 172.21.15.2:0/3256900966 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b90003750 con 0x7f8ba0077dc0 2022-03-21T13:59:58.766 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 -- 172.21.15.2:0/3256900966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 msgr2=0x7f8ba0076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:58.766 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/3256900966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009730 tx=0x7f8b900090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:58.767 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 -- 172.21.15.2:0/3256900966 shutdown_connections 2022-03-21T13:59:58.767 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/3256900966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0076220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:58.767 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 -- 172.21.15.2:0/3256900966 >> 172.21.15.2:0/3256900966 conn(0x7f8ba00fccd0 msgr2=0x7f8ba00ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:58.767 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 -- 172.21.15.2:0/3256900966 shutdown_connections 2022-03-21T13:59:58.768 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.760+0000 7f8ba58ba700 1 -- 172.21.15.2:0/3256900966 wait complete. 2022-03-21T13:59:58.768 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.761+0000 7f8ba58ba700 1 Processor -- start 2022-03-21T13:59:58.768 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.761+0000 7f8ba58ba700 1 -- start start 2022-03-21T13:59:58.768 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.762+0000 7f8ba58ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.762+0000 7f8ba58ba700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0112950 con 0x7f8ba0077dc0 2022-03-21T13:59:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.762+0000 7f8b9effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.762+0000 7f8b9effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43428/0 (socket says 172.21.15.2:43428) 2022-03-21T13:59:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.762+0000 7f8b9effd700 1 -- 172.21.15.2:0/1036177411 learned_addr learned my addr 172.21.15.2:0/1036177411 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T13:59:58.770 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8b9effd700 1 -- 172.21.15.2:0/1036177411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b90009370 con 0x7f8ba0077dc0 2022-03-21T13:59:58.770 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8b9effd700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b90004530 tx=0x7f8b900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:58.770 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b90011a60 con 0x7f8ba0077dc0 2022-03-21T13:59:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8b90011bc0 con 0x7f8ba0077dc0 2022-03-21T13:59:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b90019400 con 0x7f8ba0077dc0 2022-03-21T13:59:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba0112b50 con 0x7f8ba0077dc0 2022-03-21T13:59:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.763+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba01137f0 con 0x7f8ba0077dc0 2022-03-21T13:59:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.764+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7f8b90017020 con 0x7f8ba0077dc0 2022-03-21T13:59:58.772 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.764+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba0112fe0 con 0x7f8ba0077dc0 2022-03-21T13:59:58.773 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.765+0000 7f8b877fe700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 0x7f8b8c037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T13:59:58.773 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.765+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8b90047b10 con 0x7f8ba0077dc0 2022-03-21T13:59:58.773 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.765+0000 7f8b9e7fc700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 0x7f8b8c037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T13:59:58.773 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.766+0000 7f8b9e7fc700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 0x7f8b8c037080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009780 tx=0x7f8b94006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T13:59:58.774 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.770+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f8b9001ab30 con 0x7f8ba0077dc0 2022-03-21T13:59:58.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:58.961+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ba00fe100 con 0x7f8b8c034bd0 2022-03-21T13:59:59.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.433+0000 7f8b877fe700 1 -- 172.21.15.2:0/1036177411 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8ba00fe100 con 0x7f8b8c034bd0 2022-03-21T13:59:59.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 msgr2=0x7f8b8c037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:59.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 0x7f8b8c037080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009780 tx=0x7f8b94006cc0 comp rx=0 tx=0).stop 2022-03-21T13:59:59.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 msgr2=0x7f8ba0112410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T13:59:59.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b90004530 tx=0x7f8b900090f0 comp rx=0 tx=0).stop 2022-03-21T13:59:59.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 shutdown_connections 2022-03-21T13:59:59.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f8b8c034bd0 0x7f8b8c037080 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:59.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 --2- 172.21.15.2:0/1036177411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ba0077dc0 0x7f8ba0112410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T13:59:59.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 >> 172.21.15.2:0/1036177411 conn(0x7f8ba00fccd0 msgr2=0x7f8ba00fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T13:59:59.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 shutdown_connections 2022-03-21T13:59:59.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T13:59:59.436+0000 7f8ba58ba700 1 -- 172.21.15.2:0/1036177411 wait complete. 2022-03-21T13:59:59.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:59 smithi002 ceph-mon[17167]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T13:59:59.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:59 smithi002 ceph-mon[17167]: Generating ssh key... 2022-03-21T13:59:59.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:59 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:59.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 13:59:59 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T13:59:59.834 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: Generating public/private rsa key pair. 2022-03-21T13:59:59.835 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: Your identification has been saved in /tmp/tmp1qyvi182/key. 2022-03-21T13:59:59.835 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: Your public key has been saved in /tmp/tmp1qyvi182/key.pub. 2022-03-21T13:59:59.835 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: The key fingerprint is: 2022-03-21T13:59:59.835 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: SHA256:tFN/TiPuPRXVNaPFjue6xfAByjeGlPIWedYl+TGbW5g ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T13:59:59.835 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: The key's randomart image is: 2022-03-21T13:59:59.836 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: +---[RSA 3072]----+ 2022-03-21T13:59:59.836 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | o==| 2022-03-21T13:59:59.836 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | o ==*| 2022-03-21T13:59:59.836 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | ...= =o=*| 2022-03-21T13:59:59.837 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | . o=.*.E*o| 2022-03-21T13:59:59.837 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | S *o=*.+| 2022-03-21T13:59:59.837 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | ...o==+o| 2022-03-21T13:59:59.837 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | . o= | 2022-03-21T13:59:59.837 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | . oo | 2022-03-21T13:59:59.838 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: | ..o. | 2022-03-21T13:59:59.838 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 13:59:59 smithi002 conmon[17580]: +----[SHA256]-----+ 2022-03-21T13:59:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:00.464 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.461+0000 7fe131ee9700 1 Processor -- start 2022-03-21T14:00:00.465 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.461+0000 7fe131ee9700 1 -- start start 2022-03-21T14:00:00.465 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.461+0000 7fe131ee9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:00.465 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.461+0000 7fe131ee9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c1040e0 con 0x7fe12c103790 2022-03-21T14:00:00.466 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.462+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:00.466 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.462+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43430/0 (socket says 172.21.15.2:43430) 2022-03-21T14:00:00.466 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.462+0000 7fe12b7fe700 1 -- 172.21.15.2:0/4070530444 learned_addr learned my addr 172.21.15.2:0/4070530444 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:00.466 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.463+0000 7fe12b7fe700 1 -- 172.21.15.2:0/4070530444 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c104220 con 0x7fe12c103790 2022-03-21T14:00:00.467 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.463+0000 7fe12b7fe700 1 --2- 172.21.15.2:0/4070530444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009730 tx=0x7fe11c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89988d52d6a3ce2 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:00.467 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.463+0000 7fe12a7fc700 1 -- 172.21.15.2:0/4070530444 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe11c004030 con 0x7fe12c103790 2022-03-21T14:00:00.467 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.463+0000 7fe12a7fc700 1 -- 172.21.15.2:0/4070530444 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe11c004190 con 0x7fe12c103790 2022-03-21T14:00:00.467 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.463+0000 7fe12a7fc700 1 -- 172.21.15.2:0/4070530444 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe11c003750 con 0x7fe12c103790 2022-03-21T14:00:00.471 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 -- 172.21.15.2:0/4070530444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 msgr2=0x7fe12c103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:00.471 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 --2- 172.21.15.2:0/4070530444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009730 tx=0x7fe11c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.472 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 -- 172.21.15.2:0/4070530444 shutdown_connections 2022-03-21T14:00:00.472 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 --2- 172.21.15.2:0/4070530444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c103ba0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.473 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 -- 172.21.15.2:0/4070530444 >> 172.21.15.2:0/4070530444 conn(0x7fe12c076dd0 msgr2=0x7fe12c0771d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:00.473 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 -- 172.21.15.2:0/4070530444 shutdown_connections 2022-03-21T14:00:00.473 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.464+0000 7fe131ee9700 1 -- 172.21.15.2:0/4070530444 wait complete. 2022-03-21T14:00:00.473 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.465+0000 7fe131ee9700 1 Processor -- start 2022-03-21T14:00:00.474 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe131ee9700 1 -- start start 2022-03-21T14:00:00.474 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe131ee9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:00.474 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe131ee9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c1128a0 con 0x7fe12c103790 2022-03-21T14:00:00.474 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:00.475 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43432/0 (socket says 172.21.15.2:43432) 2022-03-21T14:00:00.475 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.466+0000 7fe12b7fe700 1 -- 172.21.15.2:0/2907852637 learned_addr learned my addr 172.21.15.2:0/2907852637 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:00.475 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe12b7fe700 1 -- 172.21.15.2:0/2907852637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c009370 con 0x7fe12c103790 2022-03-21T14:00:00.475 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe12b7fe700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe11c004540 tx=0x7fe11c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:00.475 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe11c011ae0 con 0x7fe12c103790 2022-03-21T14:00:00.476 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c112aa0 con 0x7fe12c103790 2022-03-21T14:00:00.476 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe11c011c40 con 0x7fe12c103790 2022-03-21T14:00:00.476 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.467+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe11c024570 con 0x7fe12c103790 2022-03-21T14:00:00.476 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.468+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c113740 con 0x7fe12c103790 2022-03-21T14:00:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.468+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fe11c02b060 con 0x7fe12c103790 2022-03-21T14:00:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.469+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c05e600 con 0x7fe12c103790 2022-03-21T14:00:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.469+0000 7fe113fff700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 0x7fe118037030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.469+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe11c047d20 con 0x7fe12c103790 2022-03-21T14:00:00.478 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.469+0000 7fe12affd700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 0x7fe118037030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:00.478 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.470+0000 7fe12affd700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 0x7fe118037030 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe120009780 tx=0x7fe120006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:00.478 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.474+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe11c018b00 con 0x7fe12c103790 2022-03-21T14:00:00.665 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.664+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe12c101d50 con 0x7fe118034b80 2022-03-21T14:00:00.666 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.665+0000 7fe113fff700 1 -- 172.21.15.2:0/2907852637 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe12c101d50 con 0x7fe118034b80 2022-03-21T14:00:00.666 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:00.669 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 msgr2=0x7fe118037030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:00.669 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 0x7fe118037030 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe120009780 tx=0x7fe120006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.669 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 msgr2=0x7fe12c112360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:00.670 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe11c004540 tx=0x7fe11c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.670 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 shutdown_connections 2022-03-21T14:00:00.670 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe118034b80 0x7fe118037030 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.670 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 --2- 172.21.15.2:0/2907852637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe12c103790 0x7fe12c112360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 >> 172.21.15.2:0/2907852637 conn(0x7fe12c076dd0 msgr2=0x7fe12c101640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 shutdown_connections 2022-03-21T14:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:00.667+0000 7fe131ee9700 1 -- 172.21.15.2:0/2907852637 wait complete. 2022-03-21T14:00:01.058 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-03-21T14:00:01.058 INFO:teuthology.orchestra.run.smithi002.stderr:Adding key to root@localhost authorized_keys... 2022-03-21T14:00:01.060 INFO:teuthology.orchestra.run.smithi002.stderr:Adding host smithi002... 2022-03-21T14:00:01.176 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:01.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:01 smithi002 ceph-mon[17167]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:01.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.668+0000 7fe1f4721700 1 Processor -- start 2022-03-21T14:00:01.671 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.668+0000 7fe1f4721700 1 -- start start 2022-03-21T14:00:01.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.668+0000 7fe1f4721700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:01.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.668+0000 7fe1f4721700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec076fa0 con 0x7fe1ec076b90 2022-03-21T14:00:01.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.669+0000 7fe1f24bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:01.672 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.669+0000 7fe1f24bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43434/0 (socket says 172.21.15.2:43434) 2022-03-21T14:00:01.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.669+0000 7fe1f24bd700 1 -- 172.21.15.2:0/1371761210 learned_addr learned my addr 172.21.15.2:0/1371761210 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:01.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.669+0000 7fe1f24bd700 1 -- 172.21.15.2:0/1371761210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec075670 con 0x7fe1ec076b90 2022-03-21T14:00:01.673 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f24bd700 1 --2- 172.21.15.2:0/1371761210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009730 tx=0x7fe1e80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec904e883a23c30 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:01.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f14bb700 1 -- 172.21.15.2:0/1371761210 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1e8004030 con 0x7fe1ec076b90 2022-03-21T14:00:01.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f14bb700 1 -- 172.21.15.2:0/1371761210 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe1e8004190 con 0x7fe1ec076b90 2022-03-21T14:00:01.674 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f14bb700 1 -- 172.21.15.2:0/1371761210 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1e8003750 con 0x7fe1ec076b90 2022-03-21T14:00:01.678 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f4721700 1 -- 172.21.15.2:0/1371761210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 msgr2=0x7fe1ec075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:01.678 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.670+0000 7fe1f4721700 1 --2- 172.21.15.2:0/1371761210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009730 tx=0x7fe1e80090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:01.678 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.671+0000 7fe1f4721700 1 -- 172.21.15.2:0/1371761210 shutdown_connections 2022-03-21T14:00:01.679 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.671+0000 7fe1f4721700 1 --2- 172.21.15.2:0/1371761210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec075130 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:01.679 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.671+0000 7fe1f4721700 1 -- 172.21.15.2:0/1371761210 >> 172.21.15.2:0/1371761210 conn(0x7fe1ec0fcda0 msgr2=0x7fe1ec0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:01.679 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.671+0000 7fe1f4721700 1 -- 172.21.15.2:0/1371761210 shutdown_connections 2022-03-21T14:00:01.679 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.671+0000 7fe1f4721700 1 -- 172.21.15.2:0/1371761210 wait complete. 2022-03-21T14:00:01.680 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.672+0000 7fe1f4721700 1 Processor -- start 2022-03-21T14:00:01.680 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.672+0000 7fe1f4721700 1 -- start start 2022-03-21T14:00:01.680 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.672+0000 7fe1f4721700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:01.680 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.672+0000 7fe1f4721700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1128e0 con 0x7fe1ec076b90 2022-03-21T14:00:01.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.673+0000 7fe1f24bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:01.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.673+0000 7fe1f24bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43436/0 (socket says 172.21.15.2:43436) 2022-03-21T14:00:01.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.673+0000 7fe1f24bd700 1 -- 172.21.15.2:0/3318910204 learned_addr learned my addr 172.21.15.2:0/3318910204 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:01.681 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.673+0000 7fe1f24bd700 1 -- 172.21.15.2:0/3318910204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e8009370 con 0x7fe1ec076b90 2022-03-21T14:00:01.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.673+0000 7fe1f24bd700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8004530 tx=0x7fe1e80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:01.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.674+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1e8011a60 con 0x7fe1ec076b90 2022-03-21T14:00:01.682 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.674+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fe1e8011bc0 con 0x7fe1ec076b90 2022-03-21T14:00:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.674+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec112ae0 con 0x7fe1ec076b90 2022-03-21T14:00:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.674+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1e8019400 con 0x7fe1ec076b90 2022-03-21T14:00:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.674+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec113780 con 0x7fe1ec076b90 2022-03-21T14:00:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.675+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fe1e8017020 con 0x7fe1ec076b90 2022-03-21T14:00:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.675+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec05e600 con 0x7fe1ec076b90 2022-03-21T14:00:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.676+0000 7fe1daffd700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 0x7fe1d4037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.676+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe1e8047b00 con 0x7fe1ec076b90 2022-03-21T14:00:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.676+0000 7fe1f1cbc700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 0x7fe1d4037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:01.685 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.677+0000 7fe1f1cbc700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 0x7fe1d4037080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009780 tx=0x7fe1dc006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:01.685 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.681+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe1e801ab30 con 0x7fe1ec076b90 2022-03-21T14:00:01.875 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:01.874+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1ec0fe250 con 0x7fe1d4034bd0 2022-03-21T14:00:02.445 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:02 smithi002 ceph-mon[17167]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:03.442 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:03 smithi002 ceph-mon[17167]: Deploying cephadm binary to smithi002 2022-03-21T14:00:03.452 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Added host 'smithi002' with addr '172.21.15.2' 2022-03-21T14:00:03.453 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.450+0000 7fe1daffd700 1 -- 172.21.15.2:0/3318910204 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe1ec0fe250 con 0x7fe1d4034bd0 2022-03-21T14:00:03.455 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.453+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 msgr2=0x7fe1d4037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:03.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.453+0000 7fe1f4721700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 0x7fe1d4037080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009780 tx=0x7fe1dc006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:03.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 msgr2=0x7fe1ec1123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:03.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8004530 tx=0x7fe1e80090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:03.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 shutdown_connections 2022-03-21T14:00:03.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe1d4034bd0 0x7fe1d4037080 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:03.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 --2- 172.21.15.2:0/3318910204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1ec076b90 0x7fe1ec1123a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:03.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 >> 172.21.15.2:0/3318910204 conn(0x7fe1ec0fcda0 msgr2=0x7fe1ec0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:03.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 shutdown_connections 2022-03-21T14:00:03.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:03.454+0000 7fe1f4721700 1 -- 172.21.15.2:0/3318910204 wait complete. 2022-03-21T14:00:03.953 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mon service... 2022-03-21T14:00:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:04.556 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.551+0000 7facd1aa8700 1 Processor -- start 2022-03-21T14:00:04.556 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.551+0000 7facd1aa8700 1 -- start start 2022-03-21T14:00:04.557 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.551+0000 7facd1aa8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:04.557 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.551+0000 7facd1aa8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc076fa0 con 0x7faccc076b90 2022-03-21T14:00:04.557 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.552+0000 7faccb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:04.557 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.552+0000 7faccb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43438/0 (socket says 172.21.15.2:43438) 2022-03-21T14:00:04.558 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.552+0000 7faccb7fe700 1 -- 172.21.15.2:0/1210825973 learned_addr learned my addr 172.21.15.2:0/1210825973 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:04.558 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.552+0000 7faccb7fe700 1 -- 172.21.15.2:0/1210825973 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc075670 con 0x7faccc076b90 2022-03-21T14:00:04.558 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.552+0000 7faccb7fe700 1 --2- 172.21.15.2:0/1210825973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7facbc009730 tx=0x7facbc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc4e87511dfc51e server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:04.558 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facca7fc700 1 -- 172.21.15.2:0/1210825973 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facbc004030 con 0x7faccc076b90 2022-03-21T14:00:04.559 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facca7fc700 1 -- 172.21.15.2:0/1210825973 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7facbc004190 con 0x7faccc076b90 2022-03-21T14:00:04.559 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facca7fc700 1 -- 172.21.15.2:0/1210825973 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facbc003750 con 0x7faccc076b90 2022-03-21T14:00:04.559 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facd1aa8700 1 -- 172.21.15.2:0/1210825973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 msgr2=0x7faccc075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facd1aa8700 1 --2- 172.21.15.2:0/1210825973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7facbc009730 tx=0x7facbc0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.553+0000 7facd1aa8700 1 -- 172.21.15.2:0/1210825973 shutdown_connections 2022-03-21T14:00:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.554+0000 7facd1aa8700 1 --2- 172.21.15.2:0/1210825973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc075130 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.554+0000 7facd1aa8700 1 -- 172.21.15.2:0/1210825973 >> 172.21.15.2:0/1210825973 conn(0x7faccc0fcda0 msgr2=0x7faccc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.554+0000 7facd1aa8700 1 -- 172.21.15.2:0/1210825973 shutdown_connections 2022-03-21T14:00:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.554+0000 7facd1aa8700 1 -- 172.21.15.2:0/1210825973 wait complete. 2022-03-21T14:00:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7facd1aa8700 1 Processor -- start 2022-03-21T14:00:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7facd1aa8700 1 -- start start 2022-03-21T14:00:04.562 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7facd1aa8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7facd1aa8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc1128e0 con 0x7faccc076b90 2022-03-21T14:00:04.565 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7faccb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:04.565 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7faccb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43440/0 (socket says 172.21.15.2:43440) 2022-03-21T14:00:04.565 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.555+0000 7faccb7fe700 1 -- 172.21.15.2:0/4117043284 learned_addr learned my addr 172.21.15.2:0/4117043284 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:04.565 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.556+0000 7faccb7fe700 1 -- 172.21.15.2:0/4117043284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009370 con 0x7faccc076b90 2022-03-21T14:00:04.566 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.556+0000 7faccb7fe700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7facbc004530 tx=0x7facbc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:04.566 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.556+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facbc011a60 con 0x7faccc076b90 2022-03-21T14:00:04.566 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.556+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7facbc011bc0 con 0x7faccc076b90 2022-03-21T14:00:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.556+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc112ae0 con 0x7faccc076b90 2022-03-21T14:00:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.557+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facbc019400 con 0x7faccc076b90 2022-03-21T14:00:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.557+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc113780 con 0x7faccc076b90 2022-03-21T14:00:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.558+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7facbc017020 con 0x7faccc076b90 2022-03-21T14:00:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.558+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc05e600 con 0x7faccc076b90 2022-03-21T14:00:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.558+0000 7facb3fff700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 0x7facb8037030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.558+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7facbc047b30 con 0x7faccc076b90 2022-03-21T14:00:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.558+0000 7faccaffd700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 0x7facb8037030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.559+0000 7faccaffd700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 0x7facb8037030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7facc0009780 tx=0x7facc0006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.563+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7facbc01ab30 con 0x7faccc076b90 2022-03-21T14:00:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:04 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:04 smithi002 ceph-mon[17167]: Added host smithi002 2022-03-21T14:00:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:04 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:04.785 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.784+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7faccc0fe250 con 0x7facb8034b80 2022-03-21T14:00:04.790 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.789+0000 7facb3fff700 1 -- 172.21.15.2:0/4117043284 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faccc0fe250 con 0x7facb8034b80 2022-03-21T14:00:04.791 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mon update... 2022-03-21T14:00:04.794 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 msgr2=0x7facb8037030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:04.794 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 0x7facb8037030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7facc0009780 tx=0x7facc0006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.794 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 msgr2=0x7faccc1123a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:04.795 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7facbc004530 tx=0x7facbc0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.795 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 shutdown_connections 2022-03-21T14:00:04.795 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7facb8034b80 0x7facb8037030 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.795 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 --2- 172.21.15.2:0/4117043284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faccc076b90 0x7faccc1123a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:04.796 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 >> 172.21.15.2:0/4117043284 conn(0x7faccc0fcda0 msgr2=0x7faccc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:04.796 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 shutdown_connections 2022-03-21T14:00:04.796 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:04.792+0000 7facd1aa8700 1 -- 172.21.15.2:0/4117043284 wait complete. 2022-03-21T14:00:05.246 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mgr service... 2022-03-21T14:00:05.380 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:05.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:05 smithi002 ceph-mon[17167]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:05.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:05 smithi002 ceph-mon[17167]: Saving service mon spec with placement count:5 2022-03-21T14:00:05.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:05 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:06.002 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.000+0000 7fd9b0e12700 1 Processor -- start 2022-03-21T14:00:06.002 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.000+0000 7fd9b0e12700 1 -- start start 2022-03-21T14:00:06.002 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.001+0000 7fd9b0e12700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:06.003 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.001+0000 7fd9b0e12700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac101ee0 con 0x7fd9ac101590 2022-03-21T14:00:06.004 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.001+0000 7fd9aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:06.004 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.001+0000 7fd9aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43442/0 (socket says 172.21.15.2:43442) 2022-03-21T14:00:06.005 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.001+0000 7fd9aa59c700 1 -- 172.21.15.2:0/2680418410 learned_addr learned my addr 172.21.15.2:0/2680418410 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:06.005 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.002+0000 7fd9aa59c700 1 -- 172.21.15.2:0/2680418410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac102020 con 0x7fd9ac101590 2022-03-21T14:00:06.005 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.002+0000 7fd9aa59c700 1 --2- 172.21.15.2:0/2680418410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009730 tx=0x7fd99c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6b8e7ce102de97 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:06.005 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.003+0000 7fd9a959a700 1 -- 172.21.15.2:0/2680418410 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd99c004030 con 0x7fd9ac101590 2022-03-21T14:00:06.005 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.003+0000 7fd9a959a700 1 -- 172.21.15.2:0/2680418410 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd99c004190 con 0x7fd9ac101590 2022-03-21T14:00:06.006 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.003+0000 7fd9a959a700 1 -- 172.21.15.2:0/2680418410 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd99c003750 con 0x7fd9ac101590 2022-03-21T14:00:06.008 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.003+0000 7fd9b0e12700 1 -- 172.21.15.2:0/2680418410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 msgr2=0x7fd9ac1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:06.009 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.003+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/2680418410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009730 tx=0x7fd99c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.009 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.004+0000 7fd9b0e12700 1 -- 172.21.15.2:0/2680418410 shutdown_connections 2022-03-21T14:00:06.009 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.004+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/2680418410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac1019a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.009 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.004+0000 7fd9b0e12700 1 -- 172.21.15.2:0/2680418410 >> 172.21.15.2:0/2680418410 conn(0x7fd9ac0fcda0 msgr2=0x7fd9ac0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:06.010 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.004+0000 7fd9b0e12700 1 -- 172.21.15.2:0/2680418410 shutdown_connections 2022-03-21T14:00:06.010 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.004+0000 7fd9b0e12700 1 -- 172.21.15.2:0/2680418410 wait complete. 2022-03-21T14:00:06.010 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.005+0000 7fd9b0e12700 1 Processor -- start 2022-03-21T14:00:06.010 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.005+0000 7fd9b0e12700 1 -- start start 2022-03-21T14:00:06.010 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9b0e12700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:06.011 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9b0e12700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac1106b0 con 0x7fd9ac101590 2022-03-21T14:00:06.011 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:06.011 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43444/0 (socket says 172.21.15.2:43444) 2022-03-21T14:00:06.011 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9aa59c700 1 -- 172.21.15.2:0/1609529227 learned_addr learned my addr 172.21.15.2:0/1609529227 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:06.012 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.006+0000 7fd9aa59c700 1 -- 172.21.15.2:0/1609529227 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c009370 con 0x7fd9ac101590 2022-03-21T14:00:06.012 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd9aa59c700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd99c004530 tx=0x7fd99c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:06.012 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd99c011a10 con 0x7fd9ac101590 2022-03-21T14:00:06.012 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd99c011b70 con 0x7fd9ac101590 2022-03-21T14:00:06.013 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac1108b0 con 0x7fd9ac101590 2022-03-21T14:00:06.013 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd99c024490 con 0x7fd9ac101590 2022-03-21T14:00:06.013 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.007+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac111550 con 0x7fd9ac101590 2022-03-21T14:00:06.014 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.008+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fd99c02b060 con 0x7fd9ac101590 2022-03-21T14:00:06.014 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.008+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac05e600 con 0x7fd9ac101590 2022-03-21T14:00:06.015 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.009+0000 7fd99affd700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 0x7fd990037030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:06.015 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.009+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd99c047ce0 con 0x7fd9ac101590 2022-03-21T14:00:06.015 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.009+0000 7fd9a9d9b700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 0x7fd990037030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:06.016 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.010+0000 7fd9a9d9b700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 0x7fd990037030 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009780 tx=0x7fd9a0006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:06.016 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.014+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd99c02c340 con 0x7fd9ac101590 2022-03-21T14:00:06.143 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.141+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd99c00fee0 con 0x7fd9ac101590 2022-03-21T14:00:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.227+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd9ac0fe1a0 con 0x7fd990034b80 2022-03-21T14:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.233+0000 7fd99affd700 1 -- 172.21.15.2:0/1609529227 <== mgr.14118 v2:172.21.15.2:6800/1086540904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd9ac0fe1a0 con 0x7fd990034b80 2022-03-21T14:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mgr update... 2022-03-21T14:00:06.237 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 msgr2=0x7fd990037030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:06.238 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 0x7fd990037030 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009780 tx=0x7fd9a0006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.238 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 msgr2=0x7fd9ac110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:06.238 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd99c004530 tx=0x7fd99c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.238 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 shutdown_connections 2022-03-21T14:00:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fd990034b80 0x7fd990037030 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 --2- 172.21.15.2:0/1609529227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9ac101590 0x7fd9ac110170 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.235+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 >> 172.21.15.2:0/1609529227 conn(0x7fd9ac0fcda0 msgr2=0x7fd9ac0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:06.239 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.236+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 shutdown_connections 2022-03-21T14:00:06.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:06.236+0000 7fd9b0e12700 1 -- 172.21.15.2:0/1609529227 wait complete. 2022-03-21T14:00:06.777 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:07.292 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.288+0000 7f90392ee700 1 Processor -- start 2022-03-21T14:00:07.293 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.288+0000 7f90392ee700 1 -- start start 2022-03-21T14:00:07.293 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.289+0000 7f90392ee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:07.293 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.289+0000 7f90392ee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034101f00 con 0x7f90341015b0 2022-03-21T14:00:07.294 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.289+0000 7f9032ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:07.294 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.289+0000 7f9032ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43446/0 (socket says 172.21.15.2:43446) 2022-03-21T14:00:07.294 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.289+0000 7f9032ffd700 1 -- 172.21.15.2:0/3191420268 learned_addr learned my addr 172.21.15.2:0/3191420268 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:07.295 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.290+0000 7f9032ffd700 1 -- 172.21.15.2:0/3191420268 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034102040 con 0x7f90341015b0 2022-03-21T14:00:07.295 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.290+0000 7f9032ffd700 1 --2- 172.21.15.2:0/3191420268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9024009730 tx=0x7f90240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504197fe43be83fd server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:07.295 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.290+0000 7f9031ffb700 1 -- 172.21.15.2:0/3191420268 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024004030 con 0x7f90341015b0 2022-03-21T14:00:07.296 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.290+0000 7f9031ffb700 1 -- 172.21.15.2:0/3191420268 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9024004190 con 0x7f90341015b0 2022-03-21T14:00:07.296 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.290+0000 7f9031ffb700 1 -- 172.21.15.2:0/3191420268 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90240037a0 con 0x7f90341015b0 2022-03-21T14:00:07.296 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 -- 172.21.15.2:0/3191420268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 msgr2=0x7f90341019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:07.296 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 --2- 172.21.15.2:0/3191420268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9024009730 tx=0x7f90240090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.297 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 -- 172.21.15.2:0/3191420268 shutdown_connections 2022-03-21T14:00:07.297 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 --2- 172.21.15.2:0/3191420268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341019c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.297 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 -- 172.21.15.2:0/3191420268 >> 172.21.15.2:0/3191420268 conn(0x7f90340fcda0 msgr2=0x7f90340ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:07.298 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 -- 172.21.15.2:0/3191420268 shutdown_connections 2022-03-21T14:00:07.298 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.291+0000 7f90392ee700 1 -- 172.21.15.2:0/3191420268 wait complete. 2022-03-21T14:00:07.301 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.292+0000 7f90392ee700 1 Processor -- start 2022-03-21T14:00:07.301 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.292+0000 7f90392ee700 1 -- start start 2022-03-21T14:00:07.301 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f90392ee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f90392ee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034110710 con 0x7f90341015b0 2022-03-21T14:00:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f9032ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f9032ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43448/0 (socket says 172.21.15.2:43448) 2022-03-21T14:00:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f9032ffd700 1 -- 172.21.15.2:0/704948790 learned_addr learned my addr 172.21.15.2:0/704948790 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:07.303 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.293+0000 7f9032ffd700 1 -- 172.21.15.2:0/704948790 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024009370 con 0x7f90341015b0 2022-03-21T14:00:07.303 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f9032ffd700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9024004150 tx=0x7f90240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:07.303 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90240119c0 con 0x7f90341015b0 2022-03-21T14:00:07.303 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9024011b20 con 0x7f90341015b0 2022-03-21T14:00:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9034110910 con 0x7f90341015b0 2022-03-21T14:00:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024022560 con 0x7f90341015b0 2022-03-21T14:00:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.294+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90341115b0 con 0x7f90341015b0 2022-03-21T14:00:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.295+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7f9024017020 con 0x7f90341015b0 2022-03-21T14:00:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.295+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903405e600 con 0x7f90341015b0 2022-03-21T14:00:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.296+0000 7f901b7fe700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 0x7f9020037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.296+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9024048080 con 0x7f90341015b0 2022-03-21T14:00:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.296+0000 7f90327fc700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 0x7f9020037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.297+0000 7f90327fc700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 0x7f9020037080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9028009780 tx=0x7f9028006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.301+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f902401a9e0 con 0x7f90341015b0 2022-03-21T14:00:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:07 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:07 smithi002 ceph-mon[17167]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:07 smithi002 ceph-mon[17167]: Saving service mgr spec with placement count:2 2022-03-21T14:00:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:07 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:07 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:07.493 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.492+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f903404af20 con 0x7f90341015b0 2022-03-21T14:00:07.496 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.495+0000 7f901b7fe700 1 -- 172.21.15.2:0/704948790 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f902402a800 con 0x7f90341015b0 2022-03-21T14:00:07.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 msgr2=0x7f9020037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:07.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 0x7f9020037080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9028009780 tx=0x7f9028006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.500 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 msgr2=0x7f90341101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:07.501 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9024004150 tx=0x7f90240090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.501 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 shutdown_connections 2022-03-21T14:00:07.501 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f9020034bd0 0x7f9020037080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.501 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 --2- 172.21.15.2:0/704948790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90341015b0 0x7f90341101d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:07.502 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 >> 172.21.15.2:0/704948790 conn(0x7f90340fcda0 msgr2=0x7f90340fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:07.502 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 shutdown_connections 2022-03-21T14:00:07.502 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:07.498+0000 7f90392ee700 1 -- 172.21.15.2:0/704948790 wait complete. 2022-03-21T14:00:07.975 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:08.595 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.593+0000 7eff390f4700 1 Processor -- start 2022-03-21T14:00:08.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.593+0000 7eff390f4700 1 -- start start 2022-03-21T14:00:08.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.594+0000 7eff390f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:08.596 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.594+0000 7eff390f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34076fa0 con 0x7eff34076b90 2022-03-21T14:00:08.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.594+0000 7eff32d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:08.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.594+0000 7eff32d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43450/0 (socket says 172.21.15.2:43450) 2022-03-21T14:00:08.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.594+0000 7eff32d9d700 1 -- 172.21.15.2:0/2071154613 learned_addr learned my addr 172.21.15.2:0/2071154613 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:08.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.595+0000 7eff32d9d700 1 -- 172.21.15.2:0/2071154613 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34075670 con 0x7eff34076b90 2022-03-21T14:00:08.600 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.595+0000 7eff32d9d700 1 --2- 172.21.15.2:0/2071154613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff24009730 tx=0x7eff240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730ea5a9c1039f68 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:08.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.595+0000 7eff31d9b700 1 -- 172.21.15.2:0/2071154613 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff24004030 con 0x7eff34076b90 2022-03-21T14:00:08.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff31d9b700 1 -- 172.21.15.2:0/2071154613 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7eff24004190 con 0x7eff34076b90 2022-03-21T14:00:08.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff31d9b700 1 -- 172.21.15.2:0/2071154613 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff240037a0 con 0x7eff34076b90 2022-03-21T14:00:08.601 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 -- 172.21.15.2:0/2071154613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 msgr2=0x7eff34075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:08.602 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 --2- 172.21.15.2:0/2071154613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff24009730 tx=0x7eff240090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.602 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 -- 172.21.15.2:0/2071154613 shutdown_connections 2022-03-21T14:00:08.602 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 --2- 172.21.15.2:0/2071154613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34075130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.602 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 -- 172.21.15.2:0/2071154613 >> 172.21.15.2:0/2071154613 conn(0x7eff340fcda0 msgr2=0x7eff340ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:08.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 -- 172.21.15.2:0/2071154613 shutdown_connections 2022-03-21T14:00:08.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.596+0000 7eff390f4700 1 -- 172.21.15.2:0/2071154613 wait complete. 2022-03-21T14:00:08.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.597+0000 7eff390f4700 1 Processor -- start 2022-03-21T14:00:08.603 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.597+0000 7eff390f4700 1 -- start start 2022-03-21T14:00:08.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff390f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:08.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff390f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34112940 con 0x7eff34076b90 2022-03-21T14:00:08.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff32d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:08.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff32d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43452/0 (socket says 172.21.15.2:43452) 2022-03-21T14:00:08.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff32d9d700 1 -- 172.21.15.2:0/821105623 learned_addr learned my addr 172.21.15.2:0/821105623 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:08.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.598+0000 7eff32d9d700 1 -- 172.21.15.2:0/821105623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff24009370 con 0x7eff34076b90 2022-03-21T14:00:08.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff32d9d700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:08.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff240118c0 con 0x7eff34076b90 2022-03-21T14:00:08.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7eff24011a20 con 0x7eff34076b90 2022-03-21T14:00:08.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff34112b40 con 0x7eff34076b90 2022-03-21T14:00:08.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff24022540 con 0x7eff34076b90 2022-03-21T14:00:08.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.599+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff341137e0 con 0x7eff34076b90 2022-03-21T14:00:08.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.600+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7eff24017020 con 0x7eff34076b90 2022-03-21T14:00:08.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.601+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3405e600 con 0x7eff34076b90 2022-03-21T14:00:08.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.601+0000 7eff1b7fe700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 0x7eff20037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:08.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.601+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7eff24048010 con 0x7eff34076b90 2022-03-21T14:00:08.609 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.601+0000 7eff3259c700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 0x7eff20037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:08.609 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.602+0000 7eff3259c700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 0x7eff20037080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff28009780 tx=0x7eff28006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:08.609 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.606+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7eff24011b90 con 0x7eff34076b90 2022-03-21T14:00:08.809 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.808+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7eff34107840 con 0x7eff34076b90 2022-03-21T14:00:08.813 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.811+0000 7eff1b7fe700 1 -- 172.21.15.2:0/821105623 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7eff2402a390 con 0x7eff34076b90 2022-03-21T14:00:08.816 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 msgr2=0x7eff20037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:08.816 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 0x7eff20037080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff28009780 tx=0x7eff28006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.817 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 msgr2=0x7eff34112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:08.817 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff240090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.817 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 shutdown_connections 2022-03-21T14:00:08.818 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7eff20034bd0 0x7eff20037080 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.818 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 --2- 172.21.15.2:0/821105623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff34076b90 0x7eff34112400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.814+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 >> 172.21.15.2:0/821105623 conn(0x7eff340fcda0 msgr2=0x7eff340fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.815+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 shutdown_connections 2022-03-21T14:00:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:08.815+0000 7eff390f4700 1 -- 172.21.15.2:0/821105623 wait complete. 2022-03-21T14:00:08.847 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:08 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/704948790' entity='client.admin' 2022-03-21T14:00:09.242 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling the dashboard module... 2022-03-21T14:00:09.362 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:09.847 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.844+0000 7fe518631700 1 Processor -- start 2022-03-21T14:00:09.847 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.844+0000 7fe518631700 1 -- start start 2022-03-21T14:00:09.847 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.845+0000 7fe518631700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.845+0000 7fe518631700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510102050 con 0x7fe5100700d0 2022-03-21T14:00:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.845+0000 7fe5163cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.845+0000 7fe5163cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43456/0 (socket says 172.21.15.2:43456) 2022-03-21T14:00:09.849 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.845+0000 7fe5163cd700 1 -- 172.21.15.2:0/3171636119 learned_addr learned my addr 172.21.15.2:0/3171636119 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:09.849 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.846+0000 7fe5163cd700 1 -- 172.21.15.2:0/3171636119 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510106ba0 con 0x7fe5100700d0 2022-03-21T14:00:09.854 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.846+0000 7fe5163cd700 1 --2- 172.21.15.2:0/3171636119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009730 tx=0x7fe50c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d4f534570add3 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:09.855 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.847+0000 7fe5153cb700 1 -- 172.21.15.2:0/3171636119 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50c004030 con 0x7fe5100700d0 2022-03-21T14:00:09.855 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.847+0000 7fe5153cb700 1 -- 172.21.15.2:0/3171636119 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe50c004190 con 0x7fe5100700d0 2022-03-21T14:00:09.855 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.847+0000 7fe5153cb700 1 -- 172.21.15.2:0/3171636119 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50c0037a0 con 0x7fe5100700d0 2022-03-21T14:00:09.856 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.847+0000 7fe518631700 1 -- 172.21.15.2:0/3171636119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 msgr2=0x7fe510106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:09.856 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.847+0000 7fe518631700 1 --2- 172.21.15.2:0/3171636119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009730 tx=0x7fe50c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:09.856 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.848+0000 7fe518631700 1 -- 172.21.15.2:0/3171636119 shutdown_connections 2022-03-21T14:00:09.857 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.848+0000 7fe518631700 1 --2- 172.21.15.2:0/3171636119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe510106660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:09.857 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.848+0000 7fe518631700 1 -- 172.21.15.2:0/3171636119 >> 172.21.15.2:0/3171636119 conn(0x7fe5100fbc70 msgr2=0x7fe5100fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:09.857 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.848+0000 7fe518631700 1 -- 172.21.15.2:0/3171636119 shutdown_connections 2022-03-21T14:00:09.857 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.848+0000 7fe518631700 1 -- 172.21.15.2:0/3171636119 wait complete. 2022-03-21T14:00:09.858 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.849+0000 7fe518631700 1 Processor -- start 2022-03-21T14:00:09.858 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.849+0000 7fe518631700 1 -- start start 2022-03-21T14:00:09.858 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.849+0000 7fe518631700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:09.858 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.850+0000 7fe518631700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510112910 con 0x7fe5100700d0 2022-03-21T14:00:09.859 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.850+0000 7fe5163cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:09.859 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.850+0000 7fe5163cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43458/0 (socket says 172.21.15.2:43458) 2022-03-21T14:00:09.859 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.850+0000 7fe5163cd700 1 -- 172.21.15.2:0/533262955 learned_addr learned my addr 172.21.15.2:0/533262955 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:09.859 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.850+0000 7fe5163cd700 1 -- 172.21.15.2:0/533262955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c009370 con 0x7fe5100700d0 2022-03-21T14:00:09.860 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe5163cd700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe50c006fd0 tx=0x7fe50c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:09.860 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50c011940 con 0x7fe5100700d0 2022-03-21T14:00:09.860 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe50c011aa0 con 0x7fe5100700d0 2022-03-21T14:00:09.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50c022540 con 0x7fe5100700d0 2022-03-21T14:00:09.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe510112b10 con 0x7fe5100700d0 2022-03-21T14:00:09.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.851+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5101137b0 con 0x7fe5100700d0 2022-03-21T14:00:09.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.852+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 41844+0+0 (secure 0 0 0) 0x7fe50c017020 con 0x7fe5100700d0 2022-03-21T14:00:09.861 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.853+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51005e600 con 0x7fe5100700d0 2022-03-21T14:00:09.862 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.853+0000 7fe4feffd700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 0x7fe4f80370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:09.862 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.853+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe50c048140 con 0x7fe5100700d0 2022-03-21T14:00:09.862 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.853+0000 7fe515bcc700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 0x7fe4f80370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:09.863 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.854+0000 7fe515bcc700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 0x7fe4f80370c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe500009780 tx=0x7fe500006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:09.863 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:09.862+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe50c011c10 con 0x7fe5100700d0 2022-03-21T14:00:10.108 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.107+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe51004af20 con 0x7fe5100700d0 2022-03-21T14:00:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:09 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/821105623' entity='client.admin' 2022-03-21T14:00:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:09 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:10.817 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.815+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 9) v1 ==== 41857+0+0 (secure 0 0 0) 0x7fe50c00f050 con 0x7fe5100700d0 2022-03-21T14:00:10.818 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.817+0000 7fe4feffd700 1 -- 172.21.15.2:0/533262955 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe50c005a60 con 0x7fe5100700d0 2022-03-21T14:00:10.824 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 msgr2=0x7fe4f80370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:10.824 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 0x7fe4f80370c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe500009780 tx=0x7fe500006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 msgr2=0x7fe5101123d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe50c006fd0 tx=0x7fe50c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 shutdown_connections 2022-03-21T14:00:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe4f8034c10 0x7fe4f80370c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 --2- 172.21.15.2:0/533262955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5100700d0 0x7fe5101123d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.821+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 >> 172.21.15.2:0/533262955 conn(0x7fe5100fbc70 msgr2=0x7fe5100fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.822+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 shutdown_connections 2022-03-21T14:00:10.826 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:10.822+0000 7fe518631700 1 -- 172.21.15.2:0/533262955 wait complete. 2022-03-21T14:00:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:10 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/533262955' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-03-21T14:00:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:10 smithi002 ceph-mon[17167]: from='mgr.14118 172.21.15.2:0/3452441692' entity='mgr.x' 2022-03-21T14:00:11.116 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: ignoring --setuser ceph since I am not root 2022-03-21T14:00:11.117 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: ignoring --setgroup ceph since I am not root 2022-03-21T14:00:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:11.526 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: 2022-03-21T14:00:11.181+0000 7f2c12051000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-21T14:00:11.527 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: 2022-03-21T14:00:11.411+0000 7f2c12051000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-21T14:00:11.804 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: 2022-03-21T14:00:11.568+0000 7f2c12051000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-21T14:00:11.804 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:11 smithi002 conmon[17580]: 2022-03-21T14:00:11.802+0000 7f2c12051000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-21T14:00:11.886 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.885+0000 7f96f90f0700 1 Processor -- start 2022-03-21T14:00:11.888 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.885+0000 7f96f90f0700 1 -- start start 2022-03-21T14:00:11.888 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.885+0000 7f96f90f0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:11.889 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.885+0000 7f96f90f0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f4076fa0 con 0x7f96f4076b90 2022-03-21T14:00:11.889 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.886+0000 7f96f2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:11.889 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.886+0000 7f96f2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43464/0 (socket says 172.21.15.2:43464) 2022-03-21T14:00:11.889 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.886+0000 7f96f2d9d700 1 -- 172.21.15.2:0/2243252669 learned_addr learned my addr 172.21.15.2:0/2243252669 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:11.890 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.887+0000 7f96f2d9d700 1 -- 172.21.15.2:0/2243252669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4075670 con 0x7f96f4076b90 2022-03-21T14:00:11.890 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.887+0000 7f96f2d9d700 1 --2- 172.21.15.2:0/2243252669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009730 tx=0x7f96e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c540b4ed2166933 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:11.892 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f1d9b700 1 -- 172.21.15.2:0/2243252669 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e4004030 con 0x7f96f4076b90 2022-03-21T14:00:11.893 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f1d9b700 1 -- 172.21.15.2:0/2243252669 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f96e4004190 con 0x7f96f4076b90 2022-03-21T14:00:11.893 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f1d9b700 1 -- 172.21.15.2:0/2243252669 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e40037a0 con 0x7f96f4076b90 2022-03-21T14:00:11.893 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f90f0700 1 -- 172.21.15.2:0/2243252669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 msgr2=0x7f96f4075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:11.894 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f90f0700 1 --2- 172.21.15.2:0/2243252669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009730 tx=0x7f96e40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:11.894 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f90f0700 1 -- 172.21.15.2:0/2243252669 shutdown_connections 2022-03-21T14:00:11.894 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f90f0700 1 --2- 172.21.15.2:0/2243252669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4075130 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:11.894 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.888+0000 7f96f90f0700 1 -- 172.21.15.2:0/2243252669 >> 172.21.15.2:0/2243252669 conn(0x7f96f40fcda0 msgr2=0x7f96f40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:11.895 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.889+0000 7f96f90f0700 1 -- 172.21.15.2:0/2243252669 shutdown_connections 2022-03-21T14:00:11.895 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.889+0000 7f96f90f0700 1 -- 172.21.15.2:0/2243252669 wait complete. 2022-03-21T14:00:11.895 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.890+0000 7f96f90f0700 1 Processor -- start 2022-03-21T14:00:11.895 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.890+0000 7f96f90f0700 1 -- start start 2022-03-21T14:00:11.896 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.891+0000 7f96f90f0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:11.896 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.891+0000 7f96f90f0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f4112940 con 0x7f96f4076b90 2022-03-21T14:00:11.896 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.891+0000 7f96f2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:11.897 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.891+0000 7f96f2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43466/0 (socket says 172.21.15.2:43466) 2022-03-21T14:00:11.897 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.891+0000 7f96f2d9d700 1 -- 172.21.15.2:0/3082446426 learned_addr learned my addr 172.21.15.2:0/3082446426 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:11.897 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.892+0000 7f96f2d9d700 1 -- 172.21.15.2:0/3082446426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e4009370 con 0x7f96f4076b90 2022-03-21T14:00:11.899 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.892+0000 7f96f2d9d700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96e4006fd0 tx=0x7f96e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:11.900 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.892+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e40118c0 con 0x7f96f4076b90 2022-03-21T14:00:11.900 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.893+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f4112b40 con 0x7f96f4076b90 2022-03-21T14:00:11.900 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.892+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f96e4011a20 con 0x7f96f4076b90 2022-03-21T14:00:11.900 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.893+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e4022540 con 0x7f96f4076b90 2022-03-21T14:00:11.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.893+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f41137e0 con 0x7f96f4076b90 2022-03-21T14:00:11.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.894+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 41857+0+0 (secure 0 0 0) 0x7f96e4017020 con 0x7f96f4076b90 2022-03-21T14:00:11.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.894+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f405e600 con 0x7f96f4076b90 2022-03-21T14:00:11.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.894+0000 7f96db7fe700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 0x7f96e00370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:11.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.894+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f96e4048030 con 0x7f96f4076b90 2022-03-21T14:00:11.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.895+0000 7f96f259c700 1 -- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 msgr2=0x7f96e00370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:11.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.895+0000 7f96f259c700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 0x7f96e00370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T14:00:11.903 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:11.899+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f96e4011b90 con 0x7f96f4076b90 2022-03-21T14:00:12.088 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:11 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/533262955' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-03-21T14:00:12.089 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:11 smithi002 ceph-mon[17167]: mgrmap e9: x(active, since 16s) 2022-03-21T14:00:12.097 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.096+0000 7f96f259c700 1 -- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 msgr2=0x7f96e00370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:12.097 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.096+0000 7f96f259c700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 0x7f96e00370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-03-21T14:00:12.163 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.161+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f96f4107860 con 0x7f96f4076b90 2022-03-21T14:00:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.162+0000 7f96db7fe700 1 -- 172.21.15.2:0/3082446426 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f96e402a390 con 0x7f96f4076b90 2022-03-21T14:00:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T14:00:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 9, 2022-03-21T14:00:12.165 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2022-03-21T14:00:12.165 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "x", 2022-03-21T14:00:12.165 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2022-03-21T14:00:12.166 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T14:00:12.167 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 msgr2=0x7f96e00370c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T14:00:12.168 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 0x7f96e00370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:12.168 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 msgr2=0x7f96f4112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:12.168 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96e4006fd0 tx=0x7f96e40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:12.168 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 shutdown_connections 2022-03-21T14:00:12.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7f96e0034c10 0x7f96e00370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:12.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 --2- 172.21.15.2:0/3082446426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96f4076b90 0x7f96f4112400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:12.169 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 >> 172.21.15.2:0/3082446426 conn(0x7f96f40fcda0 msgr2=0x7f96f40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 shutdown_connections 2022-03-21T14:00:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:12.164+0000 7f96f90f0700 1 -- 172.21.15.2:0/3082446426 wait complete. 2022-03-21T14:00:12.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:12 smithi002 conmon[17580]: 2022-03-21T14:00:12.178+0000 7f2c12051000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:00:12.659 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:12 smithi002 conmon[17580]: 2022-03-21T14:00:12.372+0000 7f2c12051000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:00:12.730 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2022-03-21T14:00:12.731 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 9... 2022-03-21T14:00:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:13.056 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:12 smithi002 conmon[17580]: 2022-03-21T14:00:12.853+0000 7f2c12051000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-21T14:00:13.057 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:12 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3082446426' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-03-21T14:00:13.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:13 smithi002 conmon[17580]: 2022-03-21T14:00:13.099+0000 7f2c12051000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-21T14:00:13.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.379+0000 7fe6ad6cb700 1 Processor -- start 2022-03-21T14:00:13.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.379+0000 7fe6ad6cb700 1 -- start start 2022-03-21T14:00:13.381 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.380+0000 7fe6ad6cb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:13.382 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.380+0000 7fe6ad6cb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8101f00 con 0x7fe6a81015b0 2022-03-21T14:00:13.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.380+0000 7fe6a6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:13.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.380+0000 7fe6a6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43468/0 (socket says 172.21.15.2:43468) 2022-03-21T14:00:13.384 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.380+0000 7fe6a6ffd700 1 -- 172.21.15.2:0/2545783753 learned_addr learned my addr 172.21.15.2:0/2545783753 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:13.385 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.381+0000 7fe6a6ffd700 1 -- 172.21.15.2:0/2545783753 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a8102040 con 0x7fe6a81015b0 2022-03-21T14:00:13.385 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.381+0000 7fe6a6ffd700 1 --2- 172.21.15.2:0/2545783753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe698009730 tx=0x7fe6980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2287a8226640a68b server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:13.385 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.381+0000 7fe6a5ffb700 1 -- 172.21.15.2:0/2545783753 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe698004030 con 0x7fe6a81015b0 2022-03-21T14:00:13.385 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.381+0000 7fe6a5ffb700 1 -- 172.21.15.2:0/2545783753 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe698004190 con 0x7fe6a81015b0 2022-03-21T14:00:13.386 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.381+0000 7fe6a5ffb700 1 -- 172.21.15.2:0/2545783753 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6980037a0 con 0x7fe6a81015b0 2022-03-21T14:00:13.386 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.382+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/2545783753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 msgr2=0x7fe6a81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:13.386 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.382+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/2545783753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe698009730 tx=0x7fe6980090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:13.386 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.382+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/2545783753 shutdown_connections 2022-03-21T14:00:13.386 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.382+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/2545783753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81019c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:13.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.382+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/2545783753 >> 172.21.15.2:0/2545783753 conn(0x7fe6a80fcda0 msgr2=0x7fe6a80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:13.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.383+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/2545783753 shutdown_connections 2022-03-21T14:00:13.387 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.383+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/2545783753 wait complete. 2022-03-21T14:00:13.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.384+0000 7fe6ad6cb700 1 Processor -- start 2022-03-21T14:00:13.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.384+0000 7fe6ad6cb700 1 -- start start 2022-03-21T14:00:13.390 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.385+0000 7fe6ad6cb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:13.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.385+0000 7fe6ad6cb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a81128f0 con 0x7fe6a81015b0 2022-03-21T14:00:13.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.385+0000 7fe6a6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:13.391 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.385+0000 7fe6a6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43470/0 (socket says 172.21.15.2:43470) 2022-03-21T14:00:13.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.385+0000 7fe6a6ffd700 1 -- 172.21.15.2:0/3415285109 learned_addr learned my addr 172.21.15.2:0/3415285109 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:13.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe6a6ffd700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698009370 con 0x7fe6a81015b0 2022-03-21T14:00:13.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe6a6ffd700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe698004150 tx=0x7fe6980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:13.392 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6980119c0 con 0x7fe6a81015b0 2022-03-21T14:00:13.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe698011b20 con 0x7fe6a81015b0 2022-03-21T14:00:13.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a8112af0 con 0x7fe6a81015b0 2022-03-21T14:00:13.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe698022560 con 0x7fe6a81015b0 2022-03-21T14:00:13.393 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.386+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a8113790 con 0x7fe6a81015b0 2022-03-21T14:00:13.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 41857+0+0 (secure 0 0 0) 0x7fe698017020 con 0x7fe6a81015b0 2022-03-21T14:00:13.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe68f7fe700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:13.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6940377d0 con 0x7fe694034c10 2022-03-21T14:00:13.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe698048110 con 0x7fe6a81015b0 2022-03-21T14:00:13.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:13.395 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.388+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T14:00:13.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.589+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:13.590 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.589+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-03-21T14:00:13.745 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:13 smithi002 conmon[17580]: 2022-03-21T14:00:13.450+0000 7f2c12051000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-21T14:00:13.991 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.990+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:13.991 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:13.990+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-03-21T14:00:14.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:13 smithi002 conmon[17580]: 2022-03-21T14:00:13.744+0000 7f2c12051000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-21T14:00:14.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:13 smithi002 conmon[17580]: 2022-03-21T14:00:13.970+0000 7f2c12051000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-21T14:00:14.792 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:14.791+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:14.791+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-03-21T14:00:15.043 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:14 smithi002 conmon[17580]: 2022-03-21T14:00:14.766+0000 7f2c12051000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-21T14:00:15.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:15 smithi002 conmon[17580]: 2022-03-21T14:00:15.042+0000 7f2c12051000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-21T14:00:16.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:16 smithi002 conmon[17580]: 2022-03-21T14:00:16.048+0000 7f2c12051000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-21T14:00:16.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:16.392+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:16.394 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:16.393+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-03-21T14:00:17.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:16 smithi002 conmon[17580]: 2022-03-21T14:00:16.961+0000 7f2c12051000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-21T14:00:17.366 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:17 smithi002 conmon[17580]: 2022-03-21T14:00:17.121+0000 7f2c12051000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-21T14:00:17.865 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:17 smithi002 conmon[17580]: 2022-03-21T14:00:17.585+0000 7f2c12051000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-21T14:00:17.866 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:17 smithi002 conmon[17580]: 2022-03-21T14:00:17.750+0000 7f2c12051000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-21T14:00:18.365 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:17 smithi002 conmon[17580]: 2022-03-21T14:00:17.919+0000 7f2c12051000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-21T14:00:18.838 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:18 smithi002 conmon[17580]: 2022-03-21T14:00:18.504+0000 7f2c12051000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-21T14:00:18.838 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:18 smithi002 conmon[17580]: 2022-03-21T14:00:18.668+0000 7f2c12051000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-21T14:00:19.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:18 smithi002 conmon[17580]: 2022-03-21T14:00:18.837+0000 7f2c12051000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-21T14:00:19.597 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:19.596+0000 7fe6a67fc700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1086540904 2022-03-21T14:00:19.598 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:19.596+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-03-21T14:00:20.115 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:19 smithi002 conmon[17580]: 2022-03-21T14:00:19.826+0000 7f2c12051000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-21T14:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:20.309+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 10) v1 ==== 41667+0+0 (secure 0 0 0) 0x7fe69800f050 con 0x7fe6a81015b0 2022-03-21T14:00:20.311 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:20.309+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 msgr2=0x7fe6940370c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T14:00:20.311 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:20.309+0000 7fe68f7fe700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:20.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:20 smithi002 ceph-mon[17167]: Active manager daemon x restarted 2022-03-21T14:00:20.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:20 smithi002 ceph-mon[17167]: Activating manager daemon x 2022-03-21T14:00:20.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:20 smithi002 ceph-mon[17167]: osdmap e3: 0 total, 0 up, 0 in 2022-03-21T14:00:20.574 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:20 smithi002 conmon[17580]: 2022-03-21T14:00:20.242+0000 7f2c12051000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-21T14:00:21.320 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.318+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 41777+0+0 (secure 0 0 0) 0x7fe69801a430 con 0x7fe6a81015b0 2022-03-21T14:00:21.321 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.318+0000 7fe68f7fe700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 0x7fe69403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:21.321 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.318+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6940377d0 con 0x7fe694038650 2022-03-21T14:00:21.321 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.319+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 0x7fe69403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:21.322 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.320+0000 7fe6a67fc700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 0x7fe69403aa30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe69c003850 tx=0x7fe69c009350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:21.322 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.321+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7fe6940377d0 con 0x7fe694038650 2022-03-21T14:00:21.329 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.328+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe6a8112f80 con 0x7fe694038650 2022-03-21T14:00:21.330 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.328+0000 7fe68f7fe700 1 -- 172.21.15.2:0/3415285109 <== mgr.14150 v2:172.21.15.2:6800/613532934 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe6a8112f80 con 0x7fe694038650 2022-03-21T14:00:21.330 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2022-03-21T14:00:21.330 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2022-03-21T14:00:21.330 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2022-03-21T14:00:21.331 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2022-03-21T14:00:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 msgr2=0x7fe69403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 0x7fe69403aa30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe69c003850 tx=0x7fe69c009350 comp rx=0 tx=0).stop 2022-03-21T14:00:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 msgr2=0x7fe6a81123b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe698004150 tx=0x7fe6980090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 shutdown_connections 2022-03-21T14:00:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe694038650 0x7fe69403aa30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6a81015b0 0x7fe6a81123b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 --2- 172.21.15.2:0/3415285109 >> [v2:172.21.15.2:6800/1086540904,v1:172.21.15.2:6801/1086540904] conn(0x7fe694034c10 0x7fe6940370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 >> 172.21.15.2:0/3415285109 conn(0x7fe6a80fcda0 msgr2=0x7fe6a80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 shutdown_connections 2022-03-21T14:00:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:21.329+0000 7fe6ad6cb700 1 -- 172.21.15.2:0/3415285109 wait complete. 2022-03-21T14:00:21.358 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:21 smithi002 conmon[17580]: [21/Mar/2022:14:00:21] ENGINE Bus STARTING 2022-03-21T14:00:21.359 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:21 smithi002 conmon[17580]: [21/Mar/2022:14:00:21] ENGINE Serving on https://172.21.15.2:7150 2022-03-21T14:00:21.359 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 21 14:00:21 smithi002 conmon[17580]: [21/Mar/2022:14:00:21] ENGINE Bus STARTED 2022-03-21T14:00:21.359 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: mgrmap e10: x(active, starting, since 0.0533317s) 2022-03-21T14:00:21.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:00:21.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-03-21T14:00:21.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-03-21T14:00:21.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-03-21T14:00:21.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-03-21T14:00:21.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: Manager daemon x is now available 2022-03-21T14:00:21.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:21.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:21.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:00:21.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:00:21.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:21.729 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 9 is available 2022-03-21T14:00:21.729 INFO:teuthology.orchestra.run.smithi002.stderr:Generating a dashboard self-signed certificate... 2022-03-21T14:00:21.843 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.430+0000 7fe9ebfff700 1 Processor -- start 2022-03-21T14:00:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.430+0000 7fe9ebfff700 1 -- start start 2022-03-21T14:00:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.430+0000 7fe9ebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:22.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.430+0000 7fe9ebfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec101d20 con 0x7fe9ec1013d0 2022-03-21T14:00:22.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.431+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:22.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.431+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43486/0 (socket says 172.21.15.2:43486) 2022-03-21T14:00:22.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.431+0000 7fe9eaffd700 1 -- 172.21.15.2:0/30469991 learned_addr learned my addr 172.21.15.2:0/30469991 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:22.436 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.431+0000 7fe9eaffd700 1 -- 172.21.15.2:0/30469991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec101e60 con 0x7fe9ec1013d0 2022-03-21T14:00:22.436 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.432+0000 7fe9eaffd700 1 --2- 172.21.15.2:0/30469991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009730 tx=0x7fe9dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c38243982fecc0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:22.436 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.432+0000 7fe9e9ffb700 1 -- 172.21.15.2:0/30469991 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9dc004030 con 0x7fe9ec1013d0 2022-03-21T14:00:22.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.432+0000 7fe9e9ffb700 1 -- 172.21.15.2:0/30469991 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe9dc004190 con 0x7fe9ec1013d0 2022-03-21T14:00:22.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.432+0000 7fe9e9ffb700 1 -- 172.21.15.2:0/30469991 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9dc0037a0 con 0x7fe9ec1013d0 2022-03-21T14:00:22.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 -- 172.21.15.2:0/30469991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 msgr2=0x7fe9ec1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:22.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/30469991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009730 tx=0x7fe9dc0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:22.437 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 -- 172.21.15.2:0/30469991 shutdown_connections 2022-03-21T14:00:22.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/30469991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec1017e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:22.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 -- 172.21.15.2:0/30469991 >> 172.21.15.2:0/30469991 conn(0x7fe9ec0fcbe0 msgr2=0x7fe9ec0feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:22.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 -- 172.21.15.2:0/30469991 shutdown_connections 2022-03-21T14:00:22.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.433+0000 7fe9ebfff700 1 -- 172.21.15.2:0/30469991 wait complete. 2022-03-21T14:00:22.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.434+0000 7fe9ebfff700 1 Processor -- start 2022-03-21T14:00:22.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.434+0000 7fe9ebfff700 1 -- start start 2022-03-21T14:00:22.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9ebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:22.442 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9ebfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec1104e0 con 0x7fe9ec1013d0 2022-03-21T14:00:22.442 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:22.442 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43488/0 (socket says 172.21.15.2:43488) 2022-03-21T14:00:22.442 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9eaffd700 1 -- 172.21.15.2:0/3936773101 learned_addr learned my addr 172.21.15.2:0/3936773101 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:22.443 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.435+0000 7fe9eaffd700 1 -- 172.21.15.2:0/3936773101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc009370 con 0x7fe9ec1013d0 2022-03-21T14:00:22.443 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9eaffd700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc004150 tx=0x7fe9dc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:22.443 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9dc0119c0 con 0x7fe9ec1013d0 2022-03-21T14:00:22.443 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe9dc011b20 con 0x7fe9ec1013d0 2022-03-21T14:00:22.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9dc022560 con 0x7fe9ec1013d0 2022-03-21T14:00:22.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec1106e0 con 0x7fe9ec1013d0 2022-03-21T14:00:22.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.436+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec111380 con 0x7fe9ec1013d0 2022-03-21T14:00:22.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.438+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 41857+0+0 (secure 0 0 0) 0x7fe9dc017020 con 0x7fe9ec1013d0 2022-03-21T14:00:22.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.438+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec05e600 con 0x7fe9ec1013d0 2022-03-21T14:00:22.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.438+0000 7fe9d37fe700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 0x7fe9d80370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:22.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.438+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe9dc0484a0 con 0x7fe9ec1013d0 2022-03-21T14:00:22.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.439+0000 7fe9ea7fc700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 0x7fe9d80370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:22.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.439+0000 7fe9ea7fc700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 0x7fe9d80370c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009780 tx=0x7fe9e0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:22.446 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.444+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe9dc022790 con 0x7fe9ec1013d0 2022-03-21T14:00:22.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: [21/Mar/2022:14:00:21] ENGINE Bus STARTING 2022-03-21T14:00:22.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: [21/Mar/2022:14:00:21] ENGINE Serving on https://172.21.15.2:7150 2022-03-21T14:00:22.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: [21/Mar/2022:14:00:21] ENGINE Bus STARTED 2022-03-21T14:00:22.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: mgrmap e11: x(active, since 1.06424s) 2022-03-21T14:00:22.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-03-21T14:00:22.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:22 smithi002 ceph-mon[17167]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-03-21T14:00:22.664 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.662+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9ec0fdf90 con 0x7fe9d8034c10 2022-03-21T14:00:22.917 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.915+0000 7fe9d37fe700 1 -- 172.21.15.2:0/3936773101 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe9ec0fdf90 con 0x7fe9d8034c10 2022-03-21T14:00:23.022 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Self-signed certificate created 2022-03-21T14:00:23.024 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 msgr2=0x7fe9d80370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:23.024 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 0x7fe9d80370c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009780 tx=0x7fe9e0006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.024 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 msgr2=0x7fe9ec10ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:23.024 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc004150 tx=0x7fe9dc0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 shutdown_connections 2022-03-21T14:00:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe9d8034c10 0x7fe9d80370c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 --2- 172.21.15.2:0/3936773101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9ec1013d0 0x7fe9ec10ffa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 >> 172.21.15.2:0/3936773101 conn(0x7fe9ec0fcbe0 msgr2=0x7fe9ec0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:23.026 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 shutdown_connections 2022-03-21T14:00:23.026 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:22.918+0000 7fe9ebfff700 1 -- 172.21.15.2:0/3936773101 wait complete. 2022-03-21T14:00:23.319 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial admin user... 2022-03-21T14:00:23.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:23.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:23 smithi002 ceph-mon[17167]: mgrmap e12: x(active, since 2s) 2022-03-21T14:00:23.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:23.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:23 smithi002 ceph-mon[17167]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:23.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:23.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:23.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.956+0000 7fbaa3fbb700 1 Processor -- start 2022-03-21T14:00:23.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.956+0000 7fbaa3fbb700 1 -- start start 2022-03-21T14:00:23.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.956+0000 7fbaa3fbb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:23.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.956+0000 7fbaa3fbb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c076fa0 con 0x7fba9c076b90 2022-03-21T14:00:23.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.957+0000 7fbaa1d57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:23.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.957+0000 7fbaa1d57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43490/0 (socket says 172.21.15.2:43490) 2022-03-21T14:00:23.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.957+0000 7fbaa1d57700 1 -- 172.21.15.2:0/2701957899 learned_addr learned my addr 172.21.15.2:0/2701957899 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:23.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.957+0000 7fbaa1d57700 1 -- 172.21.15.2:0/2701957899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c075670 con 0x7fba9c076b90 2022-03-21T14:00:23.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.958+0000 7fbaa1d57700 1 --2- 172.21.15.2:0/2701957899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba98009730 tx=0x7fba980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c3ca82833aa37c server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:23.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.958+0000 7fbaa0d55700 1 -- 172.21.15.2:0/2701957899 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba98004030 con 0x7fba9c076b90 2022-03-21T14:00:23.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.958+0000 7fbaa0d55700 1 -- 172.21.15.2:0/2701957899 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fba98004190 con 0x7fba9c076b90 2022-03-21T14:00:23.964 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.958+0000 7fbaa0d55700 1 -- 172.21.15.2:0/2701957899 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba980037a0 con 0x7fba9c076b90 2022-03-21T14:00:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.959+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/2701957899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 msgr2=0x7fba9c075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.959+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/2701957899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba98009730 tx=0x7fba980090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.959+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/2701957899 shutdown_connections 2022-03-21T14:00:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.959+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/2701957899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c075130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.959+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/2701957899 >> 172.21.15.2:0/2701957899 conn(0x7fba9c0fcda0 msgr2=0x7fba9c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.960+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/2701957899 shutdown_connections 2022-03-21T14:00:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.960+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/2701957899 wait complete. 2022-03-21T14:00:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.961+0000 7fbaa3fbb700 1 Processor -- start 2022-03-21T14:00:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.961+0000 7fbaa3fbb700 1 -- start start 2022-03-21T14:00:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.961+0000 7fbaa3fbb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.961+0000 7fbaa3fbb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c116d90 con 0x7fba9c076b90 2022-03-21T14:00:23.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.962+0000 7fbaa1d57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:23.968 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.962+0000 7fbaa1d57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43492/0 (socket says 172.21.15.2:43492) 2022-03-21T14:00:23.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.962+0000 7fbaa1d57700 1 -- 172.21.15.2:0/1640776692 learned_addr learned my addr 172.21.15.2:0/1640776692 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:23.970 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.962+0000 7fbaa1d57700 1 -- 172.21.15.2:0/1640776692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98009370 con 0x7fba9c076b90 2022-03-21T14:00:23.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fbaa1d57700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba98006fd0 tx=0x7fba980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:23.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba980119e0 con 0x7fba9c076b90 2022-03-21T14:00:23.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fba98011b40 con 0x7fba9c076b90 2022-03-21T14:00:23.971 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c116f90 con 0x7fba9c076b90 2022-03-21T14:00:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba980224b0 con 0x7fba9c076b90 2022-03-21T14:00:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.963+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c117c30 con 0x7fba9c076b90 2022-03-21T14:00:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.964+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c05e600 con 0x7fba9c076b90 2022-03-21T14:00:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.965+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 41857+0+0 (secure 0 0 0) 0x7fba98017020 con 0x7fba9c076b90 2022-03-21T14:00:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.965+0000 7fba8a7fc700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 0x7fba840370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.965+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fba980482c0 con 0x7fba9c076b90 2022-03-21T14:00:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.966+0000 7fbaa1556700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 0x7fba840370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.966+0000 7fbaa1556700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 0x7fba840370c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009780 tx=0x7fba8c006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:23.970+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fba9801a430 con 0x7fba9c076b90 2022-03-21T14:00:24.187 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.184+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fba9c05e420 con 0x7fba84034c10 2022-03-21T14:00:24.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.432+0000 7fba8a7fc700 1 -- 172.21.15.2:0/1640776692 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fba9c05e420 con 0x7fba84034c10 2022-03-21T14:00:24.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$7X9uemZDV5vq3Zb.oWDbaeJWl4FtlN04/gkt4blTvIXk2QoWYvyoW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1647871224, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2022-03-21T14:00:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 msgr2=0x7fba840370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 0x7fba840370c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009780 tx=0x7fba8c006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 msgr2=0x7fba9c116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba98006fd0 tx=0x7fba980090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:24.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 shutdown_connections 2022-03-21T14:00:24.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fba84034c10 0x7fba840370c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:24.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 --2- 172.21.15.2:0/1640776692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba9c076b90 0x7fba9c116850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:24.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 >> 172.21.15.2:0/1640776692 conn(0x7fba9c0fcda0 msgr2=0x7fba9c1028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 shutdown_connections 2022-03-21T14:00:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:24.435+0000 7fbaa3fbb700 1 -- 172.21.15.2:0/1640776692 wait complete. 2022-03-21T14:00:24.813 INFO:teuthology.orchestra.run.smithi002.stderr:Fetching dashboard port number... 2022-03-21T14:00:24.932 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:25 smithi002 ceph-mon[17167]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:25.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.429+0000 7fe309e7f700 1 Processor -- start 2022-03-21T14:00:25.432 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.429+0000 7fe309e7f700 1 -- start start 2022-03-21T14:00:25.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.430+0000 7fe309e7f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:25.433 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.430+0000 7fe309e7f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe304076fa0 con 0x7fe304076b90 2022-03-21T14:00:25.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.430+0000 7fe3037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:25.434 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.430+0000 7fe3037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43496/0 (socket says 172.21.15.2:43496) 2022-03-21T14:00:25.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.430+0000 7fe3037fe700 1 -- 172.21.15.2:0/1779282267 learned_addr learned my addr 172.21.15.2:0/1779282267 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:25.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.431+0000 7fe3037fe700 1 -- 172.21.15.2:0/1779282267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304075670 con 0x7fe304076b90 2022-03-21T14:00:25.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.431+0000 7fe3037fe700 1 --2- 172.21.15.2:0/1779282267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009730 tx=0x7fe2f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3445352e8638e3db server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:25.435 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.432+0000 7fe3027fc700 1 -- 172.21.15.2:0/1779282267 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2f4004030 con 0x7fe304076b90 2022-03-21T14:00:25.436 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.432+0000 7fe3027fc700 1 -- 172.21.15.2:0/1779282267 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe2f4004190 con 0x7fe304076b90 2022-03-21T14:00:25.436 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.432+0000 7fe3027fc700 1 -- 172.21.15.2:0/1779282267 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2f40037a0 con 0x7fe304076b90 2022-03-21T14:00:25.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 -- 172.21.15.2:0/1779282267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 msgr2=0x7fe304075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:25.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 --2- 172.21.15.2:0/1779282267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009730 tx=0x7fe2f40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.438 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 -- 172.21.15.2:0/1779282267 shutdown_connections 2022-03-21T14:00:25.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 --2- 172.21.15.2:0/1779282267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304075130 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 -- 172.21.15.2:0/1779282267 >> 172.21.15.2:0/1779282267 conn(0x7fe3040fcda0 msgr2=0x7fe3040ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:25.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 -- 172.21.15.2:0/1779282267 shutdown_connections 2022-03-21T14:00:25.439 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.433+0000 7fe309e7f700 1 -- 172.21.15.2:0/1779282267 wait complete. 2022-03-21T14:00:25.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.434+0000 7fe309e7f700 1 Processor -- start 2022-03-21T14:00:25.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.434+0000 7fe309e7f700 1 -- start start 2022-03-21T14:00:25.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.434+0000 7fe309e7f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:25.440 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.434+0000 7fe309e7f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe304116d90 con 0x7fe304076b90 2022-03-21T14:00:25.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.435+0000 7fe3037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:25.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.435+0000 7fe3037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43498/0 (socket says 172.21.15.2:43498) 2022-03-21T14:00:25.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.435+0000 7fe3037fe700 1 -- 172.21.15.2:0/2932664169 learned_addr learned my addr 172.21.15.2:0/2932664169 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:25.441 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe3037fe700 1 -- 172.21.15.2:0/2932664169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4009370 con 0x7fe304076b90 2022-03-21T14:00:25.442 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe3037fe700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4006fd0 tx=0x7fe2f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:25.443 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2f40118c0 con 0x7fe304076b90 2022-03-21T14:00:25.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe304116f90 con 0x7fe304076b90 2022-03-21T14:00:25.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe2f4011a20 con 0x7fe304076b90 2022-03-21T14:00:25.444 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.436+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2f4022540 con 0x7fe304076b90 2022-03-21T14:00:25.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.437+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe304117c30 con 0x7fe304076b90 2022-03-21T14:00:25.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.438+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30405e600 con 0x7fe304076b90 2022-03-21T14:00:25.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.438+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 41857+0+0 (secure 0 0 0) 0x7fe2f4017020 con 0x7fe304076b90 2022-03-21T14:00:25.445 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.439+0000 7fe2ebfff700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 0x7fe2f0037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:25.446 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.439+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe2f4048160 con 0x7fe304076b90 2022-03-21T14:00:25.446 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.439+0000 7fe302ffd700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 0x7fe2f0037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:25.446 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.441+0000 7fe302ffd700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 0x7fe2f0037080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009780 tx=0x7fe2f8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:25.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.448+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe2f4011b90 con 0x7fe304076b90 2022-03-21T14:00:25.651 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.649+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe304103430 con 0x7fe304076b90 2022-03-21T14:00:25.652 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.650+0000 7fe2ebfff700 1 -- 172.21.15.2:0/2932664169 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe2f4016260 con 0x7fe304076b90 2022-03-21T14:00:25.652 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 8443 2022-03-21T14:00:25.655 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 msgr2=0x7fe2f0037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:25.655 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 0x7fe2f0037080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009780 tx=0x7fe2f8006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.656 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 msgr2=0x7fe304116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:25.656 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4006fd0 tx=0x7fe2f40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.656 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 shutdown_connections 2022-03-21T14:00:25.656 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe2f0034bd0 0x7fe2f0037080 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.657 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 --2- 172.21.15.2:0/2932664169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe304076b90 0x7fe304116850 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:25.657 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 >> 172.21.15.2:0/2932664169 conn(0x7fe3040fcda0 msgr2=0x7fe304102940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:25.657 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.652+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 shutdown_connections 2022-03-21T14:00:25.657 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:25.653+0000 7fe309e7f700 1 -- 172.21.15.2:0/2932664169 wait complete. 2022-03-21T14:00:26.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:26 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2932664169' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-03-21T14:00:26.329 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2022-03-21T14:00:26.349 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2022-03-21T14:00:26.351 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2022-03-21T14:00:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2022-03-21T14:00:26.355 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph Dashboard is now available at: 2022-03-21T14:00:26.355 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:26.355 INFO:teuthology.orchestra.run.smithi002.stderr: URL: https://smithi002.front.sepia.ceph.com:8443/ 2022-03-21T14:00:26.355 INFO:teuthology.orchestra.run.smithi002.stderr: User: admin 2022-03-21T14:00:26.356 INFO:teuthology.orchestra.run.smithi002.stderr: Password: 2wsexf1scp 2022-03-21T14:00:26.356 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:26.356 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling autotune for osd_memory_target 2022-03-21T14:00:26.470 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:26.947 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.945+0000 7efd76086700 1 Processor -- start 2022-03-21T14:00:26.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.945+0000 7efd76086700 1 -- start start 2022-03-21T14:00:26.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.945+0000 7efd76086700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:26.948 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.945+0000 7efd76086700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70101d20 con 0x7efd701013d0 2022-03-21T14:00:26.949 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.945+0000 7efd75084700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:26.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.946+0000 7efd75084700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43502/0 (socket says 172.21.15.2:43502) 2022-03-21T14:00:26.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.946+0000 7efd75084700 1 -- 172.21.15.2:0/279425005 learned_addr learned my addr 172.21.15.2:0/279425005 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:26.952 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.946+0000 7efd75084700 1 -- 172.21.15.2:0/279425005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70101e60 con 0x7efd701013d0 2022-03-21T14:00:26.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.946+0000 7efd75084700 1 --2- 172.21.15.2:0/279425005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009730 tx=0x7efd6c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f745d02eb3ba0a5d server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:26.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd5ffff700 1 -- 172.21.15.2:0/279425005 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd6c004030 con 0x7efd701013d0 2022-03-21T14:00:26.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd5ffff700 1 -- 172.21.15.2:0/279425005 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efd6c004190 con 0x7efd701013d0 2022-03-21T14:00:26.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd5ffff700 1 -- 172.21.15.2:0/279425005 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd6c0037a0 con 0x7efd701013d0 2022-03-21T14:00:26.954 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd76086700 1 -- 172.21.15.2:0/279425005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 msgr2=0x7efd701017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:26.954 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd76086700 1 --2- 172.21.15.2:0/279425005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009730 tx=0x7efd6c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:26.954 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd76086700 1 -- 172.21.15.2:0/279425005 shutdown_connections 2022-03-21T14:00:26.954 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd76086700 1 --2- 172.21.15.2:0/279425005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd701017e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:26.955 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.947+0000 7efd76086700 1 -- 172.21.15.2:0/279425005 >> 172.21.15.2:0/279425005 conn(0x7efd700fcba0 msgr2=0x7efd700feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:26.955 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.948+0000 7efd76086700 1 -- 172.21.15.2:0/279425005 shutdown_connections 2022-03-21T14:00:26.955 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.948+0000 7efd76086700 1 -- 172.21.15.2:0/279425005 wait complete. 2022-03-21T14:00:26.955 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.948+0000 7efd76086700 1 Processor -- start 2022-03-21T14:00:26.956 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd76086700 1 -- start start 2022-03-21T14:00:26.956 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd76086700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:26.956 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd76086700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70110500 con 0x7efd701013d0 2022-03-21T14:00:26.956 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd75084700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:26.957 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd75084700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43504/0 (socket says 172.21.15.2:43504) 2022-03-21T14:00:26.957 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.949+0000 7efd75084700 1 -- 172.21.15.2:0/3003775228 learned_addr learned my addr 172.21.15.2:0/3003775228 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:26.957 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd75084700 1 -- 172.21.15.2:0/3003775228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c009370 con 0x7efd701013d0 2022-03-21T14:00:26.957 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd75084700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd6c003970 tx=0x7efd6c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:26.959 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd6c011a60 con 0x7efd701013d0 2022-03-21T14:00:26.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efd6c011bc0 con 0x7efd701013d0 2022-03-21T14:00:26.960 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd70110700 con 0x7efd701013d0 2022-03-21T14:00:26.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.950+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd6c022530 con 0x7efd701013d0 2022-03-21T14:00:26.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.951+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701113a0 con 0x7efd701013d0 2022-03-21T14:00:26.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.952+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 41857+0+0 (secure 0 0 0) 0x7efd6c017020 con 0x7efd701013d0 2022-03-21T14:00:26.961 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.952+0000 7efd5dffb700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 0x7efd58037080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.952+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efd6c0482d0 con 0x7efd701013d0 2022-03-21T14:00:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.952+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7005e600 con 0x7efd701013d0 2022-03-21T14:00:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.952+0000 7efd74883700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 0x7efd58037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.953+0000 7efd74883700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 0x7efd58037080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd60009780 tx=0x7efd60006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:26.957+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7efd6c01a430 con 0x7efd701013d0 2022-03-21T14:00:27.110 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.108+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7efd6c00f050 con 0x7efd701013d0 2022-03-21T14:00:27.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.147+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7efd7004af20 con 0x7efd701013d0 2022-03-21T14:00:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.147+0000 7efd5dffb700 1 -- 172.21.15.2:0/3003775228 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7efd6c018520 con 0x7efd701013d0 2022-03-21T14:00:27.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 msgr2=0x7efd58037080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:27.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 0x7efd58037080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd60009780 tx=0x7efd60006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:27.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 msgr2=0x7efd7010ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:27.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd6c003970 tx=0x7efd6c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:27.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 shutdown_connections 2022-03-21T14:00:27.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efd58034bd0 0x7efd58037080 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:27.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 --2- 172.21.15.2:0/3003775228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd701013d0 0x7efd7010ffc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:27.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 >> 172.21.15.2:0/3003775228 conn(0x7efd700fcba0 msgr2=0x7efd700fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:27.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 shutdown_connections 2022-03-21T14:00:27.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:27.150+0000 7efd76086700 1 -- 172.21.15.2:0/3003775228 wait complete. 2022-03-21T14:00:27.659 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.150+0000 7f4ac02a1700 1 Processor -- start 2022-03-21T14:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.150+0000 7f4ac02a1700 1 -- start start 2022-03-21T14:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.150+0000 7f4ac02a1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.150+0000 7f4ac02a1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab8076fa0 con 0x7f4ab8076b90 2022-03-21T14:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.151+0000 7f4abe03d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.151+0000 7f4abe03d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43506/0 (socket says 172.21.15.2:43506) 2022-03-21T14:00:28.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.151+0000 7f4abe03d700 1 -- 172.21.15.2:0/2638114936 learned_addr learned my addr 172.21.15.2:0/2638114936 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:28.158 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.151+0000 7f4abe03d700 1 -- 172.21.15.2:0/2638114936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab8075670 con 0x7f4ab8076b90 2022-03-21T14:00:28.158 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4abe03d700 1 --2- 172.21.15.2:0/2638114936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009730 tx=0x7f4ab40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c38a79b862d052 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:28.159 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4abd03b700 1 -- 172.21.15.2:0/2638114936 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ab4004030 con 0x7f4ab8076b90 2022-03-21T14:00:28.159 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4abd03b700 1 -- 172.21.15.2:0/2638114936 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4ab4004190 con 0x7f4ab8076b90 2022-03-21T14:00:28.159 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4abd03b700 1 -- 172.21.15.2:0/2638114936 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ab40037a0 con 0x7f4ab8076b90 2022-03-21T14:00:28.159 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4ac02a1700 1 -- 172.21.15.2:0/2638114936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 msgr2=0x7f4ab8075130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:28.160 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.152+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/2638114936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009730 tx=0x7f4ab40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.160 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.153+0000 7f4ac02a1700 1 -- 172.21.15.2:0/2638114936 shutdown_connections 2022-03-21T14:00:28.160 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.153+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/2638114936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8075130 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.161 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.153+0000 7f4ac02a1700 1 -- 172.21.15.2:0/2638114936 >> 172.21.15.2:0/2638114936 conn(0x7f4ab80fcda0 msgr2=0x7f4ab80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:28.161 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.153+0000 7f4ac02a1700 1 -- 172.21.15.2:0/2638114936 shutdown_connections 2022-03-21T14:00:28.161 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.153+0000 7f4ac02a1700 1 -- 172.21.15.2:0/2638114936 wait complete. 2022-03-21T14:00:28.161 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.154+0000 7f4ac02a1700 1 Processor -- start 2022-03-21T14:00:28.161 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.154+0000 7f4ac02a1700 1 -- start start 2022-03-21T14:00:28.162 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.154+0000 7f4ac02a1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:28.162 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.154+0000 7f4ac02a1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab8116d90 con 0x7f4ab8076b90 2022-03-21T14:00:28.162 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.155+0000 7f4abe03d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:28.163 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.155+0000 7f4abe03d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43510/0 (socket says 172.21.15.2:43510) 2022-03-21T14:00:28.163 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.155+0000 7f4abe03d700 1 -- 172.21.15.2:0/1391045934 learned_addr learned my addr 172.21.15.2:0/1391045934 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:28.163 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.155+0000 7f4abe03d700 1 -- 172.21.15.2:0/1391045934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4009370 con 0x7f4ab8076b90 2022-03-21T14:00:28.163 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.155+0000 7f4abe03d700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4006fd0 tx=0x7f4ab40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:28.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.156+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ab40118c0 con 0x7f4ab8076b90 2022-03-21T14:00:28.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.156+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4ab4011a20 con 0x7f4ab8076b90 2022-03-21T14:00:28.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.156+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab8116f90 con 0x7f4ab8076b90 2022-03-21T14:00:28.164 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.156+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ab4022540 con 0x7f4ab8076b90 2022-03-21T14:00:28.165 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.156+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab8117c30 con 0x7f4ab8076b90 2022-03-21T14:00:28.166 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.157+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab805e600 con 0x7f4ab8076b90 2022-03-21T14:00:28.166 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.157+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f4ab4017020 con 0x7f4ab8076b90 2022-03-21T14:00:28.167 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.158+0000 7f4aa6ffd700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 0x7f4aa0037160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:28.167 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.158+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4ab4048250 con 0x7f4ab8076b90 2022-03-21T14:00:28.167 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.158+0000 7f4abd83c700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 0x7f4aa0037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:28.167 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.159+0000 7f4abd83c700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 0x7f4aa0037160 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009780 tx=0x7f4aa8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:28.168 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.162+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f4ab402a980 con 0x7f4ab8076b90 2022-03-21T14:00:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:28 smithi002 ceph-mon[17167]: mgrmap e13: x(active, since 6s) 2022-03-21T14:00:28.450 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.448+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4ab8103430 con 0x7f4ab8076b90 2022-03-21T14:00:28.452 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.450+0000 7f4aa6ffd700 1 -- 172.21.15.2:0/1391045934 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4ab40205b0 con 0x7f4ab8076b90 2022-03-21T14:00:28.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 msgr2=0x7f4aa0037160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:28.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 0x7f4aa0037160 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009780 tx=0x7f4aa8006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 msgr2=0x7f4ab8116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:28.456 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4006fd0 tx=0x7f4ab40090f0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 shutdown_connections 2022-03-21T14:00:28.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4aa0034cb0 0x7f4aa0037160 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 --2- 172.21.15.2:0/1391045934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ab8076b90 0x7f4ab8116850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:28.457 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 >> 172.21.15.2:0/1391045934 conn(0x7f4ab80fcda0 msgr2=0x7f4ab8102940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:28.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 shutdown_connections 2022-03-21T14:00:28.458 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2022-03-21T14:00:28.453+0000 7f4ac02a1700 1 -- 172.21.15.2:0/1391045934 wait complete. 2022-03-21T14:00:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2022-03-21T14:00:29.036 INFO:teuthology.orchestra.run.smithi002.stderr:You can access the Ceph CLI with: 2022-03-21T14:00:29.037 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.037 INFO:teuthology.orchestra.run.smithi002.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:29.037 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.037 INFO:teuthology.orchestra.run.smithi002.stderr:Please consider enabling telemetry to help improve Ceph: 2022-03-21T14:00:29.038 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.038 INFO:teuthology.orchestra.run.smithi002.stderr: ceph telemetry on 2022-03-21T14:00:29.038 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.038 INFO:teuthology.orchestra.run.smithi002.stderr:For more information see: 2022-03-21T14:00:29.039 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.039 INFO:teuthology.orchestra.run.smithi002.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2022-03-21T14:00:29.039 INFO:teuthology.orchestra.run.smithi002.stderr: 2022-03-21T14:00:29.039 INFO:teuthology.orchestra.run.smithi002.stderr:Bootstrap complete. 2022-03-21T14:00:29.098 INFO:tasks.cephadm:Fetching config... 2022-03-21T14:00:29.099 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:00:29.099 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-03-21T14:00:29.147 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-03-21T14:00:29.147 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:00:29.148 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-03-21T14:00:29.235 INFO:tasks.cephadm:Fetching mon keyring... 2022-03-21T14:00:29.236 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:00:29.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-03-21T14:00:29.309 INFO:tasks.cephadm:Fetching pub ssh key... 2022-03-21T14:00:29.309 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:00:29.309 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-03-21T14:00:29.394 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-03-21T14:00:29.395 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-03-21T14:00:29.540 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:29.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:29 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1391045934' entity='client.admin' 2022-03-21T14:00:29.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-03-21T14:00:29.628 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:29.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-03-21T14:00:29.684 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+yQnpnaXV9drO0TAXfJRYiAWNkJCWraZTnVEXSKEMQTyGqNmqcJG9nwKhasdr/mAfIz9gN3yXq9tsyAykZncz+yNw0m/NCihTLU+Cg4PyMZBx2HdO3jfwmMAT180RLvpoW3SyVkuaQgfKxmjiUBxzYZhGxt5v5pSDS7ACv2FrbEGabtLwhdARJgj3bJb5iwv7YjZrlQINvidJQ/TEpdwO5l1FM64PACXBsNiFIwvF5dMNkcll9+ig4+GySgkHx0SrizUr1D4aq+ihwwmBEtKXpzxXl4G3FepXmBJ18I4YB+1Hk9zPxHdiubrfy7VNBEvwsGkJMrDVd8E6WwxrCNdo37tJMF1DM+BKHgPGn5JjfCXFtk8Q8q74/+jNSL5z94bjyVos20ocb/kjVD0GztcSvemt+zlblE0QicZw9VWt0WDYztHaOJAHsRqw/J5G5/NkENtV/A/57PAnK+URNyc1h5MKE1puUkJZ7LIy6pwJLWKjJvURj6WrS1QjFHSTHA8= ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:29.697 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-03-21T14:00:31.217 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:31.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 -- 172.21.15.2:0/2814858769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 msgr2=0x7f4204100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:31.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 --2- 172.21.15.2:0/2814858769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f4204100a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097e0 tx=0x7f41f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:31.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 -- 172.21.15.2:0/2814858769 shutdown_connections 2022-03-21T14:00:31.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 --2- 172.21.15.2:0/2814858769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f4204100a70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:31.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 -- 172.21.15.2:0/2814858769 >> 172.21.15.2:0/2814858769 conn(0x7f42040f6100 msgr2=0x7f42040f8510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 -- 172.21.15.2:0/2814858769 shutdown_connections 2022-03-21T14:00:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.854+0000 7f4208bb9700 1 -- 172.21.15.2:0/2814858769 wait complete. 2022-03-21T14:00:31.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.855+0000 7f4208bb9700 1 Processor -- start 2022-03-21T14:00:31.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.855+0000 7f4208bb9700 1 -- start start 2022-03-21T14:00:31.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.855+0000 7f4208bb9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:31.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.855+0000 7f4208bb9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040fcf50 con 0x7f42040657e0 2022-03-21T14:00:31.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.856+0000 7f420259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.856+0000 7f420259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43514/0 (socket says 172.21.15.2:43514) 2022-03-21T14:00:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.856+0000 7f420259c700 1 -- 172.21.15.2:0/3901856768 learned_addr learned my addr 172.21.15.2:0/3901856768 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.856+0000 7f420259c700 1 -- 172.21.15.2:0/3901856768 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f40093f0 con 0x7f42040657e0 2022-03-21T14:00:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f420259c700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097b0 tx=0x7f41f4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41f401c070 con 0x7f42040657e0 2022-03-21T14:00:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f41f4003e40 con 0x7f42040657e0 2022-03-21T14:00:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42040fd150 con 0x7f42040657e0 2022-03-21T14:00:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41f4021be0 con 0x7f42040657e0 2022-03-21T14:00:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.857+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42040fd5e0 con 0x7f42040657e0 2022-03-21T14:00:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.858+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f41f4028030 con 0x7f42040657e0 2022-03-21T14:00:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.858+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420404af20 con 0x7f42040657e0 2022-03-21T14:00:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.858+0000 7f41f2ffd700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 0x7f41ec037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.858+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f41f4047db0 con 0x7f42040657e0 2022-03-21T14:00:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.859+0000 7f4201d9b700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 0x7f41ec037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.859+0000 7f4201d9b700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 0x7f41ec037110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009780 tx=0x7f41f8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:31.864+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f41f402a590 con 0x7f42040657e0 2022-03-21T14:00:32.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.061+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f42041038f0 con 0x7f42040657e0 2022-03-21T14:00:32.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.066+0000 7f41f2ffd700 1 -- 172.21.15.2:0/3901856768 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f41f4026020 con 0x7f42040657e0 2022-03-21T14:00:32.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 msgr2=0x7f41ec037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:32.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 0x7f41ec037110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009780 tx=0x7f41f8006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:32.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 msgr2=0x7f42040fca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:32.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097b0 tx=0x7f41f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:32.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 shutdown_connections 2022-03-21T14:00:32.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f41ec034c60 0x7f41ec037110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:32.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 --2- 172.21.15.2:0/3901856768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42040657e0 0x7f42040fca10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:32.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 >> 172.21.15.2:0/3901856768 conn(0x7f42040f6100 msgr2=0x7f42040f6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:32.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 shutdown_connections 2022-03-21T14:00:32.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:32.070+0000 7f4208bb9700 1 -- 172.21.15.2:0/3901856768 wait complete. 2022-03-21T14:00:32.719 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-03-21T14:00:32.720 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-03-21T14:00:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:33 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3901856768' entity='client.admin' 2022-03-21T14:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:35.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 -- 172.21.15.2:0/963383809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 msgr2=0x7f01840fe0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:35.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 --2- 172.21.15.2:0/963383809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f01840fe0a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01780097e0 tx=0x7f0178005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:35.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 -- 172.21.15.2:0/963383809 shutdown_connections 2022-03-21T14:00:35.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 --2- 172.21.15.2:0/963383809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f01840fe0a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:35.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 -- 172.21.15.2:0/963383809 >> 172.21.15.2:0/963383809 conn(0x7f01840f95e0 msgr2=0x7f01840fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:35.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 -- 172.21.15.2:0/963383809 shutdown_connections 2022-03-21T14:00:35.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.142+0000 7f018b169700 1 -- 172.21.15.2:0/963383809 wait complete. 2022-03-21T14:00:35.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.143+0000 7f018b169700 1 Processor -- start 2022-03-21T14:00:35.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.143+0000 7f018b169700 1 -- start start 2022-03-21T14:00:35.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f018b169700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:35.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f018b169700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018410e390 con 0x7f01840fdc90 2022-03-21T14:00:35.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f0188f05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:35.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f0188f05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43518/0 (socket says 172.21.15.2:43518) 2022-03-21T14:00:35.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f0188f05700 1 -- 172.21.15.2:0/2378737788 learned_addr learned my addr 172.21.15.2:0/2378737788 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:35.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.144+0000 7f0188f05700 1 -- 172.21.15.2:0/2378737788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780093f0 con 0x7f01840fdc90 2022-03-21T14:00:35.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f0188f05700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f01780097b0 tx=0x7f0178005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:35.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f017801c070 con 0x7f01840fdc90 2022-03-21T14:00:35.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0178003e40 con 0x7f01840fdc90 2022-03-21T14:00:35.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0178021be0 con 0x7f01840fdc90 2022-03-21T14:00:35.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018410e590 con 0x7f01840fdc90 2022-03-21T14:00:35.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.145+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018410f150 con 0x7f01840fdc90 2022-03-21T14:00:35.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.146+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018405e600 con 0x7f01840fdc90 2022-03-21T14:00:35.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.147+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f0178028030 con 0x7f01840fdc90 2022-03-21T14:00:35.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.147+0000 7f01817fa700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 0x7f0174037160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:35.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.147+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0178047d30 con 0x7f01840fdc90 2022-03-21T14:00:35.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.147+0000 7f0183fff700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 0x7f0174037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:35.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.148+0000 7f0183fff700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 0x7f0174037160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0170009780 tx=0x7f0170006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:35.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.152+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f017802a590 con 0x7f01840fdc90 2022-03-21T14:00:35.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.343+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f018410fd50 con 0x7f0174034cb0 2022-03-21T14:00:35.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.347+0000 7f01817fa700 1 -- 172.21.15.2:0/2378737788 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f018410fd50 con 0x7f0174034cb0 2022-03-21T14:00:35.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 msgr2=0x7f0174037160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:35.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 0x7f0174037160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0170009780 tx=0x7f0170006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:35.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 msgr2=0x7f018410de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:35.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f01780097b0 tx=0x7f0178005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:35.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 shutdown_connections 2022-03-21T14:00:35.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0174034cb0 0x7f0174037160 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:35.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 --2- 172.21.15.2:0/2378737788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01840fdc90 0x7f018410de50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:35.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.349+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 >> 172.21.15.2:0/2378737788 conn(0x7f01840f95e0 msgr2=0x7f01840fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:35.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.350+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 shutdown_connections 2022-03-21T14:00:35.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:35.350+0000 7f018b169700 1 -- 172.21.15.2:0/2378737788 wait complete. 2022-03-21T14:00:35.639 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:35.639 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:00:35.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:35.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:35.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:35.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:35.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:35.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:35.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:00:35.838 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi016 2022-03-21T14:00:35.839 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T14:00:35.839 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/etc/ceph/ceph.conf 2022-03-21T14:00:35.858 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T14:00:35.858 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:35.916 INFO:tasks.cephadm:Adding host smithi016 to orchestrator... 2022-03-21T14:00:35.916 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch host add smithi016 2022-03-21T14:00:36.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:36 smithi002 ceph-mon[17167]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:36.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:36 smithi002 ceph-mon[17167]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T14:00:36.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:36 smithi002 ceph-mon[17167]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:36.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:36.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:37.328 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:37.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.822+0000 7fee3db10700 1 -- 172.21.15.2:0/2202217765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 msgr2=0x7fee38101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:37.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.822+0000 7fee3db10700 1 --2- 172.21.15.2:0/2202217765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38101ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee280097e0 tx=0x7fee28005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:37.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.822+0000 7fee3db10700 1 -- 172.21.15.2:0/2202217765 shutdown_connections 2022-03-21T14:00:37.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.822+0000 7fee3db10700 1 --2- 172.21.15.2:0/2202217765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38101ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:37.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.822+0000 7fee3db10700 1 -- 172.21.15.2:0/2202217765 >> 172.21.15.2:0/2202217765 conn(0x7fee380f76d0 msgr2=0x7fee380f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:37.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.823+0000 7fee3db10700 1 -- 172.21.15.2:0/2202217765 shutdown_connections 2022-03-21T14:00:37.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.823+0000 7fee3db10700 1 -- 172.21.15.2:0/2202217765 wait complete. 2022-03-21T14:00:37.827 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.823+0000 7fee3db10700 1 Processor -- start 2022-03-21T14:00:37.828 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee3db10700 1 -- start start 2022-03-21T14:00:37.828 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee3db10700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:37.828 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee3db10700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee38114350 con 0x7fee38101b10 2022-03-21T14:00:37.829 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee377fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:37.829 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee377fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43524/0 (socket says 172.21.15.2:43524) 2022-03-21T14:00:37.829 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.824+0000 7fee377fe700 1 -- 172.21.15.2:0/3338891315 learned_addr learned my addr 172.21.15.2:0/3338891315 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:37.829 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee377fe700 1 -- 172.21.15.2:0/3338891315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee280093f0 con 0x7fee38101b10 2022-03-21T14:00:37.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee377fe700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee280097b0 tx=0x7fee28005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:37.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee2801c070 con 0x7fee38101b10 2022-03-21T14:00:37.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fee28003df0 con 0x7fee38101b10 2022-03-21T14:00:37.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee38114550 con 0x7fee38101b10 2022-03-21T14:00:37.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee28021b30 con 0x7fee38101b10 2022-03-21T14:00:37.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.825+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee381149e0 con 0x7fee38101b10 2022-03-21T14:00:37.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.827+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3805e600 con 0x7fee38101b10 2022-03-21T14:00:37.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.827+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fee28028030 con 0x7fee38101b10 2022-03-21T14:00:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.827+0000 7fee1ffff700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 0x7fee20037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:37.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.827+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fee28047ce0 con 0x7fee38101b10 2022-03-21T14:00:37.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.828+0000 7fee36ffd700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 0x7fee20037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:37.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.828+0000 7fee36ffd700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 0x7fee20037110 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009780 tx=0x7fee2c006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:37.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:37.832+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fee28003f60 con 0x7fee38101b10 2022-03-21T14:00:38.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.024+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "target": ["mon-mgr", ""]}) v1 -- 0x7fee3810f530 con 0x7fee20034c60 2022-03-21T14:00:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.978+0000 7fee1ffff700 1 -- 172.21.15.2:0/3338891315 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fee2c004040 con 0x7fee20034c60 2022-03-21T14:00:38.980 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi016' with addr '172.21.15.16' 2022-03-21T14:00:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 msgr2=0x7fee20037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 0x7fee20037110 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009780 tx=0x7fee2c006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 msgr2=0x7fee38113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee280097b0 tx=0x7fee28005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 shutdown_connections 2022-03-21T14:00:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee20034c60 0x7fee20037110 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 --2- 172.21.15.2:0/3338891315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee38101b10 0x7fee38113e10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 >> 172.21.15.2:0/3338891315 conn(0x7fee380f76d0 msgr2=0x7fee38070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 shutdown_connections 2022-03-21T14:00:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:38.981+0000 7fee3db10700 1 -- 172.21.15.2:0/3338891315 wait complete. 2022-03-21T14:00:39.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch host ls --format=json 2022-03-21T14:00:40.274 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:40.274 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: Deploying cephadm binary to smithi016 2022-03-21T14:00:40.274 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:40.275 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: Added host smithi016 2022-03-21T14:00:40.275 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:40.275 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:40.275 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:00:40.276 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:40.276 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:40.934 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:41.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.360+0000 7face2203700 1 -- 172.21.15.2:0/4194032572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 msgr2=0x7facdc101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:41.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.360+0000 7face2203700 1 --2- 172.21.15.2:0/4194032572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc101ea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faccc0097e0 tx=0x7faccc005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:41.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.361+0000 7face2203700 1 -- 172.21.15.2:0/4194032572 shutdown_connections 2022-03-21T14:00:41.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.361+0000 7face2203700 1 --2- 172.21.15.2:0/4194032572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc101ea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:41.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.361+0000 7face2203700 1 -- 172.21.15.2:0/4194032572 >> 172.21.15.2:0/4194032572 conn(0x7facdc071980 msgr2=0x7facdc071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:41.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.361+0000 7face2203700 1 -- 172.21.15.2:0/4194032572 shutdown_connections 2022-03-21T14:00:41.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.361+0000 7face2203700 1 -- 172.21.15.2:0/4194032572 wait complete. 2022-03-21T14:00:41.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.362+0000 7face2203700 1 Processor -- start 2022-03-21T14:00:41.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.362+0000 7face2203700 1 -- start start 2022-03-21T14:00:41.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.362+0000 7face2203700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:41.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.362+0000 7face2203700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc114450 con 0x7facdc101ad0 2022-03-21T14:00:41.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.362+0000 7facdb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:41.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facdb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43528/0 (socket says 172.21.15.2:43528) 2022-03-21T14:00:41.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facdb7fe700 1 -- 172.21.15.2:0/3426756678 learned_addr learned my addr 172.21.15.2:0/3426756678 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:41.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facdb7fe700 1 -- 172.21.15.2:0/3426756678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0093f0 con 0x7facdc101ad0 2022-03-21T14:00:41.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facdb7fe700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faccc0097b0 tx=0x7faccc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:41.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faccc01c070 con 0x7facdc101ad0 2022-03-21T14:00:41.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7faccc003df0 con 0x7facdc101ad0 2022-03-21T14:00:41.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faccc021b30 con 0x7facdc101ad0 2022-03-21T14:00:41.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.363+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc114650 con 0x7facdc101ad0 2022-03-21T14:00:41.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.364+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc114a60 con 0x7facdc101ad0 2022-03-21T14:00:41.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.365+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7faccc028030 con 0x7facdc101ad0 2022-03-21T14:00:41.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.365+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc05e600 con 0x7facdc101ad0 2022-03-21T14:00:41.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.365+0000 7facc3fff700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 0x7facc4037160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:41.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.365+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7faccc047e20 con 0x7facdc101ad0 2022-03-21T14:00:41.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.365+0000 7facdaffd700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 0x7facc4037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:41.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.366+0000 7facdaffd700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 0x7facc4037160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facd0009780 tx=0x7facd0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:41.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.370+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7faccc02a590 con 0x7facdc101ad0 2022-03-21T14:00:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.562+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facdc10f5d0 con 0x7facc4034cb0 2022-03-21T14:00:41.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.563+0000 7facc3fff700 1 -- 172.21.15.2:0/3426756678 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7facdc10f5d0 con 0x7facc4034cb0 2022-03-21T14:00:41.565 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:00:41.565 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}] 2022-03-21T14:00:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 msgr2=0x7facc4037160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 0x7facc4037160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facd0009780 tx=0x7facd0006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 msgr2=0x7facdc113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faccc0097b0 tx=0x7faccc005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 shutdown_connections 2022-03-21T14:00:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7facc4034cb0 0x7facc4037160 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 --2- 172.21.15.2:0/3426756678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facdc101ad0 0x7facdc113f10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 >> 172.21.15.2:0/3426756678 conn(0x7facdc071980 msgr2=0x7facdc103140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 shutdown_connections 2022-03-21T14:00:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:41.566+0000 7face2203700 1 -- 172.21.15.2:0/3426756678 wait complete. 2022-03-21T14:00:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:41 smithi002 ceph-mon[17167]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:42.038 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi079 2022-03-21T14:00:42.038 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T14:00:42.039 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.conf 2022-03-21T14:00:42.056 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T14:00:42.057 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:42.114 INFO:tasks.cephadm:Adding host smithi079 to orchestrator... 2022-03-21T14:00:42.114 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch host add smithi079 2022-03-21T14:00:43.089 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:42 smithi002 ceph-mon[17167]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:00:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:43 smithi002 ceph-mon[17167]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:00:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/4114819243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 msgr2=0x7f2e08101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/4114819243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08101ee0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e040097e0 tx=0x7f2e04005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/4114819243 shutdown_connections 2022-03-21T14:00:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/4114819243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08101ee0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/4114819243 >> 172.21.15.2:0/4114819243 conn(0x7f2e080f76d0 msgr2=0x7f2e080f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/4114819243 shutdown_connections 2022-03-21T14:00:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.156+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/4114819243 wait complete. 2022-03-21T14:00:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.157+0000 7f2e0fcc1700 1 Processor -- start 2022-03-21T14:00:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.157+0000 7f2e0fcc1700 1 -- start start 2022-03-21T14:00:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.158+0000 7f2e0fcc1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.158+0000 7f2e0fcc1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08112140 con 0x7f2e08101b10 2022-03-21T14:00:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.158+0000 7f2e0da5d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.158+0000 7f2e0da5d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43532/0 (socket says 172.21.15.2:43532) 2022-03-21T14:00:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.158+0000 7f2e0da5d700 1 -- 172.21.15.2:0/1844509010 learned_addr learned my addr 172.21.15.2:0/1844509010 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2e0da5d700 1 -- 172.21.15.2:0/1844509010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e040093f0 con 0x7f2e08101b10 2022-03-21T14:00:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2e0da5d700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e040097b0 tx=0x7f2e04005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e0401c070 con 0x7f2e08101b10 2022-03-21T14:00:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2e04003df0 con 0x7f2e08101b10 2022-03-21T14:00:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e08112340 con 0x7f2e08101b10 2022-03-21T14:00:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e04021b30 con 0x7f2e08101b10 2022-03-21T14:00:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.159+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e081127d0 con 0x7f2e08101b10 2022-03-21T14:00:44.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.160+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0805e600 con 0x7f2e08101b10 2022-03-21T14:00:44.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.161+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f2e04028030 con 0x7f2e08101b10 2022-03-21T14:00:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.161+0000 7f2dfe7fc700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 0x7f2df4037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.161+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2e04047d80 con 0x7f2e08101b10 2022-03-21T14:00:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.161+0000 7f2e0d25c700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 0x7f2df4037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.162+0000 7f2e0d25c700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 0x7f2df4037110 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009780 tx=0x7f2df8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:44.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.166+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2e04003f60 con 0x7f2e08101b10 2022-03-21T14:00:44.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:44.357+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e0810bae0 con 0x7f2df4034c60 2022-03-21T14:00:45.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.268+0000 7f2dfe7fc700 1 -- 172.21.15.2:0/1844509010 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f2e0810bae0 con 0x7f2df4034c60 2022-03-21T14:00:45.271 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi079' with addr '172.21.15.79' 2022-03-21T14:00:45.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 msgr2=0x7f2df4037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:45.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 0x7f2df4037110 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009780 tx=0x7f2df8006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 msgr2=0x7f2e08111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e040097b0 tx=0x7f2e04005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 shutdown_connections 2022-03-21T14:00:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2df4034c60 0x7f2df4037110 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 --2- 172.21.15.2:0/1844509010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e08101b10 0x7f2e08111c00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 >> 172.21.15.2:0/1844509010 conn(0x7f2e080f76d0 msgr2=0x7f2e080f95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 shutdown_connections 2022-03-21T14:00:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:45.271+0000 7f2e0fcc1700 1 -- 172.21.15.2:0/1844509010 wait complete. 2022-03-21T14:00:45.733 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch host ls --format=json 2022-03-21T14:00:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: Deploying cephadm binary to smithi079 2022-03-21T14:00:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:46.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:46.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:46.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:00:46.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:46.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:47.025 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:46 smithi002 ceph-mon[17167]: Added host smithi079 2022-03-21T14:00:47.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:46 smithi002 ceph-mon[17167]: Updating smithi016:/etc/ceph/ceph.conf 2022-03-21T14:00:47.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:46 smithi002 ceph-mon[17167]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:47.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:47.241 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:47.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.721+0000 7fda13fff700 1 -- 172.21.15.2:0/2335885430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 msgr2=0x7fda14101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:47.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.721+0000 7fda13fff700 1 --2- 172.21.15.2:0/2335885430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14101d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda040097e0 tx=0x7fda04005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:47.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.721+0000 7fda13fff700 1 -- 172.21.15.2:0/2335885430 shutdown_connections 2022-03-21T14:00:47.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.721+0000 7fda13fff700 1 --2- 172.21.15.2:0/2335885430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14101d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:47.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.721+0000 7fda13fff700 1 -- 172.21.15.2:0/2335885430 >> 172.21.15.2:0/2335885430 conn(0x7fda140f74f0 msgr2=0x7fda140f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:47.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.722+0000 7fda13fff700 1 -- 172.21.15.2:0/2335885430 shutdown_connections 2022-03-21T14:00:47.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.722+0000 7fda13fff700 1 -- 172.21.15.2:0/2335885430 wait complete. 2022-03-21T14:00:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.722+0000 7fda13fff700 1 Processor -- start 2022-03-21T14:00:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda13fff700 1 -- start start 2022-03-21T14:00:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda13fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda13fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda14111fc0 con 0x7fda14101940 2022-03-21T14:00:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda12ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43536/0 (socket says 172.21.15.2:43536) 2022-03-21T14:00:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.723+0000 7fda12ffd700 1 -- 172.21.15.2:0/1458179558 learned_addr learned my addr 172.21.15.2:0/1458179558 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:47.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.724+0000 7fda12ffd700 1 -- 172.21.15.2:0/1458179558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040093f0 con 0x7fda14101940 2022-03-21T14:00:47.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.724+0000 7fda12ffd700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda040097b0 tx=0x7fda04005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:47.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.724+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0401c070 con 0x7fda14101940 2022-03-21T14:00:47.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.724+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fda04003df0 con 0x7fda14101940 2022-03-21T14:00:47.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.724+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda141121c0 con 0x7fda14101940 2022-03-21T14:00:47.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.725+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda04021b30 con 0x7fda14101940 2022-03-21T14:00:47.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.725+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda14112650 con 0x7fda14101940 2022-03-21T14:00:47.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.726+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fda04028030 con 0x7fda14101940 2022-03-21T14:00:47.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.726+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1405e600 con 0x7fda14101940 2022-03-21T14:00:47.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.726+0000 7fd9fb7fe700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 0x7fd9fc037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:47.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.726+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda04047ce0 con 0x7fda14101940 2022-03-21T14:00:47.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.727+0000 7fda127fc700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 0x7fd9fc037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:47.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.727+0000 7fda127fc700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 0x7fd9fc037110 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda08009780 tx=0x7fda08006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:47.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.731+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fda04003f60 con 0x7fda14101940 2022-03-21T14:00:47.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.924+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda1410b940 con 0x7fd9fc034c60 2022-03-21T14:00:47.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.925+0000 7fd9fb7fe700 1 -- 172.21.15.2:0/1458179558 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+237 (secure 0 0 0) 0x7fda1410b940 con 0x7fd9fc034c60 2022-03-21T14:00:47.926 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:00:47.927 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}] 2022-03-21T14:00:47.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 msgr2=0x7fd9fc037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:47.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 0x7fd9fc037110 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda08009780 tx=0x7fda08006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:47.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 msgr2=0x7fda14111a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:47.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda040097b0 tx=0x7fda04005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:47.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 shutdown_connections 2022-03-21T14:00:47.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd9fc034c60 0x7fd9fc037110 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:47.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 --2- 172.21.15.2:0/1458179558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda14101940 0x7fda14111a80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:47.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 >> 172.21.15.2:0/1458179558 conn(0x7fda140f74f0 msgr2=0x7fda140f9390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:47.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 shutdown_connections 2022-03-21T14:00:47.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:47.928+0000 7fda13fff700 1 -- 172.21.15.2:0/1458179558 wait complete. 2022-03-21T14:00:48.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:47 smithi002 ceph-mon[17167]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:48.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:48.449 INFO:tasks.cephadm:Setting crush tunables to default 2022-03-21T14:00:48.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd crush tunables default 2022-03-21T14:00:49.269 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:48 smithi002 ceph-mon[17167]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:00:49.946 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:00:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:49 smithi002 ceph-mon[17167]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:00:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 -- 172.21.15.2:0/3473932947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 msgr2=0x7f577806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:50.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 --2- 172.21.15.2:0/3473932947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f577806f6e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f57680097e0 tx=0x7f5768005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:50.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 -- 172.21.15.2:0/3473932947 shutdown_connections 2022-03-21T14:00:50.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 --2- 172.21.15.2:0/3473932947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f577806f6e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:50.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 -- 172.21.15.2:0/3473932947 >> 172.21.15.2:0/3473932947 conn(0x7f57780f76d0 msgr2=0x7f57780f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:50.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 -- 172.21.15.2:0/3473932947 shutdown_connections 2022-03-21T14:00:50.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.412+0000 7f577d0d1700 1 -- 172.21.15.2:0/3473932947 wait complete. 2022-03-21T14:00:50.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.413+0000 7f577d0d1700 1 Processor -- start 2022-03-21T14:00:50.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.413+0000 7f577d0d1700 1 -- start start 2022-03-21T14:00:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f577d0d1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f577d0d1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778114390 con 0x7f577806f310 2022-03-21T14:00:50.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f5776d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:50.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f5776d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43540/0 (socket says 172.21.15.2:43540) 2022-03-21T14:00:50.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f5776d9d700 1 -- 172.21.15.2:0/837986225 learned_addr learned my addr 172.21.15.2:0/837986225 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:00:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.414+0000 7f5776d9d700 1 -- 172.21.15.2:0/837986225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680093f0 con 0x7f577806f310 2022-03-21T14:00:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f5776d9d700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f57680097b0 tx=0x7f5768005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f576801c070 con 0x7f577806f310 2022-03-21T14:00:50.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5778114590 con 0x7f577806f310 2022-03-21T14:00:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5768003df0 con 0x7f577806f310 2022-03-21T14:00:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5768021b30 con 0x7f577806f310 2022-03-21T14:00:50.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.415+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57781149a0 con 0x7f577806f310 2022-03-21T14:00:50.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.416+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577805e600 con 0x7f577806f310 2022-03-21T14:00:50.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.416+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f5768028030 con 0x7f577806f310 2022-03-21T14:00:50.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.417+0000 7f575f7fe700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 0x7f5760037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:50.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.417+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5768047d40 con 0x7f577806f310 2022-03-21T14:00:50.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.417+0000 7f577659c700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 0x7f5760037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:50.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.418+0000 7f577659c700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 0x7f5760037110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f576c009780 tx=0x7f576c006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:50.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.422+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f5768003f60 con 0x7f577806f310 2022-03-21T14:00:50.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.616+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f577810f540 con 0x7f577806f310 2022-03-21T14:00:50.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.983+0000 7f575f7fe700 1 -- 172.21.15.2:0/837986225 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5768026030 con 0x7f577806f310 2022-03-21T14:00:50.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 msgr2=0x7f5760037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:50.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 0x7f5760037110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f576c009780 tx=0x7f576c006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 msgr2=0x7f5778113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f57680097b0 tx=0x7f5768005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 shutdown_connections 2022-03-21T14:00:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5760034c60 0x7f5760037110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:50.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 --2- 172.21.15.2:0/837986225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f577806f310 0x7f5778113e50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:50.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 >> 172.21.15.2:0/837986225 conn(0x7f57780f76d0 msgr2=0x7f57781026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:50.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 shutdown_connections 2022-03-21T14:00:50.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:00:50.986+0000 7f577d0d1700 1 -- 172.21.15.2:0/837986225 wait complete. 2022-03-21T14:00:50.990 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2022-03-21T14:00:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:50 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-03-21T14:00:51.452 INFO:tasks.cephadm:Adding mon.a on smithi002 2022-03-21T14:00:51.452 INFO:tasks.cephadm:Adding mon.b on smithi016 2022-03-21T14:00:51.453 INFO:tasks.cephadm:Adding mon.c on smithi079 2022-03-21T14:00:51.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c' 2022-03-21T14:00:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:51 smithi002 ceph-mon[17167]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:51 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-03-21T14:00:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:51 smithi002 ceph-mon[17167]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:00:52.934 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.931+0000 7fb80e23b700 1 -- 172.21.15.79:0/3102210687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 msgr2=0x7fb8080fe6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:52.935 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.931+0000 7fb80e23b700 1 --2- 172.21.15.79:0/3102210687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb8080fe6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80097e0 tx=0x7fb7f8005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:52.935 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 -- 172.21.15.79:0/3102210687 shutdown_connections 2022-03-21T14:00:52.935 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 --2- 172.21.15.79:0/3102210687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb8080fe6f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:52.935 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 -- 172.21.15.79:0/3102210687 >> 172.21.15.79:0/3102210687 conn(0x7fb8080f60e0 msgr2=0x7fb8080f8510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:52.936 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 -- 172.21.15.79:0/3102210687 shutdown_connections 2022-03-21T14:00:52.936 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 -- 172.21.15.79:0/3102210687 wait complete. 2022-03-21T14:00:52.936 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 Processor -- start 2022-03-21T14:00:52.936 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.932+0000 7fb80e23b700 1 -- start start 2022-03-21T14:00:52.937 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.933+0000 7fb80e23b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:52.937 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.933+0000 7fb80e23b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80810e2d0 con 0x7fb808064cf0 2022-03-21T14:00:52.937 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.933+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:52.937 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.933+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.79:47868/0 (socket says 172.21.15.79:47868) 2022-03-21T14:00:52.938 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.933+0000 7fb8077fe700 1 -- 172.21.15.79:0/2863520300 learned_addr learned my addr 172.21.15.79:0/2863520300 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:00:52.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb8077fe700 1 -- 172.21.15.79:0/2863520300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80093f0 con 0x7fb808064cf0 2022-03-21T14:00:52.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb8077fe700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004560 tx=0x7fb7f8005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:52.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7f801c070 con 0x7fb808064cf0 2022-03-21T14:00:52.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80810e4d0 con 0x7fb808064cf0 2022-03-21T14:00:52.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb7f8003ec0 con 0x7fb808064cf0 2022-03-21T14:00:52.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7f8021bf0 con 0x7fb808064cf0 2022-03-21T14:00:52.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.934+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80810f0c0 con 0x7fb808064cf0 2022-03-21T14:00:52.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.935+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80805e600 con 0x7fb808064cf0 2022-03-21T14:00:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.935+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fb7f8028030 con 0x7fb808064cf0 2022-03-21T14:00:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.935+0000 7fb7fffff700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 0x7fb7f4037110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.935+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb7f8047e30 con 0x7fb808064cf0 2022-03-21T14:00:52.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.936+0000 7fb806ffd700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 0x7fb7f4037110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.936+0000 7fb806ffd700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 0x7fb7f4037110 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009780 tx=0x7fb7f0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:52.943 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:52.940+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fb7f802a590 con 0x7fb808064cf0 2022-03-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.114+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fb80810fd50 con 0x7fb7f4034c60 2022-03-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.122+0000 7fb7fffff700 1 -- 172.21.15.79:0/2863520300 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb80810fd50 con 0x7fb7f4034c60 2022-03-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled mon update... 2022-03-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 msgr2=0x7fb7f4037110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 0x7fb7f4037110 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009780 tx=0x7fb7f0006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 msgr2=0x7fb80810dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004560 tx=0x7fb7f8005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 shutdown_connections 2022-03-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb7f4034c60 0x7fb7f4037110 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 --2- 172.21.15.79:0/2863520300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb808064cf0 0x7fb80810dd90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 >> 172.21.15.79:0/2863520300 conn(0x7fb8080f60e0 msgr2=0x7fb8080f6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:53.127 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 shutdown_connections 2022-03-21T14:00:53.127 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:53.123+0000 7fb80e23b700 1 -- 172.21.15.79:0/2863520300 wait complete. 2022-03-21T14:00:53.601 DEBUG:teuthology.orchestra.run.smithi016:mon.b> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.b.service 2022-03-21T14:00:53.604 DEBUG:teuthology.orchestra.run.smithi079:mon.c> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mon.c.service 2022-03-21T14:00:53.607 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-03-21T14:00:53.607 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-03-21T14:00:53.630 INFO:journalctl@ceph.mon.c.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:00:53.632 INFO:journalctl@ceph.mon.b.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:00:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:54.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:00:54.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:54.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:54.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:00:54.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:55.258 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 -- 172.21.15.79:0/893862393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 msgr2=0x7f1f1c101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:55.259 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 --2- 172.21.15.79:0/893862393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c101ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0097e0 tx=0x7f1f0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:55.259 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 -- 172.21.15.79:0/893862393 shutdown_connections 2022-03-21T14:00:55.259 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 --2- 172.21.15.79:0/893862393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c101ee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:55.259 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 -- 172.21.15.79:0/893862393 >> 172.21.15.79:0/893862393 conn(0x7f1f1c0f76d0 msgr2=0x7f1f1c0f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:55.260 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 -- 172.21.15.79:0/893862393 shutdown_connections 2022-03-21T14:00:55.260 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 -- 172.21.15.79:0/893862393 wait complete. 2022-03-21T14:00:55.262 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.257+0000 7f1f21673700 1 Processor -- start 2022-03-21T14:00:55.263 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f21673700 1 -- start start 2022-03-21T14:00:55.263 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f21673700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:55.263 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f21673700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c06f810 con 0x7f1f1c101b10 2022-03-21T14:00:55.264 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f1affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:55.264 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f1affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.79:47874/0 (socket says 172.21.15.79:47874) 2022-03-21T14:00:55.264 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f1affd700 1 -- 172.21.15.79:0/1894754687 learned_addr learned my addr 172.21.15.79:0/1894754687 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:00:55.264 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.258+0000 7f1f1affd700 1 -- 172.21.15.79:0/1894754687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0093f0 con 0x7f1f1c101b10 2022-03-21T14:00:55.265 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f1affd700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009f50 tx=0x7f1f0c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:55.265 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c01c070 con 0x7f1f1c101b10 2022-03-21T14:00:55.265 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1f0c003e40 con 0x7f1f1c101b10 2022-03-21T14:00:55.265 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c021ba0 con 0x7f1f1c101b10 2022-03-21T14:00:55.266 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c06fa10 con 0x7f1f1c101b10 2022-03-21T14:00:55.266 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.259+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c06fea0 con 0x7f1f1c101b10 2022-03-21T14:00:55.266 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.260+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c0700e0 con 0x7f1f1c101b10 2022-03-21T14:00:55.266 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.260+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f1f0c028030 con 0x7f1f1c101b10 2022-03-21T14:00:55.267 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.260+0000 7f1f037fe700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 0x7f1f04037160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:00:55.267 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.260+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f0c047d10 con 0x7f1f1c101b10 2022-03-21T14:00:55.267 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.260+0000 7f1f1a7fc700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 0x7f1f04037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:00:55.267 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.261+0000 7f1f1a7fc700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 0x7f1f04037160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009780 tx=0x7f1f10006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:00:55.268 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.266+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f1f0c003fb0 con 0x7f1f1c101b10 2022-03-21T14:00:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:54 smithi002 ceph-mon[17167]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:54 smithi002 ceph-mon[17167]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c;count:3 2022-03-21T14:00:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:54 smithi002 ceph-mon[17167]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:00:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:54 smithi002 ceph-mon[17167]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:54 smithi002 ceph-mon[17167]: Deploying daemon mon.b on smithi016 2022-03-21T14:00:55.499 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.498+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f1c05e600 con 0x7f1f1c101b10 2022-03-21T14:00:55.500 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.500+0000 7f1f037fe700 1 -- 172.21.15.79:0/1894754687 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+706 (secure 0 0 0) 0x7f1f0c026020 con 0x7f1f1c101b10 2022-03-21T14:00:55.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T14:00:55.501 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":1,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","modified":"2022-03-21T13:59:14.094890Z","created":"2022-03-21T13:59:14.094890Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-03-21T14:00:55.502 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 msgr2=0x7f1f04037160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:55.503 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 0x7f1f04037160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009780 tx=0x7f1f10006cc0 comp rx=0 tx=0).stop 2022-03-21T14:00:55.503 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 msgr2=0x7f1f1c06f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:00:55.503 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009f50 tx=0x7f1f0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:00:55.503 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 shutdown_connections 2022-03-21T14:00:55.504 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1f04034cb0 0x7f1f04037160 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:55.504 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 --2- 172.21.15.79:0/1894754687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f1c101b10 0x7f1f1c06f2d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:00:55.504 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 >> 172.21.15.79:0/1894754687 conn(0x7f1f1c0f76d0 msgr2=0x7f1f1c0f8310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:00:55.504 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 shutdown_connections 2022-03-21T14:00:55.505 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:00:55.501+0000 7f1f21673700 1 -- 172.21.15.79:0/1894754687 wait complete. 2022-03-21T14:00:55.505 INFO:teuthology.orchestra.run.smithi079.stderr:dumped monmap epoch 1 2022-03-21T14:00:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 systemd[1]: Starting Ceph mon.b for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:00:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:55 smithi002 ceph-mon[17167]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:00:55 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1894754687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:00:56.955 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-03-21T14:00:56.956 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-03-21T14:00:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: set uid:gid to 167:167 (ceph:ceph) 2022-03-21T14:00:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev), process ceph-mon, pid 7 2022-03-21T14:00:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: pidfile_write: ignore empty --pid-file 2022-03-21T14:00:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: load: jerasure load: lrc 2022-03-21T14:00:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: RocksDB version: 6.15.5 2022-03-21T14:00:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-03-21T14:00:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Compile date Mar 19 2022 2022-03-21T14:00:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: DB SUMMARY 2022-03-21T14:00:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: DB Session ID: IIM9L2KD1BTPDN1ZK9G7 2022-03-21T14:00:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: CURRENT file: CURRENT 2022-03-21T14:00:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: IDENTITY file: IDENTITY 2022-03-21T14:00:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-03-21T14:00:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-03-21T14:00:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2022-03-21T14:00:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.error_if_exists: 0 2022-03-21T14:00:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.create_if_missing: 0 2022-03-21T14:00:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.paranoid_checks: 1 2022-03-21T14:00:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-03-21T14:00:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.env: 0x55d299ffb860 2022-03-21T14:00:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.fs: Posix File System 2022-03-21T14:00:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.info_log: 0x55d29c03be60 2022-03-21T14:00:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_file_opening_threads: 16 2022-03-21T14:00:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.statistics: (nil) 2022-03-21T14:00:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.use_fsync: 0 2022-03-21T14:00:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_log_file_size: 0 2022-03-21T14:00:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-03-21T14:00:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.log_file_time_to_roll: 0 2022-03-21T14:00:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.keep_log_file_num: 1000 2022-03-21T14:00:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.recycle_log_file_num: 0 2022-03-21T14:00:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_fallocate: 1 2022-03-21T14:00:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_mmap_reads: 0 2022-03-21T14:00:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_mmap_writes: 0 2022-03-21T14:00:57.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.use_direct_reads: 0 2022-03-21T14:00:57.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-03-21T14:00:57.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.create_missing_column_families: 0 2022-03-21T14:00:57.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.db_log_dir: 2022-03-21T14:00:57.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-03-21T14:00:57.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.table_cache_numshardbits: 6 2022-03-21T14:00:57.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.WAL_ttl_seconds: 0 2022-03-21T14:00:57.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.WAL_size_limit_MB: 0 2022-03-21T14:00:57.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-03-21T14:00:57.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-03-21T14:00:57.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.is_fd_close_on_exec: 1 2022-03-21T14:00:57.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.advise_random_on_open: 1 2022-03-21T14:00:57.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.db_write_buffer_size: 0 2022-03-21T14:00:57.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.write_buffer_manager: 0x55d29c108420 2022-03-21T14:00:57.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-03-21T14:00:57.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-03-21T14:00:57.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-03-21T14:00:57.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.use_adaptive_mutex: 0 2022-03-21T14:00:57.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.rate_limiter: (nil) 2022-03-21T14:00:57.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-03-21T14:00:57.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.wal_recovery_mode: 2 2022-03-21T14:00:57.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.enable_thread_tracking: 0 2022-03-21T14:00:57.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.enable_pipelined_write: 0 2022-03-21T14:00:57.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.unordered_write: 0 2022-03-21T14:00:57.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-03-21T14:00:57.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-03-21T14:00:57.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-03-21T14:00:57.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-03-21T14:00:57.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.row_cache: None 2022-03-21T14:00:57.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.wal_filter: None 2022-03-21T14:00:57.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-03-21T14:00:57.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_ingest_behind: 0 2022-03-21T14:00:57.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.preserve_deletes: 0 2022-03-21T14:00:57.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.two_write_queues: 0 2022-03-21T14:00:57.182 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.manual_wal_flush: 0 2022-03-21T14:00:57.182 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.atomic_flush: 0 2022-03-21T14:00:57.182 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-03-21T14:00:57.182 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.persist_stats_to_disk: 0 2022-03-21T14:00:57.183 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.write_dbid_to_manifest: 0 2022-03-21T14:00:57.183 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.log_readahead_size: 0 2022-03-21T14:00:57.183 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-03-21T14:00:57.183 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.best_efforts_recovery: 0 2022-03-21T14:00:57.184 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-03-21T14:00:57.184 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-03-21T14:00:57.184 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.allow_data_in_errors: 0 2022-03-21T14:00:57.184 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.db_host_id: __hostname__ 2022-03-21T14:00:57.185 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_background_jobs: 2 2022-03-21T14:00:57.185 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_background_compactions: -1 2022-03-21T14:00:57.185 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_subcompactions: 1 2022-03-21T14:00:57.185 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-03-21T14:00:57.186 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-03-21T14:00:57.186 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.delayed_write_rate : 16777216 2022-03-21T14:00:57.186 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_total_wal_size: 0 2022-03-21T14:00:57.186 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-03-21T14:00:57.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.stats_dump_period_sec: 600 2022-03-21T14:00:57.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.stats_persist_period_sec: 600 2022-03-21T14:00:57.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-03-21T14:00:57.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_open_files: -1 2022-03-21T14:00:57.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bytes_per_sync: 0 2022-03-21T14:00:57.188 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.wal_bytes_per_sync: 0 2022-03-21T14:00:57.188 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.strict_bytes_per_sync: 0 2022-03-21T14:00:57.188 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_readahead_size: 0 2022-03-21T14:00:57.188 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_background_flushes: -1 2022-03-21T14:00:57.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Compression algorithms supported: 2022-03-21T14:00:57.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-03-21T14:00:57.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kZSTD supported: 0 2022-03-21T14:00:57.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kXpressCompression supported: 0 2022-03-21T14:00:57.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kLZ4HCCompression supported: 1 2022-03-21T14:00:57.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kLZ4Compression supported: 1 2022-03-21T14:00:57.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kBZip2Compression supported: 0 2022-03-21T14:00:57.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kZlibCompression supported: 1 2022-03-21T14:00:57.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: kSnappyCompression supported: 1 2022-03-21T14:00:57.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Fast CRC32 supported: Supported on x86 2022-03-21T14:00:57.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2022-03-21T14:00:57.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-03-21T14:00:57.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-03-21T14:00:57.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.merge_operator: 2022-03-21T14:00:57.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_filter: None 2022-03-21T14:00:57.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_filter_factory: None 2022-03-21T14:00:57.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.sst_partitioner_factory: None 2022-03-21T14:00:57.193 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.memtable_factory: SkipListFactory 2022-03-21T14:00:57.193 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.table_factory: BlockBasedTable 2022-03-21T14:00:57.193 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d29c007cc0) 2022-03-21T14:00:57.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: cache_index_and_filter_blocks: 1 2022-03-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-03-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-03-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: pin_top_level_index_and_filter: 1 2022-03-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_type: 0 2022-03-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: data_block_index_type: 0 2022-03-21T14:00:57.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_shortening: 1 2022-03-21T14:00:57.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2022-03-21T14:00:57.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: hash_index_allow_collision: 1 2022-03-21T14:00:57.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: checksum: 1 2022-03-21T14:00:57.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: no_block_cache: 0 2022-03-21T14:00:57.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache: 0x55d29c072170 2022-03-21T14:00:57.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_name: BinnedLRUCache 2022-03-21T14:00:57.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_options: 2022-03-21T14:00:57.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: capacity : 536870912 2022-03-21T14:00:57.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: num_shard_bits : 4 2022-03-21T14:00:57.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: strict_capacity_limit : 0 2022-03-21T14:00:57.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: high_pri_pool_ratio: 0.000 2022-03-21T14:00:57.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_compressed: (nil) 2022-03-21T14:00:57.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: persistent_cache: (nil) 2022-03-21T14:00:57.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_size: 4096 2022-03-21T14:00:57.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_size_deviation: 10 2022-03-21T14:00:57.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_restart_interval: 16 2022-03-21T14:00:57.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_block_restart_interval: 1 2022-03-21T14:00:57.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: metadata_block_size: 4096 2022-03-21T14:00:57.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: partition_filters: 0 2022-03-21T14:00:57.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: use_delta_encoding: 1 2022-03-21T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-03-21T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: whole_key_filtering: 1 2022-03-21T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: verify_compression: 0 2022-03-21T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: read_amp_bytes_per_bit: 0 2022-03-21T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: format_version: 4 2022-03-21T14:00:57.201 INFO:journalctl@ceph.mon.b.smithi016.stdout: enable_index_compression: 1 2022-03-21T14:00:57.201 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_align: 0 2022-03-21T14:00:57.201 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.write_buffer_size: 33554432 2022-03-21T14:00:57.201 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_write_buffer_number: 2 2022-03-21T14:00:57.202 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression: NoCompression 2022-03-21T14:00:57.202 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression: Disabled 2022-03-21T14:00:57.202 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.prefix_extractor: nullptr 2022-03-21T14:00:57.203 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-03-21T14:00:57.203 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.num_levels: 7 2022-03-21T14:00:57.203 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-03-21T14:00:57.203 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-03-21T14:00:57.203 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-03-21T14:00:57.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-03-21T14:00:57.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-03-21T14:00:57.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-03-21T14:00:57.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-03-21T14:00:57.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-03-21T14:00:57.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-03-21T14:00:57.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-03-21T14:00:57.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.window_bits: -14 2022-03-21T14:00:57.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.level: 32767 2022-03-21T14:00:57.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.strategy: 0 2022-03-21T14:00:57.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-03-21T14:00:57.207 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-03-21T14:00:57.207 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-03-21T14:00:57.207 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compression_opts.enabled: false 2022-03-21T14:00:57.207 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-03-21T14:00:57.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-03-21T14:00:57.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-03-21T14:00:57.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.target_file_size_base: 67108864 2022-03-21T14:00:57.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.target_file_size_multiplier: 1 2022-03-21T14:00:57.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-03-21T14:00:57.209 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-03-21T14:00:57.209 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-03-21T14:00:57.209 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-03-21T14:00:57.209 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-03-21T14:00:57.210 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-03-21T14:00:57.210 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-03-21T14:00:57.210 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-03-21T14:00:57.210 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-03-21T14:00:57.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-03-21T14:00:57.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-03-21T14:00:57.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-03-21T14:00:57.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.arena_block_size: 4194304 2022-03-21T14:00:57.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-03-21T14:00:57.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-03-21T14:00:57.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-03-21T14:00:57.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.disable_auto_compactions: 0 2022-03-21T14:00:57.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-03-21T14:00:57.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-03-21T14:00:57.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-03-21T14:00:57.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-03-21T14:00:57.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-03-21T14:00:57.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-03-21T14:00:57.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-03-21T14:00:57.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-03-21T14:00:57.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-03-21T14:00:57.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-03-21T14:00:57.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.table_properties_collectors: 2022-03-21T14:00:57.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.inplace_update_support: 0 2022-03-21T14:00:57.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.inplace_update_num_locks: 10000 2022-03-21T14:00:57.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-03-21T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-03-21T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.memtable_huge_page_size: 0 2022-03-21T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.bloom_locality: 0 2022-03-21T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.max_successive_merges: 0 2022-03-21T14:00:57.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.optimize_filters_for_hits: 0 2022-03-21T14:00:57.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.paranoid_file_checks: 0 2022-03-21T14:00:57.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.force_consistency_checks: 1 2022-03-21T14:00:57.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.report_bg_io_stats: 0 2022-03-21T14:00:57.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.ttl: 2592000 2022-03-21T14:00:57.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.periodic_compaction_seconds: 0 2022-03-21T14:00:57.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.enable_blob_files: false 2022-03-21T14:00:57.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.min_blob_size: 0 2022-03-21T14:00:57.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.blob_file_size: 268435456 2022-03-21T14:00:57.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.blob_compression_type: NoCompression 2022-03-21T14:00:57.219 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.enable_blob_garbage_collection: false 2022-03-21T14:00:57.219 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-03-21T14:00:57.219 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-03-21T14:00:57.219 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2022-03-21T14:00:57.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2022-03-21T14:00:57.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871256717452, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-03-21T14:00:57.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2022-03-21T14:00:57.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-03-21T14:00:57.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871256717740, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1647871256, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e5e5eff2-f7bf-47a2-997b-54e61a96d8d8", "db_session_id": "IIM9L2KD1BTPDN1ZK9G7"}} 2022-03-21T14:00:57.221 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2022-03-21T14:00:57.221 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871256718074, "job": 1, "event": "recovery_finished"} 2022-03-21T14:00:57.221 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-03-21T14:00:57.221 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d29c058a80 2022-03-21T14:00:57.222 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: DB pointer 0x55d29c068000 2022-03-21T14:00:57.222 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-03-21T14:00:57.222 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-03-21T14:00:57.222 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** DB Stats ** 2022-03-21T14:00:57.223 INFO:journalctl@ceph.mon.b.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:00:57.223 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-03-21T14:00:57.223 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-03-21T14:00:57.223 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T14:00:57.224 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-03-21T14:00:57.224 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-03-21T14:00:57.224 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T14:00:57.224 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.224 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2022-03-21T14:00:57.225 INFO:journalctl@ceph.mon.b.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:00:57.225 INFO:journalctl@ceph.mon.b.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:00:57.225 INFO:journalctl@ceph.mon.b.smithi016.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.225 INFO:journalctl@ceph.mon.b.smithi016.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.226 INFO:journalctl@ceph.mon.b.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.226 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.226 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2022-03-21T14:00:57.226 INFO:journalctl@ceph.mon.b.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:00:57.227 INFO:journalctl@ceph.mon.b.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:00:57.227 INFO:journalctl@ceph.mon.b.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.227 INFO:journalctl@ceph.mon.b.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:00:57.227 INFO:journalctl@ceph.mon.b.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T14:00:57.227 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T14:00:57.228 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T14:00:57.228 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T14:00:57.228 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T14:00:57.228 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:00:57.229 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:00:57.229 INFO:journalctl@ceph.mon.b.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T14:00:57.229 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.229 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T14:00:57.229 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.230 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2022-03-21T14:00:57.230 INFO:journalctl@ceph.mon.b.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:00:57.230 INFO:journalctl@ceph.mon.b.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:00:57.230 INFO:journalctl@ceph.mon.b.smithi016.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.231 INFO:journalctl@ceph.mon.b.smithi016.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.231 INFO:journalctl@ceph.mon.b.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-03-21T14:00:57.231 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.231 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2022-03-21T14:00:57.231 INFO:journalctl@ceph.mon.b.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:00:57.232 INFO:journalctl@ceph.mon.b.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:00:57.232 INFO:journalctl@ceph.mon.b.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:00:57.232 INFO:journalctl@ceph.mon.b.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:00:57.233 INFO:journalctl@ceph.mon.b.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T14:00:57.233 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T14:00:57.233 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T14:00:57.233 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T14:00:57.233 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T14:00:57.234 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:00:57.234 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:00:57.234 INFO:journalctl@ceph.mon.b.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T14:00:57.234 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.235 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T14:00:57.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b does not exist in monmap, will attempt to join an existing cluster 2022-03-21T14:00:57.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: using public_addr v2:172.21.15.16:0/0 -> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] 2022-03-21T14:00:57.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: starting mon.b rank -1 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:57.236 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(???) e0 preinit fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:00:57.236 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).mds e1 new map 2022-03-21T14:00:57.236 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).mds e1 print_map 2022-03-21T14:00:57.236 INFO:journalctl@ceph.mon.b.smithi016.stdout: e1 2022-03-21T14:00:57.236 INFO:journalctl@ceph.mon.b.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-03-21T14:00:57.237 INFO:journalctl@ceph.mon.b.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-03-21T14:00:57.237 INFO:journalctl@ceph.mon.b.smithi016.stdout: legacy client fscid: -1 2022-03-21T14:00:57.237 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T14:00:57.237 INFO:journalctl@ceph.mon.b.smithi016.stdout: No filesystems configured 2022-03-21T14:00:57.238 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-03-21T14:00:57.238 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-03-21T14:00:57.238 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-03-21T14:00:57.238 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-03-21T14:00:57.239 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-03-21T14:00:57.239 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-03-21T14:00:57.239 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-03-21T14:00:57.239 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:00:57.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:00:57.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:00:57.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:00:57.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:57.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:00:57.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-03-21T14:00:57.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:57.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-03-21T14:00:57.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:00:57.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:57.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:00:57.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:57.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:00:57.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:00:57.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:00:57.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:00:57.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:00:57.245 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c;count:3 2022-03-21T14:00:57.245 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:00:57.245 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:00:57.245 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: Deploying daemon mon.b on smithi016 2022-03-21T14:00:57.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:00:57.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1894754687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:00:57.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-03-21T14:00:57.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 ceph-mon[22010]: mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2022-03-21T14:00:57.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 bash[21802]: 263ca86d67cf72b6c6f08a8a229b225800035f60db46ba1f3a6d4242b7c0364c 2022-03-21T14:00:57.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:00:56 smithi016 systemd[1]: Started Ceph mon.b for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:01:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: mon.a calling monitor election 2022-03-21T14:01:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:01.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: mon.b calling monitor election 2022-03-21T14:01:01.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:01.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:01.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:01.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2022-03-21T14:01:01.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: fsmap 2022-03-21T14:01:01.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:01.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: mgrmap e13: x(active, since 41s) 2022-03-21T14:01:01.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: overall HEALTH_OK 2022-03-21T14:01:01.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:01.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:01.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:02.158 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 -- 172.21.15.79:0/2634586342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 msgr2=0x7fa058005880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:02.158 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 --2- 172.21.15.79:0/2634586342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa058005880 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0fcb60 tx=0x7fa05c0136b0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.159 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 -- 172.21.15.79:0/2634586342 shutdown_connections 2022-03-21T14:01:02.159 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 --2- 172.21.15.79:0/2634586342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa058005880 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.159 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 -- 172.21.15.79:0/2634586342 >> 172.21.15.79:0/2634586342 conn(0x7fa06c0f76d0 msgr2=0x7fa06c0f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:02.160 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 -- 172.21.15.79:0/2634586342 shutdown_connections 2022-03-21T14:01:02.160 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.156+0000 7fa071e63700 1 -- 172.21.15.79:0/2634586342 wait complete. 2022-03-21T14:01:02.163 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 Processor -- start 2022-03-21T14:01:02.163 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 -- start start 2022-03-21T14:01:02.163 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa06c111dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:02.164 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:02.164 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c112920 con 0x7fa058003400 2022-03-21T14:01:02.164 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.157+0000 7fa071e63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c10c220 con 0x7fa06c112310 2022-03-21T14:01:02.164 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:02.165 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.79:41638/0 (socket says 172.21.15.79:41638) 2022-03-21T14:01:02.165 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 -- 172.21.15.79:0/1568264651 learned_addr learned my addr 172.21.15.79:0/1568264651 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:01:02.165 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06b7fe700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa06c111dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:02.165 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 -- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 msgr2=0x7fa06c10bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-03-21T14:01:02.166 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 -- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 msgr2=0x7fa06c10bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-03-21T14:01:02.166 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-03-21T14:01:02.166 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06affd700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T14:01:02.166 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06b7fe700 1 -- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 msgr2=0x7fa06c10bce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T14:01:02.167 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06b7fe700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.167 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06b7fe700 1 -- 172.21.15.79:0/1568264651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c013950 con 0x7fa058003400 2022-03-21T14:01:02.167 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.158+0000 7fa06b7fe700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa06c111dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa05c005fd0 tx=0x7fa05c0136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:02.167 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.159+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fa05c018070 con 0x7fa058003400 2022-03-21T14:01:02.168 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.159+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c10c4a0 con 0x7fa058003400 2022-03-21T14:01:02.168 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.159+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa05c01d880 con 0x7fa058003400 2022-03-21T14:01:02.168 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.159+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fa05c011e60 con 0x7fa058003400 2022-03-21T14:01:02.168 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.159+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c10c9e0 con 0x7fa058003400 2022-03-21T14:01:02.169 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.160+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fa05c02d030 con 0x7fa058003400 2022-03-21T14:01:02.169 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.160+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c05e600 con 0x7fa058003400 2022-03-21T14:01:02.169 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.160+0000 7fa063fff700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 0x7fa04c0372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:02.169 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.160+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa05c049080 con 0x7fa058003400 2022-03-21T14:01:02.170 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.160+0000 7fa06affd700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 0x7fa04c0372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:02.170 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.161+0000 7fa06affd700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 0x7fa04c0372a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa054006fd0 tx=0x7fa054008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: mon.a calling monitor election 2022-03-21T14:01:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: mon.b calling monitor election 2022-03-21T14:01:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2022-03-21T14:01:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: fsmap 2022-03-21T14:01:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: mgrmap e13: x(active, since 41s) 2022-03-21T14:01:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: overall HEALTH_OK 2022-03-21T14:01:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:02.176 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.165+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fa05c02fd80 con 0x7fa058003400 2022-03-21T14:01:02.424 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.423+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa06c1036f0 con 0x7fa058003400 2022-03-21T14:01:02.425 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.423+0000 7fa063fff700 1 -- 172.21.15.79:0/1568264651 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+972 (secure 0 0 0) 0x7fa05c02b030 con 0x7fa058003400 2022-03-21T14:01:02.425 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T14:01:02.425 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":2,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","modified":"2022-03-21T14:00:56.741268Z","created":"2022-03-21T13:59:14.094890Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-03-21T14:01:02.427 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 msgr2=0x7fa04c0372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:02.427 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 0x7fa04c0372a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa054006fd0 tx=0x7fa054008040 comp rx=0 tx=0).stop 2022-03-21T14:01:02.427 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 msgr2=0x7fa06c111dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:02.428 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa06c111dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa05c005fd0 tx=0x7fa05c0136b0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.428 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 shutdown_connections 2022-03-21T14:01:02.428 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa04c034df0 0x7fa04c0372a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.428 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa058003400 0x7fa06c111dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.429 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.425+0000 7fa071e63700 1 --2- 172.21.15.79:0/1568264651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06c112310 0x7fa06c10bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:02.429 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.426+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 >> 172.21.15.79:0/1568264651 conn(0x7fa06c0f76d0 msgr2=0x7fa06c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:02.429 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.426+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 shutdown_connections 2022-03-21T14:01:02.429 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:02.426+0000 7fa071e63700 1 -- 172.21.15.79:0/1568264651 wait complete. 2022-03-21T14:01:02.430 INFO:teuthology.orchestra.run.smithi079.stderr:dumped monmap epoch 2 2022-03-21T14:01:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:02 smithi002 ceph-mon[17167]: Deploying daemon mon.c on smithi079 2022-03-21T14:01:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:02 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1568264651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:02 smithi016 ceph-mon[22010]: Deploying daemon mon.c on smithi079 2022-03-21T14:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:02 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1568264651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:03 smithi002 ceph-mon[17167]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:03 smithi016 ceph-mon[22010]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:04.207 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-03-21T14:01:04.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-03-21T14:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:04 smithi079 systemd[1]: Starting Ceph mon.c for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:01:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: set uid:gid to 167:167 (ceph:ceph) 2022-03-21T14:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: ceph version 17.0.0-11208-gd3799b8c (d3799b8cfa5046465710eadeea2bed7bb4681409) quincy (dev), process ceph-mon, pid 7 2022-03-21T14:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pidfile_write: ignore empty --pid-file 2022-03-21T14:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: load: jerasure load: lrc 2022-03-21T14:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: RocksDB version: 6.15.5 2022-03-21T14:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-03-21T14:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Compile date Mar 19 2022 2022-03-21T14:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: DB SUMMARY 2022-03-21T14:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: DB Session ID: EDJS2X6YBR9N27MPLMJW 2022-03-21T14:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: CURRENT file: CURRENT 2022-03-21T14:01:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: IDENTITY file: IDENTITY 2022-03-21T14:01:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-03-21T14:01:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2022-03-21T14:01:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2022-03-21T14:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.error_if_exists: 0 2022-03-21T14:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.create_if_missing: 0 2022-03-21T14:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.paranoid_checks: 1 2022-03-21T14:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-03-21T14:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.env: 0x56396edbd860 2022-03-21T14:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.fs: Posix File System 2022-03-21T14:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.info_log: 0x563970b8fe60 2022-03-21T14:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_file_opening_threads: 16 2022-03-21T14:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.statistics: (nil) 2022-03-21T14:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.use_fsync: 0 2022-03-21T14:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_log_file_size: 0 2022-03-21T14:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-03-21T14:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 bash[22844]: e4669b1f1575ce6e25b426ffe65488d3775986829425329c79088e0b18d566f7 2022-03-21T14:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.log_file_time_to_roll: 0 2022-03-21T14:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.keep_log_file_num: 1000 2022-03-21T14:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.recycle_log_file_num: 0 2022-03-21T14:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_fallocate: 1 2022-03-21T14:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_mmap_reads: 0 2022-03-21T14:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_mmap_writes: 0 2022-03-21T14:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.use_direct_reads: 0 2022-03-21T14:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-03-21T14:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.create_missing_column_families: 0 2022-03-21T14:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.db_log_dir: 2022-03-21T14:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2022-03-21T14:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.table_cache_numshardbits: 6 2022-03-21T14:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.WAL_ttl_seconds: 0 2022-03-21T14:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.WAL_size_limit_MB: 0 2022-03-21T14:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-03-21T14:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-03-21T14:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.is_fd_close_on_exec: 1 2022-03-21T14:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.advise_random_on_open: 1 2022-03-21T14:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.db_write_buffer_size: 0 2022-03-21T14:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.write_buffer_manager: 0x563970c803c0 2022-03-21T14:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-03-21T14:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-03-21T14:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-03-21T14:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.use_adaptive_mutex: 0 2022-03-21T14:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.rate_limiter: (nil) 2022-03-21T14:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-03-21T14:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.wal_recovery_mode: 2 2022-03-21T14:01:05.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.enable_thread_tracking: 0 2022-03-21T14:01:05.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.enable_pipelined_write: 0 2022-03-21T14:01:05.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.unordered_write: 0 2022-03-21T14:01:05.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-03-21T14:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-03-21T14:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-03-21T14:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-03-21T14:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.row_cache: None 2022-03-21T14:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.wal_filter: None 2022-03-21T14:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-03-21T14:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_ingest_behind: 0 2022-03-21T14:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.preserve_deletes: 0 2022-03-21T14:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.two_write_queues: 0 2022-03-21T14:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.manual_wal_flush: 0 2022-03-21T14:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.atomic_flush: 0 2022-03-21T14:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-03-21T14:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.persist_stats_to_disk: 0 2022-03-21T14:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.write_dbid_to_manifest: 0 2022-03-21T14:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.log_readahead_size: 0 2022-03-21T14:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-03-21T14:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.best_efforts_recovery: 0 2022-03-21T14:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-03-21T14:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-03-21T14:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.allow_data_in_errors: 0 2022-03-21T14:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.db_host_id: __hostname__ 2022-03-21T14:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_background_jobs: 2 2022-03-21T14:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_background_compactions: -1 2022-03-21T14:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_subcompactions: 1 2022-03-21T14:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-03-21T14:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-03-21T14:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.delayed_write_rate : 16777216 2022-03-21T14:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_total_wal_size: 0 2022-03-21T14:01:05.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-03-21T14:01:05.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.stats_dump_period_sec: 600 2022-03-21T14:01:05.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.stats_persist_period_sec: 600 2022-03-21T14:01:05.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-03-21T14:01:05.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_open_files: -1 2022-03-21T14:01:05.745 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bytes_per_sync: 0 2022-03-21T14:01:05.745 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.wal_bytes_per_sync: 0 2022-03-21T14:01:05.745 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.strict_bytes_per_sync: 0 2022-03-21T14:01:05.745 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_readahead_size: 0 2022-03-21T14:01:05.745 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_background_flushes: -1 2022-03-21T14:01:05.746 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Compression algorithms supported: 2022-03-21T14:01:05.746 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-03-21T14:01:05.746 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kZSTD supported: 0 2022-03-21T14:01:05.746 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kXpressCompression supported: 0 2022-03-21T14:01:05.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kLZ4HCCompression supported: 1 2022-03-21T14:01:05.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kLZ4Compression supported: 1 2022-03-21T14:01:05.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kBZip2Compression supported: 0 2022-03-21T14:01:05.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kZlibCompression supported: 1 2022-03-21T14:01:05.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: kSnappyCompression supported: 1 2022-03-21T14:01:05.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Fast CRC32 supported: Supported on x86 2022-03-21T14:01:05.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2022-03-21T14:01:05.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-03-21T14:01:05.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-03-21T14:01:05.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.merge_operator: 2022-03-21T14:01:05.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_filter: None 2022-03-21T14:01:05.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_filter_factory: None 2022-03-21T14:01:05.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.sst_partitioner_factory: None 2022-03-21T14:01:05.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.memtable_factory: SkipListFactory 2022-03-21T14:01:05.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.table_factory: BlockBasedTable 2022-03-21T14:01:05.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563970b5bcc0) 2022-03-21T14:01:05.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks: 1 2022-03-21T14:01:05.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-03-21T14:01:05.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-03-21T14:01:05.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_top_level_index_and_filter: 1 2022-03-21T14:01:05.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_type: 0 2022-03-21T14:01:05.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_index_type: 0 2022-03-21T14:01:05.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_shortening: 1 2022-03-21T14:01:05.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2022-03-21T14:01:05.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: hash_index_allow_collision: 1 2022-03-21T14:01:05.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: checksum: 1 2022-03-21T14:01:05.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: no_block_cache: 0 2022-03-21T14:01:05.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache: 0x563970bc6170 2022-03-21T14:01:05.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_name: BinnedLRUCache 2022-03-21T14:01:05.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_options: 2022-03-21T14:01:05.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: capacity : 536870912 2022-03-21T14:01:05.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: num_shard_bits : 4 2022-03-21T14:01:05.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: strict_capacity_limit : 0 2022-03-21T14:01:05.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: high_pri_pool_ratio: 0.000 2022-03-21T14:01:05.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_compressed: (nil) 2022-03-21T14:01:05.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: persistent_cache: (nil) 2022-03-21T14:01:05.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size: 4096 2022-03-21T14:01:05.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size_deviation: 10 2022-03-21T14:01:05.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_restart_interval: 16 2022-03-21T14:01:05.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_block_restart_interval: 1 2022-03-21T14:01:05.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: metadata_block_size: 4096 2022-03-21T14:01:05.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: partition_filters: 0 2022-03-21T14:01:05.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: use_delta_encoding: 1 2022-03-21T14:01:05.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-03-21T14:01:05.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: whole_key_filtering: 1 2022-03-21T14:01:05.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: verify_compression: 0 2022-03-21T14:01:05.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: read_amp_bytes_per_bit: 0 2022-03-21T14:01:05.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: format_version: 4 2022-03-21T14:01:05.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_index_compression: 1 2022-03-21T14:01:05.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_align: 0 2022-03-21T14:01:05.757 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.write_buffer_size: 33554432 2022-03-21T14:01:05.757 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_write_buffer_number: 2 2022-03-21T14:01:05.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression: NoCompression 2022-03-21T14:01:05.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression: Disabled 2022-03-21T14:01:05.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.prefix_extractor: nullptr 2022-03-21T14:01:05.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-03-21T14:01:05.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.num_levels: 7 2022-03-21T14:01:05.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-03-21T14:01:05.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-03-21T14:01:05.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-03-21T14:01:05.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-03-21T14:01:05.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-03-21T14:01:05.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-03-21T14:01:05.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-03-21T14:01:05.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-03-21T14:01:05.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-03-21T14:01:05.760 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-03-21T14:01:05.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.window_bits: -14 2022-03-21T14:01:05.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.level: 32767 2022-03-21T14:01:05.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.strategy: 0 2022-03-21T14:01:05.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-03-21T14:01:05.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-03-21T14:01:05.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-03-21T14:01:05.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compression_opts.enabled: false 2022-03-21T14:01:05.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-03-21T14:01:05.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-03-21T14:01:05.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-03-21T14:01:05.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.target_file_size_base: 67108864 2022-03-21T14:01:05.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.target_file_size_multiplier: 1 2022-03-21T14:01:05.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-03-21T14:01:05.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-03-21T14:01:05.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-03-21T14:01:05.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-03-21T14:01:05.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-03-21T14:01:05.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-03-21T14:01:05.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-03-21T14:01:05.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-03-21T14:01:05.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-03-21T14:01:05.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-03-21T14:01:05.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-03-21T14:01:05.766 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-03-21T14:01:05.766 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.arena_block_size: 4194304 2022-03-21T14:01:05.766 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-03-21T14:01:05.766 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-03-21T14:01:05.767 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-03-21T14:01:05.767 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.disable_auto_compactions: 0 2022-03-21T14:01:05.767 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-03-21T14:01:05.767 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-03-21T14:01:05.767 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-03-21T14:01:05.768 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-03-21T14:01:05.768 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-03-21T14:01:05.768 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-03-21T14:01:05.768 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-03-21T14:01:05.768 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-03-21T14:01:05.769 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-03-21T14:01:05.769 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-03-21T14:01:05.769 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.table_properties_collectors: 2022-03-21T14:01:05.769 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.inplace_update_support: 0 2022-03-21T14:01:05.770 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.inplace_update_num_locks: 10000 2022-03-21T14:01:05.770 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-03-21T14:01:05.770 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-03-21T14:01:05.770 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.memtable_huge_page_size: 0 2022-03-21T14:01:05.770 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.bloom_locality: 0 2022-03-21T14:01:05.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.max_successive_merges: 0 2022-03-21T14:01:05.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.optimize_filters_for_hits: 0 2022-03-21T14:01:05.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.paranoid_file_checks: 0 2022-03-21T14:01:05.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.force_consistency_checks: 1 2022-03-21T14:01:05.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.report_bg_io_stats: 0 2022-03-21T14:01:05.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.ttl: 2592000 2022-03-21T14:01:05.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.periodic_compaction_seconds: 0 2022-03-21T14:01:05.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.enable_blob_files: false 2022-03-21T14:01:05.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.min_blob_size: 0 2022-03-21T14:01:05.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.blob_file_size: 268435456 2022-03-21T14:01:05.773 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.blob_compression_type: NoCompression 2022-03-21T14:01:05.773 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.enable_blob_garbage_collection: false 2022-03-21T14:01:05.773 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-03-21T14:01:05.773 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-03-21T14:01:05.774 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2022-03-21T14:01:05.774 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2022-03-21T14:01:05.774 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871265395972, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-03-21T14:01:05.774 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2022-03-21T14:01:05.774 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-03-21T14:01:05.775 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871265396347, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1647871265, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4a55b24e-4ad0-4e17-9c3d-63d954a99436", "db_session_id": "EDJS2X6YBR9N27MPLMJW"}} 2022-03-21T14:01:05.775 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2022-03-21T14:01:05.775 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1647871265396768, "job": 1, "event": "recovery_finished"} 2022-03-21T14:01:05.775 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-03-21T14:01:05.776 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563970baca80 2022-03-21T14:01:05.776 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: DB pointer 0x563970bbc000 2022-03-21T14:01:05.776 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-03-21T14:01:05.776 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-03-21T14:01:05.776 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** DB Stats ** 2022-03-21T14:01:05.777 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:01:05.777 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-03-21T14:01:05.777 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-03-21T14:01:05.777 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T14:01:05.777 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-03-21T14:01:05.778 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-03-21T14:01:05.778 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-03-21T14:01:05.778 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.778 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2022-03-21T14:01:05.778 INFO:journalctl@ceph.mon.c.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:01:05.779 INFO:journalctl@ceph.mon.c.smithi079.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:01:05.779 INFO:journalctl@ceph.mon.c.smithi079.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.779 INFO:journalctl@ceph.mon.c.smithi079.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.779 INFO:journalctl@ceph.mon.c.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.780 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.780 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2022-03-21T14:01:05.780 INFO:journalctl@ceph.mon.c.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:01:05.780 INFO:journalctl@ceph.mon.c.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:01:05.780 INFO:journalctl@ceph.mon.c.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.781 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:01:05.781 INFO:journalctl@ceph.mon.c.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T14:01:05.781 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T14:01:05.781 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T14:01:05.781 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T14:01:05.782 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T14:01:05.782 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:01:05.782 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:01:05.782 INFO:journalctl@ceph.mon.c.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T14:01:05.782 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.783 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T14:01:05.783 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.783 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2022-03-21T14:01:05.783 INFO:journalctl@ceph.mon.c.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:01:05.784 INFO:journalctl@ceph.mon.c.smithi079.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:01:05.784 INFO:journalctl@ceph.mon.c.smithi079.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.784 INFO:journalctl@ceph.mon.c.smithi079.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.784 INFO:journalctl@ceph.mon.c.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-03-21T14:01:05.784 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.785 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2022-03-21T14:01:05.785 INFO:journalctl@ceph.mon.c.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-03-21T14:01:05.785 INFO:journalctl@ceph.mon.c.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-03-21T14:01:05.785 INFO:journalctl@ceph.mon.c.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 2022-03-21T14:01:05.786 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-03-21T14:01:05.786 INFO:journalctl@ceph.mon.c.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-03-21T14:01:05.786 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-03-21T14:01:05.786 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-03-21T14:01:05.786 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-03-21T14:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2022-03-21T14:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-03-21T14:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-03-21T14:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2022-03-21T14:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c does not exist in monmap, will attempt to join an existing cluster 2022-03-21T14:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: using public_addr v2:172.21.15.79:0/0 -> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2022-03-21T14:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: starting mon.c rank -1 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(???) e0 preinit fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).mds e1 new map 2022-03-21T14:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).mds e1 print_map 2022-03-21T14:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout: e1 2022-03-21T14:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-03-21T14:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-03-21T14:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: legacy client fscid: -1 2022-03-21T14:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T14:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: No filesystems configured 2022-03-21T14:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-03-21T14:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-03-21T14:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-03-21T14:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-03-21T14:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-03-21T14:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-03-21T14:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-03-21T14:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-03-21T14:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-03-21T14:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/837986225' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-03-21T14:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:05.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:05.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:05.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c;count:3 2022-03-21T14:01:05.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:01:05.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:01:05.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: Deploying daemon mon.b on smithi016 2022-03-21T14:01:05.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1894754687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:05.798 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:05.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.a calling monitor election 2022-03-21T14:01:05.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.b calling monitor election 2022-03-21T14:01:05.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:05.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2022-03-21T14:01:05.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: fsmap 2022-03-21T14:01:05.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:05.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mgrmap e13: x(active, since 41s) 2022-03-21T14:01:05.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: overall HEALTH_OK 2022-03-21T14:01:05.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:05.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:05.803 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:05.803 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: Deploying daemon mon.c on smithi079 2022-03-21T14:01:05.803 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1568264651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:05.803 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:05.804 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:05.804 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-03-21T14:01:05.804 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 ceph-mon[23158]: mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2022-03-21T14:01:05.804 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:05 smithi079 systemd[1]: Started Ceph mon.c for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:01:10.770 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.769+0000 7fd8e3004700 1 -- 172.21.15.79:0/53729320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8cc003400 msgr2=0x7fd8cc005880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:10.770 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.769+0000 7fd8e3004700 1 --2- 172.21.15.79:0/53729320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8cc003400 0x7fd8cc005880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc06f4c0 tx=0x7fd8d0012480 comp rx=0 tx=0).stop 2022-03-21T14:01:10.771 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.770+0000 7fd8e3004700 1 -- 172.21.15.79:0/53729320 shutdown_connections 2022-03-21T14:01:10.771 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.770+0000 7fd8e3004700 1 --2- 172.21.15.79:0/53729320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8cc003400 0x7fd8cc005880 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:10.771 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.770+0000 7fd8e3004700 1 -- 172.21.15.79:0/53729320 >> 172.21.15.79:0/53729320 conn(0x7fd8dc071980 msgr2=0x7fd8dc071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:10.771 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.770+0000 7fd8e3004700 1 -- 172.21.15.79:0/53729320 shutdown_connections 2022-03-21T14:01:10.772 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.770+0000 7fd8e3004700 1 -- 172.21.15.79:0/53729320 wait complete. 2022-03-21T14:01:10.775 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 Processor -- start 2022-03-21T14:01:10.775 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 -- start start 2022-03-21T14:01:10.775 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:10.775 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:10.776 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 0x7fd8dc1186b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:10.776 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc118bf0 con 0x7fd8dc1162d0 2022-03-21T14:01:10.776 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc118d30 con 0x7fd8cc003400 2022-03-21T14:01:10.776 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e3004700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc118e70 con 0x7fd8dc10d420 2022-03-21T14:01:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33180/0 (socket says 172.21.15.79:33180) 2022-03-21T14:01:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 -- 172.21.15.79:0/2550847632 learned_addr learned my addr 172.21.15.79:0/2550847632 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:01:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e0da0700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:10.777 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 msgr2=0x7fd8dc115d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-03-21T14:01:10.778 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e15a1700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 0x7fd8dc1186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:10.778 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 msgr2=0x7fd8dc115d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-03-21T14:01:10.778 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-03-21T14:01:10.778 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8dbfff700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T14:01:10.779 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e0da0700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 msgr2=0x7fd8dc10cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-03-21T14:01:10.779 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e0da0700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 msgr2=0x7fd8dc10cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-03-21T14:01:10.779 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e0da0700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-03-21T14:01:10.779 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e0da0700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-03-21T14:01:10.780 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e15a1700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 msgr2=0x7fd8dc115d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T14:01:10.780 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e15a1700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:10.780 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e15a1700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 msgr2=0x7fd8dc10cee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-03-21T14:01:10.780 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.771+0000 7fd8e15a1700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:10.781 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8e15a1700 1 -- 172.21.15.79:0/2550847632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0012720 con 0x7fd8dc1162d0 2022-03-21T14:01:10.781 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8e15a1700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 0x7fd8dc1186b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400bad0 tx=0x7fd8d4009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:10.781 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400c8c0 con 0x7fd8dc1162d0 2022-03-21T14:01:10.781 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc1190d0 con 0x7fd8dc1162d0 2022-03-21T14:01:10.782 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd8d4014440 con 0x7fd8dc1162d0 2022-03-21T14:01:10.782 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4013610 con 0x7fd8dc1162d0 2022-03-21T14:01:10.782 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.772+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc1195c0 con 0x7fd8dc1162d0 2022-03-21T14:01:10.782 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.773+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fd8d401b030 con 0x7fd8dc1162d0 2022-03-21T14:01:10.782 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.773+0000 7fd8d97fa700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 0x7fd8c00375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:10.783 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.773+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd8d4046040 con 0x7fd8dc1162d0 2022-03-21T14:01:10.783 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.773+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc05e600 con 0x7fd8dc1162d0 2022-03-21T14:01:10.783 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.773+0000 7fd8e0da0700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 0x7fd8c00375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:10.783 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.774+0000 7fd8e0da0700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 0x7fd8c00375e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0005e20 tx=0x7fd8d0012480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:10.784 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:10.778+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd8d4019020 con 0x7fd8dc1162d0 2022-03-21T14:01:10.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: mon.b calling monitor election 2022-03-21T14:01:10.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:10.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: mon.a calling monitor election 2022-03-21T14:01:10.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:10.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:10.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:10.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.796 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:10.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:10.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: fsmap 2022-03-21T14:01:10.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:10.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: mgrmap e13: x(active, since 50s) 2022-03-21T14:01:10.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-03-21T14:01:10.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-03-21T14:01:10.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-03-21T14:01:10.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: mon.c (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2022-03-21T14:01:10.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:10.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:10.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:10.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: mon.b calling monitor election 2022-03-21T14:01:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: mon.a calling monitor election 2022-03-21T14:01:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: fsmap 2022-03-21T14:01:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: mgrmap e13: x(active, since 50s) 2022-03-21T14:01:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-03-21T14:01:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-03-21T14:01:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-03-21T14:01:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: mon.c (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2022-03-21T14:01:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:11.036 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.035+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8dc064940 con 0x7fd8dc1162d0 2022-03-21T14:01:11.037 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.036+0000 7fd8d97fa700 1 -- 172.21.15.79:0/2550847632 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1236 (secure 0 0 0) 0x7fd8d401b8b0 con 0x7fd8dc1162d0 2022-03-21T14:01:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: 2022-03-21T14:01:11.037 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":3,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","modified":"2022-03-21T14:01:05.422222Z","created":"2022-03-21T13:59:14.094890Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-03-21T14:01:11.039 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 msgr2=0x7fd8c00375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:11.039 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 0x7fd8c00375e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0005e20 tx=0x7fd8d0012480 comp rx=0 tx=0).stop 2022-03-21T14:01:11.039 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 msgr2=0x7fd8dc1186b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:11.039 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 0x7fd8dc1186b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400bad0 tx=0x7fd8d4009760 comp rx=0 tx=0).stop 2022-03-21T14:01:11.040 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 shutdown_connections 2022-03-21T14:01:11.040 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8cc003400 0x7fd8dc10cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:11.040 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd8c0035130 0x7fd8c00375e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:11.040 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8dc1162d0 0x7fd8dc1186b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:11.041 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 --2- 172.21.15.79:0/2550847632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8dc10d420 0x7fd8dc115d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:11.041 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 >> 172.21.15.79:0/2550847632 conn(0x7fd8dc071980 msgr2=0x7fd8dc0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:11.041 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 shutdown_connections 2022-03-21T14:01:11.041 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:11.038+0000 7fd8e3004700 1 -- 172.21.15.79:0/2550847632 wait complete. 2022-03-21T14:01:11.042 INFO:teuthology.orchestra.run.smithi079.stderr:dumped monmap epoch 3 2022-03-21T14:01:11.451 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-03-21T14:01:11.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph config generate-minimal-conf 2022-03-21T14:01:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T14:01:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/2550847632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:12.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T14:01:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/2550847632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: mon.b calling monitor election 2022-03-21T14:01:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:01:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: mon.a calling monitor election 2022-03-21T14:01:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-03-21T14:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: fsmap 2022-03-21T14:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: mgrmap e13: x(active, since 50s) 2022-03-21T14:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-03-21T14:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-03-21T14:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-03-21T14:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: mon.c (rank 2) addr [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] is down (out of quorum) 2022-03-21T14:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T14:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/2550847632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-03-21T14:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:13.019 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:01:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mon.c calling monitor election 2022-03-21T14:01:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:01:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: Reconfiguring mon.a (unknown last config time)... 2022-03-21T14:01:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: Reconfiguring daemon mon.a on smithi002 2022-03-21T14:01:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mon.c calling monitor election 2022-03-21T14:01:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mon.a calling monitor election 2022-03-21T14:01:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mon.b calling monitor election 2022-03-21T14:01:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-03-21T14:01:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: fsmap 2022-03-21T14:01:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: mgrmap e13: x(active, since 52s) 2022-03-21T14:01:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-03-21T14:01:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: Cluster is now healthy 2022-03-21T14:01:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:13 smithi079 ceph-mon[23158]: overall HEALTH_OK 2022-03-21T14:01:13.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 -- 172.21.15.2:0/1936010967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a340fd7b0 msgr2=0x7f6a34105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:13.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 --2- 172.21.15.2:0/1936010967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a340fd7b0 0x7f6a34105e80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6a240098e0 tx=0x7f6a24005f70 comp rx=0 tx=0).stop 2022-03-21T14:01:13.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 -- 172.21.15.2:0/1936010967 shutdown_connections 2022-03-21T14:01:13.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 --2- 172.21.15.2:0/1936010967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fce10 0x7f6a340fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 --2- 172.21.15.2:0/1936010967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a341063c0 0x7f6a341087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 --2- 172.21.15.2:0/1936010967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a340fd7b0 0x7f6a34105e80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 -- 172.21.15.2:0/1936010967 >> 172.21.15.2:0/1936010967 conn(0x7f6a34074ab0 msgr2=0x7f6a340f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:13.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 -- 172.21.15.2:0/1936010967 shutdown_connections 2022-03-21T14:01:13.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.555+0000 7f6a3a373700 1 -- 172.21.15.2:0/1936010967 wait complete. 2022-03-21T14:01:13.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.556+0000 7f6a3a373700 1 Processor -- start 2022-03-21T14:01:13.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.556+0000 7f6a3a373700 1 -- start start 2022-03-21T14:01:13.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:13.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fd7b0 0x7f6a341152f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:13.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a341063c0 0x7f6a3410ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:13.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3410f3f0 con 0x7f6a341063c0 2022-03-21T14:01:13.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3410f530 con 0x7f6a340fd7b0 2022-03-21T14:01:13.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a3a373700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3410f6f0 con 0x7f6a340fce10 2022-03-21T14:01:13.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:13.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a38b70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fd7b0 0x7f6a341152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:13.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39392/0 (socket says 172.21.15.2:39392) 2022-03-21T14:01:13.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 -- 172.21.15.2:0/4208039116 learned_addr learned my addr 172.21.15.2:0/4208039116 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:01:13.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39b72700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a341063c0 0x7f6a3410ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:13.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 -- 172.21.15.2:0/4208039116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fd7b0 msgr2=0x7f6a341152f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:13.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fd7b0 0x7f6a341152f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 -- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a341063c0 msgr2=0x7f6a3410ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:13.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a341063c0 0x7f6a3410ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.557+0000 7f6a39371700 1 -- 172.21.15.2:0/4208039116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240093f0 con 0x7f6a340fce10 2022-03-21T14:01:13.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a39371700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000fa90 tx=0x7f6a30005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:13.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000b4f0 con 0x7f6a340fce10 2022-03-21T14:01:13.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6a3000dc80 con 0x7f6a340fce10 2022-03-21T14:01:13.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3410f9d0 con 0x7f6a340fce10 2022-03-21T14:01:13.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011dc0 con 0x7f6a340fce10 2022-03-21T14:01:13.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.558+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3411bda0 con 0x7f6a340fce10 2022-03-21T14:01:13.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.559+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f6a3000c040 con 0x7f6a340fce10 2022-03-21T14:01:13.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.559+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3404b760 con 0x7f6a340fce10 2022-03-21T14:01:13.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.560+0000 7f6a29ffb700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 0x7f6a20037350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:13.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.560+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6a30046950 con 0x7f6a340fce10 2022-03-21T14:01:13.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.560+0000 7f6a38b70700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 0x7f6a20037350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:13.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.561+0000 7f6a38b70700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 0x7f6a20037350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a34110650 tx=0x7f6a24005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:13.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.565+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f6a30019520 con 0x7f6a340fce10 2022-03-21T14:01:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mon.c calling monitor election 2022-03-21T14:01:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:01:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: Reconfiguring mon.a (unknown last config time)... 2022-03-21T14:01:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: Reconfiguring daemon mon.a on smithi002 2022-03-21T14:01:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mon.c calling monitor election 2022-03-21T14:01:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mon.a calling monitor election 2022-03-21T14:01:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mon.b calling monitor election 2022-03-21T14:01:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-03-21T14:01:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: fsmap 2022-03-21T14:01:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: mgrmap e13: x(active, since 52s) 2022-03-21T14:01:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-03-21T14:01:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: Cluster is now healthy 2022-03-21T14:01:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:13 smithi002 ceph-mon[17167]: overall HEALTH_OK 2022-03-21T14:01:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mon.c calling monitor election 2022-03-21T14:01:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T14:01:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: Reconfiguring mon.a (unknown last config time)... 2022-03-21T14:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: Reconfiguring daemon mon.a on smithi002 2022-03-21T14:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mon.c calling monitor election 2022-03-21T14:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mon.a calling monitor election 2022-03-21T14:01:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mon.b calling monitor election 2022-03-21T14:01:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-03-21T14:01:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T14:01:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: fsmap 2022-03-21T14:01:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: osdmap e4: 0 total, 0 up, 0 in 2022-03-21T14:01:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: mgrmap e13: x(active, since 52s) 2022-03-21T14:01:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-03-21T14:01:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: Cluster is now healthy 2022-03-21T14:01:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:13 smithi016 ceph-mon[22010]: overall HEALTH_OK 2022-03-21T14:01:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.764+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6a3410bbb0 con 0x7f6a340fce10 2022-03-21T14:01:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.764+0000 7f6a29ffb700 1 -- 172.21.15.2:0/4208039116 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+269 (secure 0 0 0) 0x7f6a30019340 con 0x7f6a340fce10 2022-03-21T14:01:13.767 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:01:13.767 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2022-03-21T14:01:13.767 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:01:13.768 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2022-03-21T14:01:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 msgr2=0x7f6a20037350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 0x7f6a20037350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a34110650 tx=0x7f6a24005f70 comp rx=0 tx=0).stop 2022-03-21T14:01:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 msgr2=0x7f6a34114db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000fa90 tx=0x7f6a30005eb0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 shutdown_connections 2022-03-21T14:01:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6a20034ea0 0x7f6a20037350 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.766+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a341063c0 0x7f6a3410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.767+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a340fce10 0x7f6a34114db0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.767+0000 7f6a3a373700 1 --2- 172.21.15.2:0/4208039116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a340fd7b0 0x7f6a341152f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.767+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 >> 172.21.15.2:0/4208039116 conn(0x7f6a34074ab0 msgr2=0x7f6a340faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.767+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 shutdown_connections 2022-03-21T14:01:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:13.767+0000 7f6a3a373700 1 -- 172.21.15.2:0/4208039116 wait complete. 2022-03-21T14:01:14.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:14.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4208039116' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:14.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: Reconfiguring mon.b (monmap changed)... 2022-03-21T14:01:14.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:14.467 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:14.467 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.467 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:14 smithi002 ceph-mon[17167]: Reconfiguring daemon mon.b on smithi016 2022-03-21T14:01:14.469 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-03-21T14:01:14.469 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:01:14.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:01:14.543 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:01:14.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:01:14.655 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T14:01:14.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:01:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4208039116' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: Reconfiguring mon.b (monmap changed)... 2022-03-21T14:01:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:14 smithi016 ceph-mon[22010]: Reconfiguring daemon mon.b on smithi016 2022-03-21T14:01:14.682 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T14:01:14.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:01:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:01:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4208039116' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: Reconfiguring mon.b (monmap changed)... 2022-03-21T14:01:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:14 smithi079 ceph-mon[23158]: Reconfiguring daemon mon.b on smithi016 2022-03-21T14:01:14.750 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T14:01:14.750 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:01:14.781 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T14:01:14.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-03-21T14:01:14.848 INFO:tasks.cephadm:Adding mgr.x on smithi002 2022-03-21T14:01:14.849 INFO:tasks.cephadm:Adding mgr.y on smithi016 2022-03-21T14:01:14.849 INFO:tasks.cephadm:Adding mgr.z on smithi079 2022-03-21T14:01:14.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch apply mgr '3;smithi002=x;smithi016=y;smithi079=z' 2022-03-21T14:01:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.176 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:16.176 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:16.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:16.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:16.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-03-21T14:01:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-03-21T14:01:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:16.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:16.737 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- 172.21.15.79:0/709531711 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 msgr2=0x7fd3fc104420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:16.738 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- 172.21.15.79:0/709531711 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc104420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009c70 tx=0x7fd3f40094e0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.738 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- 172.21.15.79:0/709531711 shutdown_connections 2022-03-21T14:01:16.738 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- 172.21.15.79:0/709531711 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc104420 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.738 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- 172.21.15.79:0/709531711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3fc0fbf30 0x7fd3fc0fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.739 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- 172.21.15.79:0/709531711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3fc0fd1d0 0x7fd3fc0fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.739 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- 172.21.15.79:0/709531711 >> 172.21.15.79:0/709531711 conn(0x7fd3fc0f7720 msgr2=0x7fd3fc0f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:16.739 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- 172.21.15.79:0/709531711 shutdown_connections 2022-03-21T14:01:16.739 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- 172.21.15.79:0/709531711 wait complete. 2022-03-21T14:01:16.740 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 Processor -- start 2022-03-21T14:01:16.740 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- start start 2022-03-21T14:01:16.740 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3fc0fbf30 0x7fd3fc070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:16.740 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3fc0fd1d0 0x7fd3fc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:16.740 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:16.741 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc070000 con 0x7fd3fc0fd1d0 2022-03-21T14:01:16.741 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc070170 con 0x7fd3fc0fbf30 2022-03-21T14:01:16.741 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.734+0000 7fd404373700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc116c00 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.741 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:16.742 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33204/0 (socket says 172.21.15.79:33204) 2022-03-21T14:01:16.742 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 -- 172.21.15.79:0/3883747692 learned_addr learned my addr 172.21.15.79:0/3883747692 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:01:16.742 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 -- 172.21.15.79:0/3883747692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3fc0fbf30 msgr2=0x7fd3fc070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:16.742 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3fc0fbf30 0x7fd3fc070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.742 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 -- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3fc0fd1d0 msgr2=0x7fd3fc06f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3fc0fd1d0 0x7fd3fc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 -- 172.21.15.79:0/3883747692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4009780 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd402910700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4018da0 tx=0x7fd3f40094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4005b00 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.743 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc116e30 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.744 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd3f4004e40 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.744 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40052b0 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.744 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.735+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc117200 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.744 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.736+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7fd3f4029030 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.745 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.736+0000 7fd3f2ffd700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 0x7fd3e8037350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:16.745 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.736+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd3f4048150 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.745 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.736+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc1077b0 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.746 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.737+0000 7fd40210f700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 0x7fd3e8037350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:16.746 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.737+0000 7fd40210f700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 0x7fd3e8037350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0fd030 tx=0x7fd3f8006c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:16.746 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.741+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd3f4027070 con 0x7fd3fc0fdb60 2022-03-21T14:01:16.931 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.930+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi016=y;smithi079=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3fc02ad40 con 0x7fd3e8034ea0 2022-03-21T14:01:16.938 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.936+0000 7fd3f2ffd700 1 -- 172.21.15.79:0/3883747692 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd3fc02ad40 con 0x7fd3e8034ea0 2022-03-21T14:01:16.938 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled mgr update... 2022-03-21T14:01:16.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 msgr2=0x7fd3e8037350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:16.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 0x7fd3e8037350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0fd030 tx=0x7fd3f8006c30 comp rx=0 tx=0).stop 2022-03-21T14:01:16.940 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 msgr2=0x7fd3fc06f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:16.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4018da0 tx=0x7fd3f40094e0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 shutdown_connections 2022-03-21T14:01:16.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3fc0fbf30 0x7fd3fc070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3e8034ea0 0x7fd3e8037350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.941 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3fc0fdb60 0x7fd3fc06f860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 --2- 172.21.15.79:0/3883747692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3fc0fd1d0 0x7fd3fc06f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:16.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.938+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 >> 172.21.15.79:0/3883747692 conn(0x7fd3fc0f7720 msgr2=0x7fd3fc0f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:16.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.939+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 shutdown_connections 2022-03-21T14:01:16.942 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:01:16.939+0000 7fd404373700 1 -- 172.21.15.79:0/3883747692 wait complete. 2022-03-21T14:01:17.077 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:16 smithi079 ceph-mon[23158]: Reconfiguring mon.c (monmap changed)... 2022-03-21T14:01:17.078 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:16 smithi079 ceph-mon[23158]: Reconfiguring daemon mon.c on smithi079 2022-03-21T14:01:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:16 smithi002 ceph-mon[17167]: Reconfiguring mon.c (monmap changed)... 2022-03-21T14:01:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:16 smithi002 ceph-mon[17167]: Reconfiguring daemon mon.c on smithi079 2022-03-21T14:01:17.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:16 smithi016 ceph-mon[22010]: Reconfiguring mon.c (monmap changed)... 2022-03-21T14:01:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:16 smithi016 ceph-mon[22010]: Reconfiguring daemon mon.c on smithi079 2022-03-21T14:01:17.420 DEBUG:teuthology.orchestra.run.smithi016:mgr.y> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.y.service 2022-03-21T14:01:17.422 DEBUG:teuthology.orchestra.run.smithi079:mgr.z> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mgr.z.service 2022-03-21T14:01:17.425 INFO:tasks.cephadm:Deploying OSDs... 2022-03-21T14:01:17.426 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:01:17.426 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:01:17.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:01:17.450 INFO:journalctl@ceph.mgr.z.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:01:17.475 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:01:17.476 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:01:17.567 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:01:17.568 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:17.568 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 454 Links: 1 2022-03-21T14:01:17.568 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:17.568 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:17.569 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-03-21 14:00:33.584759815 +0000 2022-03-21T14:01:17.569 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-03-21 13:53:50.352405251 +0000 2022-03-21T14:01:17.569 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-03-21 13:53:50.352405251 +0000 2022-03-21T14:01:17.569 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-03-21T14:01:17.596 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-03-21T14:01:17.597 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-03-21T14:01:17.597 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000378472 s, 1.4 MB/s 2022-03-21T14:01:17.606 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:01:17.652 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:01:17.652 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:17.653 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 460 Links: 1 2022-03-21T14:01:17.653 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:17.653 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:17.653 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-03-21 14:00:33.584759815 +0000 2022-03-21T14:01:17.653 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-03-21 13:53:50.881393164 +0000 2022-03-21T14:01:17.654 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-03-21 13:53:50.881393164 +0000 2022-03-21T14:01:17.654 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-03-21T14:01:17.679 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-03-21T14:01:17.679 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-03-21T14:01:17.679 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000301762 s, 1.7 MB/s 2022-03-21T14:01:17.688 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:01:17.774 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:01:17.774 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:17.774 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 466 Links: 1 2022-03-21T14:01:17.775 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:17.775 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:17.775 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-03-21 14:00:33.584759815 +0000 2022-03-21T14:01:17.775 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-03-21 13:53:51.431380596 +0000 2022-03-21T14:01:17.775 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-03-21 13:53:51.431380596 +0000 2022-03-21T14:01:17.776 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-03-21T14:01:17.800 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-03-21T14:01:17.801 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-03-21T14:01:17.801 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000340723 s, 1.5 MB/s 2022-03-21T14:01:17.810 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:01:17.855 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:01:17.855 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:17.856 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 472 Links: 1 2022-03-21T14:01:17.856 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:17.856 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:17.856 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-03-21 14:00:33.585759793 +0000 2022-03-21T14:01:17.857 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-03-21 13:53:51.956368600 +0000 2022-03-21T14:01:17.857 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-03-21 13:53:51.956368600 +0000 2022-03-21T14:01:17.857 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-03-21T14:01:17.881 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-03-21T14:01:17.882 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-03-21T14:01:17.882 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00029924 s, 1.7 MB/s 2022-03-21T14:01:17.891 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2022-03-21T14:01:17.891 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:01:17.907 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:01:17.908 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:01:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:01:17.963 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 59038 Links: 1 2022-03-21T14:01:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2022-03-21 14:00:56.067456930 +0000 2022-03-21T14:01:17.964 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2022-03-21 13:53:14.282274726 +0000 2022-03-21T14:01:17.964 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2022-03-21 13:53:14.282274726 +0000 2022-03-21T14:01:17.964 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2022-03-21T14:01:17.973 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2022-03-21T14:01:17.974 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2022-03-21T14:01:17.974 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000108316 s, 4.7 MB/s 2022-03-21T14:01:17.977 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:01:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:01:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.032 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 60828 Links: 1 2022-03-21T14:01:18.033 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.033 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.033 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2022-03-21 14:00:56.067456930 +0000 2022-03-21T14:01:18.033 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2022-03-21 13:53:14.623269190 +0000 2022-03-21T14:01:18.033 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2022-03-21 13:53:14.623269190 +0000 2022-03-21T14:01:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2022-03-21T14:01:18.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:18.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi016=y;smithi079=z", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:18.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: Saving service mgr spec with placement smithi002=x;smithi016=y;smithi079=z;count:3 2022-03-21T14:01:18.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.042 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:18.042 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.042 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:18.042 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:18.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:18.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:18.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2022-03-21T14:01:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2022-03-21T14:01:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000101208 s, 5.1 MB/s 2022-03-21T14:01:18.046 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:01:18.101 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:01:18.102 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.102 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 59126 Links: 1 2022-03-21T14:01:18.102 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.103 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.103 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2022-03-21 14:00:56.068456913 +0000 2022-03-21T14:01:18.103 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2022-03-21 13:53:14.965263636 +0000 2022-03-21T14:01:18.103 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2022-03-21 13:53:14.965263636 +0000 2022-03-21T14:01:18.104 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2022-03-21T14:01:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2022-03-21T14:01:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2022-03-21T14:01:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000111033 s, 4.6 MB/s 2022-03-21T14:01:18.115 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:01:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:01:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.170 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 61865 Links: 1 2022-03-21T14:01:18.171 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.171 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.171 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2022-03-21 14:00:56.068456913 +0000 2022-03-21T14:01:18.171 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2022-03-21 13:53:15.333257661 +0000 2022-03-21T14:01:18.172 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2022-03-21 13:53:15.333257661 +0000 2022-03-21T14:01:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2022-03-21T14:01:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2022-03-21T14:01:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2022-03-21T14:01:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000110864 s, 4.6 MB/s 2022-03-21T14:01:18.183 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2022-03-21T14:01:18.183 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:01:18.199 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:01:18.199 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:01:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi016=y;smithi079=z", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: Saving service mgr spec with placement smithi002=x;smithi016=y;smithi079=z;count:3 2022-03-21T14:01:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:18.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.240 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:01:18.240 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.241 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 62604 Links: 1 2022-03-21T14:01:18.241 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.241 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.241 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2022-03-21 14:01:04.585207932 +0000 2022-03-21T14:01:18.241 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2022-03-21 13:53:07.352934516 +0000 2022-03-21T14:01:18.242 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2022-03-21 13:53:07.352934516 +0000 2022-03-21T14:01:18.242 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2022-03-21T14:01:18.250 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2022-03-21T14:01:18.251 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2022-03-21T14:01:18.251 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000246484 s, 2.1 MB/s 2022-03-21T14:01:18.254 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:01:18.309 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:01:18.309 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.309 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 62640 Links: 1 2022-03-21T14:01:18.309 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.310 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.310 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2022-03-21 14:01:04.586207908 +0000 2022-03-21T14:01:18.310 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2022-03-21 13:53:07.694926394 +0000 2022-03-21T14:01:18.310 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2022-03-21 13:53:07.694926394 +0000 2022-03-21T14:01:18.311 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2022-03-21T14:01:18.318 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2022-03-21T14:01:18.319 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2022-03-21T14:01:18.319 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000219576 s, 2.3 MB/s 2022-03-21T14:01:18.322 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:01:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi016=y;smithi079=z", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: Saving service mgr spec with placement smithi002=x;smithi016=y;smithi079=z;count:3 2022-03-21T14:01:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:18.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:18.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:18.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:18.376 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:01:18.376 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.377 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 61647 Links: 1 2022-03-21T14:01:18.377 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.377 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.377 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2022-03-21 14:01:04.586207908 +0000 2022-03-21T14:01:18.378 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2022-03-21 13:53:08.036918273 +0000 2022-03-21T14:01:18.378 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2022-03-21 13:53:08.036918273 +0000 2022-03-21T14:01:18.378 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2022-03-21T14:01:18.386 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2022-03-21T14:01:18.386 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2022-03-21T14:01:18.386 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000218913 s, 2.3 MB/s 2022-03-21T14:01:18.388 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:01:18.443 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:01:18.444 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:01:18.444 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 60878 Links: 1 2022-03-21T14:01:18.444 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:01:18.445 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2022-03-21T14:01:18.445 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2022-03-21 14:01:04.586207908 +0000 2022-03-21T14:01:18.445 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2022-03-21 13:53:08.370910342 +0000 2022-03-21T14:01:18.445 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2022-03-21 13:53:08.370910342 +0000 2022-03-21T14:01:18.446 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2022-03-21T14:01:18.452 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2022-03-21T14:01:18.453 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2022-03-21T14:01:18.453 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000192587 s, 2.7 MB/s 2022-03-21T14:01:18.455 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2022-03-21T14:01:18.456 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-03-21T14:01:18.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:18 smithi016 systemd[1]: Starting Ceph mgr.y for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:01:19.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:18 smithi002 ceph-mon[17167]: Deploying daemon mgr.y on smithi016 2022-03-21T14:01:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:18 smithi079 ceph-mon[23158]: Deploying daemon mgr.y on smithi016 2022-03-21T14:01:19.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:18 smithi016 ceph-mon[22010]: Deploying daemon mgr.y on smithi016 2022-03-21T14:01:19.247 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mgr[23254]: -- 172.21.15.16:0/3475863181 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55637e9b6340 con 0x55637e98f000 2022-03-21T14:01:19.247 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 bash[23044]: e1000d7d6b1292ba915e43cb84706553528ac99d8a2e2504c60a98084b979ecb 2022-03-21T14:01:19.547 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 systemd[1]: Started Ceph mgr.y for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:01:19.547 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 conmon[23230]: 2022-03-21T14:01:19.303+0000 7f92e4748000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-21T14:01:19.547 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 conmon[23230]: 2022-03-21T14:01:19.452+0000 7f92e4748000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-21T14:01:19.548 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 conmon[23230]: 2022-03-21T14:01:19.546+0000 7f92e4748000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-21T14:01:19.851 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:01:19.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 conmon[23230]: 2022-03-21T14:01:19.714+0000 7f92e4748000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-21T14:01:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:20.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:19 smithi016 conmon[23230]: 2022-03-21T14:01:19.958+0000 7f92e4748000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:01:20.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:20 smithi016 conmon[23230]: 2022-03-21T14:01:20.090+0000 7f92e4748000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:01:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-03-21T14:01:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:20.847 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-03-21T14:01:20.847 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-03-21T14:01:20.848 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-03-21T14:01:20.848 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-03-21T14:01:20.848 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0416538 s, 252 MB/s 2022-03-21T14:01:20.848 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-03-21T14:01:20.853 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:20 smithi016 conmon[23230]: 2022-03-21T14:01:20.431+0000 7f92e4748000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-21T14:01:20.853 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:20 smithi016 conmon[23230]: 2022-03-21T14:01:20.613+0000 7f92e4748000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-21T14:01:20.853 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:20 smithi016 conmon[23230]: 2022-03-21T14:01:20.852+0000 7f92e4748000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-21T14:01:20.947 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:20 smithi079 systemd[1]: Starting Ceph mgr.z for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:01:21.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:21 smithi016 conmon[23230]: 2022-03-21T14:01:21.049+0000 7f92e4748000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-21T14:01:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:20 smithi016 ceph-mon[22010]: Deploying daemon mgr.z on smithi079 2022-03-21T14:01:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:01:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:01:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:20 smithi079 ceph-mon[23158]: Deploying daemon mgr.z on smithi079 2022-03-21T14:01:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:01:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:01:21.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:20 smithi002 ceph-mon[17167]: Deploying daemon mgr.z on smithi079 2022-03-21T14:01:21.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:01:21.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:01:21.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2022-03-21T14:01:21.615 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:21 smithi016 conmon[23230]: 2022-03-21T14:01:21.218+0000 7f92e4748000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-21T14:01:21.747 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mgr[24855]: -- 172.21.15.79:0/3358072485 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d348b9a340 con 0x55d348b73000 2022-03-21T14:01:21.747 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 bash[24646]: 7ea299894367e8012aa768c4c58464d49acbe91f7f82a14f82dbb388fda080a3 2022-03-21T14:01:21.748 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 systemd[1]: Started Ceph mgr.z for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:01:21.748 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 conmon[24831]: 2022-03-21T14:01:21.596+0000 7f689c158000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-03-21T14:01:21.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:21 smithi016 conmon[23230]: 2022-03-21T14:01:21.784+0000 7f92e4748000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-21T14:01:22.021 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 conmon[24831]: 2022-03-21T14:01:21.746+0000 7f689c158000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-03-21T14:01:22.021 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:21 smithi079 conmon[24831]: 2022-03-21T14:01:21.845+0000 7f689c158000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-03-21T14:01:22.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mon[23158]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:22.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:22.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:22.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:22.023 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:22.275 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:22 smithi079 conmon[24831]: 2022-03-21T14:01:22.019+0000 7f689c158000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-03-21T14:01:22.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:21 smithi002 ceph-mon[17167]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:22.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:22.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:22.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:22.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:22.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:21 smithi016 conmon[23230]: 2022-03-21T14:01:21.964+0000 7f92e4748000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-21T14:01:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:21 smithi016 ceph-mon[22010]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:22.722 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:22 smithi079 conmon[24831]: 2022-03-21T14:01:22.274+0000 7f689c158000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:01:22.723 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:22 smithi079 conmon[24831]: 2022-03-21T14:01:22.409+0000 7f689c158000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-03-21T14:01:22.868 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:01:23.155 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:22 smithi079 conmon[24831]: 2022-03-21T14:01:22.750+0000 7f689c158000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-03-21T14:01:23.156 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:22 smithi079 conmon[24831]: 2022-03-21T14:01:22.918+0000 7f689c158000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-03-21T14:01:23.156 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:23 smithi079 conmon[24831]: 2022-03-21T14:01:23.154+0000 7f689c158000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-03-21T14:01:23.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:22 smithi016 conmon[23230]: 2022-03-21T14:01:22.717+0000 7f92e4748000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-21T14:01:23.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.382+0000 7f5015c00700 1 -- 172.21.15.2:0/168222632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fdb80 msgr2=0x7f5010104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:23.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.382+0000 7f5015c00700 1 --2- 172.21.15.2:0/168222632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fdb80 0x7f5010104440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5008009c70 tx=0x7f50080094e0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 -- 172.21.15.2:0/168222632 shutdown_connections 2022-03-21T14:01:23.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 --2- 172.21.15.2:0/168222632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fdb80 0x7f5010104440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 --2- 172.21.15.2:0/168222632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fbf50 0x7f50100fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 --2- 172.21.15.2:0/168222632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fd1f0 0x7f50100fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 -- 172.21.15.2:0/168222632 >> 172.21.15.2:0/168222632 conn(0x7f50100f7720 msgr2=0x7f50100f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:23.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 -- 172.21.15.2:0/168222632 shutdown_connections 2022-03-21T14:01:23.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.383+0000 7f5015c00700 1 -- 172.21.15.2:0/168222632 wait complete. 2022-03-21T14:01:23.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 Processor -- start 2022-03-21T14:01:23.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 -- start start 2022-03-21T14:01:23.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fbf50 0x7f501010ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:23.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:23.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fdb80 0x7f50101163f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:23.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010116930 con 0x7f50100fbf50 2022-03-21T14:01:23.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010116aa0 con 0x7f50100fd1f0 2022-03-21T14:01:23.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f5015c00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010116c60 con 0x7f50100fdb80 2022-03-21T14:01:23.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f500effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:23.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.384+0000 7f500f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fbf50 0x7f501010ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:23.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48122/0 (socket says 172.21.15.2:48122) 2022-03-21T14:01:23.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 -- 172.21.15.2:0/270451602 learned_addr learned my addr 172.21.15.2:0/270451602 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:01:23.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500ffff700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fdb80 0x7f50101163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:23.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 -- 172.21.15.2:0/270451602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fdb80 msgr2=0x7f50101163f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:23.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fdb80 0x7f50101163f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 -- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fbf50 msgr2=0x7f501010ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:23.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fbf50 0x7f501010ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:23.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 -- 172.21.15.2:0/270451602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008009780 con 0x7f50100fd1f0 2022-03-21T14:01:23.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f500effd700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f500400fa90 tx=0x7f5004005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:23.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.385+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500400b4f0 con 0x7f50100fd1f0 2022-03-21T14:01:23.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.386+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f500400dc00 con 0x7f50100fd1f0 2022-03-21T14:01:23.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.386+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5010116f40 con 0x7f50100fd1f0 2022-03-21T14:01:23.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.386+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004011cf0 con 0x7f50100fd1f0 2022-03-21T14:01:23.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.386+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50101173f0 con 0x7f50100fd1f0 2022-03-21T14:01:23.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.387+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 41899+0+0 (secure 0 0 0) 0x7f500400c040 con 0x7f50100fd1f0 2022-03-21T14:01:23.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.387+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5010107750 con 0x7f50100fd1f0 2022-03-21T14:01:23.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.387+0000 7f4ff7fff700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 0x7f4ff8037350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:23.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.387+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f50040468b0 con 0x7f50100fd1f0 2022-03-21T14:01:23.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.388+0000 7f500f7fe700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 0x7f4ff8037350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:23.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.388+0000 7f500f7fe700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 0x7f4ff8037350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50100fd050 tx=0x7f5000005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:23.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.392+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f5004019520 con 0x7f50100fd1f0 2022-03-21T14:01:23.472 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:23 smithi079 conmon[24831]: 2022-03-21T14:01:23.341+0000 7f689c158000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-03-21T14:01:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:23.601+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5010103cc0 con 0x7f4ff8034ea0 2022-03-21T14:01:23.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:23 smithi016 conmon[23230]: 2022-03-21T14:01:23.319+0000 7f92e4748000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-21T14:01:23.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:23 smithi016 conmon[23230]: 2022-03-21T14:01:23.415+0000 7f92e4748000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-21T14:01:23.961 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:23 smithi079 conmon[24831]: 2022-03-21T14:01:23.503+0000 7f689c158000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-03-21T14:01:24.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:23 smithi016 conmon[23230]: 2022-03-21T14:01:23.713+0000 7f92e4748000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-21T14:01:24.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:23 smithi016 conmon[23230]: 2022-03-21T14:01:23.810+0000 7f92e4748000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-21T14:01:24.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:23 smithi016 conmon[23230]: 2022-03-21T14:01:23.908+0000 7f92e4748000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-21T14:01:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: Reconfiguring mgr.x (unknown last config time)... 2022-03-21T14:01:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: Reconfiguring daemon mgr.x on smithi002 2022-03-21T14:01:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.222 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:24 smithi079 conmon[24831]: 2022-03-21T14:01:24.034+0000 7f689c158000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-03-21T14:01:24.223 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:24 smithi079 conmon[24831]: 2022-03-21T14:01:24.198+0000 7f689c158000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-03-21T14:01:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: Reconfiguring mgr.x (unknown last config time)... 2022-03-21T14:01:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: Reconfiguring daemon mgr.x on smithi002 2022-03-21T14:01:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: Reconfiguring mgr.x (unknown last config time)... 2022-03-21T14:01:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-03-21T14:01:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-03-21T14:01:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: Reconfiguring daemon mgr.x on smithi002 2022-03-21T14:01:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:24.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:24.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:24.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:24 smithi016 conmon[23230]: 2022-03-21T14:01:24.320+0000 7f92e4748000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-21T14:01:24.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:24 smithi016 conmon[23230]: 2022-03-21T14:01:24.412+0000 7f92e4748000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-21T14:01:24.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:24 smithi016 conmon[23230]: 2022-03-21T14:01:24.508+0000 7f92e4748000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-21T14:01:25.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:24 smithi002 ceph-mon[17167]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:25.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:25.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:25.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:25.032 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:25.222 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:24 smithi079 conmon[24831]: 2022-03-21T14:01:24.882+0000 7f689c158000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-03-21T14:01:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:24 smithi079 ceph-mon[23158]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:25.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:25 smithi016 conmon[23230]: 2022-03-21T14:01:25.133+0000 7f92e4748000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-21T14:01:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:24 smithi016 ceph-mon[22010]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:25.722 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:25 smithi079 conmon[24831]: 2022-03-21T14:01:25.444+0000 7f689c158000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-03-21T14:01:25.722 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:25 smithi079 conmon[24831]: 2022-03-21T14:01:25.533+0000 7f689c158000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-03-21T14:01:25.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 14:01:25 smithi016 conmon[23230]: 2022-03-21T14:01:25.448+0000 7f92e4748000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-21T14:01:26.222 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:25 smithi079 conmon[24831]: 2022-03-21T14:01:25.813+0000 7f689c158000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-03-21T14:01:26.223 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:25 smithi079 conmon[24831]: 2022-03-21T14:01:25.906+0000 7f689c158000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-03-21T14:01:26.223 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:26 smithi079 conmon[24831]: 2022-03-21T14:01:26.001+0000 7f689c158000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-03-21T14:01:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: Standby manager daemon y started 2022-03-21T14:01:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-03-21T14:01:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-03-21T14:01:26.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:26.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: Standby manager daemon y started 2022-03-21T14:01:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-03-21T14:01:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-03-21T14:01:26.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:26.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:26.389+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) v1 ==== 83499+0+0 (secure 0 0 0) 0x7f5004020d40 con 0x7f50100fd1f0 2022-03-21T14:01:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: Standby manager daemon y started 2022-03-21T14:01:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-03-21T14:01:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-03-21T14:01:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:26.722 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:26 smithi079 conmon[24831]: 2022-03-21T14:01:26.391+0000 7f689c158000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-03-21T14:01:26.723 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:26 smithi079 conmon[24831]: 2022-03-21T14:01:26.478+0000 7f689c158000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-03-21T14:01:26.723 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:26 smithi079 conmon[24831]: 2022-03-21T14:01:26.569+0000 7f689c158000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-03-21T14:01:27.455 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:27 smithi079 conmon[24831]: 2022-03-21T14:01:27.155+0000 7f689c158000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-03-21T14:01:27.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:27.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: mgrmap e14: x(active, since 66s), standbys: y 2022-03-21T14:01:27.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-03-21T14:01:27.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2448265930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]': finished 2022-03-21T14:01:27.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: osdmap e5: 1 total, 0 up, 1 in 2022-03-21T14:01:27.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: mgrmap e14: x(active, since 66s), standbys: y 2022-03-21T14:01:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-03-21T14:01:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2448265930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]': finished 2022-03-21T14:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: osdmap e5: 1 total, 0 up, 1 in 2022-03-21T14:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:27.722 INFO:journalctl@ceph.mgr.z.smithi079.stdout:Mar 21 14:01:27 smithi079 conmon[24831]: 2022-03-21T14:01:27.454+0000 7f689c158000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-03-21T14:01:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: mgrmap e14: x(active, since 66s), standbys: y 2022-03-21T14:01:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-03-21T14:01:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2448265930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]: dispatch 2022-03-21T14:01:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13405266-d8e8-4542-bf91-9450e11d2ecd"}]': finished 2022-03-21T14:01:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: osdmap e5: 1 total, 0 up, 1 in 2022-03-21T14:01:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:28.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:28.400+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mon.1 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 15) v1 ==== 125099+0+0 (secure 0 0 0) 0x7f50040243f0 con 0x7f50100fd1f0 2022-03-21T14:01:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: Standby manager daemon z started 2022-03-21T14:01:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-03-21T14:01:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-03-21T14:01:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:28 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1892988242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: Standby manager daemon z started 2022-03-21T14:01:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-03-21T14:01:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-03-21T14:01:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:28 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1892988242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: Standby manager daemon z started 2022-03-21T14:01:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-03-21T14:01:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-03-21T14:01:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-03-21T14:01:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: from='mgr.? 172.21.15.79:0/1965886040' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-03-21T14:01:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:28 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1892988242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:29 smithi016 ceph-mon[22010]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:29 smithi016 ceph-mon[22010]: mgrmap e15: x(active, since 68s), standbys: y, z 2022-03-21T14:01:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-03-21T14:01:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:29 smithi079 ceph-mon[23158]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:29 smithi079 ceph-mon[23158]: mgrmap e15: x(active, since 68s), standbys: y, z 2022-03-21T14:01:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-03-21T14:01:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:29 smithi002 ceph-mon[17167]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:29 smithi002 ceph-mon[17167]: mgrmap e15: x(active, since 68s), standbys: y, z 2022-03-21T14:01:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-03-21T14:01:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:31 smithi016 ceph-mon[22010]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:31 smithi079 ceph-mon[23158]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:31.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:31 smithi002 ceph-mon[17167]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:33 smithi016 ceph-mon[22010]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:33 smithi079 ceph-mon[23158]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:33 smithi002 ceph-mon[17167]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:35 smithi002 ceph-mon[17167]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-03-21T14:01:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:35 smithi016 ceph-mon[22010]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:35.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-03-21T14:01:35.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:35 smithi079 ceph-mon[23158]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-03-21T14:01:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:36.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:36 smithi002 ceph-mon[17167]: Deploying daemon osd.0 on smithi002 2022-03-21T14:01:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:36 smithi016 ceph-mon[22010]: Deploying daemon osd.0 on smithi002 2022-03-21T14:01:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:36 smithi079 ceph-mon[23158]: Deploying daemon osd.0 on smithi002 2022-03-21T14:01:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:37 smithi016 ceph-mon[22010]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:37 smithi079 ceph-mon[23158]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:37.841 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:37 smithi002 ceph-mon[17167]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:39 smithi002 ceph-mon[17167]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:39 smithi079 ceph-mon[23158]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:39 smithi016 ceph-mon[22010]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:41 smithi079 ceph-mon[23158]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:41.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:41 smithi002 ceph-mon[17167]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:41 smithi016 ceph-mon[22010]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:43 smithi002 ceph-mon[17167]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:43 smithi016 ceph-mon[22010]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:43 smithi079 ceph-mon[23158]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:45.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.858+0000 7f4ff7fff700 1 -- 172.21.15.2:0/270451602 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5010103cc0 con 0x7f4ff8034ea0 2022-03-21T14:01:45.860 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2022-03-21T14:01:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.861+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 msgr2=0x7f4ff8037350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.861+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 0x7f4ff8037350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f50100fd050 tx=0x7f5000005eb0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 msgr2=0x7f501010f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f500400fa90 tx=0x7f5004005eb0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 shutdown_connections 2022-03-21T14:01:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4ff8034ea0 0x7f4ff8037350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50100fdb80 0x7f50101163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50100fbf50 0x7f501010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 --2- 172.21.15.2:0/270451602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50100fd1f0 0x7f501010f360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:45.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 >> 172.21.15.2:0/270451602 conn(0x7f50100f7720 msgr2=0x7f50100f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:45.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 shutdown_connections 2022-03-21T14:01:45.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:01:45.862+0000 7f5015c00700 1 -- 172.21.15.2:0/270451602 wait complete. 2022-03-21T14:01:46.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:45 smithi002 ceph-mon[17167]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:46.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:45 smithi016 ceph-mon[22010]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:45 smithi079 ceph-mon[23158]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:46.315 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.0.service 2022-03-21T14:01:46.320 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_4... 2022-03-21T14:01:46.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-03-21T14:01:46.395 INFO:journalctl@ceph.osd.0.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:01:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:46 smithi002 ceph-mon[17167]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:46 smithi016 ceph-mon[22010]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:01:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:01:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:46 smithi079 ceph-mon[23158]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-03-21T14:01:48.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:48.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-03-21T14:01:48.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: osdmap e6: 1 total, 0 up, 1 in 2022-03-21T14:01:48.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:48.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:47 smithi002 ceph-mon[17167]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.144 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:01:47 smithi002 conmon[30406]: 2022-03-21T14:01:47.871+0000 7f306eea0700 -1 osd.0 0 waiting for initial osdmap 2022-03-21T14:01:48.144 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:01:47 smithi002 conmon[30406]: 2022-03-21T14:01:47.876+0000 7f3068033700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:01:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-03-21T14:01:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: osdmap e6: 1 total, 0 up, 1 in 2022-03-21T14:01:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:47 smithi016 ceph-mon[22010]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-03-21T14:01:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: osdmap e6: 1 total, 0 up, 1 in 2022-03-21T14:01:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: from='osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:47 smithi079 ceph-mon[23158]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:01:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-03-21T14:01:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-03-21T14:01:48.225 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2022-03-21T14:01:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2022-03-21T14:01:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00793351 s, 1.3 GB/s 2022-03-21T14:01:48.225 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 2022-03-21T14:01:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2022-03-21T14:01:48.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2022-03-21T14:01:49.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:48 smithi016 ceph-mon[22010]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:01:49.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:48 smithi016 ceph-mon[22010]: osdmap e7: 1 total, 0 up, 1 in 2022-03-21T14:01:49.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:49.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:48 smithi079 ceph-mon[23158]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:01:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:48 smithi079 ceph-mon[23158]: osdmap e7: 1 total, 0 up, 1 in 2022-03-21T14:01:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:48 smithi002 ceph-mon[17167]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:01:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:48 smithi002 ceph-mon[17167]: osdmap e7: 1 total, 0 up, 1 in 2022-03-21T14:01:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:50.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:01:50.092 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:01:50.092 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:50.092 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] boot 2022-03-21T14:01:50.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: osdmap e8: 1 total, 1 up, 1 in 2022-03-21T14:01:50.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:01:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:01:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] boot 2022-03-21T14:01:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: osdmap e8: 1 total, 1 up, 1 in 2022-03-21T14:01:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:01:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:01:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-03-21T14:01:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: osd.0 [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] boot 2022-03-21T14:01:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: osdmap e8: 1 total, 1 up, 1 in 2022-03-21T14:01:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T14:01:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 -- 172.21.15.16:0/1678746754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 msgr2=0x7f07000fd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 --2- 172.21.15.16:0/1678746754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f07000fd640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06f40098e0 tx=0x7f06f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:01:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 -- 172.21.15.16:0/1678746754 shutdown_connections 2022-03-21T14:01:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 --2- 172.21.15.16:0/1678746754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07000fdb80 0x7f0700104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 --2- 172.21.15.16:0/1678746754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fbf50 0x7f07000fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 --2- 172.21.15.16:0/1678746754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f07000fd640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 -- 172.21.15.16:0/1678746754 >> 172.21.15.16:0/1678746754 conn(0x7f07000f7720 msgr2=0x7f07000f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:01:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 -- 172.21.15.16:0/1678746754 shutdown_connections 2022-03-21T14:01:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.287+0000 7f0705af1700 1 -- 172.21.15.16:0/1678746754 wait complete. 2022-03-21T14:01:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 Processor -- start 2022-03-21T14:01:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 -- start start 2022-03-21T14:01:50.292 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07000fbf50 0x7f070010ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fdb80 0x7f07001163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700116910 con 0x7f07000fbf50 2022-03-21T14:01:50.293 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700116a80 con 0x7f07000fd1f0 2022-03-21T14:01:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f0705af1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700116c40 con 0x7f07000fdb80 2022-03-21T14:01:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42648/0 (socket says 172.21.15.16:42648) 2022-03-21T14:01:50.294 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 -- 172.21.15.16:0/299763565 learned_addr learned my addr 172.21.15.16:0/299763565 (peer_addr_for_me v2:172.21.15.16:0/0) 2022-03-21T14:01:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 -- 172.21.15.16:0/299763565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fdb80 msgr2=0x7f07001163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06fffff700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fdb80 0x7f07001163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fdb80 0x7f07001163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 -- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07000fbf50 msgr2=0x7f070010ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:01:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07000fbf50 0x7f070010ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:01:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.288+0000 7f06feffd700 1 -- 172.21.15.16:0/299763565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40093f0 con 0x7f07000fd1f0 2022-03-21T14:01:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f06feffd700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06f4006010 tx=0x7f06f4005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f401d070 con 0x7f07000fd1f0 2022-03-21T14:01:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f06f40054f0 con 0x7f07000fd1f0 2022-03-21T14:01:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0700116ec0 con 0x7f07000fd1f0 2022-03-21T14:01:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4017d10 con 0x7f07000fd1f0 2022-03-21T14:01:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.289+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07001172e0 con 0x7f07000fd1f0 2022-03-21T14:01:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.290+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07001076a0 con 0x7f07000fd1f0 2022-03-21T14:01:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.291+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 125099+0+0 (secure 0 0 0) 0x7f06f4029030 con 0x7f07000fd1f0 2022-03-21T14:01:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.292+0000 7f06e7fff700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 0x7f06e8099170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:01:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.292+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1763+0+0 (secure 0 0 0) 0x7f06f40bf990 con 0x7f07000fd1f0 2022-03-21T14:01:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.292+0000 7f06ff7fe700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 0x7f06e8099170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:01:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.293+0000 7f06ff7fe700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 0x7f06e8099170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07000fd050 tx=0x7f06f0005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:01:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.295+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f06f408e640 con 0x7f07000fd1f0 2022-03-21T14:01:50.479 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:01:50.478+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0700103cd0 con 0x7f06e8096cc0 2022-03-21T14:01:51.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: osdmap e9: 1 total, 1 up, 1 in 2022-03-21T14:01:51.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T14:01:51.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:01:51.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 17277M 2022-03-21T14:01:51.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:51.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:51.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: osdmap e9: 1 total, 1 up, 1 in 2022-03-21T14:01:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T14:01:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:01:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 17277M 2022-03-21T14:01:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: osdmap e9: 1 total, 1 up, 1 in 2022-03-21T14:01:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T14:01:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:01:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 17277M 2022-03-21T14:01:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:01:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:01:51.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:01:51.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:52 smithi002 ceph-mon[17167]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:52 smithi002 ceph-mon[17167]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:52.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:52 smithi016 ceph-mon[22010]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:52 smithi016 ceph-mon[22010]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:52 smithi079 ceph-mon[23158]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:52 smithi079 ceph-mon[23158]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:01:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:53 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/1275038102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:53 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:53 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]': finished 2022-03-21T14:01:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:53 smithi002 ceph-mon[17167]: osdmap e10: 2 total, 1 up, 2 in 2022-03-21T14:01:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:01:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:53 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/1275038102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:53 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:53 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]': finished 2022-03-21T14:01:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:53 smithi016 ceph-mon[22010]: osdmap e10: 2 total, 1 up, 2 in 2022-03-21T14:01:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:01:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:53 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/1275038102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:53 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]: dispatch 2022-03-21T14:01:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:53 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1d0a7a0-443f-4117-aed8-0078291c7c93"}]': finished 2022-03-21T14:01:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:53 smithi079 ceph-mon[23158]: osdmap e10: 2 total, 1 up, 2 in 2022-03-21T14:01:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:01:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:54 smithi002 ceph-mon[17167]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:54 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/402518049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:54 smithi016 ceph-mon[22010]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:54 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/402518049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:54 smithi079 ceph-mon[23158]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:54 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/402518049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:01:56.358 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:56 smithi016 ceph-mon[22010]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:56 smithi002 ceph-mon[17167]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:56 smithi079 ceph-mon[23158]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:58 smithi016 ceph-mon[22010]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:58 smithi002 ceph-mon[17167]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:58 smithi079 ceph-mon[23158]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:59 smithi016 ceph-mon[22010]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-03-21T14:01:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:01:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:59 smithi079 ceph-mon[23158]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-03-21T14:01:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:01:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:01:59.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:59 smithi002 ceph-mon[17167]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:01:59.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-03-21T14:01:59.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:01:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:00 smithi016 ceph-mon[22010]: Deploying daemon osd.1 on smithi016 2022-03-21T14:02:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:00 smithi079 ceph-mon[23158]: Deploying daemon osd.1 on smithi016 2022-03-21T14:02:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:00 smithi002 ceph-mon[17167]: Deploying daemon osd.1 on smithi016 2022-03-21T14:02:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:01 smithi016 ceph-mon[22010]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:01 smithi079 ceph-mon[23158]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:01 smithi002 ceph-mon[17167]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:03 smithi016 ceph-mon[22010]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:03 smithi079 ceph-mon[23158]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:03 smithi002 ceph-mon[17167]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:05.998 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:05 smithi016 ceph-mon[22010]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:05 smithi002 ceph-mon[17167]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:05 smithi079 ceph-mon[23158]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:06.759 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.758+0000 7f06e7fff700 1 -- 172.21.15.16:0/299763565 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0700103cd0 con 0x7f06e8096cc0 2022-03-21T14:02:06.759 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2022-03-21T14:02:06.761 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 msgr2=0x7f06e8099170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:06.761 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 0x7f06e8099170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07000fd050 tx=0x7f06f0005d60 comp rx=0 tx=0).stop 2022-03-21T14:02:06.761 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 msgr2=0x7f070010f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06f4006010 tx=0x7f06f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:02:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 shutdown_connections 2022-03-21T14:02:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f06e8096cc0 0x7f06e8099170 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07000fdb80 0x7f07001163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07000fbf50 0x7f070010ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 --2- 172.21.15.16:0/299763565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fd1f0 0x7f070010f340 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 >> 172.21.15.16:0/299763565 conn(0x7f07000f7720 msgr2=0x7f07000f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:02:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 shutdown_connections 2022-03-21T14:02:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:02:06.760+0000 7f0705af1700 1 -- 172.21.15.16:0/299763565 wait complete. 2022-03-21T14:02:07.302 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:07.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:07.305 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.1.service 2022-03-21T14:02:07.308 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_4... 2022-03-21T14:02:07.308 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-03-21T14:02:07.333 INFO:journalctl@ceph.osd.1.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:02:07.334 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:07.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:07.336 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:07.914 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:07 smithi016 conmon[27157]: 2022-03-21T14:02:07.738+0000 7f2dbf8123c0 -1 osd.1 0 log_to_monitors true 2022-03-21T14:02:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:08 smithi079 ceph-mon[23158]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:08 smithi079 ceph-mon[23158]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:08 smithi079 ceph-mon[23158]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:08 smithi002 ceph-mon[17167]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:08 smithi002 ceph-mon[17167]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:08 smithi002 ceph-mon[17167]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:08 smithi016 ceph-mon[22010]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:08 smithi016 ceph-mon[22010]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:08 smithi016 ceph-mon[22010]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-03-21T14:02:09.043 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-03-21T14:02:09.043 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-03-21T14:02:09.043 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2022-03-21T14:02:09.044 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2022-03-21T14:02:09.044 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109689 s, 956 MB/s 2022-03-21T14:02:09.044 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2022-03-21T14:02:09.338 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:09 smithi016 conmon[27157]: 2022-03-21T14:02:09.023+0000 7f2db6217700 -1 osd.1 0 waiting for initial osdmap 2022-03-21T14:02:09.338 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:09 smithi016 conmon[27157]: 2022-03-21T14:02:09.027+0000 7f2db0bad700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:02:09.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:09 smithi016 ceph-mon[22010]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-03-21T14:02:09.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:09 smithi016 ceph-mon[22010]: osdmap e11: 2 total, 1 up, 2 in 2022-03-21T14:02:09.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:09 smithi016 ceph-mon[22010]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:09.340 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:09 smithi016 ceph-mon[22010]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:09 smithi002 ceph-mon[17167]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-03-21T14:02:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:09 smithi002 ceph-mon[17167]: osdmap e11: 2 total, 1 up, 2 in 2022-03-21T14:02:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:09 smithi002 ceph-mon[17167]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:09 smithi002 ceph-mon[17167]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:09 smithi079 ceph-mon[23158]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-03-21T14:02:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:09 smithi079 ceph-mon[23158]: osdmap e11: 2 total, 1 up, 2 in 2022-03-21T14:02:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:09 smithi079 ceph-mon[23158]: from='osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:09 smithi079 ceph-mon[23158]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:02:09.530 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2022-03-21T14:02:10.033 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:10.033 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:02:10.034 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: osdmap e12: 2 total, 1 up, 2 in 2022-03-21T14:02:10.034 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.034 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.034 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.035 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:10.035 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.035 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.308 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:10.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:02:10.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: osdmap e12: 2 total, 1 up, 2 in 2022-03-21T14:02:10.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:02:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: osdmap e12: 2 total, 1 up, 2 in 2022-03-21T14:02:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:10.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:11.061 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- 172.21.15.79:0/3027169532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fc7c0 msgr2=0x7fd518103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:11.061 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/3027169532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fc7c0 0x7fd518103350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd510009c70 tx=0x7fd5100094e0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.062 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- 172.21.15.79:0/3027169532 shutdown_connections 2022-03-21T14:02:11.062 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/3027169532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fade0 0x7fd5180fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.062 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/3027169532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fc7c0 0x7fd518103350 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.062 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/3027169532 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5180fb730 0x7fd5180fc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.063 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- 172.21.15.79:0/3027169532 >> 172.21.15.79:0/3027169532 conn(0x7fd5180f6620 msgr2=0x7fd5180f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:02:11.063 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- 172.21.15.79:0/3027169532 shutdown_connections 2022-03-21T14:02:11.063 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- 172.21.15.79:0/3027169532 wait complete. 2022-03-21T14:02:11.064 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 Processor -- start 2022-03-21T14:02:11.065 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- start start 2022-03-21T14:02:11.065 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:11.065 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fb730 0x7fd518111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:11.065 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5180fc7c0 0x7fd518113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:11.066 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518114530 con 0x7fd5180fc7c0 2022-03-21T14:02:11.066 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5181146a0 con 0x7fd5180fb730 2022-03-21T14:02:11.066 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51cd4e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518114860 con 0x7fd5180fade0 2022-03-21T14:02:11.066 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.058+0000 7fd51659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:11.067 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33232/0 (socket says 172.21.15.79:33232) 2022-03-21T14:02:11.067 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 -- 172.21.15.79:0/21034013 learned_addr learned my addr 172.21.15.79:0/21034013 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:02:11.067 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd515d9b700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fb730 0x7fd518111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:11.067 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 -- 172.21.15.79:0/21034013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fb730 msgr2=0x7fd518111820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:11.068 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fb730 0x7fd518111820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.068 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 -- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5180fc7c0 msgr2=0x7fd518113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:11.068 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5180fc7c0 0x7fd518113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:11.069 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 -- 172.21.15.79:0/21034013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510009780 con 0x7fd5180fade0 2022-03-21T14:02:11.069 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51659c700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd50800d790 tx=0x7fd50800b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:02:11.069 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508004020 con 0x7fd5180fade0 2022-03-21T14:02:11.069 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd508004180 con 0x7fd5180fade0 2022-03-21T14:02:11.070 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508003990 con 0x7fd5180fade0 2022-03-21T14:02:11.070 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd518114b40 con 0x7fd5180fade0 2022-03-21T14:02:11.070 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.059+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd518115060 con 0x7fd5180fade0 2022-03-21T14:02:11.070 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.060+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd518106630 con 0x7fd5180fade0 2022-03-21T14:02:11.072 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.061+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 15) v1 ==== 125099+0+0 (secure 0 0 0) 0x7fd508022070 con 0x7fd5180fade0 2022-03-21T14:02:11.072 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.061+0000 7fd506ffd700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 0x7fd500099170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:11.072 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.061+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2305+0+0 (secure 0 0 0) 0x7fd508016030 con 0x7fd5180fade0 2022-03-21T14:02:11.073 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.062+0000 7fd515d9b700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 0x7fd500099170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:11.073 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.062+0000 7fd515d9b700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 0x7fd500099170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd5180fabb0 tx=0x7fd50c00a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:02:11.073 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.066+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd50808c6f0 con 0x7fd5180fade0 2022-03-21T14:02:11.240 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:11.239+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd51805d410 con 0x7fd500096cc0 2022-03-21T14:02:11.297 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:02:11.297 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:02:11.297 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T14:02:11.298 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 17176M 2022-03-21T14:02:11.298 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] boot 2022-03-21T14:02:11.298 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: osdmap e13: 2 total, 2 up, 2 in 2022-03-21T14:02:11.298 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:02:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:02:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T14:02:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 17176M 2022-03-21T14:02:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] boot 2022-03-21T14:02:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: osdmap e13: 2 total, 2 up, 2 in 2022-03-21T14:02:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:02:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:02:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T14:02:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 17176M 2022-03-21T14:02:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: osd.1 [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] boot 2022-03-21T14:02:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: osdmap e13: 2 total, 2 up, 2 in 2022-03-21T14:02:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T14:02:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:12 smithi079 ceph-mon[23158]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:12 smithi079 ceph-mon[23158]: osdmap e14: 2 total, 2 up, 2 in 2022-03-21T14:02:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:12 smithi002 ceph-mon[17167]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:12 smithi002 ceph-mon[17167]: osdmap e14: 2 total, 2 up, 2 in 2022-03-21T14:02:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:12 smithi016 ceph-mon[22010]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-03-21T14:02:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:12 smithi016 ceph-mon[22010]: osdmap e14: 2 total, 2 up, 2 in 2022-03-21T14:02:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:13 smithi002 ceph-mon[17167]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:13 smithi016 ceph-mon[22010]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:13 smithi079 ceph-mon[23158]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/181772420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]': finished 2022-03-21T14:02:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: osdmap e15: 3 total, 2 up, 3 in 2022-03-21T14:02:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:14 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1804064793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/181772420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]': finished 2022-03-21T14:02:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: osdmap e15: 3 total, 2 up, 3 in 2022-03-21T14:02:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:14 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1804064793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/181772420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]: dispatch 2022-03-21T14:02:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f5778a-ca84-43c0-8233-005fbbf6da43"}]': finished 2022-03-21T14:02:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: osdmap e15: 3 total, 2 up, 3 in 2022-03-21T14:02:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:14 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1804064793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:15 smithi002 ceph-mon[17167]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:15 smithi016 ceph-mon[22010]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:15.552 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:15 smithi079 ceph-mon[23158]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:17.588 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:17 smithi079 ceph-mon[23158]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:17 smithi016 ceph-mon[22010]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:17 smithi002 ceph-mon[17167]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:19.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:19 smithi079 ceph-mon[23158]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:19 smithi016 ceph-mon[22010]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:19 smithi002 ceph-mon[17167]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-03-21T14:02:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:20 smithi016 ceph-mon[22010]: Deploying daemon osd.2 on smithi079 2022-03-21T14:02:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:02:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:02:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-03-21T14:02:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:20 smithi079 ceph-mon[23158]: Deploying daemon osd.2 on smithi079 2022-03-21T14:02:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:02:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:02:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-03-21T14:02:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:20 smithi002 ceph-mon[17167]: Deploying daemon osd.2 on smithi079 2022-03-21T14:02:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:02:20.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:02:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:21 smithi016 ceph-mon[22010]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:21 smithi079 ceph-mon[23158]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:21 smithi002 ceph-mon[17167]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:23 smithi016 ceph-mon[22010]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:23 smithi079 ceph-mon[23158]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:23 smithi002 ceph-mon[17167]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:25.606 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:25 smithi079 ceph-mon[23158]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:25.607 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:25.607 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:25.607 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:25.608 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:25 smithi002 ceph-mon[17167]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:25 smithi016 ceph-mon[22010]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:27.046 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.045+0000 7fd506ffd700 1 -- 172.21.15.79:0/21034013 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd51805d410 con 0x7fd500096cc0 2022-03-21T14:02:27.047 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2022-03-21T14:02:27.048 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 msgr2=0x7fd500099170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:27.049 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 0x7fd500099170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd5180fabb0 tx=0x7fd50c00a250 comp rx=0 tx=0).stop 2022-03-21T14:02:27.049 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 msgr2=0x7fd518065e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:27.049 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd50800d790 tx=0x7fd50800b3a0 comp rx=0 tx=0).stop 2022-03-21T14:02:27.049 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 shutdown_connections 2022-03-21T14:02:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd500096cc0 0x7fd500099170 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5180fc7c0 0x7fd518113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd5180fade0 0x7fd518065e10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 --2- 172.21.15.79:0/21034013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5180fb730 0x7fd518111820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:27.050 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 >> 172.21.15.79:0/21034013 conn(0x7fd5180f6620 msgr2=0x7fd5180f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:02:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 shutdown_connections 2022-03-21T14:02:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:02:27.047+0000 7fd51cd4e700 1 -- 172.21.15.79:0/21034013 wait complete. 2022-03-21T14:02:27.514 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.2.service 2022-03-21T14:02:27.517 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_3... 2022-03-21T14:02:27.517 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-03-21T14:02:27.548 INFO:journalctl@ceph.osd.2.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:02:27.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:27.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:27.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:27.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:28 smithi002 ceph-mon[17167]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:28 smithi002 ceph-mon[17167]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:28 smithi016 ceph-mon[22010]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:28 smithi016 ceph-mon[22010]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:28.722 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:28 smithi079 conmon[28754]: 2022-03-21T14:02:28.269+0000 7f36105503c0 -1 osd.2 0 log_to_monitors true 2022-03-21T14:02:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:28 smithi079 ceph-mon[23158]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:28 smithi079 ceph-mon[23158]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-03-21T14:02:29.041 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:02:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-03-21T14:02:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: osdmap e16: 3 total, 2 up, 3 in 2022-03-21T14:02:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:29 smithi002 ceph-mon[17167]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-03-21T14:02:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: osdmap e16: 3 total, 2 up, 3 in 2022-03-21T14:02:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:29 smithi016 ceph-mon[22010]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:29.722 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:29 smithi079 conmon[28754]: 2022-03-21T14:02:29.379+0000 7f3606f55700 -1 osd.2 0 waiting for initial osdmap 2022-03-21T14:02:29.723 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:29 smithi079 conmon[28754]: 2022-03-21T14:02:29.384+0000 7f36018eb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:02:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-03-21T14:02:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: osdmap e16: 3 total, 2 up, 3 in 2022-03-21T14:02:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: from='osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:29 smithi079 ceph-mon[23158]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:02:30.032 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-03-21T14:02:30.032 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-03-21T14:02:30.032 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-03-21T14:02:30.033 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-03-21T14:02:30.033 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0420657 s, 249 MB/s 2022-03-21T14:02:30.033 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-03-21T14:02:30.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:02:30.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: osdmap e17: 3 total, 2 up, 3 in 2022-03-21T14:02:30.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:30.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.650 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2022-03-21T14:02:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:02:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: osdmap e17: 3 total, 2 up, 3 in 2022-03-21T14:02:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:02:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: osdmap e17: 3 total, 2 up, 3 in 2022-03-21T14:02:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:02:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:02:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:02:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T14:02:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 17172M 2022-03-21T14:02:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] boot 2022-03-21T14:02:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: osdmap e18: 3 total, 3 up, 3 in 2022-03-21T14:02:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:02:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:02:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T14:02:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 17172M 2022-03-21T14:02:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] boot 2022-03-21T14:02:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: osdmap e18: 3 total, 3 up, 3 in 2022-03-21T14:02:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:31.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:02:31.803 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:02:31.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T14:02:31.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 17172M 2022-03-21T14:02:31.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-03-21T14:02:31.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: osd.2 [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] boot 2022-03-21T14:02:31.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: osdmap e18: 3 total, 3 up, 3 in 2022-03-21T14:02:31.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T14:02:31.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:32.097 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:02:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:32 smithi002 ceph-mon[17167]: osdmap e19: 3 total, 3 up, 3 in 2022-03-21T14:02:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 -- 172.21.15.2:0/514800702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fcfe0 msgr2=0x7f5c540fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:32.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/514800702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fcfe0 0x7f5c540fd450 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5c440098e0 tx=0x7f5c44005f70 comp rx=0 tx=0).stop 2022-03-21T14:02:32.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 -- 172.21.15.2:0/514800702 shutdown_connections 2022-03-21T14:02:32.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/514800702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fd990 0x7f5c541042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:32.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/514800702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fbde0 0x7f5c540fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:32.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/514800702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fcfe0 0x7f5c540fd450 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:32.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 -- 172.21.15.2:0/514800702 >> 172.21.15.2:0/514800702 conn(0x7f5c540f75f0 msgr2=0x7f5c540f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:02:32.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 -- 172.21.15.2:0/514800702 shutdown_connections 2022-03-21T14:02:32.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.616+0000 7f5c5a89c700 1 -- 172.21.15.2:0/514800702 wait complete. 2022-03-21T14:02:32.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 Processor -- start 2022-03-21T14:02:32.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 -- start start 2022-03-21T14:02:32.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 0x7f5c5410ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:32.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fcfe0 0x7f5c5410f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:32.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:32.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541168a0 con 0x7f5c540fbde0 2022-03-21T14:02:32.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54116a10 con 0x7f5c540fd990 2022-03-21T14:02:32.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.617+0000 7f5c5a89c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54116bd0 con 0x7f5c540fcfe0 2022-03-21T14:02:32.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:32.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48144/0 (socket says 172.21.15.2:48144) 2022-03-21T14:02:32.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 -- 172.21.15.2:0/3172042077 learned_addr learned my addr 172.21.15.2:0/3172042077 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:02:32.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5989a700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 0x7f5c5410ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:32.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c59099700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fcfe0 0x7f5c5410f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:32.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 -- 172.21.15.2:0/3172042077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fcfe0 msgr2=0x7f5c5410f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:32.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fcfe0 0x7f5c5410f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 -- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 msgr2=0x7f5c5410ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 0x7f5c5410ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5a09b700 1 -- 172.21.15.2:0/3172042077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c440093f0 con 0x7f5c540fd990 2022-03-21T14:02:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.618+0000 7f5c5989a700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 0x7f5c5410ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:02:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c5a09b700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00ec50 tx=0x7f5c4c009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:02:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00cb50 con 0x7f5c540fd990 2022-03-21T14:02:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5c4c00ccb0 con 0x7f5c540fd990 2022-03-21T14:02:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c54116eb0 con 0x7f5c540fd990 2022-03-21T14:02:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c018690 con 0x7f5c540fd990 2022-03-21T14:02:32.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.619+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c541172c0 con 0x7f5c540fd990 2022-03-21T14:02:32.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.620+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54107670 con 0x7f5c540fd990 2022-03-21T14:02:32.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.622+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 125099+0+0 (secure 0 0 0) 0x7f5c4c00c040 con 0x7f5c540fd990 2022-03-21T14:02:32.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.623+0000 7f5c4a7fc700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 0x7f5c40099170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:02:32.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.623+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3233+0+0 (secure 0 0 0) 0x7f5c4c014070 con 0x7f5c540fd990 2022-03-21T14:02:32.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.624+0000 7f5c5989a700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 0x7f5c40099170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:02:32.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.624+0000 7f5c5989a700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 0x7f5c40099170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005fd0 tx=0x7f5c50005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:02:32.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.627+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f5c4c08d170 con 0x7f5c540fd990 2022-03-21T14:02:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:32 smithi016 ceph-mon[22010]: osdmap e19: 3 total, 3 up, 3 in 2022-03-21T14:02:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:32 smithi079 ceph-mon[23158]: osdmap e19: 3 total, 3 up, 3 in 2022-03-21T14:02:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-03-21T14:02:32.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:32.822+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5402ac70 con 0x7f5c40096cc0 2022-03-21T14:02:33.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:33.416+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f5c4c0bd800 con 0x7f5c540fd990 2022-03-21T14:02:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: osdmap e20: 3 total, 3 up, 3 in 2022-03-21T14:02:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:33.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:33 smithi002 sudo[33186]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:33.618 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:33 smithi002 sudo[33186]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:33.618 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:33 smithi002 sudo[33186]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:33.618 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:33 smithi002 sudo[33186]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: osdmap e20: 3 total, 3 up, 3 in 2022-03-21T14:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-03-21T14:02:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: osdmap e20: 3 total, 3 up, 3 in 2022-03-21T14:02:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:02:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:02:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:02:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:34.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:34 smithi002 ceph-mon[17167]: osdmap e21: 3 total, 3 up, 3 in 2022-03-21T14:02:34.526 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:34 smithi002 ceph-mon[17167]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-03-21T14:02:34.526 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:34.526 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:34 smithi002 sudo[33314]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:34.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:34 smithi002 sudo[33314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:34.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:34 smithi002 sudo[33314]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:34.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 21 14:02:34 smithi002 sudo[33314]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:34.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:34 smithi016 ceph-mon[22010]: osdmap e21: 3 total, 3 up, 3 in 2022-03-21T14:02:34.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:34 smithi016 ceph-mon[22010]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-03-21T14:02:34.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:34 smithi079 ceph-mon[23158]: osdmap e21: 3 total, 3 up, 3 in 2022-03-21T14:02:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:34 smithi079 ceph-mon[23158]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-03-21T14:02:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:35.122 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:34 smithi016 sudo[29197]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:35.123 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:34 smithi016 sudo[29197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:35.123 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:34 smithi016 sudo[29197]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:35.123 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:34 smithi016 sudo[29197]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:35.413 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:35 smithi016 sudo[29200]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:35.414 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:35 smithi016 sudo[29200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:35.414 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:35 smithi016 sudo[29200]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:35.414 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 14:02:35 smithi016 sudo[29200]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:35.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:35 smithi079 ceph-mon[23158]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:35.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:35.647 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30777]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:35.648 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30777]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:35.648 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30777]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:35.648 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30777]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:35 smithi002 ceph-mon[17167]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:35 smithi016 ceph-mon[22010]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:35.972 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30780]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:35.972 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:35.973 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30780]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:35.973 INFO:journalctl@ceph.osd.2.smithi079.stdout:Mar 21 14:02:35 smithi079 sudo[30780]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:36 smithi002 sudo[33459]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:36 smithi002 sudo[33459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:36 smithi002 sudo[33459]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:36 smithi002 sudo[33459]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:37 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 sudo[33462]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 sudo[33462]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 sudo[33462]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 sudo[33462]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:37 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:37.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:37.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 sudo[29203]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 sudo[29203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 sudo[29203]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:37 smithi016 sudo[29203]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 sudo[29206]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 sudo[29206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 sudo[29206]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 sudo[29206]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:38.580 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30783]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-03-21T14:02:38.580 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30783]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:38.580 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30783]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:38.581 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30783]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3379078000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]': finished 2022-03-21T14:02:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: osdmap e22: 4 total, 3 up, 4 in 2022-03-21T14:02:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3379078000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]': finished 2022-03-21T14:02:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: osdmap e22: 4 total, 3 up, 4 in 2022-03-21T14:02:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30786]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-03-21T14:02:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30786]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-03-21T14:02:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30786]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-03-21T14:02:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 sudo[30786]: pam_unix(sudo:session): session closed for user root 2022-03-21T14:02:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3379078000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]: dispatch 2022-03-21T14:02:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c84afd54-8463-4dc6-ac84-f8c56d385bc4"}]': finished 2022-03-21T14:02:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: osdmap e22: 4 total, 3 up, 4 in 2022-03-21T14:02:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T14:02:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T14:02:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T14:02:39.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:39 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:39.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:39 smithi002 ceph-mon[17167]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:39.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:39 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2306854116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:39.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:39 smithi002 ceph-mon[17167]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:39 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:39 smithi016 ceph-mon[22010]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:39 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2306854116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:39 smithi016 ceph-mon[22010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:39 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-03-21T14:02:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:39 smithi079 ceph-mon[23158]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:39 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2306854116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:02:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:39 smithi079 ceph-mon[23158]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-03-21T14:02:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:41 smithi002 ceph-mon[17167]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:41 smithi016 ceph-mon[22010]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:41 smithi079 ceph-mon[23158]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:43 smithi002 ceph-mon[17167]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-03-21T14:02:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:43 smithi016 ceph-mon[22010]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-03-21T14:02:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:43 smithi079 ceph-mon[23158]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-03-21T14:02:45.809 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:45 smithi002 ceph-mon[17167]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-03-21T14:02:45.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:45 smithi016 ceph-mon[22010]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-03-21T14:02:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:45 smithi079 ceph-mon[23158]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-03-21T14:02:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-03-21T14:02:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-03-21T14:02:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-03-21T14:02:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:47 smithi002 ceph-mon[17167]: Deploying daemon osd.3 on smithi002 2022-03-21T14:02:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:47 smithi002 ceph-mon[17167]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-03-21T14:02:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:47 smithi016 ceph-mon[22010]: Deploying daemon osd.3 on smithi002 2022-03-21T14:02:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:47 smithi016 ceph-mon[22010]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-03-21T14:02:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:47 smithi079 ceph-mon[23158]: Deploying daemon osd.3 on smithi002 2022-03-21T14:02:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:47 smithi079 ceph-mon[23158]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-03-21T14:02:49.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:49 smithi002 ceph-mon[17167]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 827 B/s rd, 24 KiB/s wr, 2 op/s 2022-03-21T14:02:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:49 smithi016 ceph-mon[22010]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 827 B/s rd, 24 KiB/s wr, 2 op/s 2022-03-21T14:02:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:49 smithi079 ceph-mon[23158]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 827 B/s rd, 24 KiB/s wr, 2 op/s 2022-03-21T14:02:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:51 smithi002 ceph-mon[17167]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:51 smithi016 ceph-mon[22010]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:51 smithi079 ceph-mon[23158]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:53 smithi002 ceph-mon[17167]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:53 smithi016 ceph-mon[22010]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:53 smithi079 ceph-mon[23158]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-03-21T14:02:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:55 smithi016 ceph-mon[22010]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:55 smithi079 ceph-mon[23158]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:55 smithi002 ceph-mon[17167]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:56.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.669+0000 7f5c4a7fc700 1 -- 172.21.15.2:0/3172042077 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5c5402ac70 con 0x7f5c40096cc0 2022-03-21T14:02:56.670 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2022-03-21T14:02:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 msgr2=0x7f5c40099170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 0x7f5c40099170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005fd0 tx=0x7f5c50005eb0 comp rx=0 tx=0).stop 2022-03-21T14:02:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 msgr2=0x7f5c54116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:02:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00ec50 tx=0x7f5c4c009430 comp rx=0 tx=0).stop 2022-03-21T14:02:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 shutdown_connections 2022-03-21T14:02:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c540fd990 0x7f5c54116360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5c40096cc0 0x7f5c40099170 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:56.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c540fbde0 0x7f5c5410ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:56.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 --2- 172.21.15.2:0/3172042077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c540fcfe0 0x7f5c5410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:02:56.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 >> 172.21.15.2:0/3172042077 conn(0x7f5c540f75f0 msgr2=0x7f5c54102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:02:56.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 shutdown_connections 2022-03-21T14:02:56.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:02:56.672+0000 7f5c5a89c700 1 -- 172.21.15.2:0/3172042077 wait complete. 2022-03-21T14:02:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:57.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:57.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:57.141 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.3.service 2022-03-21T14:02:57.146 INFO:tasks.cephadm:Deploying osd.4 on smithi016 with /dev/vg_nvme/lv_3... 2022-03-21T14:02:57.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-03-21T14:02:57.220 INFO:journalctl@ceph.osd.3.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:02:57.668 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 21 14:02:57 smithi002 conmon[36028]: 2022-03-21T14:02:57.393+0000 7f098fb7a3c0 -1 osd.3 0 log_to_monitors true 2022-03-21T14:02:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:57 smithi079 ceph-mon[23158]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:57.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:57.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:57.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:57.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:57.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:57.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:57.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:57 smithi002 ceph-mon[17167]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:58.024 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:58.024 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:02:58.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:02:58.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:02:58.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:02:58.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:58.026 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:57 smithi016 ceph-mon[22010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-03-21T14:02:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:58 smithi002 ceph-mon[17167]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-03-21T14:02:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:58 smithi002 ceph-mon[17167]: osdmap e23: 4 total, 3 up, 4 in 2022-03-21T14:02:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:58 smithi002 ceph-mon[17167]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:58 smithi002 ceph-mon[17167]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:58.868 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 21 14:02:58 smithi002 conmon[36028]: 2022-03-21T14:02:58.683+0000 7f098657f700 -1 osd.3 0 waiting for initial osdmap 2022-03-21T14:02:58.868 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 21 14:02:58 smithi002 conmon[36028]: 2022-03-21T14:02:58.692+0000 7f097ef11700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:02:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:58 smithi016 ceph-mon[22010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-03-21T14:02:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:58 smithi016 ceph-mon[22010]: osdmap e23: 4 total, 3 up, 4 in 2022-03-21T14:02:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:58 smithi016 ceph-mon[22010]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:58 smithi016 ceph-mon[22010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:58.968 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-03-21T14:02:58.968 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-03-21T14:02:58.969 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2022-03-21T14:02:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2022-03-21T14:02:58.969 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00802401 s, 1.3 GB/s 2022-03-21T14:02:58.969 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2022-03-21T14:02:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:58 smithi079 ceph-mon[23158]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-03-21T14:02:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:58 smithi079 ceph-mon[23158]: osdmap e23: 4 total, 3 up, 4 in 2022-03-21T14:02:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:58 smithi079 ceph-mon[23158]: from='osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:58 smithi079 ceph-mon[23158]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:02:59.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2022-03-21T14:02:59.946 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:59 smithi016 ceph-mon[22010]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:59.947 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:59 smithi016 ceph-mon[22010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:02:59.947 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:59 smithi016 ceph-mon[22010]: osdmap e24: 4 total, 3 up, 4 in 2022-03-21T14:02:59.947 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:59.947 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:02:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:59 smithi079 ceph-mon[23158]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:02:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:59 smithi079 ceph-mon[23158]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:02:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:59 smithi079 ceph-mon[23158]: osdmap e24: 4 total, 3 up, 4 in 2022-03-21T14:02:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:02:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:02:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:59 smithi002 ceph-mon[17167]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-03-21T14:03:00.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:59 smithi002 ceph-mon[17167]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:03:00.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:59 smithi002 ceph-mon[17167]: osdmap e24: 4 total, 3 up, 4 in 2022-03-21T14:03:00.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:00.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:02:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:00 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:03:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:00 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:03:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:00 smithi002 ceph-mon[17167]: osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] boot 2022-03-21T14:03:00.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:00 smithi002 ceph-mon[17167]: osdmap e25: 4 total, 4 up, 4 in 2022-03-21T14:03:00.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:00 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:03:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:00 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:03:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:00 smithi079 ceph-mon[23158]: osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] boot 2022-03-21T14:03:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:00 smithi079 ceph-mon[23158]: osdmap e25: 4 total, 4 up, 4 in 2022-03-21T14:03:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 -- 172.21.15.16:0/1806029386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 msgr2=0x7f2688104420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 --2- 172.21.15.16:0/1806029386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688104420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2680009c70 tx=0x7f26800094e0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 -- 172.21.15.16:0/1806029386 shutdown_connections 2022-03-21T14:03:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 --2- 172.21.15.16:0/1806029386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688104420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 --2- 172.21.15.16:0/1806029386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fbf30 0x7f26880fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 --2- 172.21.15.16:0/1806029386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fd1d0 0x7f26880fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 -- 172.21.15.16:0/1806029386 >> 172.21.15.16:0/1806029386 conn(0x7f26880f7720 msgr2=0x7f26880f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:03:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 -- 172.21.15.16:0/1806029386 shutdown_connections 2022-03-21T14:03:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.974+0000 7f269074b700 1 -- 172.21.15.16:0/1806029386 wait complete. 2022-03-21T14:03:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 Processor -- start 2022-03-21T14:03:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 -- start start 2022-03-21T14:03:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fbf30 0x7f268810c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fd1d0 0x7f268810cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26881144e0 con 0x7f26880fd1d0 2022-03-21T14:03:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688114650 con 0x7f26880fdb60 2022-03-21T14:03:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f269074b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688114810 con 0x7f26880fbf30 2022-03-21T14:03:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42730/0 (socket says 172.21.15.16:42730) 2022-03-21T14:03:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 -- 172.21.15.16:0/3272235775 learned_addr learned my addr 172.21.15.16:0/3272235775 (peer_addr_for_me v2:172.21.15.16:0/0) 2022-03-21T14:03:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268e4e7700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fbf30 0x7f268810c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268dce6700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fd1d0 0x7f268810cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 -- 172.21.15.16:0/3272235775 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fbf30 msgr2=0x7f268810c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fbf30 0x7f268810c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 -- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fd1d0 msgr2=0x7f268810cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fd1d0 0x7f268810cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.975+0000 7f268ece8700 1 -- 172.21.15.16:0/3272235775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009780 con 0x7f26880fdb60 2022-03-21T14:03:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f268ece8700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f268000b950 tx=0x7f26800094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680005ac0 con 0x7f26880fdb60 2022-03-21T14:03:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688114a90 con 0x7f26880fdb60 2022-03-21T14:03:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2680003cf0 con 0x7f26880fdb60 2022-03-21T14:03:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26800052b0 con 0x7f26880fdb60 2022-03-21T14:03:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.976+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2688114f20 con 0x7f26880fdb60 2022-03-21T14:03:00.992 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.977+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26881076e0 con 0x7f26880fdb60 2022-03-21T14:03:00.992 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.978+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f268000c040 con 0x7f26880fdb60 2022-03-21T14:03:00.993 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.978+0000 7f267effd700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 0x7f26740991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:00.993 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.978+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3694+0+0 (secure 0 0 0) 0x7f26800c0c10 con 0x7f26880fdb60 2022-03-21T14:03:00.994 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.979+0000 7f268e4e7700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 0x7f26740991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:00.994 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.979+0000 7f268e4e7700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 0x7f26740991c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2684005fd0 tx=0x7f268400b2f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:00.994 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:00.982+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f268008f0f0 con 0x7f26880fdb60 2022-03-21T14:03:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:00 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:00 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:00 smithi016 ceph-mon[22010]: osd.3 [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] boot 2022-03-21T14:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:00 smithi016 ceph-mon[22010]: osdmap e25: 4 total, 4 up, 4 in 2022-03-21T14:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T14:03:01.166 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:01.166+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f268802abf0 con 0x7f2674096d10 2022-03-21T14:03:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: osdmap e26: 4 total, 4 up, 4 in 2022-03-21T14:03:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T14:03:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 8638M 2022-03-21T14:03:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:02.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:02.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: osdmap e26: 4 total, 4 up, 4 in 2022-03-21T14:03:02.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T14:03:02.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:02.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:02.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 8638M 2022-03-21T14:03:02.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:02.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:02.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: osdmap e26: 4 total, 4 up, 4 in 2022-03-21T14:03:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T14:03:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:02.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:02.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 8638M 2022-03-21T14:03:02.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:02.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:02.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:02.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:02.755 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:02 smithi016 ceph-mon[22010]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:02.756 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:02 smithi016 ceph-mon[22010]: osdmap e27: 4 total, 4 up, 4 in 2022-03-21T14:03:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:02 smithi079 ceph-mon[23158]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:02 smithi079 ceph-mon[23158]: osdmap e27: 4 total, 4 up, 4 in 2022-03-21T14:03:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:02 smithi002 ceph-mon[17167]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:02 smithi002 ceph-mon[17167]: osdmap e27: 4 total, 4 up, 4 in 2022-03-21T14:03:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: osdmap e28: 4 total, 4 up, 4 in 2022-03-21T14:03:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/4094336058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]': finished 2022-03-21T14:03:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: osdmap e29: 5 total, 4 up, 5 in 2022-03-21T14:03:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: osdmap e28: 4 total, 4 up, 4 in 2022-03-21T14:03:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/4094336058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]': finished 2022-03-21T14:03:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: osdmap e29: 5 total, 4 up, 5 in 2022-03-21T14:03:04.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: osdmap e28: 4 total, 4 up, 4 in 2022-03-21T14:03:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/4094336058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]: dispatch 2022-03-21T14:03:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7283335-3a3f-4840-b028-f85250ba113e"}]': finished 2022-03-21T14:03:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: osdmap e29: 5 total, 4 up, 5 in 2022-03-21T14:03:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:04 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/1428399052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:04 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/1428399052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:04 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/1428399052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:05 smithi079 ceph-mon[23158]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:05 smithi002 ceph-mon[17167]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:05 smithi016 ceph-mon[22010]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:07 smithi079 ceph-mon[23158]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2022-03-21T14:03:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:07 smithi002 ceph-mon[17167]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2022-03-21T14:03:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:07 smithi016 ceph-mon[22010]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2022-03-21T14:03:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:09 smithi016 ceph-mon[22010]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2022-03-21T14:03:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:09 smithi079 ceph-mon[23158]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2022-03-21T14:03:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:09 smithi002 ceph-mon[17167]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2022-03-21T14:03:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-03-21T14:03:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:10 smithi016 ceph-mon[22010]: Deploying daemon osd.4 on smithi016 2022-03-21T14:03:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-03-21T14:03:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:10 smithi002 ceph-mon[17167]: Deploying daemon osd.4 on smithi016 2022-03-21T14:03:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-03-21T14:03:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:10 smithi079 ceph-mon[23158]: Deploying daemon osd.4 on smithi016 2022-03-21T14:03:11.874 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:11 smithi016 ceph-mon[22010]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:03:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:11 smithi002 ceph-mon[17167]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:03:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:11 smithi079 ceph-mon[23158]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:03:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:13 smithi016 ceph-mon[22010]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2022-03-21T14:03:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:13 smithi002 ceph-mon[17167]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2022-03-21T14:03:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:13 smithi079 ceph-mon[23158]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2022-03-21T14:03:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:15 smithi079 ceph-mon[23158]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-03-21T14:03:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:15 smithi002 ceph-mon[17167]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-03-21T14:03:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:15 smithi016 ceph-mon[22010]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2022-03-21T14:03:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.765+0000 7f267effd700 1 -- 172.21.15.16:0/3272235775 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f268802abf0 con 0x7f2674096d10 2022-03-21T14:03:17.767 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 4 on host 'smithi016' 2022-03-21T14:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 msgr2=0x7f26740991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 0x7f26740991c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2684005fd0 tx=0x7f268400b2f0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 msgr2=0x7f2688113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f268000b950 tx=0x7f26800094e0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 shutdown_connections 2022-03-21T14:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26880fdb60 0x7f2688113fa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2674096d10 0x7f26740991c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f26880fbf30 0x7f268810c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 --2- 172.21.15.16:0/3272235775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26880fd1d0 0x7f268810cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.767+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 >> 172.21.15.16:0/3272235775 conn(0x7f26880f7720 msgr2=0x7f26880f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.768+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 shutdown_connections 2022-03-21T14:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:03:17.768+0000 7f269074b700 1 -- 172.21.15.16:0/3272235775 wait complete. 2022-03-21T14:03:18.005 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:03:18.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:18.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:18.007 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:18.221 DEBUG:teuthology.orchestra.run.smithi016:osd.4> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.4.service 2022-03-21T14:03:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:03:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:18.224 INFO:tasks.cephadm:Deploying osd.5 on smithi079 with /dev/vg_nvme/lv_3... 2022-03-21T14:03:18.225 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-03-21T14:03:18.252 INFO:journalctl@ceph.osd.4.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:03:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 14:03:18 smithi016 conmon[32727]: 2022-03-21T14:03:18.908+0000 7ff71ad1f3c0 -1 osd.4 0 log_to_monitors true 2022-03-21T14:03:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:19 smithi016 ceph-mon[22010]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:19 smithi016 ceph-mon[22010]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:19 smithi016 ceph-mon[22010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:19.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:19 smithi079 ceph-mon[23158]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:19.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:19 smithi079 ceph-mon[23158]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:19.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:19 smithi079 ceph-mon[23158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:19 smithi002 ceph-mon[17167]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:19 smithi002 ceph-mon[17167]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:19.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:19 smithi002 ceph-mon[17167]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-03-21T14:03:20.125 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-03-21T14:03:20.125 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-03-21T14:03:20.125 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2022-03-21T14:03:20.125 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2022-03-21T14:03:20.126 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126745 s, 827 MB/s 2022-03-21T14:03:20.126 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2022-03-21T14:03:20.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2022-03-21T14:03:20.656 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 14:03:20 smithi016 conmon[32727]: 2022-03-21T14:03:20.427+0000 7ff711724700 -1 osd.4 0 waiting for initial osdmap 2022-03-21T14:03:20.657 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 14:03:20 smithi016 conmon[32727]: 2022-03-21T14:03:20.434+0000 7ff70a8b7700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:03:20.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-03-21T14:03:20.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: osdmap e30: 5 total, 4 up, 5 in 2022-03-21T14:03:20.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:20.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:03:20.659 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:03:20.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-03-21T14:03:20.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: osdmap e30: 5 total, 4 up, 5 in 2022-03-21T14:03:20.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:20.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.679 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.679 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:03:20.679 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:03:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-03-21T14:03:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: osdmap e30: 5 total, 4 up, 5 in 2022-03-21T14:03:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:03:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:03:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:03:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:03:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: osdmap e31: 5 total, 4 up, 5 in 2022-03-21T14:03:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T14:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 8588M 2022-03-21T14:03:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:03:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: osdmap e31: 5 total, 4 up, 5 in 2022-03-21T14:03:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T14:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 8588M 2022-03-21T14:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:03:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: osdmap e31: 5 total, 4 up, 5 in 2022-03-21T14:03:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T14:03:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 8588M 2022-03-21T14:03:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:22.186 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- 172.21.15.79:0/4140164599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fbf30 msgr2=0x7fd81c0fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:22.186 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 --2- 172.21.15.79:0/4140164599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fbf30 0x7fd81c0fc340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8180098e0 tx=0x7fd818005f70 comp rx=0 tx=0).stop 2022-03-21T14:03:22.186 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- 172.21.15.79:0/4140164599 shutdown_connections 2022-03-21T14:03:22.187 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 --2- 172.21.15.79:0/4140164599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fdb60 0x7fd81c104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:22.187 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 --2- 172.21.15.79:0/4140164599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fbf30 0x7fd81c0fc340 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:22.187 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 --2- 172.21.15.79:0/4140164599 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c0fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:22.187 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- 172.21.15.79:0/4140164599 >> 172.21.15.79:0/4140164599 conn(0x7fd81c0f7720 msgr2=0x7fd81c0f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:03:22.188 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- 172.21.15.79:0/4140164599 shutdown_connections 2022-03-21T14:03:22.188 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- 172.21.15.79:0/4140164599 wait complete. 2022-03-21T14:03:22.188 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 Processor -- start 2022-03-21T14:03:22.188 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 -- start start 2022-03-21T14:03:22.189 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.184+0000 7fd82444a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fbf30 0x7fd81c10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:22.189 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:22.189 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd81c0fdb60 0x7fd81c114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:22.189 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c114b70 con 0x7fd81c0fdb60 2022-03-21T14:03:22.189 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c114ce0 con 0x7fd81c0fbf30 2022-03-21T14:03:22.190 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c114ea0 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.190 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:22.190 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33318/0 (socket says 172.21.15.79:33318) 2022-03-21T14:03:22.464 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 -- 172.21.15.79:0/1600829343 learned_addr learned my addr 172.21.15.79:0/1600829343 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:03:22.464 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8221e6700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fbf30 0x7fd81c10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:22.465 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 -- 172.21.15.79:0/1600829343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fbf30 msgr2=0x7fd81c10cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:22.465 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fbf30 0x7fd81c10cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:22.465 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 -- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd81c0fdb60 msgr2=0x7fd81c114510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:22.466 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd81c0fdb60 0x7fd81c114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:22.466 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 -- 172.21.15.79:0/1600829343 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180093f0 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.466 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd8219e5700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00b530 tx=0x7fd80c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:22.466 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c011630 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.467 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c115180 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.467 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd80c011c40 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.467 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c010c80 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.467 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.185+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c115670 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.469 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.186+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c107770 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.469 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.187+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd80c021070 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.469 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.187+0000 7fd812ffd700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 0x7fd8080991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.188+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4097+0+0 (secure 0 0 0) 0x7fd80c0be320 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.188+0000 7fd8221e6700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 0x7fd8080991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.189+0000 7fd8221e6700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 0x7fd8080991c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd81800b5c0 tx=0x7fd818005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.192+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd80c08c570 con 0x7fd81c0fd1d0 2022-03-21T14:03:22.471 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:22.365+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81c05d410 con 0x7fd808096d10 2022-03-21T14:03:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:03:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:03:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] boot 2022-03-21T14:03:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: osdmap e32: 5 total, 5 up, 5 in 2022-03-21T14:03:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:03:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:03:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] boot 2022-03-21T14:03:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: osdmap e32: 5 total, 5 up, 5 in 2022-03-21T14:03:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:03:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:03:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: osd.4 [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] boot 2022-03-21T14:03:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: osdmap e32: 5 total, 5 up, 5 in 2022-03-21T14:03:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T14:03:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:23 smithi079 ceph-mon[23158]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:23 smithi079 ceph-mon[23158]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:23 smithi079 ceph-mon[23158]: osdmap e33: 5 total, 5 up, 5 in 2022-03-21T14:03:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:23 smithi002 ceph-mon[17167]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:23 smithi002 ceph-mon[17167]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:23 smithi002 ceph-mon[17167]: osdmap e33: 5 total, 5 up, 5 in 2022-03-21T14:03:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:23 smithi016 ceph-mon[22010]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-03-21T14:03:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:23 smithi016 ceph-mon[22010]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:23 smithi016 ceph-mon[22010]: osdmap e33: 5 total, 5 up, 5 in 2022-03-21T14:03:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1999089485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]': finished 2022-03-21T14:03:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: osdmap e34: 6 total, 5 up, 6 in 2022-03-21T14:03:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:25 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1891958271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1999089485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]': finished 2022-03-21T14:03:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: osdmap e34: 6 total, 5 up, 6 in 2022-03-21T14:03:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:25 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1891958271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1999089485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]: dispatch 2022-03-21T14:03:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574f01bb-e554-4aa2-bb51-627091c73352"}]': finished 2022-03-21T14:03:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: osdmap e34: 6 total, 5 up, 6 in 2022-03-21T14:03:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:25 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1891958271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:27 smithi079 ceph-mon[23158]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:27 smithi002 ceph-mon[17167]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:27 smithi016 ceph-mon[22010]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:29 smithi079 ceph-mon[23158]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:29 smithi002 ceph-mon[17167]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:29 smithi016 ceph-mon[22010]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:31 smithi079 ceph-mon[23158]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-03-21T14:03:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:31 smithi002 ceph-mon[17167]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-03-21T14:03:31.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:31.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:31 smithi016 ceph-mon[22010]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-03-21T14:03:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:32 smithi079 ceph-mon[23158]: Deploying daemon osd.5 on smithi079 2022-03-21T14:03:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:32 smithi002 ceph-mon[17167]: Deploying daemon osd.5 on smithi079 2022-03-21T14:03:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:32 smithi016 ceph-mon[22010]: Deploying daemon osd.5 on smithi079 2022-03-21T14:03:33.689 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:33 smithi079 ceph-mon[23158]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:33 smithi002 ceph-mon[17167]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:33 smithi016 ceph-mon[22010]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:35.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:35 smithi079 ceph-mon[23158]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:35 smithi002 ceph-mon[17167]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:35 smithi016 ceph-mon[22010]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:36.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:36.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:36.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:36.791 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:37 smithi079 ceph-mon[23158]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:37 smithi002 ceph-mon[17167]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:37 smithi016 ceph-mon[22010]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:38.651 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.650+0000 7fd812ffd700 1 -- 172.21.15.79:0/1600829343 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd81c05d410 con 0x7fd808096d10 2022-03-21T14:03:38.652 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 5 on host 'smithi079' 2022-03-21T14:03:38.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 msgr2=0x7fd8080991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:38.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 0x7fd8080991c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd81800b5c0 tx=0x7fd818005f70 comp rx=0 tx=0).stop 2022-03-21T14:03:38.654 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 msgr2=0x7fd81c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:38.654 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00b530 tx=0x7fd80c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:03:38.654 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 shutdown_connections 2022-03-21T14:03:38.654 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c0fbf30 0x7fd81c10cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:38.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd808096d10 0x7fd8080991c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:38.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd81c0fdb60 0x7fd81c114510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:38.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 --2- 172.21.15.79:0/1600829343 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd81c0fd1d0 0x7fd81c10d480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:38.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 >> 172.21.15.79:0/1600829343 conn(0x7fd81c0f7720 msgr2=0x7fd81c100400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:03:38.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 shutdown_connections 2022-03-21T14:03:38.656 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:03:38.652+0000 7fd82444a700 1 -- 172.21.15.79:0/1600829343 wait complete. 2022-03-21T14:03:39.107 DEBUG:teuthology.orchestra.run.smithi079:osd.5> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.5.service 2022-03-21T14:03:39.150 INFO:tasks.cephadm:Deploying osd.6 on smithi002 with /dev/vg_nvme/lv_2... 2022-03-21T14:03:39.150 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-03-21T14:03:39.179 INFO:journalctl@ceph.osd.5.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:03:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:39.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:39.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:39.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:39.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:03:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:03:40.074 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 14:03:39 smithi079 conmon[34304]: 2022-03-21T14:03:39.852+0000 7f8e89e3b3c0 -1 osd.5 0 log_to_monitors true 2022-03-21T14:03:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:40 smithi016 ceph-mon[22010]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:40 smithi016 ceph-mon[22010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.458 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:40 smithi002 ceph-mon[17167]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.458 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:40 smithi002 ceph-mon[17167]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:40 smithi079 ceph-mon[23158]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:40 smithi079 ceph-mon[23158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-03-21T14:03:40.655 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:03:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-03-21T14:03:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: osdmap e35: 6 total, 5 up, 6 in 2022-03-21T14:03:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:41 smithi016 ceph-mon[22010]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:41.464 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 14:03:41 smithi079 conmon[34304]: 2022-03-21T14:03:41.165+0000 7f8e80840700 -1 osd.5 0 waiting for initial osdmap 2022-03-21T14:03:41.464 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 14:03:41 smithi079 conmon[34304]: 2022-03-21T14:03:41.173+0000 7f8e7b1d6700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:03:41.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-03-21T14:03:41.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: osdmap e35: 6 total, 5 up, 6 in 2022-03-21T14:03:41.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:41.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:41 smithi079 ceph-mon[23158]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-03-21T14:03:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: osdmap e35: 6 total, 5 up, 6 in 2022-03-21T14:03:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: from='osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:03:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:41 smithi002 ceph-mon[17167]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-03-21T14:03:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-03-21T14:03:41.636 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-03-21T14:03:41.636 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-03-21T14:03:41.636 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0443292 s, 237 MB/s 2022-03-21T14:03:41.636 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-03-21T14:03:42.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2022-03-21T14:03:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:03:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: osdmap e36: 6 total, 5 up, 6 in 2022-03-21T14:03:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T14:03:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 8586M 2022-03-21T14:03:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:03:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: osdmap e36: 6 total, 5 up, 6 in 2022-03-21T14:03:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T14:03:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 8586M 2022-03-21T14:03:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:03:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: osdmap e36: 6 total, 5 up, 6 in 2022-03-21T14:03:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T14:03:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:03:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 8586M 2022-03-21T14:03:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:03:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:03:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:03:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] boot 2022-03-21T14:03:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: osdmap e37: 6 total, 6 up, 6 in 2022-03-21T14:03:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:43 smithi079 ceph-mon[23158]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:43.576 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:03:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:03:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:03:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] boot 2022-03-21T14:03:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: osdmap e37: 6 total, 6 up, 6 in 2022-03-21T14:03:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:43 smithi002 ceph-mon[17167]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:03:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:03:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: osd.5 [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] boot 2022-03-21T14:03:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: osdmap e37: 6 total, 6 up, 6 in 2022-03-21T14:03:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T14:03:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:43 smithi016 ceph-mon[22010]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-03-21T14:03:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.041+0000 7f8c64b0a700 1 -- 172.21.15.2:0/1692796476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fd1f0 msgr2=0x7f8c600fd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.041+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/1692796476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fd1f0 0x7f8c600fd640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c540098e0 tx=0x7f8c54005f70 comp rx=0 tx=0).stop 2022-03-21T14:03:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 -- 172.21.15.2:0/1692796476 shutdown_connections 2022-03-21T14:03:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/1692796476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 0x7f8c60104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/1692796476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fbf50 0x7f8c600fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/1692796476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fd1f0 0x7f8c600fd640 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 -- 172.21.15.2:0/1692796476 >> 172.21.15.2:0/1692796476 conn(0x7f8c600f7720 msgr2=0x7f8c600f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:03:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 -- 172.21.15.2:0/1692796476 shutdown_connections 2022-03-21T14:03:44.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.042+0000 7f8c64b0a700 1 -- 172.21.15.2:0/1692796476 wait complete. 2022-03-21T14:03:44.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 Processor -- start 2022-03-21T14:03:44.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 -- start start 2022-03-21T14:03:44.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:44.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fd1f0 0x7f8c6010f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 0x7f8c60116510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60116a50 con 0x7f8c600fdb80 2022-03-21T14:03:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60116bc0 con 0x7f8c600fbf50 2022-03-21T14:03:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.043+0000 7f8c64b0a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60116d80 con 0x7f8c600fd1f0 2022-03-21T14:03:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:44.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 0x7f8c60116510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:44.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5dd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fd1f0 0x7f8c6010f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:44.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48168/0 (socket says 172.21.15.2:48168) 2022-03-21T14:03:44.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 -- 172.21.15.2:0/696894962 learned_addr learned my addr 172.21.15.2:0/696894962 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:03:44.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 -- 172.21.15.2:0/696894962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fd1f0 msgr2=0x7f8c6010f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:44.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fd1f0 0x7f8c6010f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:44.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 -- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 msgr2=0x7f8c60116510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:03:44.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 0x7f8c60116510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:03:44.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 -- 172.21.15.2:0/696894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540093f0 con 0x7f8c600fbf50 2022-03-21T14:03:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.044+0000 7f8c5e59c700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000f860 tx=0x7f8c50005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.045+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50004020 con 0x7f8c600fbf50 2022-03-21T14:03:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.045+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8c50004180 con 0x7f8c600fbf50 2022-03-21T14:03:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.045+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c60117060 con 0x7f8c600fbf50 2022-03-21T14:03:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.045+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c500039f0 con 0x7f8c600fbf50 2022-03-21T14:03:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.045+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c60117550 con 0x7f8c600fbf50 2022-03-21T14:03:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.046+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c60107740 con 0x7f8c600fbf50 2022-03-21T14:03:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.048+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f8c50022060 con 0x7f8c600fbf50 2022-03-21T14:03:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.049+0000 7f8c4effd700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 0x7f8c480991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:03:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.049+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4558+0+0 (secure 0 0 0) 0x7f8c500be6c0 con 0x7f8c600fbf50 2022-03-21T14:03:44.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.049+0000 7f8c5dd9b700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 0x7f8c480991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:03:44.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.050+0000 7f8c5dd9b700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 0x7f8c480991c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009f50 tx=0x7f8c54005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:03:44.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.053+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f8c5008c790 con 0x7f8c600fbf50 2022-03-21T14:03:44.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:03:44.254+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c60070920 con 0x7f8c48096d10 2022-03-21T14:03:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:44 smithi079 ceph-mon[23158]: osdmap e38: 6 total, 6 up, 6 in 2022-03-21T14:03:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:44 smithi002 ceph-mon[17167]: osdmap e38: 6 total, 6 up, 6 in 2022-03-21T14:03:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:44 smithi016 ceph-mon[22010]: osdmap e38: 6 total, 6 up, 6 in 2022-03-21T14:03:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: osdmap e39: 6 total, 6 up, 6 in 2022-03-21T14:03:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:45 smithi079 ceph-mon[23158]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:45.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: osdmap e39: 6 total, 6 up, 6 in 2022-03-21T14:03:45.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:45.486 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:45.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:45.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:45.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:45 smithi002 ceph-mon[17167]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: osdmap e39: 6 total, 6 up, 6 in 2022-03-21T14:03:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:03:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:03:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:03:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:45 smithi016 ceph-mon[22010]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:46 smithi079 ceph-mon[23158]: osdmap e40: 6 total, 6 up, 6 in 2022-03-21T14:03:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:46 smithi002 ceph-mon[17167]: osdmap e40: 6 total, 6 up, 6 in 2022-03-21T14:03:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:46 smithi016 ceph-mon[22010]: osdmap e40: 6 total, 6 up, 6 in 2022-03-21T14:03:47.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:47 smithi079 ceph-mon[23158]: osdmap e41: 6 total, 6 up, 6 in 2022-03-21T14:03:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:47 smithi079 ceph-mon[23158]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:03:47.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:47 smithi002 ceph-mon[17167]: osdmap e41: 6 total, 6 up, 6 in 2022-03-21T14:03:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:47 smithi002 ceph-mon[17167]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:03:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:47 smithi016 ceph-mon[22010]: osdmap e41: 6 total, 6 up, 6 in 2022-03-21T14:03:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:47 smithi016 ceph-mon[22010]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:03:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:48 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]: dispatch 2022-03-21T14:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:48 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]': finished 2022-03-21T14:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:48 smithi079 ceph-mon[23158]: osdmap e42: 7 total, 6 up, 7 in 2022-03-21T14:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:48 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3092079976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:48 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]: dispatch 2022-03-21T14:03:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:48 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]': finished 2022-03-21T14:03:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:48 smithi002 ceph-mon[17167]: osdmap e42: 7 total, 6 up, 7 in 2022-03-21T14:03:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:03:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:48 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3092079976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:48 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]: dispatch 2022-03-21T14:03:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:48 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/163912538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9e16547-b033-47a4-a8a6-8b17c953ce5b"}]': finished 2022-03-21T14:03:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:48 smithi016 ceph-mon[22010]: osdmap e42: 7 total, 6 up, 7 in 2022-03-21T14:03:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:03:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:48 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3092079976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:03:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:49 smithi079 ceph-mon[23158]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-03-21T14:03:49.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:49 smithi002 ceph-mon[17167]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-03-21T14:03:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:49 smithi016 ceph-mon[22010]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 124 KiB/s, 0 objects/s recovering 2022-03-21T14:03:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:51 smithi079 ceph-mon[23158]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:03:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:51 smithi002 ceph-mon[17167]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:03:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:51 smithi016 ceph-mon[22010]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:03:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:53 smithi002 ceph-mon[17167]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-03-21T14:03:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:53 smithi079 ceph-mon[23158]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-03-21T14:03:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:53 smithi016 ceph-mon[22010]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-03-21T14:03:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:55 smithi002 ceph-mon[17167]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-03-21T14:03:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:55 smithi079 ceph-mon[23158]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-03-21T14:03:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:55 smithi016 ceph-mon[22010]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-03-21T14:03:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:03:56.686 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:56 smithi002 ceph-mon[17167]: Deploying daemon osd.6 on smithi002 2022-03-21T14:03:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:56 smithi079 ceph-mon[23158]: Deploying daemon osd.6 on smithi002 2022-03-21T14:03:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:56 smithi016 ceph-mon[22010]: Deploying daemon osd.6 on smithi002 2022-03-21T14:03:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:57 smithi079 ceph-mon[23158]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:57 smithi002 ceph-mon[17167]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:57 smithi016 ceph-mon[22010]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:03:59 smithi079 ceph-mon[23158]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:59.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:03:59 smithi002 ceph-mon[17167]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:03:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:03:59 smithi016 ceph-mon[22010]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:01 smithi079 ceph-mon[23158]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:01.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:01 smithi002 ceph-mon[17167]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:01 smithi016 ceph-mon[22010]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:03 smithi079 ceph-mon[23158]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:03.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:03.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:03.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:03.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:03.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:03 smithi002 ceph-mon[17167]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:03 smithi016 ceph-mon[22010]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:05.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:05 smithi002 ceph-mon[17167]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:05.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:05.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:05 smithi079 ceph-mon[23158]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:05.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.843+0000 7f8c4effd700 1 -- 172.21.15.2:0/696894962 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c60070920 con 0x7f8c48096d10 2022-03-21T14:04:05.845 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 6 on host 'smithi002' 2022-03-21T14:04:05.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.845+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 msgr2=0x7f8c480991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:05.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.845+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 0x7f8c480991c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009f50 tx=0x7f8c54005f70 comp rx=0 tx=0).stop 2022-03-21T14:04:05.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.845+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 msgr2=0x7f8c6010ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:05.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.845+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000f860 tx=0x7f8c50005eb0 comp rx=0 tx=0).stop 2022-03-21T14:04:05.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 shutdown_connections 2022-03-21T14:04:05.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c600fbf50 0x7f8c6010ef40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:05.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8c48096d10 0x7f8c480991c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:05.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c600fdb80 0x7f8c60116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:05.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 --2- 172.21.15.2:0/696894962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c600fd1f0 0x7f8c6010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:05.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 >> 172.21.15.2:0/696894962 conn(0x7f8c600f7720 msgr2=0x7f8c6006f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:05.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 shutdown_connections 2022-03-21T14:04:05.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:05.846+0000 7f8c64b0a700 1 -- 172.21.15.2:0/696894962 wait complete. 2022-03-21T14:04:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:05 smithi016 ceph-mon[22010]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:06.314 DEBUG:teuthology.orchestra.run.smithi002:osd.6> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.6.service 2022-03-21T14:04:06.319 INFO:tasks.cephadm:Deploying osd.7 on smithi016 with /dev/vg_nvme/lv_2... 2022-03-21T14:04:06.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-03-21T14:04:06.392 INFO:journalctl@ceph.osd.6.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:04:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:07.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:07.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:06 smithi002 ceph-mon[17167]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:06 smithi016 ceph-mon[22010]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:06 smithi079 ceph-mon[23158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-03-21T14:04:08.115 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 21 14:04:07 smithi002 conmon[41671]: 2022-03-21T14:04:07.864+0000 7fa7fbfb0700 -1 osd.6 0 waiting for initial osdmap 2022-03-21T14:04:08.116 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 21 14:04:07 smithi002 conmon[41671]: 2022-03-21T14:04:07.879+0000 7fa7f6946700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:04:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-03-21T14:04:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: osdmap e43: 7 total, 6 up, 7 in 2022-03-21T14:04:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:07 smithi002 ceph-mon[17167]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-03-21T14:04:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: osdmap e43: 7 total, 6 up, 7 in 2022-03-21T14:04:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:07 smithi016 ceph-mon[22010]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-03-21T14:04:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-03-21T14:04:08.193 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2022-03-21T14:04:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2022-03-21T14:04:08.194 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00807706 s, 1.3 GB/s 2022-03-21T14:04:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2022-03-21T14:04:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-03-21T14:04:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: osdmap e43: 7 total, 6 up, 7 in 2022-03-21T14:04:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: from='osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:07 smithi079 ceph-mon[23158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:04:08.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2022-03-21T14:04:09.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:08 smithi016 ceph-mon[22010]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:04:09.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:08 smithi016 ceph-mon[22010]: osdmap e44: 7 total, 6 up, 7 in 2022-03-21T14:04:09.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:09.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:09.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:08 smithi002 ceph-mon[17167]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:04:09.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:08 smithi002 ceph-mon[17167]: osdmap e44: 7 total, 6 up, 7 in 2022-03-21T14:04:09.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:09.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:08 smithi079 ceph-mon[23158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:04:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:08 smithi079 ceph-mon[23158]: osdmap e44: 7 total, 6 up, 7 in 2022-03-21T14:04:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:10.106 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 -- 172.21.15.16:0/3015122084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fc420 msgr2=0x7fb2280fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 --2- 172.21.15.16:0/3015122084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fc420 0x7fb2280fc830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb2180098e0 tx=0x7fb218005f70 comp rx=0 tx=0).stop 2022-03-21T14:04:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 -- 172.21.15.16:0/3015122084 shutdown_connections 2022-03-21T14:04:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 --2- 172.21.15.16:0/3015122084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fc420 0x7fb2280fc830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:10.107 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 --2- 172.21.15.16:0/3015122084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb228103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 --2- 172.21.15.16:0/3015122084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fcd70 0x7fb2280fb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.103+0000 7fb22de28700 1 -- 172.21.15.16:0/3015122084 >> 172.21.15.16:0/3015122084 conn(0x7fb2280f5dc0 msgr2=0x7fb2280f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.104+0000 7fb22de28700 1 -- 172.21.15.16:0/3015122084 shutdown_connections 2022-03-21T14:04:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.104+0000 7fb22de28700 1 -- 172.21.15.16:0/3015122084 wait complete. 2022-03-21T14:04:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.104+0000 7fb22de28700 1 Processor -- start 2022-03-21T14:04:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.104+0000 7fb22de28700 1 -- start start 2022-03-21T14:04:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.104+0000 7fb22de28700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb22810bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fc420 0x7fb22810c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb228113880 con 0x7fb2280fb8f0 2022-03-21T14:04:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2281139f0 con 0x7fb2280fcd70 2022-03-21T14:04:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb228113bb0 con 0x7fb2280fc420 2022-03-21T14:04:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb226ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fc420 0x7fb22810c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42830/0 (socket says 172.21.15.16:42830) 2022-03-21T14:04:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 -- 172.21.15.16:0/1012246638 learned_addr learned my addr 172.21.15.16:0/1012246638 (peer_addr_for_me v2:172.21.15.16:0/0) 2022-03-21T14:04:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb2277fe700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb22810bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 -- 172.21.15.16:0/1012246638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fc420 msgr2=0x7fb22810c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fc420 0x7fb22810c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 -- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 msgr2=0x7fb22810bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb22810bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 -- 172.21.15.16:0/1012246638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2180093f0 con 0x7fb2280fcd70 2022-03-21T14:04:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb2277fe700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb22810bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:04:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb227fff700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb22000de20 tx=0x7fb220009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2200092e0 con 0x7fb2280fcd70 2022-03-21T14:04:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb22000adf0 con 0x7fb2280fcd70 2022-03-21T14:04:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2280fa0b0 con 0x7fb2280fcd70 2022-03-21T14:04:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220010d60 con 0x7fb2280fcd70 2022-03-21T14:04:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.105+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2280fa5a0 con 0x7fb2280fcd70 2022-03-21T14:04:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:04:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:04:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] boot 2022-03-21T14:04:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: osdmap e45: 7 total, 7 up, 7 in 2022-03-21T14:04:10.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.106+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2280faf90 con 0x7fb2280fcd70 2022-03-21T14:04:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.107+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fb22000e040 con 0x7fb2280fcd70 2022-03-21T14:04:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.108+0000 7fb20bfff700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 0x7fb210099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.108+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4961+0+0 (secure 0 0 0) 0x7fb2200be5b0 con 0x7fb2280fcd70 2022-03-21T14:04:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.108+0000 7fb2277fe700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 0x7fb210099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.109+0000 7fb2277fe700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 0x7fb210099240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2180098b0 tx=0x7fb21801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.112+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fb22008c590 con 0x7fb2280fcd70 2022-03-21T14:04:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:04:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:04:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] boot 2022-03-21T14:04:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: osdmap e45: 7 total, 7 up, 7 in 2022-03-21T14:04:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:04:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:04:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-03-21T14:04:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: osd.6 [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] boot 2022-03-21T14:04:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: osdmap e45: 7 total, 7 up, 7 in 2022-03-21T14:04:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:10.300+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb228101e60 con 0x7fb210096d90 2022-03-21T14:04:11.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T14:04:11.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: osdmap e46: 7 total, 7 up, 7 in 2022-03-21T14:04:11.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 5759M 2022-03-21T14:04:11.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:11.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:11.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T14:04:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: osdmap e46: 7 total, 7 up, 7 in 2022-03-21T14:04:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 5759M 2022-03-21T14:04:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T14:04:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: osdmap e46: 7 total, 7 up, 7 in 2022-03-21T14:04:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 5759M 2022-03-21T14:04:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:11.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:11.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:11.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:11 smithi016 ceph-mon[22010]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:11 smithi016 ceph-mon[22010]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:11 smithi079 ceph-mon[23158]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:11 smithi079 ceph-mon[23158]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:11 smithi002 ceph-mon[17167]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:11 smithi002 ceph-mon[17167]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:12 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/1025444586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:12 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:12 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]': finished 2022-03-21T14:04:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:12 smithi016 ceph-mon[22010]: osdmap e47: 8 total, 7 up, 8 in 2022-03-21T14:04:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:12 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/1025444586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:12 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:12 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]': finished 2022-03-21T14:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:12 smithi079 ceph-mon[23158]: osdmap e47: 8 total, 7 up, 8 in 2022-03-21T14:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:12 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/1025444586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:12 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]: dispatch 2022-03-21T14:04:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:12 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65e72a50-adfb-492a-b31e-68e1fa96fb5e"}]': finished 2022-03-21T14:04:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:12 smithi002 ceph-mon[17167]: osdmap e47: 8 total, 7 up, 8 in 2022-03-21T14:04:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:13 smithi016 ceph-mon[22010]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:13 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/2578655078' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:13 smithi079 ceph-mon[23158]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:13 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/2578655078' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:13 smithi002 ceph-mon[17167]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:13 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/2578655078' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:15 smithi016 ceph-mon[22010]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:15 smithi079 ceph-mon[23158]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:15 smithi002 ceph-mon[17167]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:17 smithi016 ceph-mon[22010]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:17 smithi079 ceph-mon[23158]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:17 smithi002 ceph-mon[17167]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:20.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:19 smithi016 ceph-mon[22010]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:20.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-03-21T14:04:20.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:19 smithi079 ceph-mon[23158]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-03-21T14:04:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:19 smithi002 ceph-mon[17167]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-03-21T14:04:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:20 smithi016 ceph-mon[22010]: Deploying daemon osd.7 on smithi016 2022-03-21T14:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:04:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:04:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:20 smithi079 ceph-mon[23158]: Deploying daemon osd.7 on smithi016 2022-03-21T14:04:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:04:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:04:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:20 smithi002 ceph-mon[17167]: Deploying daemon osd.7 on smithi016 2022-03-21T14:04:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:04:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:04:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:21 smithi016 ceph-mon[22010]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:21 smithi079 ceph-mon[23158]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:21 smithi002 ceph-mon[17167]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:23 smithi016 ceph-mon[22010]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:23 smithi079 ceph-mon[23158]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:23 smithi002 ceph-mon[17167]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:25 smithi079 ceph-mon[23158]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:26.337 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:25 smithi016 ceph-mon[22010]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:25 smithi002 ceph-mon[17167]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.635+0000 7fb20bfff700 1 -- 172.21.15.16:0/1012246638 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb228101e60 con 0x7fb210096d90 2022-03-21T14:04:26.636 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 7 on host 'smithi016' 2022-03-21T14:04:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 msgr2=0x7fb210099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 0x7fb210099240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2180098b0 tx=0x7fb21801a040 comp rx=0 tx=0).stop 2022-03-21T14:04:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 msgr2=0x7fb2281131d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb22000de20 tx=0x7fb220009840 comp rx=0 tx=0).stop 2022-03-21T14:04:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 shutdown_connections 2022-03-21T14:04:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2280fcd70 0x7fb2281131d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb210096d90 0x7fb210099240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2280fb8f0 0x7fb22810bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 --2- 172.21.15.16:0/1012246638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2280fc420 0x7fb22810c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 >> 172.21.15.16:0/1012246638 conn(0x7fb2280f5dc0 msgr2=0x7fb2280ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 shutdown_connections 2022-03-21T14:04:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:04:26.637+0000 7fb22de28700 1 -- 172.21.15.16:0/1012246638 wait complete. 2022-03-21T14:04:27.130 DEBUG:teuthology.orchestra.run.smithi016:osd.7> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.7.service 2022-03-21T14:04:27.132 INFO:tasks.cephadm:Deploying osd.8 on smithi079 with /dev/vg_nvme/lv_2... 2022-03-21T14:04:27.133 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-03-21T14:04:27.159 INFO:journalctl@ceph.osd.7.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:04:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:28.075 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 14:04:27 smithi016 conmon[38333]: 2022-03-21T14:04:27.811+0000 7f7e3f8183c0 -1 osd.7 0 log_to_monitors true 2022-03-21T14:04:28.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:28 smithi079 ceph-mon[23158]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:28.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:28 smithi079 ceph-mon[23158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:28 smithi016 ceph-mon[22010]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:28 smithi016 ceph-mon[22010]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:28 smithi002 ceph-mon[17167]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:28 smithi002 ceph-mon[17167]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-03-21T14:04:29.008 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-03-21T14:04:29.008 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-03-21T14:04:29.009 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2022-03-21T14:04:29.009 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2022-03-21T14:04:29.009 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112691 s, 930 MB/s 2022-03-21T14:04:29.009 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2022-03-21T14:04:29.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-03-21T14:04:29.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: osdmap e48: 8 total, 7 up, 8 in 2022-03-21T14:04:29.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:29.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:29 smithi079 ceph-mon[23158]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:29.458 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2022-03-21T14:04:29.471 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-03-21T14:04:29.471 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: osdmap e48: 8 total, 7 up, 8 in 2022-03-21T14:04:29.471 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:29.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:29 smithi016 ceph-mon[22010]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:29.472 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 14:04:29 smithi016 conmon[38333]: 2022-03-21T14:04:29.198+0000 7f7e3621d700 -1 osd.7 0 waiting for initial osdmap 2022-03-21T14:04:29.473 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 14:04:29 smithi016 conmon[38333]: 2022-03-21T14:04:29.210+0000 7f7e30bb3700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:04:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-03-21T14:04:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: osdmap e48: 8 total, 7 up, 8 in 2022-03-21T14:04:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: from='osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:04:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:29 smithi002 ceph-mon[17167]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-03-21T14:04:30.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:04:30.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: osdmap e49: 8 total, 7 up, 8 in 2022-03-21T14:04:30.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T14:04:30.419 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.420 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 5725M 2022-03-21T14:04:30.421 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.421 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:04:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: osdmap e49: 8 total, 7 up, 8 in 2022-03-21T14:04:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T14:04:30.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 5725M 2022-03-21T14:04:30.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:04:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: osdmap e49: 8 total, 7 up, 8 in 2022-03-21T14:04:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T14:04:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 5725M 2022-03-21T14:04:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:30.987 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 -- 172.21.15.79:0/3811216912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fbf50 msgr2=0x7f52b00fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:30.987 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3811216912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fbf50 0x7f52b00fc360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f52a0009c70 tx=0x7f52a00094e0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.988 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 -- 172.21.15.79:0/3811216912 shutdown_connections 2022-03-21T14:04:30.988 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3811216912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fdb80 0x7f52b0104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.988 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3811216912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fbf50 0x7f52b00fc360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.988 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3811216912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52b00fd1f0 0x7f52b00fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.989 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 -- 172.21.15.79:0/3811216912 >> 172.21.15.79:0/3811216912 conn(0x7f52b00f7720 msgr2=0x7f52b00f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:30.989 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 -- 172.21.15.79:0/3811216912 shutdown_connections 2022-03-21T14:04:30.989 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 -- 172.21.15.79:0/3811216912 wait complete. 2022-03-21T14:04:30.989 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.984+0000 7f52b5c94700 1 Processor -- start 2022-03-21T14:04:30.990 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 -- start start 2022-03-21T14:04:30.990 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52b00fbf50 0x7f52b010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:30.990 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fd1f0 0x7f52b010d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:30.990 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:30.991 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b01149d0 con 0x7f52b00fbf50 2022-03-21T14:04:30.991 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0114b40 con 0x7f52b00fd1f0 2022-03-21T14:04:30.991 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52b5c94700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0114d00 con 0x7f52b00fdb80 2022-03-21T14:04:30.991 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:30.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33432/0 (socket says 172.21.15.79:33432) 2022-03-21T14:04:30.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 -- 172.21.15.79:0/3655182582 learned_addr learned my addr 172.21.15.79:0/3655182582 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:04:30.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52aeffd700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fd1f0 0x7f52b010d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:30.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 -- 172.21.15.79:0/3655182582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fd1f0 msgr2=0x7f52b010d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:30.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fd1f0 0x7f52b010d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 -- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52b00fbf50 msgr2=0x7f52b010cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:30.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52b00fbf50 0x7f52b010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:30.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 -- 172.21.15.79:0/3655182582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0009780 con 0x7f52b00fdb80 2022-03-21T14:04:30.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52affff700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52a800b5e0 tx=0x7f52a8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:30.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.985+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8011630 con 0x7f52b00fdb80 2022-03-21T14:04:30.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.986+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f52a8011c40 con 0x7f52b00fdb80 2022-03-21T14:04:30.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.986+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8010d30 con 0x7f52b00fdb80 2022-03-21T14:04:30.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.986+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b0114fe0 con 0x7f52b00fdb80 2022-03-21T14:04:30.995 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.986+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b01153f0 con 0x7f52b00fdb80 2022-03-21T14:04:30.996 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.986+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b01077e0 con 0x7f52b00fdb80 2022-03-21T14:04:30.996 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.987+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f52a8021070 con 0x7f52b00fdb80 2022-03-21T14:04:30.996 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.988+0000 7f52a7fff700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 0x7f529c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:30.997 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.988+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f52a80be780 con 0x7f52b00fdb80 2022-03-21T14:04:30.997 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.988+0000 7f52af7fe700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 0x7f529c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:30.997 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.989+0000 7f52af7fe700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 0x7f529c0991c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f52a0009b10 tx=0x7f52a00094e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:30.998 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:30.992+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f52a808c590 con 0x7f52b00fdb80 2022-03-21T14:04:31.168 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:31.167+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f52b005d410 con 0x7f529c096d10 2022-03-21T14:04:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:04:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:04:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] boot 2022-03-21T14:04:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: osdmap e50: 8 total, 8 up, 8 in 2022-03-21T14:04:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:04:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:04:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] boot 2022-03-21T14:04:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: osdmap e50: 8 total, 8 up, 8 in 2022-03-21T14:04:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:04:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:04:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: osd.7 [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] boot 2022-03-21T14:04:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: osdmap e50: 8 total, 8 up, 8 in 2022-03-21T14:04:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T14:04:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:32.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:32 smithi079 ceph-mon[23158]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:32.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:32 smithi079 ceph-mon[23158]: osdmap e51: 8 total, 8 up, 8 in 2022-03-21T14:04:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:32 smithi002 ceph-mon[17167]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:32 smithi002 ceph-mon[17167]: osdmap e51: 8 total, 8 up, 8 in 2022-03-21T14:04:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:32 smithi016 ceph-mon[22010]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:32 smithi016 ceph-mon[22010]: osdmap e51: 8 total, 8 up, 8 in 2022-03-21T14:04:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:33 smithi079 ceph-mon[23158]: osdmap e52: 8 total, 8 up, 8 in 2022-03-21T14:04:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:33 smithi079 ceph-mon[23158]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:33 smithi002 ceph-mon[17167]: osdmap e52: 8 total, 8 up, 8 in 2022-03-21T14:04:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:33 smithi002 ceph-mon[17167]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:33 smithi016 ceph-mon[22010]: osdmap e52: 8 total, 8 up, 8 in 2022-03-21T14:04:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:33 smithi016 ceph-mon[22010]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: osdmap e53: 8 total, 8 up, 8 in 2022-03-21T14:04:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/3218197766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]': finished 2022-03-21T14:04:34.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: osdmap e54: 9 total, 8 up, 9 in 2022-03-21T14:04:34.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:34.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:34 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/3826341387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: osdmap e53: 8 total, 8 up, 8 in 2022-03-21T14:04:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/3218197766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]': finished 2022-03-21T14:04:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: osdmap e54: 9 total, 8 up, 9 in 2022-03-21T14:04:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:34 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/3826341387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: osdmap e53: 8 total, 8 up, 8 in 2022-03-21T14:04:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/3218197766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]: dispatch 2022-03-21T14:04:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "680e2f3a-ec15-482d-9734-1691c5c013d6"}]': finished 2022-03-21T14:04:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: osdmap e54: 9 total, 8 up, 9 in 2022-03-21T14:04:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:34 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/3826341387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:35 smithi002 ceph-mon[17167]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:35.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:35 smithi016 ceph-mon[22010]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:35 smithi079 ceph-mon[23158]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:37 smithi079 ceph-mon[23158]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:37 smithi002 ceph-mon[17167]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:37 smithi016 ceph-mon[22010]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:39 smithi079 ceph-mon[23158]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:39.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:39 smithi002 ceph-mon[17167]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:39.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:39 smithi016 ceph-mon[22010]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-03-21T14:04:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-03-21T14:04:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-03-21T14:04:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:41 smithi079 ceph-mon[23158]: Deploying daemon osd.8 on smithi079 2022-03-21T14:04:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:41 smithi079 ceph-mon[23158]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:41 smithi002 ceph-mon[17167]: Deploying daemon osd.8 on smithi079 2022-03-21T14:04:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:41 smithi002 ceph-mon[17167]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:41 smithi016 ceph-mon[22010]: Deploying daemon osd.8 on smithi079 2022-03-21T14:04:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:41 smithi016 ceph-mon[22010]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:43 smithi079 ceph-mon[23158]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:43 smithi002 ceph-mon[17167]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:43 smithi016 ceph-mon[22010]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:45 smithi002 ceph-mon[17167]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:46.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:45 smithi016 ceph-mon[22010]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:45 smithi079 ceph-mon[23158]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:47.990 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 8 on host 'smithi079' 2022-03-21T14:04:47.991 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.989+0000 7f52a7fff700 1 -- 172.21.15.79:0/3655182582 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f52b005d410 con 0x7f529c096d10 2022-03-21T14:04:47.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.991+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 msgr2=0x7f529c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:47.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.991+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 0x7f529c0991c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f52a0009b10 tx=0x7f52a00094e0 comp rx=0 tx=0).stop 2022-03-21T14:04:47.992 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 msgr2=0x7f52b0114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:47.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52a800b5e0 tx=0x7f52a8009060 comp rx=0 tx=0).stop 2022-03-21T14:04:47.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 shutdown_connections 2022-03-21T14:04:47.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f529c096d10 0x7f529c0991c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:47.993 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52b00fdb80 0x7f52b0114320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:47.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52b00fbf50 0x7f52b010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:47.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 --2- 172.21.15.79:0/3655182582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52b00fd1f0 0x7f52b010d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:47.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 >> 172.21.15.79:0/3655182582 conn(0x7f52b00f7720 msgr2=0x7f52b00f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:47.994 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 shutdown_connections 2022-03-21T14:04:47.995 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:04:47.992+0000 7f52b5c94700 1 -- 172.21.15.79:0/3655182582 wait complete. 2022-03-21T14:04:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:47 smithi002 ceph-mon[17167]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:47 smithi016 ceph-mon[22010]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:47 smithi079 ceph-mon[23158]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:48.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:48.490 DEBUG:teuthology.orchestra.run.smithi079:osd.8> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.8.service 2022-03-21T14:04:48.492 INFO:tasks.cephadm:Deploying osd.9 on smithi002 with /dev/vg_nvme/lv_1... 2022-03-21T14:04:48.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-03-21T14:04:48.517 INFO:journalctl@ceph.osd.8.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:04:49.155 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 14:04:48 smithi079 conmon[39896]: 2022-03-21T14:04:48.873+0000 7f72e91e33c0 -1 osd.8 0 log_to_monitors true 2022-03-21T14:04:49.156 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:49.156 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:49.156 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:49.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:49.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:49.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:48 smithi079 ceph-mon[23158]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:49.280 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:49.281 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:49.281 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:49.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:49.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:49.283 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:48 smithi002 ceph-mon[17167]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:04:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:04:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:48 smithi016 ceph-mon[22010]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-03-21T14:04:50.011 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:04:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-03-21T14:04:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: osdmap e55: 9 total, 8 up, 9 in 2022-03-21T14:04:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:49 smithi002 ceph-mon[17167]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-03-21T14:04:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: osdmap e55: 9 total, 8 up, 9 in 2022-03-21T14:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:49 smithi016 ceph-mon[22010]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.454 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:50.454 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-03-21T14:04:50.454 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: osdmap e55: 9 total, 8 up, 9 in 2022-03-21T14:04:50.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:50.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: from='osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:49 smithi079 ceph-mon[23158]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:04:50.722 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 14:04:50 smithi079 conmon[39896]: 2022-03-21T14:04:50.452+0000 7f72dfbe8700 -1 osd.8 0 waiting for initial osdmap 2022-03-21T14:04:50.723 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 14:04:50 smithi079 conmon[39896]: 2022-03-21T14:04:50.465+0000 7f72d8d7b700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:04:50.920 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-03-21T14:04:50.921 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-03-21T14:04:50.921 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-03-21T14:04:50.921 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-03-21T14:04:50.921 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0454274 s, 231 MB/s 2022-03-21T14:04:50.922 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-03-21T14:04:51.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2022-03-21T14:04:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:04:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: osdmap e56: 9 total, 8 up, 9 in 2022-03-21T14:04:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T14:04:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 5724M 2022-03-21T14:04:51.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:04:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: osdmap e56: 9 total, 8 up, 9 in 2022-03-21T14:04:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T14:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 5724M 2022-03-21T14:04:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-03-21T14:04:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:04:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: osdmap e56: 9 total, 8 up, 9 in 2022-03-21T14:04:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T14:04:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:04:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 5724M 2022-03-21T14:04:51.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:51.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:04:52.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:52 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:04:52.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:52 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:04:52.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:52 smithi002 ceph-mon[17167]: osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] boot 2022-03-21T14:04:52.711 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:52 smithi002 ceph-mon[17167]: osdmap e57: 9 total, 9 up, 9 in 2022-03-21T14:04:52.711 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:52 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:04:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:52 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:04:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:52 smithi079 ceph-mon[23158]: osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] boot 2022-03-21T14:04:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:52 smithi079 ceph-mon[23158]: osdmap e57: 9 total, 9 up, 9 in 2022-03-21T14:04:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:52.837 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:04:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:52 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:04:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:52 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:04:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:52 smithi016 ceph-mon[22010]: osd.8 [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] boot 2022-03-21T14:04:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:52 smithi016 ceph-mon[22010]: osdmap e57: 9 total, 9 up, 9 in 2022-03-21T14:04:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T14:04:53.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 -- 172.21.15.2:0/2053976095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fd1f0 msgr2=0x7f97b80fd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:53.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 --2- 172.21.15.2:0/2053976095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fd1f0 0x7f97b80fd640 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0098e0 tx=0x7f97ac005f70 comp rx=0 tx=0).stop 2022-03-21T14:04:53.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 -- 172.21.15.2:0/2053976095 shutdown_connections 2022-03-21T14:04:53.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 --2- 172.21.15.2:0/2053976095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b80fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:53.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 --2- 172.21.15.2:0/2053976095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fdb80 0x7f97b8104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:53.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 --2- 172.21.15.2:0/2053976095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fd1f0 0x7f97b80fd640 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:53.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 -- 172.21.15.2:0/2053976095 >> 172.21.15.2:0/2053976095 conn(0x7f97b80f7720 msgr2=0x7f97b80f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:04:53.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 -- 172.21.15.2:0/2053976095 shutdown_connections 2022-03-21T14:04:53.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.296+0000 7f97bd046700 1 -- 172.21.15.2:0/2053976095 wait complete. 2022-03-21T14:04:53.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 Processor -- start 2022-03-21T14:04:53.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 -- start start 2022-03-21T14:04:53.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:53.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fd1f0 0x7f97b810f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:53.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fdb80 0x7f97b81164e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:53.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8116a20 con 0x7f97b80fd1f0 2022-03-21T14:04:53.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8116b90 con 0x7f97b80fbf50 2022-03-21T14:04:53.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.297+0000 7f97bd046700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8116d50 con 0x7f97b80fdb80 2022-03-21T14:04:53.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:53.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48188/0 (socket says 172.21.15.2:48188) 2022-03-21T14:04:53.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 -- 172.21.15.2:0/3718824566 learned_addr learned my addr 172.21.15.2:0/3718824566 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:04:53.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b659c700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fd1f0 0x7f97b810f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:53.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b759e700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fdb80 0x7f97b81164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:53.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 -- 172.21.15.2:0/3718824566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fdb80 msgr2=0x7f97b81164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:53.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fdb80 0x7f97b81164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:53.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 -- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fd1f0 msgr2=0x7f97b810f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:04:53.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fd1f0 0x7f97b810f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:04:53.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 -- 172.21.15.2:0/3718824566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac0093f0 con 0x7f97b80fbf50 2022-03-21T14:04:53.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.298+0000 7f97b6d9d700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97a800f860 tx=0x7f97a8005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:53.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.299+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8004020 con 0x7f97b80fbf50 2022-03-21T14:04:53.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.299+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f97a8004180 con 0x7f97b80fbf50 2022-03-21T14:04:53.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.299+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b8116fd0 con 0x7f97b80fbf50 2022-03-21T14:04:53.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.299+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a80039f0 con 0x7f97b80fbf50 2022-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.299+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b8117500 con 0x7f97b80fbf50 2022-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.300+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b81076e0 con 0x7f97b80fbf50 2022-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.302+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f97a8022060 con 0x7f97b80fbf50 2022-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.303+0000 7f979f7fe700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 0x7f97a0099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.303+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f97a80bed70 con 0x7f97b80fbf50 2022-03-21T14:04:53.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.303+0000 7f97b659c700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 0x7f97a0099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:04:53.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.304+0000 7f97b659c700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 0x7f97a0099240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005b40 tx=0x7f97ac005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:04:53.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.305+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f97a808c8f0 con 0x7f97b80fbf50 2022-03-21T14:04:53.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:04:53.504+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f97b802ad20 con 0x7f97a0096d90 2022-03-21T14:04:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:53 smithi079 ceph-mon[23158]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:53 smithi079 ceph-mon[23158]: osdmap e58: 9 total, 9 up, 9 in 2022-03-21T14:04:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:53 smithi002 ceph-mon[17167]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:53 smithi002 ceph-mon[17167]: osdmap e58: 9 total, 9 up, 9 in 2022-03-21T14:04:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:53 smithi016 ceph-mon[22010]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:53 smithi016 ceph-mon[22010]: osdmap e58: 9 total, 9 up, 9 in 2022-03-21T14:04:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:54 smithi002 ceph-mon[17167]: osdmap e59: 9 total, 9 up, 9 in 2022-03-21T14:04:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:54 smithi002 ceph-mon[17167]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:54 smithi016 ceph-mon[22010]: osdmap e59: 9 total, 9 up, 9 in 2022-03-21T14:04:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:54 smithi016 ceph-mon[22010]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:54 smithi079 ceph-mon[23158]: osdmap e59: 9 total, 9 up, 9 in 2022-03-21T14:04:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:54 smithi079 ceph-mon[23158]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:04:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:04:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:04:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:04:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:55 smithi002 ceph-mon[17167]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:55.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:55 smithi002 ceph-mon[17167]: osdmap e60: 9 total, 9 up, 9 in 2022-03-21T14:04:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:55 smithi016 ceph-mon[22010]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:55 smithi016 ceph-mon[22010]: osdmap e60: 9 total, 9 up, 9 in 2022-03-21T14:04:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:55 smithi079 ceph-mon[23158]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:04:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:55 smithi079 ceph-mon[23158]: osdmap e60: 9 total, 9 up, 9 in 2022-03-21T14:04:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:56 smithi002 ceph-mon[17167]: osdmap e61: 9 total, 9 up, 9 in 2022-03-21T14:04:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:56 smithi016 ceph-mon[22010]: osdmap e61: 9 total, 9 up, 9 in 2022-03-21T14:04:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:56 smithi079 ceph-mon[23158]: osdmap e61: 9 total, 9 up, 9 in 2022-03-21T14:04:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:04:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]: dispatch 2022-03-21T14:04:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]': finished 2022-03-21T14:04:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: osdmap e62: 10 total, 9 up, 10 in 2022-03-21T14:04:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:04:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:57 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2248901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:04:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]: dispatch 2022-03-21T14:04:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]': finished 2022-03-21T14:04:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: osdmap e62: 10 total, 9 up, 10 in 2022-03-21T14:04:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:04:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:57 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2248901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2022-03-21T14:04:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]: dispatch 2022-03-21T14:04:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3480118649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c640ec62-e489-4b6d-bd4e-3e4b609505ea"}]': finished 2022-03-21T14:04:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: osdmap e62: 10 total, 9 up, 10 in 2022-03-21T14:04:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:04:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:57 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2248901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:04:59.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:04:59 smithi002 ceph-mon[17167]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-03-21T14:04:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:04:59 smithi016 ceph-mon[22010]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-03-21T14:04:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:04:59 smithi079 ceph-mon[23158]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-03-21T14:05:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:01 smithi002 ceph-mon[17167]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:01 smithi016 ceph-mon[22010]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:01 smithi079 ceph-mon[23158]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:03 smithi002 ceph-mon[17167]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:05:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:03 smithi016 ceph-mon[22010]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:05:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:03 smithi079 ceph-mon[23158]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-03-21T14:05:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:05 smithi002 ceph-mon[17167]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-03-21T14:05:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:05 smithi002 ceph-mon[17167]: Deploying daemon osd.9 on smithi002 2022-03-21T14:05:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:05 smithi016 ceph-mon[22010]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-03-21T14:05:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:05 smithi016 ceph-mon[22010]: Deploying daemon osd.9 on smithi002 2022-03-21T14:05:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:05 smithi079 ceph-mon[23158]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-03-21T14:05:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:05 smithi079 ceph-mon[23158]: Deploying daemon osd.9 on smithi002 2022-03-21T14:05:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:07 smithi002 ceph-mon[17167]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:07 smithi016 ceph-mon[22010]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:07 smithi079 ceph-mon[23158]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:09.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:09 smithi002 ceph-mon[17167]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:09.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:09 smithi016 ceph-mon[22010]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:09 smithi079 ceph-mon[23158]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:11.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:11 smithi002 ceph-mon[17167]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:11 smithi016 ceph-mon[22010]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:11 smithi079 ceph-mon[23158]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:13.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:13.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:13.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:13.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:13 smithi002 ceph-mon[17167]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:13 smithi016 ceph-mon[22010]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:13 smithi079 ceph-mon[23158]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:15.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.977+0000 7f979f7fe700 1 -- 172.21.15.2:0/3718824566 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f97b802ad20 con 0x7f97a0096d90 2022-03-21T14:05:15.979 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 9 on host 'smithi002' 2022-03-21T14:05:15.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 msgr2=0x7f97a0099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:15.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 0x7f97a0099240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005b40 tx=0x7f97ac005f70 comp rx=0 tx=0).stop 2022-03-21T14:05:15.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 msgr2=0x7f97b810ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:15.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97a800f860 tx=0x7f97a8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:05:15.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 shutdown_connections 2022-03-21T14:05:15.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97b80fbf50 0x7f97b810ef10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:15.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f97a0096d90 0x7f97a0099240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:15.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97b80fdb80 0x7f97b81164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:15.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 --2- 172.21.15.2:0/3718824566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97b80fd1f0 0x7f97b810f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:15.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 >> 172.21.15.2:0/3718824566 conn(0x7f97b80f7720 msgr2=0x7f97b8102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:05:15.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 shutdown_connections 2022-03-21T14:05:15.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:05:15.980+0000 7f97bd046700 1 -- 172.21.15.2:0/3718824566 wait complete. 2022-03-21T14:05:16.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:16.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.470 DEBUG:teuthology.orchestra.run.smithi002:osd.9> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.9.service 2022-03-21T14:05:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-03-21T14:05:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:16.476 INFO:tasks.cephadm:Deploying osd.10 on smithi016 with /dev/vg_nvme/lv_1... 2022-03-21T14:05:16.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-03-21T14:05:16.547 INFO:journalctl@ceph.osd.9.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:05:17.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:17.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:17.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:17.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-03-21T14:05:17.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: osdmap e63: 10 total, 9 up, 10 in 2022-03-21T14:05:17.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:17.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:16 smithi002 ceph-mon[17167]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:17.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:17.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:17.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-03-21T14:05:17.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: osdmap e63: 10 total, 9 up, 10 in 2022-03-21T14:05:17.382 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:17.382 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.382 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:16 smithi016 ceph-mon[22010]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-03-21T14:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: osdmap e63: 10 total, 9 up, 10 in 2022-03-21T14:05:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:16 smithi079 ceph-mon[23158]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-03-21T14:05:17.615 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 21 14:05:17 smithi002 conmon[47363]: 2022-03-21T14:05:17.352+0000 7f42be857700 -1 osd.9 0 waiting for initial osdmap 2022-03-21T14:05:17.616 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 21 14:05:17 smithi002 conmon[47363]: 2022-03-21T14:05:17.373+0000 7f42bb1f1700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:05:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:17 smithi016 ceph-mon[22010]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:17 smithi016 ceph-mon[22010]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:05:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:17 smithi016 ceph-mon[22010]: osdmap e64: 10 total, 9 up, 10 in 2022-03-21T14:05:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-03-21T14:05:18.361 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-03-21T14:05:18.361 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2022-03-21T14:05:18.361 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2022-03-21T14:05:18.361 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0080444 s, 1.3 GB/s 2022-03-21T14:05:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2022-03-21T14:05:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:17 smithi002 ceph-mon[17167]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:17 smithi002 ceph-mon[17167]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:05:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:17 smithi002 ceph-mon[17167]: osdmap e64: 10 total, 9 up, 10 in 2022-03-21T14:05:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:17 smithi079 ceph-mon[23158]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:17 smithi079 ceph-mon[23158]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-03-21T14:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:17 smithi079 ceph-mon[23158]: osdmap e64: 10 total, 9 up, 10 in 2022-03-21T14:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:18.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2022-03-21T14:05:19.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:05:19.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:05:19.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] boot 2022-03-21T14:05:19.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: osdmap e65: 10 total, 10 up, 10 in 2022-03-21T14:05:19.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:19.612 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:19 smithi002 ceph-mon[17167]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:19.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:05:19.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:05:19.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] boot 2022-03-21T14:05:19.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: osdmap e65: 10 total, 10 up, 10 in 2022-03-21T14:05:19.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:19.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:19 smithi016 ceph-mon[22010]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:05:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:05:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: osd.9 [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] boot 2022-03-21T14:05:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: osdmap e65: 10 total, 10 up, 10 in 2022-03-21T14:05:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T14:05:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:19 smithi079 ceph-mon[23158]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-03-21T14:05:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- 172.21.15.16:0/2580606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fbf30 msgr2=0x7f50bc0fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/2580606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fbf30 0x7f50bc0fc340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50b80098e0 tx=0x7f50b8005f70 comp rx=0 tx=0).stop 2022-03-21T14:05:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- 172.21.15.16:0/2580606618 shutdown_connections 2022-03-21T14:05:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/2580606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fbf30 0x7f50bc0fc340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/2580606618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fdb60 0x7f50bc104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/2580606618 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc0fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- 172.21.15.16:0/2580606618 >> 172.21.15.16:0/2580606618 conn(0x7f50bc0f7720 msgr2=0x7f50bc0f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:05:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- 172.21.15.16:0/2580606618 shutdown_connections 2022-03-21T14:05:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- 172.21.15.16:0/2580606618 wait complete. 2022-03-21T14:05:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 Processor -- start 2022-03-21T14:05:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.309+0000 7f50c3bbd700 1 -- start start 2022-03-21T14:05:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 0x7f50bc070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 0x7f50bc06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc070000 con 0x7f50bc0fbf30 2022-03-21T14:05:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc070170 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc116da0 con 0x7f50bc0fdb60 2022-03-21T14:05:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42970/0 (socket says 172.21.15.16:42970) 2022-03-21T14:05:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 -- 172.21.15.16:0/1639148035 learned_addr learned my addr 172.21.15.16:0/1639148035 (peer_addr_for_me v2:172.21.15.16:0/0) 2022-03-21T14:05:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1959700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 0x7f50bc070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c215a700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 0x7f50bc06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 -- 172.21.15.16:0/1639148035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 msgr2=0x7f50bc06f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 0x7f50bc06f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 -- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 msgr2=0x7f50bc070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 0x7f50bc070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 -- 172.21.15.16:0/1639148035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b80093f0 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1959700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 0x7f50bc070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-03-21T14:05:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c215a700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 0x7f50bc06f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-03-21T14:05:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c1158700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00ca00 tx=0x7f50ac00a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:05:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac00a350 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc116fd0 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f50ac0054d0 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac004a40 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.321 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.310+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc117510 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.321 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.311+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc1076f0 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.312+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f50ac01e060 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.313+0000 7f50b27fc700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 0x7f50a80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.313+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f50ac013070 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.313+0000 7f50c1959700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 0x7f50a80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.314+0000 7f50c1959700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 0x7f50a80991c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f50b800b5c0 tx=0x7f50b8005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:05:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.316+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f50ac08cdd0 con 0x7f50bc0fd1d0 2022-03-21T14:05:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:20.501+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f50bc05d410 con 0x7f50a8096d10 2022-03-21T14:05:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: osdmap e66: 10 total, 10 up, 10 in 2022-03-21T14:05:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T14:05:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 4319M 2022-03-21T14:05:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:05:20.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:05:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: osdmap e66: 10 total, 10 up, 10 in 2022-03-21T14:05:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T14:05:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 4319M 2022-03-21T14:05:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:05:20.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:05:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: osdmap e66: 10 total, 10 up, 10 in 2022-03-21T14:05:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T14:05:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 4319M 2022-03-21T14:05:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:05:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:05:21.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:21 smithi016 ceph-mon[22010]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:21.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:21 smithi016 ceph-mon[22010]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:21.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:21.614 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:21.614 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:21 smithi079 ceph-mon[23158]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:21 smithi079 ceph-mon[23158]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:21 smithi002 ceph-mon[17167]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:21 smithi002 ceph-mon[17167]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/1586979820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]': finished 2022-03-21T14:05:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: osdmap e67: 11 total, 10 up, 11 in 2022-03-21T14:05:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:23 smithi079 ceph-mon[23158]: from='client.? 172.21.15.16:0/2004762438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/1586979820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]': finished 2022-03-21T14:05:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: osdmap e67: 11 total, 10 up, 11 in 2022-03-21T14:05:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:23 smithi002 ceph-mon[17167]: from='client.? 172.21.15.16:0/2004762438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/1586979820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]: dispatch 2022-03-21T14:05:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0"}]': finished 2022-03-21T14:05:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: osdmap e67: 11 total, 10 up, 11 in 2022-03-21T14:05:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:23 smithi016 ceph-mon[22010]: from='client.? 172.21.15.16:0/2004762438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:25 smithi079 ceph-mon[23158]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:25 smithi002 ceph-mon[17167]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:25 smithi016 ceph-mon[22010]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:27 smithi079 ceph-mon[23158]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:27.737 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:27 smithi016 ceph-mon[22010]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:27 smithi002 ceph-mon[17167]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:29 smithi016 ceph-mon[22010]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:29 smithi079 ceph-mon[23158]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:29 smithi002 ceph-mon[17167]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-03-21T14:05:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:30 smithi016 ceph-mon[22010]: Deploying daemon osd.10 on smithi016 2022-03-21T14:05:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-03-21T14:05:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:30.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:30 smithi002 ceph-mon[17167]: Deploying daemon osd.10 on smithi016 2022-03-21T14:05:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-03-21T14:05:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:30 smithi079 ceph-mon[23158]: Deploying daemon osd.10 on smithi016 2022-03-21T14:05:31.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:31 smithi016 ceph-mon[22010]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:31 smithi002 ceph-mon[17167]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:31 smithi079 ceph-mon[23158]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:33 smithi016 ceph-mon[22010]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:33 smithi002 ceph-mon[17167]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:33 smithi079 ceph-mon[23158]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:35 smithi002 ceph-mon[17167]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:35 smithi016 ceph-mon[22010]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:35 smithi079 ceph-mon[23158]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:37.022 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 10 on host 'smithi016' 2022-03-21T14:05:37.023 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.020+0000 7f50b27fc700 1 -- 172.21.15.16:0/1639148035 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f50bc05d410 con 0x7f50a8096d10 2022-03-21T14:05:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 msgr2=0x7f50a80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 0x7f50a80991c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f50b800b5c0 tx=0x7f50b8005f70 comp rx=0 tx=0).stop 2022-03-21T14:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 msgr2=0x7f50bc06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00ca00 tx=0x7f50ac00a2e0 comp rx=0 tx=0).stop 2022-03-21T14:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 shutdown_connections 2022-03-21T14:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f50a8096d10 0x7f50a80991c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f50bc0fdb60 0x7f50bc06f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50bc0fbf30 0x7f50bc070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 --2- 172.21.15.16:0/1639148035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50bc0fd1d0 0x7f50bc06f320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 >> 172.21.15.16:0/1639148035 conn(0x7f50bc0f7720 msgr2=0x7f50bc0f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 shutdown_connections 2022-03-21T14:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2022-03-21T14:05:37.023+0000 7f50c3bbd700 1 -- 172.21.15.16:0/1639148035 wait complete. 2022-03-21T14:05:37.552 DEBUG:teuthology.orchestra.run.smithi016:osd.10> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.10.service 2022-03-21T14:05:37.555 INFO:tasks.cephadm:Deploying osd.11 on smithi079 with /dev/vg_nvme/lv_1... 2022-03-21T14:05:37.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-03-21T14:05:37.584 INFO:journalctl@ceph.osd.10.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:05:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:38 smithi016 ceph-mon[22010]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.315 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:38 smithi079 ceph-mon[23158]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:38.315 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.316 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:38 smithi002 ceph-mon[17167]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:38.664 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 14:05:38 smithi016 conmon[43966]: 2022-03-21T14:05:38.284+0000 7f40cc7ae3c0 -1 osd.10 0 log_to_monitors true 2022-03-21T14:05:39.101 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:39 smithi079 ceph-mon[23158]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.101 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:39 smithi079 ceph-mon[23158]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:39 smithi016 ceph-mon[22010]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:39 smithi016 ceph-mon[22010]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:39 smithi002 ceph-mon[17167]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:39 smithi002 ceph-mon[17167]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-03-21T14:05:39.542 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-03-21T14:05:39.543 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-03-21T14:05:39.543 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2022-03-21T14:05:39.543 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2022-03-21T14:05:39.544 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111885 s, 937 MB/s 2022-03-21T14:05:39.544 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2022-03-21T14:05:39.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2022-03-21T14:05:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-03-21T14:05:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: osdmap e68: 11 total, 10 up, 11 in 2022-03-21T14:05:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:40 smithi079 ceph-mon[23158]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-03-21T14:05:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: osdmap e68: 11 total, 10 up, 11 in 2022-03-21T14:05:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:40.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:40.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:40 smithi002 ceph-mon[17167]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:40.413 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 14:05:40 smithi016 conmon[43966]: 2022-03-21T14:05:40.035+0000 7f40c31b3700 -1 osd.10 0 waiting for initial osdmap 2022-03-21T14:05:40.414 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 14:05:40 smithi016 conmon[43966]: 2022-03-21T14:05:40.053+0000 7f40bbb45700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:05:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-03-21T14:05:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: osdmap e68: 11 total, 10 up, 11 in 2022-03-21T14:05:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: from='osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:40 smithi016 ceph-mon[22010]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2022-03-21T14:05:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:05:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: osdmap e69: 11 total, 10 up, 11 in 2022-03-21T14:05:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:05:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: osdmap e69: 11 total, 10 up, 11 in 2022-03-21T14:05:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:41.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:41.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:41.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2022-03-21T14:05:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: osdmap e69: 11 total, 10 up, 11 in 2022-03-21T14:05:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:05:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:41.646 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1088100216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fbf30 msgr2=0x7f9f600fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:41.647 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1088100216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fbf30 0x7f9f600fc340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9f540098e0 tx=0x7f9f54005f70 comp rx=0 tx=0).stop 2022-03-21T14:05:41.647 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1088100216 shutdown_connections 2022-03-21T14:05:41.647 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1088100216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fdb60 0x7f9f60104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:41.647 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1088100216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fbf30 0x7f9f600fc340 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1088100216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fd1d0 0x7f9f600fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1088100216 >> 172.21.15.79:0/1088100216 conn(0x7f9f600f7720 msgr2=0x7f9f600f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:05:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1088100216 shutdown_connections 2022-03-21T14:05:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.643+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1088100216 wait complete. 2022-03-21T14:05:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 Processor -- start 2022-03-21T14:05:41.649 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 -- start start 2022-03-21T14:05:41.649 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fbf30 0x7f9f60070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:41.649 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:41.649 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fdb60 0x7f9f6006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6006ff70 con 0x7f9f600fdb60 2022-03-21T14:05:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600700e0 con 0x7f9f600fbf30 2022-03-21T14:05:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f66c6e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60117010 con 0x7f9f600fd1d0 2022-03-21T14:05:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33568/0 (socket says 172.21.15.79:33568) 2022-03-21T14:05:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 -- 172.21.15.79:0/1053807495 learned_addr learned my addr 172.21.15.79:0/1053807495 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:05:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f64a0a700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fbf30 0x7f9f60070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f6520b700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fdb60 0x7f9f6006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 -- 172.21.15.79:0/1053807495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fbf30 msgr2=0x7f9f60070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:41.652 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fbf30 0x7f9f60070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:41.652 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 -- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fdb60 msgr2=0x7f9f6006f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:41.652 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fdb60 0x7f9f6006f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:41.652 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.644+0000 7f9f5ffff700 1 -- 172.21.15.79:0/1053807495 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540093f0 con 0x7f9f600fd1d0 2022-03-21T14:05:41.652 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f5ffff700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00b530 tx=0x7f9f4c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:05:41.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c011630 con 0x7f9f600fd1d0 2022-03-21T14:05:41.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f601171b0 con 0x7f9f600fd1d0 2022-03-21T14:05:41.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9f4c011c40 con 0x7f9f600fd1d0 2022-03-21T14:05:41.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c010cf0 con 0x7f9f600fd1d0 2022-03-21T14:05:41.653 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.645+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f601176f0 con 0x7f9f600fd1d0 2022-03-21T14:05:41.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.646+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f60107740 con 0x7f9f600fd1d0 2022-03-21T14:05:41.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.647+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f9f4c00f050 con 0x7f9f600fd1d0 2022-03-21T14:05:41.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.647+0000 7f9f5d7fa700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 0x7f9f500991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:05:41.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.647+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f9f4c0be900 con 0x7f9f600fd1d0 2022-03-21T14:05:41.655 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.647+0000 7f9f64a0a700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 0x7f9f500991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:05:41.656 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.648+0000 7f9f64a0a700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 0x7f9f500991c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400b5c0 tx=0x7f9f54005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:05:41.656 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.651+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f9f4c08c200 con 0x7f9f600fd1d0 2022-03-21T14:05:41.828 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:41.827+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f6005d410 con 0x7f9f50096d10 2022-03-21T14:05:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:05:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:05:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T14:05:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 4294M 2022-03-21T14:05:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] boot 2022-03-21T14:05:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: osdmap e70: 11 total, 11 up, 11 in 2022-03-21T14:05:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:42.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:05:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:05:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T14:05:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 4294M 2022-03-21T14:05:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] boot 2022-03-21T14:05:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: osdmap e70: 11 total, 11 up, 11 in 2022-03-21T14:05:42.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:42.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:05:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:05:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T14:05:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 4294M 2022-03-21T14:05:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-03-21T14:05:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: osd.10 [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] boot 2022-03-21T14:05:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: osdmap e70: 11 total, 11 up, 11 in 2022-03-21T14:05:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T14:05:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-03-21T14:05:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-03-21T14:05:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:43 smithi002 ceph-mon[17167]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:43 smithi002 ceph-mon[17167]: osdmap e71: 11 total, 11 up, 11 in 2022-03-21T14:05:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:43 smithi016 ceph-mon[22010]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:43 smithi016 ceph-mon[22010]: osdmap e71: 11 total, 11 up, 11 in 2022-03-21T14:05:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:43 smithi079 ceph-mon[23158]: from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:05:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:43 smithi079 ceph-mon[23158]: osdmap e71: 11 total, 11 up, 11 in 2022-03-21T14:05:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:44 smithi002 ceph-mon[17167]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:44 smithi002 ceph-mon[17167]: osdmap e72: 11 total, 11 up, 11 in 2022-03-21T14:05:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:44 smithi002 ceph-mon[17167]: osdmap e73: 11 total, 11 up, 11 in 2022-03-21T14:05:44.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:44 smithi016 ceph-mon[22010]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:44 smithi016 ceph-mon[22010]: osdmap e72: 11 total, 11 up, 11 in 2022-03-21T14:05:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:44 smithi016 ceph-mon[22010]: osdmap e73: 11 total, 11 up, 11 in 2022-03-21T14:05:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:44 smithi079 ceph-mon[23158]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:44 smithi079 ceph-mon[23158]: osdmap e72: 11 total, 11 up, 11 in 2022-03-21T14:05:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:44 smithi079 ceph-mon[23158]: osdmap e73: 11 total, 11 up, 11 in 2022-03-21T14:05:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1427690696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]': finished 2022-03-21T14:05:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: osdmap e74: 12 total, 11 up, 12 in 2022-03-21T14:05:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:05:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:45 smithi002 ceph-mon[17167]: from='client.? 172.21.15.79:0/1997140383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1427690696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]': finished 2022-03-21T14:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: osdmap e74: 12 total, 11 up, 12 in 2022-03-21T14:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:45 smithi016 ceph-mon[22010]: from='client.? 172.21.15.79:0/1997140383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1427690696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]: dispatch 2022-03-21T14:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31a5878a-28c3-42a5-80c4-a032b07fc71b"}]': finished 2022-03-21T14:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: osdmap e74: 12 total, 11 up, 12 in 2022-03-21T14:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:05:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:45 smithi079 ceph-mon[23158]: from='client.? 172.21.15.79:0/1997140383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-03-21T14:05:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:46 smithi002 ceph-mon[17167]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:46 smithi016 ceph-mon[22010]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:46 smithi079 ceph-mon[23158]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:48 smithi002 ceph-mon[17167]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2022-03-21T14:05:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:48 smithi016 ceph-mon[22010]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2022-03-21T14:05:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:48 smithi079 ceph-mon[23158]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2022-03-21T14:05:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:50 smithi002 ceph-mon[17167]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:50 smithi016 ceph-mon[22010]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:50 smithi079 ceph-mon[23158]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-03-21T14:05:52.349 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:52 smithi079 ceph-mon[23158]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-03-21T14:05:52.350 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-03-21T14:05:52.350 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:52 smithi002 ceph-mon[17167]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-03-21T14:05:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-03-21T14:05:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:52 smithi016 ceph-mon[22010]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-03-21T14:05:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-03-21T14:05:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:53 smithi002 ceph-mon[17167]: Deploying daemon osd.11 on smithi079 2022-03-21T14:05:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:53 smithi016 ceph-mon[22010]: Deploying daemon osd.11 on smithi079 2022-03-21T14:05:53.414 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:53 smithi079 ceph-mon[23158]: Deploying daemon osd.11 on smithi079 2022-03-21T14:05:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:54 smithi002 ceph-mon[17167]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-03-21T14:05:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:54 smithi016 ceph-mon[22010]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-03-21T14:05:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:54 smithi079 ceph-mon[23158]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-03-21T14:05:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:55 smithi002 ceph-mon[17167]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2022-03-21T14:05:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:55 smithi016 ceph-mon[22010]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2022-03-21T14:05:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:55 smithi079 ceph-mon[23158]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2022-03-21T14:05:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:56.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:56.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:05:56.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:05:56.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:05:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:57 smithi079 ceph-mon[23158]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:05:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:57 smithi016 ceph-mon[22010]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:05:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:57 smithi002 ceph-mon[17167]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-03-21T14:05:58.439 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.438+0000 7f9f5d7fa700 1 -- 172.21.15.79:0/1053807495 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9f6005d410 con 0x7f9f50096d10 2022-03-21T14:05:58.440 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 11 on host 'smithi079' 2022-03-21T14:05:58.442 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 msgr2=0x7f9f500991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:58.442 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 0x7f9f500991c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400b5c0 tx=0x7f9f54005f70 comp rx=0 tx=0).stop 2022-03-21T14:05:58.442 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 msgr2=0x7f9f6006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:05:58.443 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00b530 tx=0x7f9f4c0090f0 comp rx=0 tx=0).stop 2022-03-21T14:05:58.443 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 shutdown_connections 2022-03-21T14:05:58.443 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f600fbf30 0x7f9f60070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:58.443 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9f50096d10 0x7f9f500991c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:58.444 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f600fdb60 0x7f9f6006f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:58.444 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 --2- 172.21.15.79:0/1053807495 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f600fd1d0 0x7f9f6006f320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:05:58.444 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.440+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 >> 172.21.15.79:0/1053807495 conn(0x7f9f600f7720 msgr2=0x7f9f60100400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:05:58.444 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.441+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 shutdown_connections 2022-03-21T14:05:58.445 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:05:58.441+0000 7f9f66c6e700 1 -- 172.21.15.79:0/1053807495 wait complete. 2022-03-21T14:05:58.943 DEBUG:teuthology.orchestra.run.smithi079:osd.11> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@osd.11.service 2022-03-21T14:05:58.945 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2022-03-21T14:05:58.946 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd stat -f json 2022-03-21T14:05:58.972 INFO:journalctl@ceph.osd.11.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:05:59.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:59 smithi079 ceph-mon[23158]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:59.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:59.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:05:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:59.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:59 smithi002 ceph-mon[17167]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:59.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:59.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:05:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:59.908 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 14:05:59 smithi079 conmon[45530]: 2022-03-21T14:05:59.626+0000 7f23a71153c0 -1 osd.11 0 log_to_monitors true 2022-03-21T14:05:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:59 smithi016 ceph-mon[22010]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:05:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:05:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:05:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:00 smithi002 ceph-mon[17167]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:00 smithi002 ceph-mon[17167]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:00 smithi079 ceph-mon[23158]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:00 smithi079 ceph-mon[23158]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:00 smithi016 ceph-mon[22010]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:00 smithi016 ceph-mon[22010]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-03-21T14:06:01.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 -- 172.21.15.2:0/4119036398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 msgr2=0x7f0cdc106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:01.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 --2- 172.21.15.2:0/4119036398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc106230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009c70 tx=0x7f0cd40094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 -- 172.21.15.2:0/4119036398 shutdown_connections 2022-03-21T14:06:01.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 --2- 172.21.15.2:0/4119036398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 0x7f0cdc100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 --2- 172.21.15.2:0/4119036398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc106230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 --2- 172.21.15.2:0/4119036398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 0x7f0cdc1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 -- 172.21.15.2:0/4119036398 >> 172.21.15.2:0/4119036398 conn(0x7f0cdc0f6620 msgr2=0x7f0cdc0f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:01.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 -- 172.21.15.2:0/4119036398 shutdown_connections 2022-03-21T14:06:01.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.044+0000 7f0ce4305700 1 -- 172.21.15.2:0/4119036398 wait complete. 2022-03-21T14:06:01.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.045+0000 7f0ce4305700 1 Processor -- start 2022-03-21T14:06:01.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.045+0000 7f0ce4305700 1 -- start start 2022-03-21T14:06:01.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 0x7f0cdc10ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:01.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 0x7f0cdc10f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:01.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:01.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1167f0 con 0x7f0cdc1014d0 2022-03-21T14:06:01.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc116960 con 0x7f0cdc065f90 2022-03-21T14:06:01.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce4305700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc116b20 con 0x7f0cdc103df0 2022-03-21T14:06:01.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce28a2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:01.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce18a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 0x7f0cdc10f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:01.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce28a2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39488/0 (socket says 172.21.15.2:39488) 2022-03-21T14:06:01.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce20a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 0x7f0cdc10ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:01.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce28a2700 1 -- 172.21.15.2:0/1286378230 learned_addr learned my addr 172.21.15.2:0/1286378230 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:01.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.046+0000 7f0ce28a2700 1 -- 172.21.15.2:0/1286378230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 msgr2=0x7f0cdc10ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:01.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce28a2700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 0x7f0cdc10ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce28a2700 1 -- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 msgr2=0x7f0cdc10f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:01.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce28a2700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 0x7f0cdc10f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce28a2700 1 -- 172.21.15.2:0/1286378230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4009780 con 0x7f0cdc103df0 2022-03-21T14:06:01.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce28a2700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40095d0 tx=0x7f0cd40094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:01.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd401c070 con 0x7f0cdc103df0 2022-03-21T14:06:01.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0cd400f910 con 0x7f0cdc103df0 2022-03-21T14:06:01.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc116da0 con 0x7f0cdc103df0 2022-03-21T14:06:01.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4003de0 con 0x7f0cdc103df0 2022-03-21T14:06:01.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.047+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc1171b0 con 0x7f0cdc103df0 2022-03-21T14:06:01.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.048+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc1099b0 con 0x7f0cdc103df0 2022-03-21T14:06:01.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.050+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f0cd4027030 con 0x7f0cdc103df0 2022-03-21T14:06:01.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.051+0000 7f0cd2ffd700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 0x7f0cc8099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:01.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.051+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6827+0+0 (secure 0 0 0) 0x7f0cd40c05b0 con 0x7f0cdc103df0 2022-03-21T14:06:01.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.051+0000 7f0ce20a1700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 0x7f0cc8099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:01.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.052+0000 7f0ce20a1700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 0x7f0cc8099240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc1101d0 tx=0x7f0cd8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:01.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.055+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f0cd408de50 con 0x7f0cdc103df0 2022-03-21T14:06:01.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.242+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0cdc0f7fd0 con 0x7f0cdc103df0 2022-03-21T14:06:01.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.242+0000 7f0cd2ffd700 1 -- 172.21.15.2:0/1286378230 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f0cd4091470 con 0x7f0cdc103df0 2022-03-21T14:06:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 msgr2=0x7f0cc8099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 0x7f0cc8099240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc1101d0 tx=0x7f0cd8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 msgr2=0x7f0cdc116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40095d0 tx=0x7f0cd40094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 shutdown_connections 2022-03-21T14:06:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc065f90 0x7f0cdc10ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0cc8096d90 0x7f0cc8099240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0cdc103df0 0x7f0cdc116270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 --2- 172.21.15.2:0/1286378230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cdc1014d0 0x7f0cdc10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 >> 172.21.15.2:0/1286378230 conn(0x7f0cdc0f6620 msgr2=0x7f0cdc102110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 shutdown_connections 2022-03-21T14:06:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:01.245+0000 7f0ce4305700 1 -- 172.21.15.2:0/1286378230 wait complete. 2022-03-21T14:06:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-03-21T14:06:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: osdmap e75: 12 total, 11 up, 12 in 2022-03-21T14:06:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1286378230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:01.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:01.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:01.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.722 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 14:06:01 smithi079 conmon[45530]: 2022-03-21T14:06:01.457+0000 7f239db1a700 -1 osd.11 0 waiting for initial osdmap 2022-03-21T14:06:01.723 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 14:06:01 smithi079 conmon[45530]: 2022-03-21T14:06:01.475+0000 7f2398cb1700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-03-21T14:06:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-03-21T14:06:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: osdmap e75: 12 total, 11 up, 12 in 2022-03-21T14:06:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:01.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1286378230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:01.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:01.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:01.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:01.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.783 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1647871541,"num_in_osds":12,"osd_in_since":1647871544,"num_remapped_pgs":0} 2022-03-21T14:06:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-03-21T14:06:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: osdmap e75: 12 total, 11 up, 12 in 2022-03-21T14:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2022-03-21T14:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1286378230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:06:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:01.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:01.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:01.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T14:06:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 4293M 2022-03-21T14:06:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:06:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: osdmap e76: 12 total, 11 up, 12 in 2022-03-21T14:06:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:02.786 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd stat -f json 2022-03-21T14:06:02.853 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T14:06:02.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 4293M 2022-03-21T14:06:02.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:06:02.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: osdmap e76: 12 total, 11 up, 12 in 2022-03-21T14:06:02.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:02.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T14:06:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 4293M 2022-03-21T14:06:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2022-03-21T14:06:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: osdmap e76: 12 total, 11 up, 12 in 2022-03-21T14:06:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: purged_snaps scrub starts 2022-03-21T14:06:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: purged_snaps scrub ok 2022-03-21T14:06:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] boot 2022-03-21T14:06:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: osdmap e77: 12 total, 12 up, 12 in 2022-03-21T14:06:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: purged_snaps scrub starts 2022-03-21T14:06:03.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: purged_snaps scrub ok 2022-03-21T14:06:03.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:03.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] boot 2022-03-21T14:06:03.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: osdmap e77: 12 total, 12 up, 12 in 2022-03-21T14:06:03.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: purged_snaps scrub starts 2022-03-21T14:06:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: purged_snaps scrub ok 2022-03-21T14:06:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-03-21T14:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: osd.11 [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] boot 2022-03-21T14:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: osdmap e77: 12 total, 12 up, 12 in 2022-03-21T14:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T14:06:04.407 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:04 smithi079 ceph-mon[23158]: osdmap e78: 12 total, 12 up, 12 in 2022-03-21T14:06:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:04 smithi002 ceph-mon[17167]: osdmap e78: 12 total, 12 up, 12 in 2022-03-21T14:06:04.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 -- 172.21.15.2:0/2779849247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 msgr2=0x7f59040fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:04.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 --2- 172.21.15.2:0/2779849247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f59040fc360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f58f40098e0 tx=0x7f58f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:04.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 -- 172.21.15.2:0/2779849247 shutdown_connections 2022-03-21T14:06:04.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 --2- 172.21.15.2:0/2779849247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fdb80 0x7f5904104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:04.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 --2- 172.21.15.2:0/2779849247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f59040fc360 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:04.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 --2- 172.21.15.2:0/2779849247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fd1f0 0x7f59040fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:04.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 -- 172.21.15.2:0/2779849247 >> 172.21.15.2:0/2779849247 conn(0x7f59040f7720 msgr2=0x7f59040f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:04.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 -- 172.21.15.2:0/2779849247 shutdown_connections 2022-03-21T14:06:04.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.902+0000 7f590aa0d700 1 -- 172.21.15.2:0/2779849247 wait complete. 2022-03-21T14:06:04.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 Processor -- start 2022-03-21T14:06:04.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 -- start start 2022-03-21T14:06:04.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:04.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 0x7f590410f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:04.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fdb80 0x7f5904116450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:04.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5904116990 con 0x7f59040fbf50 2022-03-21T14:06:04.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5904116b00 con 0x7f59040fd1f0 2022-03-21T14:06:04.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.903+0000 7f590aa0d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5904116cc0 con 0x7f59040fdb80 2022-03-21T14:06:04.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:04.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43638/0 (socket says 172.21.15.2:43638) 2022-03-21T14:06:04.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 -- 172.21.15.2:0/809606236 learned_addr learned my addr 172.21.15.2:0/809606236 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:04.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f59037fe700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 0x7f590410f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:04.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5908faa700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fdb80 0x7f5904116450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:04.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 -- 172.21.15.2:0/809606236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fdb80 msgr2=0x7f5904116450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:04.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fdb80 0x7f5904116450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:04.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 -- 172.21.15.2:0/809606236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 msgr2=0x7f590410f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:04.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 0x7f590410f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:04.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f5903fff700 1 -- 172.21.15.2:0/809606236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40093f0 con 0x7f59040fbf50 2022-03-21T14:06:04.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.904+0000 7f59037fe700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 0x7f590410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:04.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f5903fff700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f58f4000e50 tx=0x7f58f4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:04.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f401d070 con 0x7f59040fbf50 2022-03-21T14:06:04.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f58f4005040 con 0x7f59040fbf50 2022-03-21T14:06:04.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5904116f40 con 0x7f59040fbf50 2022-03-21T14:06:04.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40179e0 con 0x7f59040fbf50 2022-03-21T14:06:04.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.905+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5904117310 con 0x7f59040fbf50 2022-03-21T14:06:04.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.906+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59041076e0 con 0x7f59040fbf50 2022-03-21T14:06:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:04 smithi016 ceph-mon[22010]: osdmap e78: 12 total, 12 up, 12 in 2022-03-21T14:06:04.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.908+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f58f4028030 con 0x7f59040fbf50 2022-03-21T14:06:04.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.909+0000 7f5900ff9700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 0x7f58ec0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:04.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.909+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f58f40c0590 con 0x7f59040fbf50 2022-03-21T14:06:04.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.910+0000 7f59037fe700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 0x7f58ec0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:04.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.910+0000 7f59037fe700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 0x7f58ec0991c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005fd0 tx=0x7f58f8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:04.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:04.913+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f58f408dc10 con 0x7f59040fbf50 2022-03-21T14:06:05.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.101+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f5904117a70 con 0x7f59040fbf50 2022-03-21T14:06:05.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.102+0000 7f5900ff9700 1 -- 172.21.15.2:0/809606236 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7f58f4091230 con 0x7f59040fbf50 2022-03-21T14:06:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:05.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.104+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 msgr2=0x7f58ec0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:05.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.104+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 0x7f58ec0991c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005fd0 tx=0x7f58f8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:05.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.104+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 msgr2=0x7f590410ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:05.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.104+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f58f4000e50 tx=0x7f58f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:05.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.104+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 shutdown_connections 2022-03-21T14:06:05.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f58ec096d10 0x7f58ec0991c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:05.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59040fdb80 0x7f5904116450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:05.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59040fbf50 0x7f590410ee80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:05.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 --2- 172.21.15.2:0/809606236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59040fd1f0 0x7f590410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:05.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 >> 172.21.15.2:0/809606236 conn(0x7f59040f7720 msgr2=0x7f59040f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:05.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 shutdown_connections 2022-03-21T14:06:05.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:05.105+0000 7f590aa0d700 1 -- 172.21.15.2:0/809606236 wait complete. 2022-03-21T14:06:05.613 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1647871562,"num_in_osds":12,"osd_in_since":1647871544,"num_remapped_pgs":0} 2022-03-21T14:06:05.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-03-21T14:06:05.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:05 smithi002 ceph-mon[17167]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:05.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:05 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/809606236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:05 smithi079 ceph-mon[23158]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:05 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/809606236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:05 smithi016 ceph-mon[22010]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:05 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/809606236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-03-21T14:06:06.263 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:06:07.415 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:07.587 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:07 smithi002 ceph-mon[17167]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:07.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 -- 172.21.15.2:0/1721428241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fd280 msgr2=0x7f82ec0fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:07.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 --2- 172.21.15.2:0/1721428241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fd280 0x7f82ec0fd6d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f82e00098e0 tx=0x7f82e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:07.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 -- 172.21.15.2:0/1721428241 shutdown_connections 2022-03-21T14:06:07.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 --2- 172.21.15.2:0/1721428241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 0x7f82ec1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:07.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 --2- 172.21.15.2:0/1721428241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fbfe0 0x7f82ec0fc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:07.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 --2- 172.21.15.2:0/1721428241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fd280 0x7f82ec0fd6d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:07.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 -- 172.21.15.2:0/1721428241 >> 172.21.15.2:0/1721428241 conn(0x7f82ec0f77b0 msgr2=0x7f82ec0f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:07.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 -- 172.21.15.2:0/1721428241 shutdown_connections 2022-03-21T14:06:07.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.885+0000 7f82f16aa700 1 -- 172.21.15.2:0/1721428241 wait complete. 2022-03-21T14:06:07.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 Processor -- start 2022-03-21T14:06:07.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 -- start start 2022-03-21T14:06:07.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fbfe0 0x7f82ec10efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:07.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:07.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 0x7f82ec116580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:07.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec116ac0 con 0x7f82ec0fbfe0 2022-03-21T14:06:07.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec116c30 con 0x7f82ec0fdc10 2022-03-21T14:06:07.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.886+0000 7f82f16aa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec116df0 con 0x7f82ec0fd280 2022-03-21T14:06:07.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:07.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fbfe0 0x7f82ec10efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:07.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39496/0 (socket says 172.21.15.2:39496) 2022-03-21T14:06:07.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 -- 172.21.15.2:0/831499342 learned_addr learned my addr 172.21.15.2:0/831499342 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:07.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82eb7fe700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 0x7f82ec116580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:07.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 -- 172.21.15.2:0/831499342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 msgr2=0x7f82ec116580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:07.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 0x7f82ec116580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:07.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 -- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fbfe0 msgr2=0x7f82ec10efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:07.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fbfe0 0x7f82ec10efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:07.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 -- 172.21.15.2:0/831499342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00093f0 con 0x7f82ec0fd280 2022-03-21T14:06:07.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.887+0000 7f82ea7fc700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82e0006010 tx=0x7f82e0005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:07.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.888+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e001d070 con 0x7f82ec0fd280 2022-03-21T14:06:07.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.888+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec117070 con 0x7f82ec0fd280 2022-03-21T14:06:07.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.888+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f82e0005040 con 0x7f82ec0fd280 2022-03-21T14:06:07.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.888+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0017a00 con 0x7f82ec0fd280 2022-03-21T14:06:07.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.888+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec117470 con 0x7f82ec0fd280 2022-03-21T14:06:07.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.889+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec107830 con 0x7f82ec0fd280 2022-03-21T14:06:07.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.891+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f82e0029030 con 0x7f82ec0fd280 2022-03-21T14:06:07.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.892+0000 7f82d37fe700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 0x7f82d40991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:07.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.892+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f82e00c0cb0 con 0x7f82ec0fd280 2022-03-21T14:06:07.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.892+0000 7f82eaffd700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 0x7f82d40991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:07.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.893+0000 7f82eaffd700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 0x7f82d40991c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0fd0e0 tx=0x7f82dc005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:07.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:07.894+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f82e008e330 con 0x7f82ec0fd280 2022-03-21T14:06:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:07 smithi016 ceph-mon[22010]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:07 smithi079 ceph-mon[23158]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.081+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82ec109690 con 0x7f82ec0fd280 2022-03-21T14:06:08.083 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.083+0000 7f82d37fe700 1 -- 172.21.15.2:0/831499342 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17985 (secure 0 0 0) 0x7f82e0091950 con 0x7f82ec0fd280 2022-03-21T14:06:08.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:08.085 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:03.460664+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:06:08.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 msgr2=0x7f82d40991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:08.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 0x7f82d40991c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0fd0e0 tx=0x7f82dc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:08.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 msgr2=0x7f82ec10f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:08.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82e0006010 tx=0x7f82e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:08.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 shutdown_connections 2022-03-21T14:06:08.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec0fdc10 0x7f82ec116580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:08.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f82d4096d10 0x7f82d40991c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:08.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82ec0fbfe0 0x7f82ec10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:08.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 --2- 172.21.15.2:0/831499342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82ec0fd280 0x7f82ec10f4f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:08.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 >> 172.21.15.2:0/831499342 conn(0x7f82ec0f77b0 msgr2=0x7f82ec102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:08.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 shutdown_connections 2022-03-21T14:06:08.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:08.086+0000 7f82f16aa700 1 -- 172.21.15.2:0/831499342 wait complete. 2022-03-21T14:06:08.522 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-21T14:06:08.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd pool get .mgr pg_num 2022-03-21T14:06:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:08 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/831499342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:08 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/831499342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:08 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/831499342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:06:09.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:09 smithi002 ceph-mon[17167]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:09 smithi016 ceph-mon[22010]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:09 smithi079 ceph-mon[23158]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:10.212 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 -- 172.21.15.2:0/2720702392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c104020 msgr2=0x7fe61c106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 --2- 172.21.15.2:0/2720702392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c104020 0x7fe61c106400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe614009c70 tx=0x7fe6140094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 -- 172.21.15.2:0/2720702392 shutdown_connections 2022-03-21T14:06:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 --2- 172.21.15.2:0/2720702392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c104020 0x7fe61c106400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 --2- 172.21.15.2:0/2720702392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c070e40 0x7fe61c06f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 --2- 172.21.15.2:0/2720702392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c06fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 -- 172.21.15.2:0/2720702392 >> 172.21.15.2:0/2720702392 conn(0x7fe61c0f7880 msgr2=0x7fe61c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:10.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 -- 172.21.15.2:0/2720702392 shutdown_connections 2022-03-21T14:06:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.720+0000 7fe621d3e700 1 -- 172.21.15.2:0/2720702392 wait complete. 2022-03-21T14:06:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.721+0000 7fe621d3e700 1 Processor -- start 2022-03-21T14:06:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.721+0000 7fe621d3e700 1 -- start start 2022-03-21T14:06:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c10edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:10.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:10.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 0x7fe61c116390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:10.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1168d0 con 0x7fe61c070e40 2022-03-21T14:06:10.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c116a40 con 0x7fe61c06f880 2022-03-21T14:06:10.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe621d3e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c116c00 con 0x7fe61c104020 2022-03-21T14:06:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe61affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe61affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43646/0 (socket says 172.21.15.2:43646) 2022-03-21T14:06:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe61affd700 1 -- 172.21.15.2:0/985941991 learned_addr learned my addr 172.21.15.2:0/985941991 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe61b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c10edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.722+0000 7fe61bfff700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 0x7fe61c116390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 -- 172.21.15.2:0/985941991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 msgr2=0x7fe61c116390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 0x7fe61c116390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 -- 172.21.15.2:0/985941991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 msgr2=0x7fe61c10edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c10edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 -- 172.21.15.2:0/985941991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614009780 con 0x7fe61c070e40 2022-03-21T14:06:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61bfff700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 0x7fe61c116390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61b7fe700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c10edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:10.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe61affd700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe61000fa90 tx=0x7fe610005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:10.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000b4f0 con 0x7fe61c070e40 2022-03-21T14:06:10.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe61000dc80 con 0x7fe61c070e40 2022-03-21T14:06:10.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.723+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c116ee0 con 0x7fe61c070e40 2022-03-21T14:06:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.724+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610011d50 con 0x7fe61c070e40 2022-03-21T14:06:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.724+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c117410 con 0x7fe61c070e40 2022-03-21T14:06:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.725+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61c109ae0 con 0x7fe61c070e40 2022-03-21T14:06:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.726+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fe61000c040 con 0x7fe61c070e40 2022-03-21T14:06:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.728+0000 7fe603fff700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 0x7fe604099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.728+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe6100bf890 con 0x7fe61c070e40 2022-03-21T14:06:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.728+0000 7fe61b7fe700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 0x7fe604099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.729+0000 7fe61b7fe700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 0x7fe604099240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe60c005fd0 tx=0x7fe60c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:10.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.732+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe61008cf10 con 0x7fe61c070e40 2022-03-21T14:06:10.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.919+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe61c02ac00 con 0x7fe61c070e40 2022-03-21T14:06:10.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.919+0000 7fe603fff700 1 -- 172.21.15.2:0/985941991 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe610090530 con 0x7fe61c070e40 2022-03-21T14:06:10.921 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:06:10.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.921+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 msgr2=0x7fe604099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:10.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.921+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 0x7fe604099240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe60c005fd0 tx=0x7fe60c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 msgr2=0x7fe61c10f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe61000fa90 tx=0x7fe610005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 shutdown_connections 2022-03-21T14:06:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe61c06f880 0x7fe61c10edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe604096d90 0x7fe604099240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe61c104020 0x7fe61c116390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 --2- 172.21.15.2:0/985941991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe61c070e40 0x7fe61c10f300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 >> 172.21.15.2:0/985941991 conn(0x7fe61c0f7880 msgr2=0x7fe61c106170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 shutdown_connections 2022-03-21T14:06:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:10.922+0000 7fe621d3e700 1 -- 172.21.15.2:0/985941991 wait complete. 2022-03-21T14:06:11.419 INFO:tasks.cephadm:Adding mds.a on smithi002 2022-03-21T14:06:11.420 INFO:tasks.cephadm:Adding mds.d on smithi002 2022-03-21T14:06:11.420 INFO:tasks.cephadm:Adding mds.g on smithi002 2022-03-21T14:06:11.420 INFO:tasks.cephadm:Adding mds.j on smithi002 2022-03-21T14:06:11.421 INFO:tasks.cephadm:Adding mds.b on smithi016 2022-03-21T14:06:11.421 INFO:tasks.cephadm:Adding mds.e on smithi016 2022-03-21T14:06:11.421 INFO:tasks.cephadm:Adding mds.h on smithi016 2022-03-21T14:06:11.421 INFO:tasks.cephadm:Adding mds.k on smithi016 2022-03-21T14:06:11.422 INFO:tasks.cephadm:Adding mds.c on smithi079 2022-03-21T14:06:11.422 INFO:tasks.cephadm:Adding mds.f on smithi079 2022-03-21T14:06:11.423 INFO:tasks.cephadm:Adding mds.i on smithi079 2022-03-21T14:06:11.423 INFO:tasks.cephadm:Adding mds.l on smithi079 2022-03-21T14:06:11.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph orch apply mds all '12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l' 2022-03-21T14:06:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:11 smithi079 ceph-mon[23158]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:11 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/985941991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:11 smithi002 ceph-mon[17167]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:11 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/985941991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:11 smithi016 ceph-mon[22010]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:11 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/985941991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:12.916 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- 172.21.15.79:0/373666772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fbf30 msgr2=0x7f2e580fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:12.916 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/373666772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fbf30 0x7f2e580fc340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2e540098e0 tx=0x7f2e54005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:12.917 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- 172.21.15.79:0/373666772 shutdown_connections 2022-03-21T14:06:12.917 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/373666772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e580fdb60 0x7f2e58104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:12.917 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/373666772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fbf30 0x7f2e580fc340 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:12.917 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/373666772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fd1d0 0x7f2e580fd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:12.918 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- 172.21.15.79:0/373666772 >> 172.21.15.79:0/373666772 conn(0x7f2e580f7720 msgr2=0x7f2e580f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:12.918 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- 172.21.15.79:0/373666772 shutdown_connections 2022-03-21T14:06:12.918 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- 172.21.15.79:0/373666772 wait complete. 2022-03-21T14:06:12.918 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 Processor -- start 2022-03-21T14:06:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- start start 2022-03-21T14:06:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fbf30 0x7f2e58070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e580fdb60 0x7f2e5806f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:12.920 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58070000 con 0x7f2e580fdb60 2022-03-21T14:06:12.920 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58070170 con 0x7f2e580fbf30 2022-03-21T14:06:12.920 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5fc82700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58117030 con 0x7f2e580fd1d0 2022-03-21T14:06:12.920 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5d21d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:12.921 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5d21d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33694/0 (socket says 172.21.15.79:33694) 2022-03-21T14:06:12.921 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.914+0000 7f2e5d21d700 1 -- 172.21.15.79:0/799769271 learned_addr learned my addr 172.21.15.79:0/799769271 (peer_addr_for_me v2:172.21.15.79:0/0) 2022-03-21T14:06:12.921 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5da1e700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fbf30 0x7f2e58070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:12.921 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 -- 172.21.15.79:0/799769271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fbf30 msgr2=0x7f2e58070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:12.922 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fbf30 0x7f2e58070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:12.922 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 -- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e580fdb60 msgr2=0x7f2e5806f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:12.922 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e580fdb60 0x7f2e5806f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:12.922 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 -- 172.21.15.79:0/799769271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e540093f0 con 0x7f2e580fd1d0 2022-03-21T14:06:12.922 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5d21d700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800b530 tx=0x7f2e480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:12.923 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48011630 con 0x7f2e580fd1d0 2022-03-21T14:06:12.923 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2e48011c40 con 0x7f2e580fd1d0 2022-03-21T14:06:12.923 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48010cf0 con 0x7f2e580fd1d0 2022-03-21T14:06:12.923 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e58117260 con 0x7f2e580fd1d0 2022-03-21T14:06:12.924 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.915+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e58117750 con 0x7f2e580fd1d0 2022-03-21T14:06:12.924 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.916+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e58107760 con 0x7f2e580fd1d0 2022-03-21T14:06:12.925 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.917+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2e4800f050 con 0x7f2e580fd1d0 2022-03-21T14:06:12.925 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.918+0000 7f2e4e7fc700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 0x7f2e44099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:12.925 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.918+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f2e480beec0 con 0x7f2e580fd1d0 2022-03-21T14:06:12.926 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.918+0000 7f2e5da1e700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 0x7f2e44099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:12.926 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.919+0000 7f2e5da1e700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 0x7f2e44099240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b5c0 tx=0x7f2e54005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:12.926 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:12.921+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2e4808c540 con 0x7f2e580fd1d0 2022-03-21T14:06:13.097 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.097+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e5805d510 con 0x7f2e44096d90 2022-03-21T14:06:13.105 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.105+0000 7f2e4e7fc700 1 -- 172.21.15.79:0/799769271 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2e5805d510 con 0x7f2e44096d90 2022-03-21T14:06:13.106 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled mds.all update... 2022-03-21T14:06:13.108 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 msgr2=0x7f2e44099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:13.108 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 0x7f2e44099240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b5c0 tx=0x7f2e54005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:13.108 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 msgr2=0x7f2e5806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:13.108 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800b530 tx=0x7f2e480090f0 comp rx=0 tx=0).stop 2022-03-21T14:06:13.109 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 shutdown_connections 2022-03-21T14:06:13.109 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e580fbf30 0x7f2e58070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:13.109 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2e44096d90 0x7f2e44099240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:13.109 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e580fdb60 0x7f2e5806f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:13.110 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 --2- 172.21.15.79:0/799769271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e580fd1d0 0x7f2e5806f320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:13.110 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.107+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 >> 172.21.15.79:0/799769271 conn(0x7f2e580f7720 msgr2=0x7f2e58100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:13.110 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.108+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 shutdown_connections 2022-03-21T14:06:13.110 INFO:teuthology.orchestra.run.smithi079.stderr:2022-03-21T14:06:13.108+0000 7f2e5fc82700 1 -- 172.21.15.79:0/799769271 wait complete. 2022-03-21T14:06:13.563 DEBUG:teuthology.orchestra.run.smithi002:mds.a> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.a.service 2022-03-21T14:06:13.569 DEBUG:teuthology.orchestra.run.smithi002:mds.d> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.d.service 2022-03-21T14:06:13.574 DEBUG:teuthology.orchestra.run.smithi002:mds.g> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.g.service 2022-03-21T14:06:13.620 DEBUG:teuthology.orchestra.run.smithi002:mds.j> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.j.service 2022-03-21T14:06:13.626 DEBUG:teuthology.orchestra.run.smithi016:mds.b> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.b.service 2022-03-21T14:06:13.628 DEBUG:teuthology.orchestra.run.smithi016:mds.e> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.e.service 2022-03-21T14:06:13.654 INFO:journalctl@ceph.mds.b.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:06:13.655 DEBUG:teuthology.orchestra.run.smithi016:mds.h> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.h.service 2022-03-21T14:06:13.666 INFO:journalctl@ceph.mds.a.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:06:13.667 INFO:journalctl@ceph.mds.d.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:06:13.679 INFO:journalctl@ceph.mds.e.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:06:13.681 DEBUG:teuthology.orchestra.run.smithi016:mds.k> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.k.service 2022-03-21T14:06:13.705 INFO:journalctl@ceph.mds.h.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:06:13.706 DEBUG:teuthology.orchestra.run.smithi079:mds.c> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.c.service 2022-03-21T14:06:13.708 INFO:journalctl@ceph.mds.j.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:06:13.709 DEBUG:teuthology.orchestra.run.smithi079:mds.f> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.f.service 2022-03-21T14:06:13.710 INFO:journalctl@ceph.mds.g.smithi002.stdout:-- Logs begin at Mon 2022-03-21 13:48:17 UTC. -- 2022-03-21T14:06:13.730 INFO:journalctl@ceph.mds.k.smithi016.stdout:-- Logs begin at Mon 2022-03-21 13:44:08 UTC. -- 2022-03-21T14:06:13.735 INFO:journalctl@ceph.mds.c.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:06:13.736 DEBUG:teuthology.orchestra.run.smithi079:mds.i> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.i.service 2022-03-21T14:06:13.754 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:13 smithi079 ceph-mon[23158]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:13.755 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:13.755 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:13.755 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:13.755 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:13.756 DEBUG:teuthology.orchestra.run.smithi079:mds.l> sudo journalctl -f -n 0 -u ceph-ee5e78f2-a91e-11ec-8c35-001a4aab830c@mds.l.service 2022-03-21T14:06:13.759 INFO:journalctl@ceph.mds.f.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:06:13.760 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-03-21T14:06:13.761 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:06:13.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:13 smithi002 ceph-mon[17167]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:13.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:13.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:13.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:13.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:13.779 INFO:journalctl@ceph.mds.i.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:06:13.784 INFO:journalctl@ceph.mds.l.smithi079.stdout:-- Logs begin at Mon 2022-03-21 13:44:32 UTC. -- 2022-03-21T14:06:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:13 smithi016 ceph-mon[22010]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:14.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 -- 172.21.15.2:0/3456991606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 msgr2=0x7f0fe80fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3456991606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe80fc3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40098e0 tx=0x7f0fe4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 -- 172.21.15.2:0/3456991606 shutdown_connections 2022-03-21T14:06:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3456991606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe81044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3456991606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe80fc3c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3456991606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 0x7f0fe80fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.013+0000 7f0fee43f700 1 -- 172.21.15.2:0/3456991606 >> 172.21.15.2:0/3456991606 conn(0x7f0fe80f7760 msgr2=0x7f0fe80f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.014+0000 7f0fee43f700 1 -- 172.21.15.2:0/3456991606 shutdown_connections 2022-03-21T14:06:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.014+0000 7f0fee43f700 1 -- 172.21.15.2:0/3456991606 wait complete. 2022-03-21T14:06:14.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.014+0000 7f0fee43f700 1 Processor -- start 2022-03-21T14:06:14.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 -- start start 2022-03-21T14:06:14.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 0x7f0fe806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe806f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806ff70 con 0x7f0fe80fdbe0 2022-03-21T14:06:14.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80700e0 con 0x7f0fe80fd250 2022-03-21T14:06:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fee43f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8117060 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fedc3e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe806f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fed43d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fed43d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39504/0 (socket says 172.21.15.2:39504) 2022-03-21T14:06:14.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fed43d700 1 -- 172.21.15.2:0/3792337404 learned_addr learned my addr 172.21.15.2:0/3792337404 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:14.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fedc3e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe806f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43650/0 (socket says 172.21.15.2:43650) 2022-03-21T14:06:14.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.015+0000 7f0fecc3c700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 0x7f0fe806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 -- 172.21.15.2:0/3792337404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 msgr2=0x7f0fe806f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 0x7f0fe806f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 -- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 msgr2=0x7f0fe806f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe806f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 -- 172.21.15.2:0/3792337404 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe40093f0 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.016+0000 7f0fed43d700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4010df0 tx=0x7f0fe4005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:14.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.017+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe401d070 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.017+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8117200 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.017+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0fe4005740 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.017+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8117600 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.017+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4017a00 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.018+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe8107790 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.021+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f0fe4028030 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.022+0000 7f0fddffb700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 0x7f0fd40991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.023+0000 7f0fecc3c700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 0x7f0fd40991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.023+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f0fe40c0930 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.024+0000 7f0fecc3c700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 0x7f0fd40991c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005fd0 tx=0x7f0fd8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:14.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.024+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f0fe4091a40 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.220+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe81031e0 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.221+0000 7f0fddffb700 1 -- 172.21.15.2:0/3792337404 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17985 (secure 0 0 0) 0x7f0fe40915d0 con 0x7f0fe80fbfb0 2022-03-21T14:06:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:14.222 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:03.460664+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:06:14.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.223+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 msgr2=0x7f0fd40991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.223+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 0x7f0fd40991c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005fd0 tx=0x7f0fd8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.223+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 msgr2=0x7f0fe8070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.223+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4010df0 tx=0x7f0fe4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:14.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 shutdown_connections 2022-03-21T14:06:14.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0fd4096d10 0x7f0fd40991c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fe80fdbe0 0x7f0fe806f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fe80fbfb0 0x7f0fe8070cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 --2- 172.21.15.2:0/3792337404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe80fd250 0x7f0fe806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 >> 172.21.15.2:0/3792337404 conn(0x7f0fe80f7760 msgr2=0x7f0fe8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:14.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 shutdown_connections 2022-03-21T14:06:14.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.224+0000 7f0fee43f700 1 -- 172.21.15.2:0/3792337404 wait complete. 2022-03-21T14:06:14.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-21T14:06:14.251 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-03-21T14:06:14.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1035931919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fbfd0 msgr2=0x7fcfc40fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1035931919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc40fc3e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00098e0 tx=0x7fcfc0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:14.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1035931919 shutdown_connections 2022-03-21T14:06:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1035931919 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc41044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1035931919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc40fc3e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1035931919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fd270 0x7fcfc40fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1035931919 >> 172.21.15.2:0/1035931919 conn(0x7fcfc40f7780 msgr2=0x7fcfc40f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1035931919 shutdown_connections 2022-03-21T14:06:14.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.527+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1035931919 wait complete. 2022-03-21T14:06:14.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.528+0000 7fcfca0e2700 1 Processor -- start 2022-03-21T14:06:14.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.528+0000 7fcfca0e2700 1 -- start start 2022-03-21T14:06:14.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.528+0000 7fcfca0e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc4101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.528+0000 7fcfca0e2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfca0e2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc40fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfca0e2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40fefb0 con 0x7fcfc40fd270 2022-03-21T14:06:14.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfca0e2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40ff120 con 0x7fcfc40fbfd0 2022-03-21T14:06:14.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfca0e2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40ff2e0 con 0x7fcfc40fdc00 2022-03-21T14:06:14.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc90e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc4101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43654/0 (socket says 172.21.15.2:43654) 2022-03-21T14:06:14.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 -- 172.21.15.2:0/1920885750 learned_addr learned my addr 172.21.15.2:0/1920885750 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:14.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc98e1700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc40fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 -- 172.21.15.2:0/1920885750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 msgr2=0x7fcfc40fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc40fe8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 -- 172.21.15.2:0/1920885750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 msgr2=0x7fcfc4101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc4101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc88df700 1 -- 172.21.15.2:0/1920885750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00093f0 con 0x7fcfc40fd270 2022-03-21T14:06:14.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc98e1700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc40fe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:14.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.529+0000 7fcfc90e0700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:14.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfc88df700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400fa90 tx=0x7fcfb4005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:14.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400b4f0 con 0x7fcfc40fd270 2022-03-21T14:06:14.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fcfb400dc80 con 0x7fcfc40fd270 2022-03-21T14:06:14.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4011dc0 con 0x7fcfc40fd270 2022-03-21T14:06:14.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc411b4b0 con 0x7fcfc40fd270 2022-03-21T14:06:14.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.530+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc411b9d0 con 0x7fcfc40fd270 2022-03-21T14:06:14.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.532+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc4107820 con 0x7fcfc40fd270 2022-03-21T14:06:14.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.533+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fcfb400c040 con 0x7fcfc40fd270 2022-03-21T14:06:14.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.535+0000 7fcfb9ffb700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 0x7fcfb00991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.535+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fcfb40bf4d0 con 0x7fcfc40fd270 2022-03-21T14:06:14.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.535+0000 7fcfc90e0700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 0x7fcfb00991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.536+0000 7fcfc90e0700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 0x7fcfb00991c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000fa00 tx=0x7fcfc0005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:14.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.539+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fcfb408cb50 con 0x7fcfc40fd270 2022-03-21T14:06:14.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.737+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfc411c050 con 0x7fcfc40fd270 2022-03-21T14:06:14.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.738+0000 7fcfb9ffb700 1 -- 172.21.15.2:0/1920885750 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfb4090170 con 0x7fcfc40fd270 2022-03-21T14:06:14.739 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:06:14.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 msgr2=0x7fcfb00991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 0x7fcfb00991c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000fa00 tx=0x7fcfc0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:14.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 msgr2=0x7fcfc41022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400fa90 tx=0x7fcfb4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 shutdown_connections 2022-03-21T14:06:14.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfc40fbfd0 0x7fcfc4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfb0096d10 0x7fcfb00991c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfc40fdc00 0x7fcfc40fe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 --2- 172.21.15.2:0/1920885750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfc40fd270 0x7fcfc41022a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 >> 172.21.15.2:0/1920885750 conn(0x7fcfc40f7780 msgr2=0x7fcfc406f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:14.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 shutdown_connections 2022-03-21T14:06:14.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.741+0000 7fcfca0e2700 1 -- 172.21.15.2:0/1920885750 wait complete. 2022-03-21T14:06:14.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:06:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l;count:12 2022-03-21T14:06:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:14.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:14.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: Deploying daemon mds.a on smithi002 2022-03-21T14:06:14.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:14 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3792337404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:06:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l;count:12 2022-03-21T14:06:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: Deploying daemon mds.a on smithi002 2022-03-21T14:06:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:14 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3792337404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l;count:12 2022-03-21T14:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: Deploying daemon mds.a on smithi002 2022-03-21T14:06:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:14 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3792337404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:14.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 -- 172.21.15.2:0/3536987581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 msgr2=0x7efeac0fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 --2- 172.21.15.2:0/3536987581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac0fd6a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0098e0 tx=0x7efe9c005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:14.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 -- 172.21.15.2:0/3536987581 shutdown_connections 2022-03-21T14:06:14.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 --2- 172.21.15.2:0/3536987581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 0x7efeac1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 --2- 172.21.15.2:0/3536987581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 0x7efeac0fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 --2- 172.21.15.2:0/3536987581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac0fd6a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 -- 172.21.15.2:0/3536987581 >> 172.21.15.2:0/3536987581 conn(0x7efeac0f7760 msgr2=0x7efeac0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:14.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 -- 172.21.15.2:0/3536987581 shutdown_connections 2022-03-21T14:06:14.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.990+0000 7efeb2eac700 1 -- 172.21.15.2:0/3536987581 wait complete. 2022-03-21T14:06:14.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 Processor -- start 2022-03-21T14:06:14.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 -- start start 2022-03-21T14:06:14.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 0x7efeac070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 0x7efeac06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:14.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac06ff70 con 0x7efeac0fd250 2022-03-21T14:06:14.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0700e0 con 0x7efeac0fdbe0 2022-03-21T14:06:14.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb2eac700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac116e80 con 0x7efeac0fbfb0 2022-03-21T14:06:14.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.991+0000 7efeb16a9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43658/0 (socket says 172.21.15.2:43658) 2022-03-21T14:06:14.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 -- 172.21.15.2:0/60952485 learned_addr learned my addr 172.21.15.2:0/60952485 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:14.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb26ab700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 0x7efeac06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb1eaa700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 0x7efeac070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:14.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 -- 172.21.15.2:0/60952485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 msgr2=0x7efeac070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 0x7efeac070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 -- 172.21.15.2:0/60952485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 msgr2=0x7efeac06f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:14.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 0x7efeac06f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:14.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 -- 172.21.15.2:0/60952485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40096b0 con 0x7efeac0fd250 2022-03-21T14:06:15.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.992+0000 7efeb16a9700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0093c0 tx=0x7efe9c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:15.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.993+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c01d070 con 0x7efeac0fd250 2022-03-21T14:06:15.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.993+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efe9c005040 con 0x7efeac0fd250 2022-03-21T14:06:15.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.993+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c017a20 con 0x7efeac0fd250 2022-03-21T14:06:15.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.993+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c0093f0 con 0x7efeac0fd250 2022-03-21T14:06:15.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.993+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac117260 con 0x7efeac0fd250 2022-03-21T14:06:15.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.994+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac1077a0 con 0x7efeac0fd250 2022-03-21T14:06:15.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.995+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7efe9c029030 con 0x7efeac0fd250 2022-03-21T14:06:15.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.997+0000 7efea27fc700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 0x7efe980991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:15.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.997+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efe9c0c0740 con 0x7efeac0fd250 2022-03-21T14:06:15.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.997+0000 7efeb1eaa700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 0x7efe980991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:15.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:14.997+0000 7efeb1eaa700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 0x7efe980991c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efeac0fd0b0 tx=0x7efea8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:15.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.001+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7efe9c058f40 con 0x7efeac0fd250 2022-03-21T14:06:15.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.236+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeac103180 con 0x7efeac0fd250 2022-03-21T14:06:15.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.237+0000 7efea27fc700 1 -- 172.21.15.2:0/60952485 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7efe9c08deb0 con 0x7efeac0fd250 2022-03-21T14:06:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:15.243 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-03-21T14:06:15.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 msgr2=0x7efe980991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 0x7efe980991c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efeac0fd0b0 tx=0x7efea8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 msgr2=0x7efeac06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:15.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0093c0 tx=0x7efe9c005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:15.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 shutdown_connections 2022-03-21T14:06:15.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeac0fdbe0 0x7efeac06f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:15.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efe98096d10 0x7efe980991c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:15.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efeac0fbfb0 0x7efeac070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 --2- 172.21.15.2:0/60952485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efeac0fd250 0x7efeac06f320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 >> 172.21.15.2:0/60952485 conn(0x7efeac0f7760 msgr2=0x7efeac100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 shutdown_connections 2022-03-21T14:06:15.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:15.239+0000 7efeb2eac700 1 -- 172.21.15.2:0/60952485 wait complete. 2022-03-21T14:06:15.248 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 1 2022-03-21T14:06:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:15 smithi016 ceph-mon[22010]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:15 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1920885750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:15 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/60952485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:15 smithi079 ceph-mon[23158]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:15 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1920885750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:15 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/60952485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:16.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:15 smithi002 ceph-mon[17167]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:16.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:15 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1920885750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:16.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:15 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/60952485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:16.366 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:16 smithi002 systemd[1]: Starting Ceph mds.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:17.040 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:16 smithi002 bash[51447]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:17.266 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:17.292 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:17 smithi002 bash[51447]: d91edd3edd029ae3641f1e45a4ad0d18af182ac150a0c7fb009dedb061f5ec5b 2022-03-21T14:06:17.293 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:17 smithi002 ceph-mds[51660]: -- 172.21.15.2:0/4234327429 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560dc8550340 con 0x560dc8528800 2022-03-21T14:06:17.293 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:17 smithi002 conmon[51636]: starting mds.a at 2022-03-21T14:06:17.293 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 21 14:06:17 smithi002 systemd[1]: Started Ceph mds.a for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:17.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 -- 172.21.15.2:0/3825404756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fbfb0 msgr2=0x7fd3c80fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:17.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 --2- 172.21.15.2:0/3825404756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c80fc3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800b820 tx=0x7fd3b8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 -- 172.21.15.2:0/3825404756 shutdown_connections 2022-03-21T14:06:17.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 --2- 172.21.15.2:0/3825404756 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c81044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 --2- 172.21.15.2:0/3825404756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c80fc3c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 --2- 172.21.15.2:0/3825404756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 0x7fd3c80fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 -- 172.21.15.2:0/3825404756 >> 172.21.15.2:0/3825404756 conn(0x7fd3c80f7760 msgr2=0x7fd3c80f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:17.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 -- 172.21.15.2:0/3825404756 shutdown_connections 2022-03-21T14:06:17.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.478+0000 7fd3c759e700 1 -- 172.21.15.2:0/3825404756 wait complete. 2022-03-21T14:06:17.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.479+0000 7fd3c759e700 1 Processor -- start 2022-03-21T14:06:17.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 -- start start 2022-03-21T14:06:17.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:17.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 0x7fd3c810d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:17.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c8114540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:17.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8114a80 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8114bf0 con 0x7fd3c80fd250 2022-03-21T14:06:17.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c759e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8114db0 con 0x7fd3c80fbfb0 2022-03-21T14:06:17.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:17.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c6d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c8114540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:17.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39520/0 (socket says 172.21.15.2:39520) 2022-03-21T14:06:17.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c659c700 1 -- 172.21.15.2:0/346541975 learned_addr learned my addr 172.21.15.2:0/346541975 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:17.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.480+0000 7fd3c5d9b700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 0x7fd3c810d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:17.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 -- 172.21.15.2:0/346541975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 msgr2=0x7fd3c810cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:17.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 -- 172.21.15.2:0/346541975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 msgr2=0x7fd3c810d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:17.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 0x7fd3c810d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 -- 172.21.15.2:0/346541975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc0093f0 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c659c700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:17.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c6d9d700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c8114540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000eb50 tx=0x7fd3c0009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:17.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000c9d0 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd3c0003840 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.481+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b800b360 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.482+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c00044d0 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.482+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c81153f0 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.483+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c8107790 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.485+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd3c000c040 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.486+0000 7fd3b6ffd700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 0x7fd3b0099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:17.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.486+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd3c0014070 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.486+0000 7fd3c659c700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 0x7fd3b0099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:17.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.487+0000 7fd3c659c700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 0x7fd3b0099240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800be90 tx=0x7fd3b8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:17.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.490+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd3c008d040 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.730+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3c8115670 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.731+0000 7fd3b6ffd700 1 -- 172.21.15.2:0/346541975 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1254 (secure 0 0 0) 0x7fd3c0090660 con 0x7fd3c80fdbe0 2022-03-21T14:06:17.733 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:17.733 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-03-21T14:06:17.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 msgr2=0x7fd3b0099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:17.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 0x7fd3b0099240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800be90 tx=0x7fd3b8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 msgr2=0x7fd3c8114540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:17.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c8114540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000eb50 tx=0x7fd3c0009430 comp rx=0 tx=0).stop 2022-03-21T14:06:17.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 shutdown_connections 2022-03-21T14:06:17.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd3b0096d90 0x7fd3b0099240 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3c80fdbe0 0x7fd3c8114540 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c80fbfb0 0x7fd3c810cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 --2- 172.21.15.2:0/346541975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c80fd250 0x7fd3c810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:17.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.733+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 >> 172.21.15.2:0/346541975 conn(0x7fd3c80f7760 msgr2=0x7fd3c81004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:17.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.734+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 shutdown_connections 2022-03-21T14:06:17.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:17.734+0000 7fd3c759e700 1 -- 172.21.15.2:0/346541975 wait complete. 2022-03-21T14:06:17.741 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2022-03-21T14:06:18.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:18.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.2:6834/2676603000,v1:172.21.15.2:6835/2676603000] up:boot 2022-03-21T14:06:18.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: fsmap 1 up:standby 2022-03-21T14:06:18.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-03-21T14:06:18.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:18.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:18.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:18.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:18.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:18 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/346541975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.2:6834/2676603000,v1:172.21.15.2:6835/2676603000] up:boot 2022-03-21T14:06:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: fsmap 1 up:standby 2022-03-21T14:06:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-03-21T14:06:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:18 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/346541975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.2:6834/2676603000,v1:172.21.15.2:6835/2676603000] up:boot 2022-03-21T14:06:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: fsmap 1 up:standby 2022-03-21T14:06:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-03-21T14:06:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:18 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/346541975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:19 smithi002 ceph-mon[17167]: Deploying daemon mds.d on smithi002 2022-03-21T14:06:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:19 smithi016 ceph-mon[22010]: Deploying daemon mds.d on smithi002 2022-03-21T14:06:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:19 smithi079 ceph-mon[23158]: Deploying daemon mds.d on smithi002 2022-03-21T14:06:19.768 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:19.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.989+0000 7f53fffff700 1 -- 172.21.15.2:0/2286050003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fd250 msgr2=0x7f54000fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:19.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.989+0000 7f53fffff700 1 --2- 172.21.15.2:0/2286050003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fd250 0x7f54000fd6a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f53f40098e0 tx=0x7f53f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:19.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 -- 172.21.15.2:0/2286050003 shutdown_connections 2022-03-21T14:06:19.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 --2- 172.21.15.2:0/2286050003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fbfb0 0x7f54000fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:19.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 --2- 172.21.15.2:0/2286050003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 0x7f54001044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:19.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 --2- 172.21.15.2:0/2286050003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fd250 0x7f54000fd6a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:19.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 -- 172.21.15.2:0/2286050003 >> 172.21.15.2:0/2286050003 conn(0x7f54000f7760 msgr2=0x7f54000f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:19.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 -- 172.21.15.2:0/2286050003 shutdown_connections 2022-03-21T14:06:19.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.990+0000 7f53fffff700 1 -- 172.21.15.2:0/2286050003 wait complete. 2022-03-21T14:06:19.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 Processor -- start 2022-03-21T14:06:19.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 -- start start 2022-03-21T14:06:19.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:19.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fd250 0x7f540010f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:19.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 0x7f54001164c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:19.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400116a00 con 0x7f54000fdbe0 2022-03-21T14:06:19.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400116b70 con 0x7f54000fd250 2022-03-21T14:06:19.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.991+0000 7f53fffff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400116d30 con 0x7f54000fbfb0 2022-03-21T14:06:19.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:19.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53ff7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 0x7f54001164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:19.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39524/0 (socket says 172.21.15.2:39524) 2022-03-21T14:06:19.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 -- 172.21.15.2:0/3349131826 learned_addr learned my addr 172.21.15.2:0/3349131826 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:19.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53fe7fc700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fd250 0x7f540010f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:19.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 -- 172.21.15.2:0/3349131826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fd250 msgr2=0x7f540010f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:19.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fd250 0x7f540010f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 -- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 msgr2=0x7f54001164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:20.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 0x7f54001164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 -- 172.21.15.2:0/3349131826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40093f0 con 0x7f54000fbfb0 2022-03-21T14:06:20.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.992+0000 7f53feffd700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53f000f860 tx=0x7f53f0005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:20.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.993+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0004020 con 0x7f54000fbfb0 2022-03-21T14:06:20.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.993+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5400116fb0 con 0x7f54000fbfb0 2022-03-21T14:06:20.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.993+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f53f0004180 con 0x7f54000fbfb0 2022-03-21T14:06:20.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.993+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00039f0 con 0x7f54000fbfb0 2022-03-21T14:06:20.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.993+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5400117490 con 0x7f54000fbfb0 2022-03-21T14:06:20.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.994+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54001077a0 con 0x7f54000fbfb0 2022-03-21T14:06:20.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.996+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f53f0022060 con 0x7f54000fbfb0 2022-03-21T14:06:20.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.997+0000 7f53e77fe700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 0x7f53e80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:20.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.997+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f53f00bf060 con 0x7f54000fbfb0 2022-03-21T14:06:20.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.997+0000 7f53fe7fc700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 0x7f53e80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:20.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.998+0000 7f53fe7fc700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 0x7f53e80991c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009f50 tx=0x7f53f4005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:20.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:19.999+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f53f008c770 con 0x7f54000fbfb0 2022-03-21T14:06:20.080 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:19 smithi002 systemd[1]: Starting Ceph mds.d for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:20.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.252+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5400117740 con 0x7f54000fbfb0 2022-03-21T14:06:20.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.253+0000 7f53e77fe700 1 -- 172.21.15.2:0/3349131826 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1254 (secure 0 0 0) 0x7f53f008fd90 con 0x7f54000fbfb0 2022-03-21T14:06:20.254 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:20.255 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-03-21T14:06:20.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 msgr2=0x7f53e80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:20.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 0x7f53e80991c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009f50 tx=0x7f53f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:20.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 msgr2=0x7f540010eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:20.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53f000f860 tx=0x7f53f0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 shutdown_connections 2022-03-21T14:06:20.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f53e8096d10 0x7f53e80991c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54000fdbe0 0x7f54001164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f54000fbfb0 0x7f540010eef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 --2- 172.21.15.2:0/3349131826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54000fd250 0x7f540010f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:20.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 >> 172.21.15.2:0/3349131826 conn(0x7f54000f7760 msgr2=0x7f540006f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:20.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 shutdown_connections 2022-03-21T14:06:20.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:20.257+0000 7f53fffff700 1 -- 172.21.15.2:0/3349131826 wait complete. 2022-03-21T14:06:20.262 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2022-03-21T14:06:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:20 smithi002 ceph-mon[17167]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:20 smithi016 ceph-mon[22010]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:20 smithi079 ceph-mon[23158]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:20.664 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:20 smithi002 bash[52036]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:21.061 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:20 smithi002 ceph-mds[52293]: -- 172.21.15.2:0/202701836 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c3e0488340 con 0x55c3e0460800 2022-03-21T14:06:21.062 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:20 smithi002 conmon[52269]: starting mds.d at 2022-03-21T14:06:21.062 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:20 smithi002 bash[52036]: 6bc40f5ef0d7d614521ce4f8547feb173e9981c8f249f497b5196d2d95f65b7c 2022-03-21T14:06:21.062 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 21 14:06:20 smithi002 systemd[1]: Started Ceph mds.d for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3349131826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:06:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3349131826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3349131826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:06:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:06:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:22.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:22.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:22 smithi002 ceph-mon[17167]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:22.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:22 smithi002 ceph-mon[17167]: Deploying daemon mds.g on smithi002 2022-03-21T14:06:22.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:22 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.2:6836/595788412,v1:172.21.15.2:6837/595788412] up:boot 2022-03-21T14:06:22.340 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:22 smithi002 ceph-mon[17167]: fsmap 2 up:standby 2022-03-21T14:06:22.340 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-03-21T14:06:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:22 smithi016 ceph-mon[22010]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:22 smithi016 ceph-mon[22010]: Deploying daemon mds.g on smithi002 2022-03-21T14:06:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:22 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.2:6836/595788412,v1:172.21.15.2:6837/595788412] up:boot 2022-03-21T14:06:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:22 smithi016 ceph-mon[22010]: fsmap 2 up:standby 2022-03-21T14:06:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-03-21T14:06:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:22 smithi079 ceph-mon[23158]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:22 smithi079 ceph-mon[23158]: Deploying daemon mds.g on smithi002 2022-03-21T14:06:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:22 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.2:6836/595788412,v1:172.21.15.2:6837/595788412] up:boot 2022-03-21T14:06:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:22 smithi079 ceph-mon[23158]: fsmap 2 up:standby 2022-03-21T14:06:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-03-21T14:06:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 -- 172.21.15.2:0/3498264812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79000ff130 msgr2=0x7f7900101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 --2- 172.21.15.2:0/3498264812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79000ff130 0x7f7900101570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f78f00098e0 tx=0x7f78f0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 -- 172.21.15.2:0/3498264812 shutdown_connections 2022-03-21T14:06:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 --2- 172.21.15.2:0/3498264812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900065180 0x7f79000febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 --2- 172.21.15.2:0/3498264812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7900101c00 0x7f7900104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 --2- 172.21.15.2:0/3498264812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79000ff130 0x7f7900101570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 -- 172.21.15.2:0/3498264812 >> 172.21.15.2:0/3498264812 conn(0x7f79000f6640 msgr2=0x7f79000f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 -- 172.21.15.2:0/3498264812 shutdown_connections 2022-03-21T14:06:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.526+0000 7f79070e5700 1 -- 172.21.15.2:0/3498264812 wait complete. 2022-03-21T14:06:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 Processor -- start 2022-03-21T14:06:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 -- start start 2022-03-21T14:06:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79000ff130 0x7f790010f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900101c00 0x7f7900116680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900116bc0 con 0x7f79000ff130 2022-03-21T14:06:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900116d30 con 0x7f7900101c00 2022-03-21T14:06:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.527+0000 7f79070e5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900116ef0 con 0x7f7900065180 2022-03-21T14:06:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79058e2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79000ff130 0x7f790010f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39532/0 (socket says 172.21.15.2:39532) 2022-03-21T14:06:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 -- 172.21.15.2:0/2277894989 learned_addr learned my addr 172.21.15.2:0/2277894989 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79068e4700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900101c00 0x7f7900116680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 -- 172.21.15.2:0/2277894989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900101c00 msgr2=0x7f7900116680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900101c00 0x7f7900116680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 -- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79000ff130 msgr2=0x7f790010f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79000ff130 0x7f790010f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.528+0000 7f79060e3700 1 -- 172.21.15.2:0/2277894989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f00093f0 con 0x7f7900065180 2022-03-21T14:06:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f79060e3700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00fa90 tx=0x7f78fc005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00b4f0 con 0x7f7900065180 2022-03-21T14:06:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f78fc00dc80 con 0x7f7900065180 2022-03-21T14:06:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79001171d0 con 0x7f7900065180 2022-03-21T14:06:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc011dc0 con 0x7f7900065180 2022-03-21T14:06:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.529+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7900117710 con 0x7f7900065180 2022-03-21T14:06:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.530+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7900107760 con 0x7f7900065180 2022-03-21T14:06:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.532+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f78fc00c040 con 0x7f7900065180 2022-03-21T14:06:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.533+0000 7f78f6ffd700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 0x7f78ec099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.533+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f78fc0bf340 con 0x7f7900065180 2022-03-21T14:06:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.534+0000 7f79058e2700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 0x7f78ec099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.534+0000 7f79058e2700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 0x7f78ec099240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78f000f8e0 tx=0x7f78f0005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.535+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f78fc08cab0 con 0x7f7900065180 2022-03-21T14:06:22.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.769+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7900117990 con 0x7f7900065180 2022-03-21T14:06:22.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.770+0000 7f78f6ffd700 1 -- 172.21.15.2:0/2277894989 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2005 (secure 0 0 0) 0x7f78fc0900d0 con 0x7f7900065180 2022-03-21T14:06:22.771 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:22.772 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-03-21T14:06:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 msgr2=0x7f78ec099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 0x7f78ec099240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78f000f8e0 tx=0x7f78f0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 msgr2=0x7f790010f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:22.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00fa90 tx=0x7f78fc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 shutdown_connections 2022-03-21T14:06:22.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900101c00 0x7f7900116680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f78ec096d90 0x7f78ec099240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7900065180 0x7f790010f0b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 --2- 172.21.15.2:0/2277894989 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79000ff130 0x7f790010f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 >> 172.21.15.2:0/2277894989 conn(0x7f79000f6640 msgr2=0x7f79000fa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 shutdown_connections 2022-03-21T14:06:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:22.773+0000 7f79070e5700 1 -- 172.21.15.2:0/2277894989 wait complete. 2022-03-21T14:06:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2022-03-21T14:06:23.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:23 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2277894989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:23 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2277894989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:23 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2277894989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:24.093 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:23 smithi002 systemd[1]: Starting Ceph mds.g for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:24 smithi002 ceph-mon[17167]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:24.366 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:24 smithi002 bash[52665]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:24 smithi016 ceph-mon[22010]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:24 smithi079 ceph-mon[23158]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:24.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:24.922 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:24 smithi002 ceph-mds[52882]: -- 172.21.15.2:0/448272557 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556dcef2c340 con 0x556dcef04c00 2022-03-21T14:06:24.923 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:24 smithi002 conmon[52857]: starting mds.g at 2022-03-21T14:06:24.923 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:24 smithi002 bash[52665]: 23ee8a62fac3328fbbf7f37f71a82241d56339093e563200492948639a63e23d 2022-03-21T14:06:24.923 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 21 14:06:24 smithi002 systemd[1]: Started Ceph mds.g for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:25.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 -- 172.21.15.2:0/1899660998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 msgr2=0x7fec3406f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:25.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 --2- 172.21.15.2:0/1899660998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3406f360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec300098e0 tx=0x7fec30005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:25.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 -- 172.21.15.2:0/1899660998 shutdown_connections 2022-03-21T14:06:25.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 --2- 172.21.15.2:0/1899660998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3406f360 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 --2- 172.21.15.2:0/1899660998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 0x7fec34106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 --2- 172.21.15.2:0/1899660998 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 0x7fec3406fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 -- 172.21.15.2:0/1899660998 >> 172.21.15.2:0/1899660998 conn(0x7fec340f7760 msgr2=0x7fec340f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:25.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 -- 172.21.15.2:0/1899660998 shutdown_connections 2022-03-21T14:06:25.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.014+0000 7fec3a9de700 1 -- 172.21.15.2:0/1899660998 wait complete. 2022-03-21T14:06:25.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 Processor -- start 2022-03-21T14:06:25.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 -- start start 2022-03-21T14:06:25.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 0x7fec3410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:25.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:25.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 0x7fec34116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:25.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.015+0000 7fec3a9de700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34116a60 con 0x7fec34103f40 2022-03-21T14:06:25.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec3a9de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34116bd0 con 0x7fec34070e60 2022-03-21T14:06:25.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec3a9de700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34116d90 con 0x7fec3406f8a0 2022-03-21T14:06:25.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:25.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec3a1dd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 0x7fec34116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:25.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48268/0 (socket says 172.21.15.2:48268) 2022-03-21T14:06:25.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 -- 172.21.15.2:0/2503732120 learned_addr learned my addr 172.21.15.2:0/2503732120 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:25.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec399dc700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 0x7fec3410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:25.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 -- 172.21.15.2:0/2503732120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 msgr2=0x7fec3410ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:25.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 0x7fec3410ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 -- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 msgr2=0x7fec34116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:25.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 0x7fec34116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.016+0000 7fec391db700 1 -- 172.21.15.2:0/2503732120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec300093f0 con 0x7fec34070e60 2022-03-21T14:06:25.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec391db700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec2400fa90 tx=0x7fec24005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:25.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2400b4f0 con 0x7fec34070e60 2022-03-21T14:06:25.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fec2400dc80 con 0x7fec34070e60 2022-03-21T14:06:25.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec34117070 con 0x7fec34070e60 2022-03-21T14:06:25.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24011dc0 con 0x7fec34070e60 2022-03-21T14:06:25.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.017+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec341175e0 con 0x7fec34070e60 2022-03-21T14:06:25.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.018+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec34109a50 con 0x7fec34070e60 2022-03-21T14:06:25.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.020+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fec2400c040 con 0x7fec34070e60 2022-03-21T14:06:25.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.021+0000 7fec2a7fc700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 0x7fec200991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:25.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.021+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fec240bf4d0 con 0x7fec34070e60 2022-03-21T14:06:25.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.021+0000 7fec399dc700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 0x7fec200991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:25.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.022+0000 7fec399dc700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 0x7fec200991c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fec30009f50 tx=0x7fec30005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:25.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.024+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fec2408cb50 con 0x7fec34070e60 2022-03-21T14:06:25.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.265+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec34109be0 con 0x7fec34070e60 2022-03-21T14:06:25.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.266+0000 7fec2a7fc700 1 -- 172.21.15.2:0/2503732120 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2759 (secure 0 0 0) 0x7fec34109be0 con 0x7fec34070e60 2022-03-21T14:06:25.268 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:25.269 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-03-21T14:06:25.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 msgr2=0x7fec200991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:25.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 0x7fec200991c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fec30009f50 tx=0x7fec30005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:25.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 msgr2=0x7fec3410f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:25.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec2400fa90 tx=0x7fec24005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 shutdown_connections 2022-03-21T14:06:25.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fec20096d10 0x7fec200991c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec34103f40 0x7fec34116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec3406f8a0 0x7fec3410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 --2- 172.21.15.2:0/2503732120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec34070e60 0x7fec3410f490 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:25.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 >> 172.21.15.2:0/2503732120 conn(0x7fec340f7760 msgr2=0x7fec340f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:25.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 shutdown_connections 2022-03-21T14:06:25.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:25.269+0000 7fec3a9de700 1 -- 172.21.15.2:0/2503732120 wait complete. 2022-03-21T14:06:25.274 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2022-03-21T14:06:26.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:26.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:26.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:26.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:26.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:26.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: Deploying daemon mds.j on smithi002 2022-03-21T14:06:26.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.2:6838/2330341423,v1:172.21.15.2:6839/2330341423] up:boot 2022-03-21T14:06:26.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: fsmap 3 up:standby 2022-03-21T14:06:26.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-03-21T14:06:26.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:25 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2503732120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: Deploying daemon mds.j on smithi002 2022-03-21T14:06:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.2:6838/2330341423,v1:172.21.15.2:6839/2330341423] up:boot 2022-03-21T14:06:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: fsmap 3 up:standby 2022-03-21T14:06:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-03-21T14:06:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:25 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2503732120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: Deploying daemon mds.j on smithi002 2022-03-21T14:06:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.2:6838/2330341423,v1:172.21.15.2:6839/2330341423] up:boot 2022-03-21T14:06:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: fsmap 3 up:standby 2022-03-21T14:06:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-03-21T14:06:26.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:25 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2503732120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:27.297 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:27.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.520+0000 7f34b3fff700 1 -- 172.21.15.2:0/1805867503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fd250 msgr2=0x7f34b40fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:27.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.520+0000 7f34b3fff700 1 --2- 172.21.15.2:0/1805867503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fd250 0x7f34b40fd6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34a80098e0 tx=0x7f34a8005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:27.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 -- 172.21.15.2:0/1805867503 shutdown_connections 2022-03-21T14:06:27.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 --2- 172.21.15.2:0/1805867503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fdbe0 0x7f34b41044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 --2- 172.21.15.2:0/1805867503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fbfb0 0x7f34b40fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 --2- 172.21.15.2:0/1805867503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fd250 0x7f34b40fd6a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 -- 172.21.15.2:0/1805867503 >> 172.21.15.2:0/1805867503 conn(0x7f34b40f7760 msgr2=0x7f34b40f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:27.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 -- 172.21.15.2:0/1805867503 shutdown_connections 2022-03-21T14:06:27.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.521+0000 7f34b3fff700 1 -- 172.21.15.2:0/1805867503 wait complete. 2022-03-21T14:06:27.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 Processor -- start 2022-03-21T14:06:27.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 -- start start 2022-03-21T14:06:27.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 0x7f34b410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:27.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 0x7f34b410d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:27.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 0x7f34b4114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:27.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4114890 con 0x7f34b40fdbe0 2022-03-21T14:06:27.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4114a00 con 0x7f34b40fbfb0 2022-03-21T14:06:27.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b3fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4114bc0 con 0x7f34b40fd250 2022-03-21T14:06:27.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.522+0000 7f34b27fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 0x7f34b410d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:27.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 0x7f34b4114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:27.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b27fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 0x7f34b410d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39546/0 (socket says 172.21.15.2:39546) 2022-03-21T14:06:27.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 0x7f34b410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:27.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b27fc700 1 -- 172.21.15.2:0/288184994 learned_addr learned my addr 172.21.15.2:0/288184994 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:27.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 -- 172.21.15.2:0/288184994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 msgr2=0x7f34b410d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 0x7f34b410d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 -- 172.21.15.2:0/288184994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 msgr2=0x7f34b410cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 0x7f34b410cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 -- 172.21.15.2:0/288184994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac0096b0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b2ffd700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 0x7f34b410cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:27.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.523+0000 7f34b37fe700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 0x7f34b4114310 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00eca0 tx=0x7f34ac009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:27.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.524+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00caa0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.524+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a80093f0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.524+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f34ac00cc00 con 0x7f34b40fdbe0 2022-03-21T14:06:27.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.524+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac004020 con 0x7f34b40fdbe0 2022-03-21T14:06:27.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.524+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b41151b0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.525+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b41077a0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.527+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f34ac01e020 con 0x7f34b40fdbe0 2022-03-21T14:06:27.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.529+0000 7f349b7fe700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 0x7f349c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:27.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.529+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f34ac014070 con 0x7f34b40fdbe0 2022-03-21T14:06:27.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.529+0000 7f34b2ffd700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 0x7f349c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:27.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.530+0000 7f34b2ffd700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 0x7f349c0991c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34b410e270 tx=0x7f34a4005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:27.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.534+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f34ac08d210 con 0x7f34b40fdbe0 2022-03-21T14:06:27.615 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:27 smithi002 systemd[1]: Starting Ceph mds.j for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:27.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.788+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b402abc0 con 0x7f34b40fdbe0 2022-03-21T14:06:27.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.789+0000 7f349b7fe700 1 -- 172.21.15.2:0/288184994 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2759 (secure 0 0 0) 0x7f34ac090830 con 0x7f34b40fdbe0 2022-03-21T14:06:27.791 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:27.792 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-03-21T14:06:27.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 msgr2=0x7f349c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 0x7f349c0991c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34b410e270 tx=0x7f34a4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 msgr2=0x7f34b4114310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 0x7f34b4114310 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00eca0 tx=0x7f34ac009430 comp rx=0 tx=0).stop 2022-03-21T14:06:27.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 shutdown_connections 2022-03-21T14:06:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34b40fbfb0 0x7f34b410cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f349c096d10 0x7f349c0991c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34b40fdbe0 0x7f34b4114310 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 --2- 172.21.15.2:0/288184994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34b40fd250 0x7f34b410d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:27.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.793+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 >> 172.21.15.2:0/288184994 conn(0x7f34b40f7760 msgr2=0x7f34b4100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:27.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.794+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 shutdown_connections 2022-03-21T14:06:27.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:27.794+0000 7f34b3fff700 1 -- 172.21.15.2:0/288184994 wait complete. 2022-03-21T14:06:27.798 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2022-03-21T14:06:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:27 smithi002 ceph-mon[17167]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:27 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/288184994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:28.116 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:27 smithi002 bash[53270]: Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:06:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:27 smithi016 ceph-mon[22010]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:27 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/288184994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:27 smithi079 ceph-mon[23158]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:27 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/288184994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:28.865 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:28 smithi002 ceph-mds[53508]: -- 172.21.15.2:0/2968993770 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d9ffda340 con 0x556d9ffb3000 2022-03-21T14:06:28.866 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:28 smithi002 conmon[53483]: starting mds.j at 2022-03-21T14:06:28.866 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:28 smithi002 bash[53270]: d4ba25ec22affc96ab2407a1aef845257fd9263a9e25a9e00b2b8413d3976725 2022-03-21T14:06:28.866 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 21 14:06:28 smithi002 systemd[1]: Started Ceph mds.j for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:29.825 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: Deploying daemon mds.b on smithi016 2022-03-21T14:06:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.2:6840/1904085966,v1:172.21.15.2:6841/1904085966] up:boot 2022-03-21T14:06:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: fsmap 4 up:standby 2022-03-21T14:06:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-03-21T14:06:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: Deploying daemon mds.b on smithi016 2022-03-21T14:06:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.2:6840/1904085966,v1:172.21.15.2:6841/1904085966] up:boot 2022-03-21T14:06:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: fsmap 4 up:standby 2022-03-21T14:06:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-03-21T14:06:29.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: Deploying daemon mds.b on smithi016 2022-03-21T14:06:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.2:6840/1904085966,v1:172.21.15.2:6841/1904085966] up:boot 2022-03-21T14:06:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: fsmap 4 up:standby 2022-03-21T14:06:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-03-21T14:06:30.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 -- 172.21.15.2:0/914587662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266006f8a0 msgr2=0x7f266006fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:30.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 --2- 172.21.15.2:0/914587662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266006f8a0 0x7f266006fd10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f26540098e0 tx=0x7f2654005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:30.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 -- 172.21.15.2:0/914587662 shutdown_connections 2022-03-21T14:06:30.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 --2- 172.21.15.2:0/914587662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2660070e60 0x7f266006f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 --2- 172.21.15.2:0/914587662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660103f40 0x7f2660106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 --2- 172.21.15.2:0/914587662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266006f8a0 0x7f266006fd10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 -- 172.21.15.2:0/914587662 >> 172.21.15.2:0/914587662 conn(0x7f26600f7760 msgr2=0x7f26600f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 -- 172.21.15.2:0/914587662 shutdown_connections 2022-03-21T14:06:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.042+0000 7f2664b06700 1 -- 172.21.15.2:0/914587662 wait complete. 2022-03-21T14:06:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 Processor -- start 2022-03-21T14:06:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 -- start start 2022-03-21T14:06:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 0x7f266010f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2660103f40 0x7f2660116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660116a40 con 0x7f2660103f40 2022-03-21T14:06:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660116bb0 con 0x7f266006f8a0 2022-03-21T14:06:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.043+0000 7f2664b06700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660116d70 con 0x7f2660070e60 2022-03-21T14:06:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2660103f40 0x7f2660116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:30.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48282/0 (socket says 172.21.15.2:48282) 2022-03-21T14:06:30.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 -- 172.21.15.2:0/1843381404 learned_addr learned my addr 172.21.15.2:0/1843381404 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:30.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265effd700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 0x7f266010f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:30.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 -- 172.21.15.2:0/1843381404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 msgr2=0x7f266010f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 0x7f266010f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 -- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2660103f40 msgr2=0x7f2660116500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2660103f40 0x7f2660116500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 -- 172.21.15.2:0/1843381404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265000b360 con 0x7f266006f8a0 2022-03-21T14:06:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265effd700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 0x7f266010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.044+0000 7f265f7fe700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f265000ead0 tx=0x7f2650005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.045+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265000b6e0 con 0x7f266006f8a0 2022-03-21T14:06:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.045+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f265000c9d0 con 0x7f266006f8a0 2022-03-21T14:06:30.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.045+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26540093f0 con 0x7f266006f8a0 2022-03-21T14:06:30.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.045+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650003c00 con 0x7f266006f8a0 2022-03-21T14:06:30.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.045+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660117340 con 0x7f266006f8a0 2022-03-21T14:06:30.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.046+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660109a50 con 0x7f266006f8a0 2022-03-21T14:06:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.048+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f265000c040 con 0x7f266006f8a0 2022-03-21T14:06:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.049+0000 7f2647fff700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 0x7f26480991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.049+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f2650014070 con 0x7f266006f8a0 2022-03-21T14:06:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.049+0000 7f265effd700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 0x7f26480991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.050+0000 7f265effd700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 0x7f26480991c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2654009f50 tx=0x7f265401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.052+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f265008cf40 con 0x7f266006f8a0 2022-03-21T14:06:30.280 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.278+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2660102510 con 0x7f266006f8a0 2022-03-21T14:06:30.281 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.279+0000 7f2647fff700 1 -- 172.21.15.2:0/1843381404 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3513 (secure 0 0 0) 0x7f2650090560 con 0x7f266006f8a0 2022-03-21T14:06:30.282 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:30.282 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-03-21T14:06:30.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 msgr2=0x7f26480991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:30.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 0x7f26480991c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2654009f50 tx=0x7f265401a040 comp rx=0 tx=0).stop 2022-03-21T14:06:30.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 msgr2=0x7f266010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f265000ead0 tx=0x7f2650005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 shutdown_connections 2022-03-21T14:06:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f266006f8a0 0x7f266010ef30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2648096d10 0x7f26480991c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2660103f40 0x7f2660116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 --2- 172.21.15.2:0/1843381404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2660070e60 0x7f266010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:30.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 >> 172.21.15.2:0/1843381404 conn(0x7f26600f7760 msgr2=0x7f26600f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:30.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 shutdown_connections 2022-03-21T14:06:30.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:30.282+0000 7f2664b06700 1 -- 172.21.15.2:0/1843381404 wait complete. 2022-03-21T14:06:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 5 2022-03-21T14:06:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:30 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1843381404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:30.664 INFO:journalctl@ceph.mds.b.smithi016.stdout:Mar 21 14:06:30 smithi016 systemd[1]: Starting Ceph mds.b for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:30 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1843381404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:30 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1843381404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:31.414 INFO:journalctl@ceph.mds.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mds[46797]: -- 172.21.15.16:0/2112893708 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e5d9b6a340 con 0x55e5d9b42c00 2022-03-21T14:06:31.414 INFO:journalctl@ceph.mds.b.smithi016.stdout:Mar 21 14:06:31 smithi016 conmon[46772]: starting mds.b at 2022-03-21T14:06:31.414 INFO:journalctl@ceph.mds.b.smithi016.stdout:Mar 21 14:06:31 smithi016 bash[46588]: 1162df6bc57ec5abeccf4056b3d0d57c7a639f95c5e6faeec2790701ee4c885d 2022-03-21T14:06:31.414 INFO:journalctl@ceph.mds.b.smithi016.stdout:Mar 21 14:06:31 smithi016 systemd[1]: Started Ceph mds.b for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mon[22010]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:32.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:32.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:32.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:32.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:32.140 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:31 smithi002 ceph-mon[17167]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:32.140 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:32.140 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:32.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:32.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:32.141 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:31 smithi079 ceph-mon[23158]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:32.141 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:32.142 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:32.142 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:32.142 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:32.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:32.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.523+0000 7f257ecdd700 1 -- 172.21.15.2:0/920443082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fd250 msgr2=0x7f25780fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:32.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.523+0000 7f257ecdd700 1 --2- 172.21.15.2:0/920443082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fd250 0x7f25780fd6a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25680098e0 tx=0x7f2568005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:32.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 -- 172.21.15.2:0/920443082 shutdown_connections 2022-03-21T14:06:32.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 --2- 172.21.15.2:0/920443082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 0x7f25781044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 --2- 172.21.15.2:0/920443082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fbfb0 0x7f25780fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 --2- 172.21.15.2:0/920443082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fd250 0x7f25780fd6a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 -- 172.21.15.2:0/920443082 >> 172.21.15.2:0/920443082 conn(0x7f25780f77a0 msgr2=0x7f25780f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:32.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 -- 172.21.15.2:0/920443082 shutdown_connections 2022-03-21T14:06:32.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.524+0000 7f257ecdd700 1 -- 172.21.15.2:0/920443082 wait complete. 2022-03-21T14:06:32.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 Processor -- start 2022-03-21T14:06:32.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 -- start start 2022-03-21T14:06:32.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:32.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 0x7f257810f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:32.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 0x7f2578116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:32.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578116a60 con 0x7f25780fdbe0 2022-03-21T14:06:32.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578116bd0 con 0x7f25780fbfb0 2022-03-21T14:06:32.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257ecdd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578116d90 con 0x7f25780fd250 2022-03-21T14:06:32.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257dcdb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:32.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257dcdb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48286/0 (socket says 172.21.15.2:48286) 2022-03-21T14:06:32.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257dcdb700 1 -- 172.21.15.2:0/3856191980 learned_addr learned my addr 172.21.15.2:0/3856191980 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:32.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.525+0000 7f257d4da700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 0x7f257810f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:32.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257e4dc700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 0x7f2578116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:32.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 -- 172.21.15.2:0/3856191980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 msgr2=0x7f257810f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:32.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 0x7f257810f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 -- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 msgr2=0x7f2578116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:32.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 0x7f2578116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 -- 172.21.15.2:0/3856191980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25680093f0 con 0x7f25780fbfb0 2022-03-21T14:06:32.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257d4da700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 0x7f257810f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:32.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257dcdb700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f257400f860 tx=0x7f2574005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:32.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574004020 con 0x7f25780fbfb0 2022-03-21T14:06:32.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2574004180 con 0x7f25780fbfb0 2022-03-21T14:06:32.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2578117070 con 0x7f25780fbfb0 2022-03-21T14:06:32.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.526+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740039f0 con 0x7f25780fbfb0 2022-03-21T14:06:32.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.527+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2578117630 con 0x7f25780fbfb0 2022-03-21T14:06:32.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.527+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25781077a0 con 0x7f25780fbfb0 2022-03-21T14:06:32.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.529+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2574022060 con 0x7f25780fbfb0 2022-03-21T14:06:32.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.531+0000 7f256e7fc700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 0x7f2564099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:32.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.531+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f25740bf2d0 con 0x7f25780fbfb0 2022-03-21T14:06:32.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.531+0000 7f257d4da700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 0x7f2564099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:32.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.532+0000 7f257d4da700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 0x7f2564099240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2568009f50 tx=0x7f256801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:32.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.533+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f257408c950 con 0x7f25780fbfb0 2022-03-21T14:06:32.753 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:32 smithi016 ceph-mon[22010]: Deploying daemon mds.e on smithi016 2022-03-21T14:06:32.753 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:32 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.16:6832/3142985186,v1:172.21.15.16:6833/3142985186] up:boot 2022-03-21T14:06:32.754 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:32 smithi016 ceph-mon[22010]: fsmap 5 up:standby 2022-03-21T14:06:32.754 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-03-21T14:06:32.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.767+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25781178b0 con 0x7f25780fbfb0 2022-03-21T14:06:32.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.767+0000 7f256e7fc700 1 -- 172.21.15.2:0/3856191980 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4270 (secure 0 0 0) 0x7f257408ff70 con 0x7f25780fbfb0 2022-03-21T14:06:32.769 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:32.770 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2022-03-21T14:06:32.772 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 msgr2=0x7f2564099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:32.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 0x7f2564099240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2568009f50 tx=0x7f256801a040 comp rx=0 tx=0).stop 2022-03-21T14:06:32.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 msgr2=0x7f257810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:32.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f257400f860 tx=0x7f2574005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 shutdown_connections 2022-03-21T14:06:32.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25780fbfb0 0x7f257810efa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2564096d90 0x7f2564099240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25780fdbe0 0x7f2578116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 --2- 172.21.15.2:0/3856191980 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25780fd250 0x7f257810f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:32.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.770+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 >> 172.21.15.2:0/3856191980 conn(0x7f25780f77a0 msgr2=0x7f257806f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:32.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.771+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 shutdown_connections 2022-03-21T14:06:32.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:32.771+0000 7f257ecdd700 1 -- 172.21.15.2:0/3856191980 wait complete. 2022-03-21T14:06:32.776 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 6 2022-03-21T14:06:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:32 smithi079 ceph-mon[23158]: Deploying daemon mds.e on smithi016 2022-03-21T14:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:32 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.16:6832/3142985186,v1:172.21.15.16:6833/3142985186] up:boot 2022-03-21T14:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:32 smithi079 ceph-mon[23158]: fsmap 5 up:standby 2022-03-21T14:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-03-21T14:06:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:32 smithi002 ceph-mon[17167]: Deploying daemon mds.e on smithi016 2022-03-21T14:06:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:32 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.16:6832/3142985186,v1:172.21.15.16:6833/3142985186] up:boot 2022-03-21T14:06:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:32 smithi002 ceph-mon[17167]: fsmap 5 up:standby 2022-03-21T14:06:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-03-21T14:06:33.163 INFO:journalctl@ceph.mds.e.smithi016.stdout:Mar 21 14:06:32 smithi016 systemd[1]: Starting Ceph mds.e for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:33 smithi016 ceph-mon[22010]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:33 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3856191980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:33.914 INFO:journalctl@ceph.mds.e.smithi016.stdout:Mar 21 14:06:33 smithi016 ceph-mds[47340]: -- 172.21.15.16:0/3256191948 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b79a294340 con 0x55b79a26cc00 2022-03-21T14:06:33.915 INFO:journalctl@ceph.mds.e.smithi016.stdout:Mar 21 14:06:33 smithi016 conmon[47316]: starting mds.e at 2022-03-21T14:06:33.915 INFO:journalctl@ceph.mds.e.smithi016.stdout:Mar 21 14:06:33 smithi016 bash[47127]: 71cfba31687577858d55b0dd5dfaee2e0ba56530d251f6f072dcffb137d262ef 2022-03-21T14:06:33.915 INFO:journalctl@ceph.mds.e.smithi016.stdout:Mar 21 14:06:33 smithi016 systemd[1]: Started Ceph mds.e for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:33 smithi079 ceph-mon[23158]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:33 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3856191980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:33 smithi002 ceph-mon[17167]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:33 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3856191980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:34.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: Deploying daemon mds.h on smithi016 2022-03-21T14:06:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.16:6834/2493160775,v1:172.21.15.16:6835/2493160775] up:boot 2022-03-21T14:06:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: fsmap 6 up:standby 2022-03-21T14:06:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-03-21T14:06:35.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.013+0000 7fd8053b8700 1 -- 172.21.15.2:0/289108592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fdbe0 msgr2=0x7fd8001044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:35.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.013+0000 7fd8053b8700 1 --2- 172.21.15.2:0/289108592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fdbe0 0x7fd8001044a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009ba0 tx=0x7fd7f8009430 comp rx=0 tx=0).stop 2022-03-21T14:06:35.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 -- 172.21.15.2:0/289108592 shutdown_connections 2022-03-21T14:06:35.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 --2- 172.21.15.2:0/289108592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fdbe0 0x7fd8001044a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 --2- 172.21.15.2:0/289108592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fbfb0 0x7fd8000fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 --2- 172.21.15.2:0/289108592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fd250 0x7fd8000fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 -- 172.21.15.2:0/289108592 >> 172.21.15.2:0/289108592 conn(0x7fd8000f7760 msgr2=0x7fd8000f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 -- 172.21.15.2:0/289108592 shutdown_connections 2022-03-21T14:06:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 -- 172.21.15.2:0/289108592 wait complete. 2022-03-21T14:06:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.014+0000 7fd8053b8700 1 Processor -- start 2022-03-21T14:06:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 -- start start 2022-03-21T14:06:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 0x7fd80010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fdbe0 0x7fd8001163a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001168e0 con 0x7fd8000fbfb0 2022-03-21T14:06:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800116a50 con 0x7fd8000fdbe0 2022-03-21T14:06:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd8053b8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800116c10 con 0x7fd8000fd250 2022-03-21T14:06:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd7fffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 0x7fd80010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd7ff7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd7fffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 0x7fd80010edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43706/0 (socket says 172.21.15.2:43706) 2022-03-21T14:06:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd7fffff700 1 -- 172.21.15.2:0/2258953263 learned_addr learned my addr 172.21.15.2:0/2258953263 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd7ff7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39562/0 (socket says 172.21.15.2:39562) 2022-03-21T14:06:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.015+0000 7fd804bb7700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fdbe0 0x7fd8001163a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 -- 172.21.15.2:0/2258953263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fdbe0 msgr2=0x7fd8001163a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fdbe0 0x7fd8001163a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 -- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 msgr2=0x7fd80010edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 0x7fd80010edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 -- 172.21.15.2:0/2258953263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f40093f0 con 0x7fd8000fd250 2022-03-21T14:06:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7ff7fe700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400ea00 tx=0x7fd7f4005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f40096d0 con 0x7fd8000fd250 2022-03-21T14:06:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd7f400c820 con 0x7fd8000fd250 2022-03-21T14:06:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f80096b0 con 0x7fd8000fd250 2022-03-21T14:06:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4003b30 con 0x7fd8000fd250 2022-03-21T14:06:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.016+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001171f0 con 0x7fd8000fd250 2022-03-21T14:06:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.017+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800107760 con 0x7fd8000fd250 2022-03-21T14:06:35.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.019+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd7f400c040 con 0x7fd8000fd250 2022-03-21T14:06:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.020+0000 7fd7fcff9700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 0x7fd7e80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.020+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd7f4014070 con 0x7fd8000fd250 2022-03-21T14:06:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.021+0000 7fd7fffff700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 0x7fd7e80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:35.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.021+0000 7fd7fffff700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 0x7fd7e80991c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000bdd0 tx=0x7fd7f0005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:35.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.024+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd7f408cf00 con 0x7fd8000fd250 2022-03-21T14:06:35.068 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:35.068 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:35.068 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:35.068 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:35.069 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: Deploying daemon mds.h on smithi016 2022-03-21T14:06:35.069 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.16:6834/2493160775,v1:172.21.15.16:6835/2493160775] up:boot 2022-03-21T14:06:35.069 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: fsmap 6 up:standby 2022-03-21T14:06:35.069 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-03-21T14:06:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: Deploying daemon mds.h on smithi016 2022-03-21T14:06:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.16:6834/2493160775,v1:172.21.15.16:6835/2493160775] up:boot 2022-03-21T14:06:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: fsmap 6 up:standby 2022-03-21T14:06:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-03-21T14:06:35.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.255+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd800117440 con 0x7fd8000fd250 2022-03-21T14:06:35.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.256+0000 7fd7fcff9700 1 -- 172.21.15.2:0/2258953263 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5027 (secure 0 0 0) 0x7fd7f4090520 con 0x7fd8000fd250 2022-03-21T14:06:35.257 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:35.257 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2022-03-21T14:06:35.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 msgr2=0x7fd7e80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:35.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 0x7fd7e80991c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000bdd0 tx=0x7fd7f0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 msgr2=0x7fd80010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:35.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400ea00 tx=0x7fd7f4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:35.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 shutdown_connections 2022-03-21T14:06:35.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8000fdbe0 0x7fd8001163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7e8096d10 0x7fd7e80991c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8000fbfb0 0x7fd80010edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 --2- 172.21.15.2:0/2258953263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8000fd250 0x7fd80010f310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.258+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 >> 172.21.15.2:0/2258953263 conn(0x7fd8000f7760 msgr2=0x7fd8000f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.259+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 shutdown_connections 2022-03-21T14:06:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:35.259+0000 7fd8053b8700 1 -- 172.21.15.2:0/2258953263 wait complete. 2022-03-21T14:06:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2022-03-21T14:06:35.664 INFO:journalctl@ceph.mds.h.smithi016.stdout:Mar 21 14:06:35 smithi016 systemd[1]: Starting Ceph mds.h for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:35.952 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:35 smithi016 ceph-mon[22010]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:35.952 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:35 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2258953263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:35.953 INFO:journalctl@ceph.mds.h.smithi016.stdout:Mar 21 14:06:35 smithi016 ceph-mds[47882]: -- 172.21.15.16:0/3960750162 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556ef5980340 con 0x556ef5958800 2022-03-21T14:06:35.953 INFO:journalctl@ceph.mds.h.smithi016.stdout:Mar 21 14:06:35 smithi016 conmon[47857]: starting mds.h at 2022-03-21T14:06:35.953 INFO:journalctl@ceph.mds.h.smithi016.stdout:Mar 21 14:06:35 smithi016 bash[47672]: 5949428548d3e8e459c557127991e73c41effa62d2cf31473900a6497efc8178 2022-03-21T14:06:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:35 smithi079 ceph-mon[23158]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:35 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2258953263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:35 smithi002 ceph-mon[17167]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:36.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:35 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2258953263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:36.413 INFO:journalctl@ceph.mds.h.smithi016.stdout:Mar 21 14:06:35 smithi016 systemd[1]: Started Ceph mds.h for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:37.285 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:37.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:37.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:37.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:37.349 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:37.349 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: Deploying daemon mds.k on smithi016 2022-03-21T14:06:37.349 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.16:6836/2678336191,v1:172.21.15.16:6837/2678336191] up:boot 2022-03-21T14:06:37.349 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: fsmap 7 up:standby 2022-03-21T14:06:37.350 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-03-21T14:06:37.352 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:37.352 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:37.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:37.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:37.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: Deploying daemon mds.k on smithi016 2022-03-21T14:06:37.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.16:6836/2678336191,v1:172.21.15.16:6837/2678336191] up:boot 2022-03-21T14:06:37.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: fsmap 7 up:standby 2022-03-21T14:06:37.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-03-21T14:06:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: Deploying daemon mds.k on smithi016 2022-03-21T14:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.16:6836/2678336191,v1:172.21.15.16:6837/2678336191] up:boot 2022-03-21T14:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: fsmap 7 up:standby 2022-03-21T14:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-03-21T14:06:37.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 -- 172.21.15.2:0/2189932035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 msgr2=0x7f9e1c0feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:37.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 --2- 172.21.15.2:0/2189932035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0feb70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e180098e0 tx=0x7f9e18005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:37.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 -- 172.21.15.2:0/2189932035 shutdown_connections 2022-03-21T14:06:37.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 --2- 172.21.15.2:0/2189932035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 --2- 172.21.15.2:0/2189932035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0feb70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 --2- 172.21.15.2:0/2189932035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.499+0000 7f9e23089700 1 -- 172.21.15.2:0/2189932035 >> 172.21.15.2:0/2189932035 conn(0x7f9e1c0f6640 msgr2=0x7f9e1c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:37.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.500+0000 7f9e23089700 1 -- 172.21.15.2:0/2189932035 shutdown_connections 2022-03-21T14:06:37.503 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.500+0000 7f9e23089700 1 -- 172.21.15.2:0/2189932035 wait complete. 2022-03-21T14:06:37.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.500+0000 7f9e23089700 1 Processor -- start 2022-03-21T14:06:37.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 -- start start 2022-03-21T14:06:37.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0fc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:37.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c0faa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:37.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:37.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0fb750 con 0x7f9e1c0ff140 2022-03-21T14:06:37.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0fb8c0 con 0x7f9e1c101c10 2022-03-21T14:06:37.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e23089700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c1170d0 con 0x7f9e1c065200 2022-03-21T14:06:37.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e21886700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c0faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:37.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e22888700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:37.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e22888700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48298/0 (socket says 172.21.15.2:48298) 2022-03-21T14:06:37.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e22888700 1 -- 172.21.15.2:0/3440617836 learned_addr learned my addr 172.21.15.2:0/3440617836 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:37.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.501+0000 7f9e22087700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:37.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 -- 172.21.15.2:0/3440617836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 msgr2=0x7f9e1c0fc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:37.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0fc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 -- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 msgr2=0x7f9e1c0faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:37.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c0faa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 -- 172.21.15.2:0/3440617836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e180093f0 con 0x7f9e1c101c10 2022-03-21T14:06:37.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e21886700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c0faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:37.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e22888700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400bc30 tx=0x7f9e14009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:37.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e140092e0 con 0x7f9e1c101c10 2022-03-21T14:06:37.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c117300 con 0x7f9e1c101c10 2022-03-21T14:06:37.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9e1400ca80 con 0x7f9e1c101c10 2022-03-21T14:06:37.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14007b60 con 0x7f9e1c101c10 2022-03-21T14:06:37.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.502+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c1177a0 con 0x7f9e1c101c10 2022-03-21T14:06:37.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.503+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c107770 con 0x7f9e1c101c10 2022-03-21T14:06:37.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.505+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f9e1400cbf0 con 0x7f9e1c101c10 2022-03-21T14:06:37.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.506+0000 7f9e12ffd700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 0x7f9e080991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:37.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.506+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f9e140bea40 con 0x7f9e1c101c10 2022-03-21T14:06:37.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.506+0000 7f9e22087700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 0x7f9e080991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:37.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.507+0000 7f9e22087700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 0x7f9e080991c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800fa00 tx=0x7f9e18005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:37.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.508+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f9e14049f40 con 0x7f9e1c101c10 2022-03-21T14:06:37.656 INFO:journalctl@ceph.mds.k.smithi016.stdout:Mar 21 14:06:37 smithi016 systemd[1]: Starting Ceph mds.k for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.739+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e1c0f7ff0 con 0x7f9e1c101c10 2022-03-21T14:06:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.740+0000 7f9e12ffd700 1 -- 172.21.15.2:0/3440617836 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5784 (secure 0 0 0) 0x7f9e1408c0c0 con 0x7f9e1c101c10 2022-03-21T14:06:37.741 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:37.741 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-03-21T14:06:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 msgr2=0x7f9e080991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 0x7f9e080991c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800fa00 tx=0x7f9e18005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 msgr2=0x7f9e1c0fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400bc30 tx=0x7f9e14009840 comp rx=0 tx=0).stop 2022-03-21T14:06:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 shutdown_connections 2022-03-21T14:06:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c101c10 0x7f9e1c0fafb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9e08096d10 0x7f9e080991c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e1c065200 0x7f9e1c0fc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.742+0000 7f9e23089700 1 --2- 172.21.15.2:0/3440617836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e1c0ff140 0x7f9e1c0faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.743+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 >> 172.21.15.2:0/3440617836 conn(0x7f9e1c0f6640 msgr2=0x7f9e1c0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.743+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 shutdown_connections 2022-03-21T14:06:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:37.743+0000 7f9e23089700 1 -- 172.21.15.2:0/3440617836 wait complete. 2022-03-21T14:06:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 8 2022-03-21T14:06:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:38 smithi002 ceph-mon[17167]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:38 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3440617836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:38.269 INFO:journalctl@ceph.mds.k.smithi016.stdout:Mar 21 14:06:38 smithi016 ceph-mds[48423]: -- 172.21.15.16:0/1599336210 <== mon.1 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559d6e0fe340 con 0x559d6e0d7000 2022-03-21T14:06:38.270 INFO:journalctl@ceph.mds.k.smithi016.stdout:Mar 21 14:06:38 smithi016 conmon[48399]: starting mds.k at 2022-03-21T14:06:38.270 INFO:journalctl@ceph.mds.k.smithi016.stdout:Mar 21 14:06:38 smithi016 bash[48212]: 19b39b3e297207b309057b6daed34e5b7ae255b55fc1c1eac0c09a8929a5d516 2022-03-21T14:06:38.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:38 smithi016 ceph-mon[22010]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:38.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:38 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3440617836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:38.350 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:38 smithi079 ceph-mon[23158]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:38.351 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:38 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3440617836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:38.663 INFO:journalctl@ceph.mds.k.smithi016.stdout:Mar 21 14:06:38 smithi016 systemd[1]: Started Ceph mds.k for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: Deploying daemon mds.c on smithi079 2022-03-21T14:06:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.16:6838/4173795056,v1:172.21.15.16:6839/4173795056] up:boot 2022-03-21T14:06:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: fsmap 8 up:standby 2022-03-21T14:06:39.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-03-21T14:06:39.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:39.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:39.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:39.621 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:39.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: Deploying daemon mds.c on smithi079 2022-03-21T14:06:39.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:39.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.16:6838/4173795056,v1:172.21.15.16:6839/4173795056] up:boot 2022-03-21T14:06:39.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: fsmap 8 up:standby 2022-03-21T14:06:39.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-03-21T14:06:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: Deploying daemon mds.c on smithi079 2022-03-21T14:06:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.16:6838/4173795056,v1:172.21.15.16:6839/4173795056] up:boot 2022-03-21T14:06:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: fsmap 8 up:standby 2022-03-21T14:06:39.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-03-21T14:06:39.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:39.972 INFO:journalctl@ceph.mds.c.smithi079.stdout:Mar 21 14:06:39 smithi079 systemd[1]: Starting Ceph mds.c for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:39.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1446584968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c103f40 msgr2=0x7f4c1c106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:39.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1446584968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c103f40 0x7f4c1c106320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009c70 tx=0x7f4c140094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1446584968 shutdown_connections 2022-03-21T14:06:39.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1446584968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c103f40 0x7f4c1c106320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1446584968 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 0x7f4c1c06f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1446584968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c06fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1446584968 >> 172.21.15.2:0/1446584968 conn(0x7f4c1c0f77a0 msgr2=0x7f4c1c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:39.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1446584968 shutdown_connections 2022-03-21T14:06:39.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.985+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1446584968 wait complete. 2022-03-21T14:06:39.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.986+0000 7f4c1bfff700 1 Processor -- start 2022-03-21T14:06:39.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.986+0000 7f4c1bfff700 1 -- start start 2022-03-21T14:06:39.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.986+0000 7f4c1bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.986+0000 7f4c1bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 0x7f4c1c10f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c103f40 0x7f4c1c116300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1bfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c116840 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1bfff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c1169b0 con 0x7f4c1c103f40 2022-03-21T14:06:39.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1bfff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c116b70 con 0x7f4c1c070e60 2022-03-21T14:06:39.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:39.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43714/0 (socket says 172.21.15.2:43714) 2022-03-21T14:06:39.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 -- 172.21.15.2:0/1060968592 learned_addr learned my addr 172.21.15.2:0/1060968592 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:39.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1a7fc700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 0x7f4c1c10f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:39.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1b7fe700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c103f40 0x7f4c1c116300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:39.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 -- 172.21.15.2:0/1060968592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 msgr2=0x7f4c1c10f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 0x7f4c1c10f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 -- 172.21.15.2:0/1060968592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c103f40 msgr2=0x7f4c1c116300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c103f40 0x7f4c1c116300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.987+0000 7f4c1affd700 1 -- 172.21.15.2:0/1060968592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14009780 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c1affd700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00fa90 tx=0x7f4c0c005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:39.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c00be00 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4c0c00de90 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c116e50 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c011e80 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.988+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c117260 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.989+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c109a00 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.991+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f4c0c00c040 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.993+0000 7f4c037fe700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 0x7f4c04099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:39.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.993+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f4c0c0bf520 con 0x7f4c1c06f8a0 2022-03-21T14:06:39.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.993+0000 7f4c1a7fc700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 0x7f4c04099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:40.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.994+0000 7f4c1a7fc700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 0x7f4c04099240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005fd0 tx=0x7f4c10005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:40.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:39.997+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f4c0c08cba0 con 0x7f4c1c06f8a0 2022-03-21T14:06:40.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.227+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c1c02acd0 con 0x7f4c1c06f8a0 2022-03-21T14:06:40.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.228+0000 7f4c037fe700 1 -- 172.21.15.2:0/1060968592 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6541 (secure 0 0 0) 0x7f4c0c0901c0 con 0x7f4c1c06f8a0 2022-03-21T14:06:40.229 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:40.230 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2022-03-21T14:06:40.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 msgr2=0x7f4c04099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:40.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 0x7f4c04099240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005fd0 tx=0x7f4c10005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 msgr2=0x7f4c1c10ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:40.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00fa90 tx=0x7f4c0c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 shutdown_connections 2022-03-21T14:06:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c1c103f40 0x7f4c1c116300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4c04096d90 0x7f4c04099240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c1c06f8a0 0x7f4c1c10ed30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 --2- 172.21.15.2:0/1060968592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c1c070e60 0x7f4c1c10f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 >> 172.21.15.2:0/1060968592 conn(0x7f4c1c0f77a0 msgr2=0x7f4c1c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 shutdown_connections 2022-03-21T14:06:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:40.231+0000 7f4c1bfff700 1 -- 172.21.15.2:0/1060968592 wait complete. 2022-03-21T14:06:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2022-03-21T14:06:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:40 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1060968592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:40.473 INFO:journalctl@ceph.mds.c.smithi079.stdout:Mar 21 14:06:40 smithi079 ceph-mds[48580]: -- 172.21.15.79:0/38390026 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ecdca68340 con 0x55ecdca40800 2022-03-21T14:06:40.473 INFO:journalctl@ceph.mds.c.smithi079.stdout:Mar 21 14:06:40 smithi079 conmon[48556]: starting mds.c at 2022-03-21T14:06:40.473 INFO:journalctl@ceph.mds.c.smithi079.stdout:Mar 21 14:06:40 smithi079 bash[48368]: 2cc4d73ec67921237ba56dbc1d7483112861fe3ced8d16e7eeb1b1ac3a95e559 2022-03-21T14:06:40.474 INFO:journalctl@ceph.mds.c.smithi079.stdout:Mar 21 14:06:40 smithi079 systemd[1]: Started Ceph mds.c for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:40 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1060968592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:40 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1060968592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: Deploying daemon mds.f on smithi079 2022-03-21T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6832/2500439348,v1:172.21.15.79:6833/2500439348] up:boot 2022-03-21T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: fsmap 9 up:standby 2022-03-21T14:06:41.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-03-21T14:06:41.882 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:41.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:41.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:41.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:41.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:41.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: Deploying daemon mds.f on smithi079 2022-03-21T14:06:41.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6832/2500439348,v1:172.21.15.79:6833/2500439348] up:boot 2022-03-21T14:06:41.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: fsmap 9 up:standby 2022-03-21T14:06:41.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-03-21T14:06:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: Deploying daemon mds.f on smithi079 2022-03-21T14:06:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6832/2500439348,v1:172.21.15.79:6833/2500439348] up:boot 2022-03-21T14:06:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: fsmap 9 up:standby 2022-03-21T14:06:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-03-21T14:06:42.223 INFO:journalctl@ceph.mds.f.smithi079.stdout:Mar 21 14:06:42 smithi079 systemd[1]: Starting Ceph mds.f for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:42.258 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 -- 172.21.15.2:0/2221392205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 msgr2=0x7f20300feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 --2- 172.21.15.2:0/2221392205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f20300feb70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f202c0098e0 tx=0x7f202c005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 -- 172.21.15.2:0/2221392205 shutdown_connections 2022-03-21T14:06:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 --2- 172.21.15.2:0/2221392205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f2030104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 --2- 172.21.15.2:0/2221392205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f20300feb70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 --2- 172.21.15.2:0/2221392205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 0x7f2030101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.473+0000 7f2037286700 1 -- 172.21.15.2:0/2221392205 >> 172.21.15.2:0/2221392205 conn(0x7f20300f6640 msgr2=0x7f20300f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.474+0000 7f2037286700 1 -- 172.21.15.2:0/2221392205 shutdown_connections 2022-03-21T14:06:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.474+0000 7f2037286700 1 -- 172.21.15.2:0/2221392205 wait complete. 2022-03-21T14:06:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.474+0000 7f2037286700 1 Processor -- start 2022-03-21T14:06:42.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 -- start start 2022-03-21T14:06:42.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:42.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 0x7f203010d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:42.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f20301144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:42.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301149e0 con 0x7f2030065200 2022-03-21T14:06:42.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2030114b50 con 0x7f2030101c10 2022-03-21T14:06:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2037286700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2030114d10 con 0x7f20300ff140 2022-03-21T14:06:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2036284700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2036284700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43718/0 (socket says 172.21.15.2:43718) 2022-03-21T14:06:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2036284700 1 -- 172.21.15.2:0/433975931 learned_addr learned my addr 172.21.15.2:0/433975931 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2035a83700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 0x7f203010d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.475+0000 7f2036a85700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f20301144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 -- 172.21.15.2:0/433975931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 msgr2=0x7f203010d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 0x7f203010d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 -- 172.21.15.2:0/433975931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 msgr2=0x7f20301144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f20301144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 -- 172.21.15.2:0/433975931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280096b0 con 0x7f2030065200 2022-03-21T14:06:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036a85700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f20301144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2036284700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f202c000e50 tx=0x7f202c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c01d070 con 0x7f2030065200 2022-03-21T14:06:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f202c005040 con 0x7f2030065200 2022-03-21T14:06:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c0093f0 con 0x7f2030065200 2022-03-21T14:06:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c017a40 con 0x7f2030065200 2022-03-21T14:06:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.476+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20301151b0 con 0x7f2030065200 2022-03-21T14:06:42.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.478+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2030107740 con 0x7f2030065200 2022-03-21T14:06:42.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.481+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f202c028030 con 0x7f2030065200 2022-03-21T14:06:42.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.482+0000 7f2026ffd700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 0x7f201c099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:42.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.482+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f202c0c0840 con 0x7f2030065200 2022-03-21T14:06:42.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.482+0000 7f2035a83700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 0x7f201c099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:42.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.483+0000 7f2035a83700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 0x7f201c099240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2030064fd0 tx=0x7f2020005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:42.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.484+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f202c08dfb0 con 0x7f2030065200 2022-03-21T14:06:42.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.718+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20300f7ff0 con 0x7f2030065200 2022-03-21T14:06:42.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.718+0000 7f2026ffd700 1 -- 172.21.15.2:0/433975931 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7300 (secure 0 0 0) 0x7f202c0915d0 con 0x7f2030065200 2022-03-21T14:06:42.720 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:42.720 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-03-21T14:06:42.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 msgr2=0x7f201c099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:42.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 0x7f201c099240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2030064fd0 tx=0x7f2020005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 msgr2=0x7f203010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:42.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f202c000e50 tx=0x7f202c005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:42.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 shutdown_connections 2022-03-21T14:06:42.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2030101c10 0x7f20301144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f201c096d90 0x7f201c099240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2030065200 0x7f203010ced0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 --2- 172.21.15.2:0/433975931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f20300ff140 0x7f203010d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:42.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 >> 172.21.15.2:0/433975931 conn(0x7f20300f6640 msgr2=0x7f20300fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:42.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 shutdown_connections 2022-03-21T14:06:42.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:42.721+0000 7f2037286700 1 -- 172.21.15.2:0/433975931 wait complete. 2022-03-21T14:06:42.725 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 10 2022-03-21T14:06:42.972 INFO:journalctl@ceph.mds.f.smithi079.stdout:Mar 21 14:06:42 smithi079 ceph-mds[49122]: -- 172.21.15.79:0/170135792 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b511be340 con 0x557b51197000 2022-03-21T14:06:42.973 INFO:journalctl@ceph.mds.f.smithi079.stdout:Mar 21 14:06:42 smithi079 conmon[49098]: starting mds.f at 2022-03-21T14:06:42.973 INFO:journalctl@ceph.mds.f.smithi079.stdout:Mar 21 14:06:42 smithi079 bash[48911]: d3558fe48edf5b2a17a9bd71a2cbce4c8ffad8c97a1658024ce5d0ed90798086 2022-03-21T14:06:42.973 INFO:journalctl@ceph.mds.f.smithi079.stdout:Mar 21 14:06:42 smithi079 systemd[1]: Started Ceph mds.f for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/433975931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: Deploying daemon mds.i on smithi079 2022-03-21T14:06:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6834/805336294,v1:172.21.15.79:6835/805336294] up:boot 2022-03-21T14:06:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: fsmap 10 up:standby 2022-03-21T14:06:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-03-21T14:06:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/433975931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:44.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:44.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:44.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: Deploying daemon mds.i on smithi079 2022-03-21T14:06:44.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6834/805336294,v1:172.21.15.79:6835/805336294] up:boot 2022-03-21T14:06:44.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: fsmap 10 up:standby 2022-03-21T14:06:44.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-03-21T14:06:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/433975931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: Deploying daemon mds.i on smithi079 2022-03-21T14:06:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6834/805336294,v1:172.21.15.79:6835/805336294] up:boot 2022-03-21T14:06:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: fsmap 10 up:standby 2022-03-21T14:06:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-03-21T14:06:44.292 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 14:06:44 smithi079 systemd[1]: Starting Ceph mds.i for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:44.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:44.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 -- 172.21.15.2:0/3074195066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fd250 msgr2=0x7fc1ec0fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:44.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/3074195066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec0fd6a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00098e0 tx=0x7fc1e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:44.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 -- 172.21.15.2:0/3074195066 shutdown_connections 2022-03-21T14:06:44.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/3074195066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec0fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:44.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/3074195066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:44.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/3074195066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec0fd6a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:44.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 -- 172.21.15.2:0/3074195066 >> 172.21.15.2:0/3074195066 conn(0x7fc1ec0f7760 msgr2=0x7fc1ec0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:44.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 -- 172.21.15.2:0/3074195066 shutdown_connections 2022-03-21T14:06:44.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.962+0000 7fc1ebfff700 1 -- 172.21.15.2:0/3074195066 wait complete. 2022-03-21T14:06:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 Processor -- start 2022-03-21T14:06:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 -- start start 2022-03-21T14:06:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec10cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec10d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:44.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1142b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:44.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec114830 con 0x7fc1ec0fbfb0 2022-03-21T14:06:44.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1149a0 con 0x7fc1ec0fd250 2022-03-21T14:06:44.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.963+0000 7fc1ebfff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec114b60 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec10cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:44.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec10cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43724/0 (socket says 172.21.15.2:43724) 2022-03-21T14:06:44.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eaffd700 1 -- 172.21.15.2:0/147092908 learned_addr learned my addr 172.21.15.2:0/147092908 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:44.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1ea7fc700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec10d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:44.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:44.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 -- 172.21.15.2:0/147092908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fd250 msgr2=0x7fc1ec10d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:44.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec10d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:44.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 -- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 msgr2=0x7fc1ec10cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:44.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec10cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:44.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 -- 172.21.15.2:0/147092908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e00093f0 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.964+0000 7fc1eb7fe700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1142b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400eb50 tx=0x7fc1e4009430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:44.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.965+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e400c9d0 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.965+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc1e4003840 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.965+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec114e40 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.965+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40044d0 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.965+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec115320 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.966+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec1077a0 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.967+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc1e400c040 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.969+0000 7fc1d37fe700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 0x7fc1d4099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:44.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.969+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc1e4014070 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.969+0000 7fc1eaffd700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 0x7fc1d4099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:44.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.970+0000 7fc1eaffd700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 0x7fc1d4099240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec10e210 tx=0x7fc1dc005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:44.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:44.973+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc1e408d070 con 0x7fc1ec0fdbe0 2022-03-21T14:06:44.999 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 14:06:44 smithi079 ceph-mds[49672]: -- 172.21.15.79:0/727255983 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56274f888340 con 0x56274f860c00 2022-03-21T14:06:44.999 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 14:06:44 smithi079 conmon[49648]: starting mds.i at 2022-03-21T14:06:44.999 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 14:06:44 smithi079 bash[49464]: 9cdefcb8a7ea5c63b7493feb2ee56bc31d40c0f73c7e875fcf602696b1c8a624 2022-03-21T14:06:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.206+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ec103180 con 0x7fc1ec0fdbe0 2022-03-21T14:06:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.207+0000 7fc1d37fe700 1 -- 172.21.15.2:0/147092908 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8055 (secure 0 0 0) 0x7fc1e4090690 con 0x7fc1ec0fdbe0 2022-03-21T14:06:45.208 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:45.209 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[]} 2022-03-21T14:06:45.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 msgr2=0x7fc1d4099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:45.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 0x7fc1d4099240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec10e210 tx=0x7fc1dc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:45.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 msgr2=0x7fc1ec1142b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:45.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1142b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400eb50 tx=0x7fc1e4009430 comp rx=0 tx=0).stop 2022-03-21T14:06:45.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 shutdown_connections 2022-03-21T14:06:45.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc1d4096d90 0x7fc1d4099240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:45.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1ec0fdbe0 0x7fc1ec1142b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:45.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ec0fbfb0 0x7fc1ec10cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:45.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 --2- 172.21.15.2:0/147092908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1ec0fd250 0x7fc1ec10d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:45.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 >> 172.21.15.2:0/147092908 conn(0x7fc1ec0f7760 msgr2=0x7fc1ec100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:45.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 shutdown_connections 2022-03-21T14:06:45.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:45.210+0000 7fc1ebfff700 1 -- 172.21.15.2:0/147092908 wait complete. 2022-03-21T14:06:45.214 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2022-03-21T14:06:45.472 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 14:06:44 smithi079 systemd[1]: Started Ceph mds.i for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:46.331 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:46.331 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:46.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:46.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:46.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:46.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: Deploying daemon mds.l on smithi079 2022-03-21T14:06:46.333 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/147092908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:46.333 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6836/2188072757,v1:172.21.15.79:6837/2188072757] up:boot 2022-03-21T14:06:46.333 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: fsmap 11 up:standby 2022-03-21T14:06:46.333 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T14:06:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:46.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:46.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:46.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:46.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: Deploying daemon mds.l on smithi079 2022-03-21T14:06:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/147092908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6836/2188072757,v1:172.21.15.79:6837/2188072757] up:boot 2022-03-21T14:06:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: fsmap 11 up:standby 2022-03-21T14:06:46.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T14:06:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-03-21T14:06:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-03-21T14:06:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: Deploying daemon mds.l on smithi079 2022-03-21T14:06:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/147092908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6836/2188072757,v1:172.21.15.79:6837/2188072757] up:boot 2022-03-21T14:06:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: fsmap 11 up:standby 2022-03-21T14:06:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T14:06:46.723 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:46 smithi079 systemd[1]: Starting Ceph mds.l for ee5e78f2-a91e-11ec-8c35-001a4aab830c... 2022-03-21T14:06:47.237 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 -- 172.21.15.2:0/3944930951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 msgr2=0x7f42500fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 --2- 172.21.15.2:0/3944930951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f42500fd6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42440098e0 tx=0x7f4244005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 -- 172.21.15.2:0/3944930951 shutdown_connections 2022-03-21T14:06:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 --2- 172.21.15.2:0/3944930951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 0x7f42501044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 --2- 172.21.15.2:0/3944930951 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f42500fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 --2- 172.21.15.2:0/3944930951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f42500fd6a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 -- 172.21.15.2:0/3944930951 >> 172.21.15.2:0/3944930951 conn(0x7f42500f7760 msgr2=0x7f42500f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 -- 172.21.15.2:0/3944930951 shutdown_connections 2022-03-21T14:06:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.461+0000 7f424ffff700 1 -- 172.21.15.2:0/3944930951 wait complete. 2022-03-21T14:06:47.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.462+0000 7f424ffff700 1 Processor -- start 2022-03-21T14:06:47.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 -- start start 2022-03-21T14:06:47.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:47.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f425010d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:47.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 0x7f42501143e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:47.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250114920 con 0x7f42500fd250 2022-03-21T14:06:47.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250114a90 con 0x7f42500fdbe0 2022-03-21T14:06:47.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424ffff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250114c50 con 0x7f42500fbfb0 2022-03-21T14:06:47.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:47.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39582/0 (socket says 172.21.15.2:39582) 2022-03-21T14:06:47.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424effd700 1 -- 172.21.15.2:0/478182897 learned_addr learned my addr 172.21.15.2:0/478182897 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:47.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424f7fe700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 0x7f42501143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:47.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.463+0000 7f424e7fc700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f425010d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:47.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 -- 172.21.15.2:0/478182897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 msgr2=0x7f42501143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:47.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 0x7f42501143e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 -- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 msgr2=0x7f425010d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:47.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f425010d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 -- 172.21.15.2:0/478182897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440093f0 con 0x7f42500fbfb0 2022-03-21T14:06:47.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424e7fc700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f425010d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:47.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424effd700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f424000f860 tx=0x7f4240005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:47.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240004020 con 0x7f42500fbfb0 2022-03-21T14:06:47.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4240004180 con 0x7f42500fbfb0 2022-03-21T14:06:47.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400039e0 con 0x7f42500fbfb0 2022-03-21T14:06:47.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.464+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4250114f30 con 0x7f42500fbfb0 2022-03-21T14:06:47.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.465+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4250115280 con 0x7f42500fbfb0 2022-03-21T14:06:47.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.466+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250107750 con 0x7f42500fbfb0 2022-03-21T14:06:47.473 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:47 smithi079 ceph-mds[50217]: -- 172.21.15.79:0/173790227 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56289ec02340 con 0x56289ebdac00 2022-03-21T14:06:47.473 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:47 smithi079 conmon[50193]: starting mds.l at 2022-03-21T14:06:47.474 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:47 smithi079 bash[50006]: f62ac7f0d07ee070bc3d81b0d847307b186c5d186c29abb19862914a711dce39 2022-03-21T14:06:47.474 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:47 smithi079 systemd[1]: Started Ceph mds.l for ee5e78f2-a91e-11ec-8c35-001a4aab830c. 2022-03-21T14:06:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.467+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f4240022060 con 0x7f42500fbfb0 2022-03-21T14:06:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.468+0000 7f42377fe700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 0x7f42380991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.469+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f42400befe0 con 0x7f42500fbfb0 2022-03-21T14:06:47.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.469+0000 7f424e7fc700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 0x7f42380991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:47.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.470+0000 7f424e7fc700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 0x7f42380991c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4244009f50 tx=0x7f424401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:47.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.473+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f424008c750 con 0x7f42500fbfb0 2022-03-21T14:06:47.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.706+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f425002ac40 con 0x7f42500fbfb0 2022-03-21T14:06:47.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.708+0000 7f42377fe700 1 -- 172.21.15.2:0/478182897 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8813 (secure 0 0 0) 0x7f424008fd70 con 0x7f42500fbfb0 2022-03-21T14:06:47.709 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:47.709 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24575,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-03-21T14:06:47.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.710+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 msgr2=0x7f42380991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:47.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.710+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 0x7f42380991c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4244009f50 tx=0x7f424401a040 comp rx=0 tx=0).stop 2022-03-21T14:06:47.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.710+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 msgr2=0x7f425010ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:47.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.710+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f424000f860 tx=0x7f4240005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 shutdown_connections 2022-03-21T14:06:47.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42500fdbe0 0x7f42501143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4238096d10 0x7f42380991c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42500fbfb0 0x7f425010ce10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 --2- 172.21.15.2:0/478182897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42500fd250 0x7f425010d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:47.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 >> 172.21.15.2:0/478182897 conn(0x7f42500f7760 msgr2=0x7f4250100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:47.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 shutdown_connections 2022-03-21T14:06:47.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:47.711+0000 7f424ffff700 1 -- 172.21.15.2:0/478182897 wait complete. 2022-03-21T14:06:47.715 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 12 2022-03-21T14:06:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:48 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/478182897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:48 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/478182897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:48 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/478182897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:49 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:boot 2022-03-21T14:06:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:49 smithi002 ceph-mon[17167]: fsmap 12 up:standby 2022-03-21T14:06:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-03-21T14:06:49.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:boot 2022-03-21T14:06:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:49 smithi016 ceph-mon[22010]: fsmap 12 up:standby 2022-03-21T14:06:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-03-21T14:06:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:boot 2022-03-21T14:06:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:49 smithi079 ceph-mon[23158]: fsmap 12 up:standby 2022-03-21T14:06:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-03-21T14:06:49.737 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:49.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 -- 172.21.15.2:0/2113289323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fd250 msgr2=0x7f6f7c0fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:49.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 --2- 172.21.15.2:0/2113289323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c0fd6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6f700098e0 tx=0x7f6f70005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:49.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 -- 172.21.15.2:0/2113289323 shutdown_connections 2022-03-21T14:06:49.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 --2- 172.21.15.2:0/2113289323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:49.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 --2- 172.21.15.2:0/2113289323 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c0fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:49.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 --2- 172.21.15.2:0/2113289323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c0fd6a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:49.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 -- 172.21.15.2:0/2113289323 >> 172.21.15.2:0/2113289323 conn(0x7f6f7c0f7760 msgr2=0x7f6f7c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:49.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 -- 172.21.15.2:0/2113289323 shutdown_connections 2022-03-21T14:06:49.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.969+0000 7f6f813c1700 1 -- 172.21.15.2:0/2113289323 wait complete. 2022-03-21T14:06:49.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.970+0000 7f6f813c1700 1 Processor -- start 2022-03-21T14:06:49.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.970+0000 7f6f813c1700 1 -- start start 2022-03-21T14:06:49.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:49.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c10f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:49.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c116510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:49.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c116a50 con 0x7f6f7c0fdbe0 2022-03-21T14:06:49.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c116bc0 con 0x7f6f7c0fd250 2022-03-21T14:06:49.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f813c1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c116d80 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:49.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39586/0 (socket says 172.21.15.2:39586) 2022-03-21T14:06:49.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7bfff700 1 -- 172.21.15.2:0/3010361668 learned_addr learned my addr 172.21.15.2:0/3010361668 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:49.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f80bc0700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c116510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:49.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7b7fe700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c10f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:49.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7bfff700 1 -- 172.21.15.2:0/3010361668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 msgr2=0x7f6f7c10f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:49.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.971+0000 7f6f7bfff700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c10f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:49.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f7bfff700 1 -- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 msgr2=0x7f6f7c116510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:49.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f7bfff700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c116510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:49.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f7bfff700 1 -- 172.21.15.2:0/3010361668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700093f0 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f80bc0700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:49.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f7b7fe700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c10f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:49.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f7bfff700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00f860 tx=0x7f6f6c005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:49.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c004020 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6f6c00dcb0 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c117060 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c003b40 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.972+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c117620 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.973+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c1077a0 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.975+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f6f6c022060 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.976+0000 7f6f78ff9700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 0x7f6f64099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:49.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.977+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6f6c0bf0e0 con 0x7f6f7c0fbfb0 2022-03-21T14:06:49.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.977+0000 7f6f7b7fe700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 0x7f6f64099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:49.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.977+0000 7f6f7b7fe700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 0x7f6f64099240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009f50 tx=0x7f6f7001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:49.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:49.979+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f6f6c08c760 con 0x7f6f7c0fbfb0 2022-03-21T14:06:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:50 smithi002 ceph-mon[17167]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:50 smithi016 ceph-mon[22010]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.222+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f7c1178a0 con 0x7f6f7c0fbfb0 2022-03-21T14:06:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.223+0000 7f6f78ff9700 1 -- 172.21.15.2:0/3010361668 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9571 (secure 0 0 0) 0x7f6f6c08fd80 con 0x7f6f7c0fbfb0 2022-03-21T14:06:50.224 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:50.224 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24575,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24587,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1907413140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1907413140},{"type":"v1","addr":"172.21.15.79:6839","nonce":1907413140}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2022-03-21T14:06:50.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 msgr2=0x7f6f64099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 0x7f6f64099240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009f50 tx=0x7f6f7001a040 comp rx=0 tx=0).stop 2022-03-21T14:06:50.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 msgr2=0x7f6f7c10ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00f860 tx=0x7f6f6c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 shutdown_connections 2022-03-21T14:06:50.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.225+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f64096d90 0x7f6f64099240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c0fdbe0 0x7f6f7c116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f7c0fbfb0 0x7f6f7c10ef40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 --2- 172.21.15.2:0/3010361668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f7c0fd250 0x7f6f7c10f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 >> 172.21.15.2:0/3010361668 conn(0x7f6f7c0f7760 msgr2=0x7f6f7c0f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:50.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 shutdown_connections 2022-03-21T14:06:50.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.226+0000 7f6f813c1700 1 -- 172.21.15.2:0/3010361668 wait complete. 2022-03-21T14:06:50.229 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 13 2022-03-21T14:06:50.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:06:50.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 -- 172.21.15.2:0/2553750694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b406f8a0 msgr2=0x7f36b406fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 --2- 172.21.15.2:0/2553750694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b406f8a0 0x7f36b406fd10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f36a00098e0 tx=0x7f36a0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 -- 172.21.15.2:0/2553750694 shutdown_connections 2022-03-21T14:06:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 --2- 172.21.15.2:0/2553750694 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b4103f40 0x7f36b4106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 --2- 172.21.15.2:0/2553750694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 0x7f36b406f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 --2- 172.21.15.2:0/2553750694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b406f8a0 0x7f36b406fd10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 -- 172.21.15.2:0/2553750694 >> 172.21.15.2:0/2553750694 conn(0x7f36b40f7760 msgr2=0x7f36b40f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 -- 172.21.15.2:0/2553750694 shutdown_connections 2022-03-21T14:06:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.468+0000 7f36b9e44700 1 -- 172.21.15.2:0/2553750694 wait complete. 2022-03-21T14:06:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:50 smithi079 ceph-mon[23158]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:50.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 Processor -- start 2022-03-21T14:06:50.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 -- start start 2022-03-21T14:06:50.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 0x7f36b410f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b4103f40 0x7f36b4116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4116a70 con 0x7f36b4070e60 2022-03-21T14:06:50.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4116be0 con 0x7f36b4103f40 2022-03-21T14:06:50.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.469+0000 7f36b9e44700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4116da0 con 0x7f36b406f8a0 2022-03-21T14:06:50.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39590/0 (socket says 172.21.15.2:39590) 2022-03-21T14:06:50.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 -- 172.21.15.2:0/932972268 learned_addr learned my addr 172.21.15.2:0/932972268 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:50.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b3fff700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 0x7f36b410f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b9643700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b4103f40 0x7f36b4116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 -- 172.21.15.2:0/932972268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b4103f40 msgr2=0x7f36b4116530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b4103f40 0x7f36b4116530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 -- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 msgr2=0x7f36b410f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 0x7f36b410f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.470+0000 7f36b8e42700 1 -- 172.21.15.2:0/932972268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac0096b0 con 0x7f36b406f8a0 2022-03-21T14:06:50.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b8e42700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f36a800fa90 tx=0x7f36a8005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:50.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a800be00 con 0x7f36b406f8a0 2022-03-21T14:06:50.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f36a800de90 con 0x7f36b406f8a0 2022-03-21T14:06:50.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8011e80 con 0x7f36b406f8a0 2022-03-21T14:06:50.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a00093f0 con 0x7f36b406f8a0 2022-03-21T14:06:50.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.471+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b4117410 con 0x7f36b406f8a0 2022-03-21T14:06:50.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.472+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b4109a50 con 0x7f36b406f8a0 2022-03-21T14:06:50.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.475+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f36a800c040 con 0x7f36b406f8a0 2022-03-21T14:06:50.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.477+0000 7f36b17fa700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 0x7f36a4099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.477+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f36a80bf490 con 0x7f36b406f8a0 2022-03-21T14:06:50.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.477+0000 7f36b3fff700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 0x7f36a4099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.478+0000 7f36b3fff700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 0x7f36a4099240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009f50 tx=0x7f36a0005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:50.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.480+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f36a808cb10 con 0x7f36b406f8a0 2022-03-21T14:06:50.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.680+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b4102510 con 0x7f36b406f8a0 2022-03-21T14:06:50.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.681+0000 7f36b17fa700 1 -- 172.21.15.2:0/932972268 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+17985 (secure 0 0 0) 0x7f36a8090130 con 0x7f36b406f8a0 2022-03-21T14:06:50.682 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:50.683 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":78,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:03.460664+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:06:50.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 msgr2=0x7f36a4099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 0x7f36a4099240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009f50 tx=0x7f36a0005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:50.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 msgr2=0x7f36b410ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f36a800fa90 tx=0x7f36a8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 shutdown_connections 2022-03-21T14:06:50.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36b4103f40 0x7f36b4116530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36a4096d90 0x7f36a4099240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36b406f8a0 0x7f36b410ef60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 --2- 172.21.15.2:0/932972268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36b4070e60 0x7f36b410f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 >> 172.21.15.2:0/932972268 conn(0x7f36b40f7760 msgr2=0x7f36b40f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:50.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 shutdown_connections 2022-03-21T14:06:50.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.684+0000 7f36b9e44700 1 -- 172.21.15.2:0/932972268 wait complete. 2022-03-21T14:06:50.712 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-21T14:06:50.712 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-03-21T14:06:50.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 -- 172.21.15.2:0/3843846189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fbfd0 msgr2=0x7f1c040fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 --2- 172.21.15.2:0/3843846189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fbfd0 0x7f1c040fc3e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40098e0 tx=0x7f1bf4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:50.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 -- 172.21.15.2:0/3843846189 shutdown_connections 2022-03-21T14:06:50.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 --2- 172.21.15.2:0/3843846189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fdc00 0x7f1c041044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 --2- 172.21.15.2:0/3843846189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fbfd0 0x7f1c040fc3e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 --2- 172.21.15.2:0/3843846189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fd270 0x7f1c040fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 -- 172.21.15.2:0/3843846189 >> 172.21.15.2:0/3843846189 conn(0x7f1c040f7780 msgr2=0x7f1c040f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:50.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 -- 172.21.15.2:0/3843846189 shutdown_connections 2022-03-21T14:06:50.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.930+0000 7f1c08afa700 1 -- 172.21.15.2:0/3843846189 wait complete. 2022-03-21T14:06:50.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.931+0000 7f1c08afa700 1 Processor -- start 2022-03-21T14:06:50.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.931+0000 7f1c08afa700 1 -- start start 2022-03-21T14:06:50.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fbfd0 0x7f1c0410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fdc00 0x7f1c04114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041148a0 con 0x7f1c040fdc00 2022-03-21T14:06:50.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04114a10 con 0x7f1c040fbfd0 2022-03-21T14:06:50.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c08afa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04114bd0 con 0x7f1c040fd270 2022-03-21T14:06:50.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fdc00 0x7f1c04114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fbfd0 0x7f1c0410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39594/0 (socket says 172.21.15.2:39594) 2022-03-21T14:06:50.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 -- 172.21.15.2:0/2189909188 learned_addr learned my addr 172.21.15.2:0/2189909188 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:50.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 -- 172.21.15.2:0/2189909188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fbfd0 msgr2=0x7f1c0410cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fbfd0 0x7f1c0410cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 -- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fdc00 msgr2=0x7f1c04114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:50.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.932+0000 7f1c02ffd700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fdc00 0x7f1c04114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:50.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1c02ffd700 1 -- 172.21.15.2:0/2189909188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40093f0 con 0x7f1c040fd270 2022-03-21T14:06:50.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1c02ffd700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800fa90 tx=0x7f1bf8005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:50.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800b4f0 con 0x7f1c040fd270 2022-03-21T14:06:50.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1bf800dc80 con 0x7f1c040fd270 2022-03-21T14:06:50.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c04114eb0 con 0x7f1c040fd270 2022-03-21T14:06:50.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8011dc0 con 0x7f1c040fd270 2022-03-21T14:06:50.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.933+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c04115390 con 0x7f1c040fd270 2022-03-21T14:06:50.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.935+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c04107810 con 0x7f1c040fd270 2022-03-21T14:06:50.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.936+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f1bf800c040 con 0x7f1c040fd270 2022-03-21T14:06:50.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.938+0000 7f1bebfff700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 0x7f1bec099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:50.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.939+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1bf80bf880 con 0x7f1c040fd270 2022-03-21T14:06:50.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.939+0000 7f1c037fe700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 0x7f1bec099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:50.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.939+0000 7f1c037fe700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 0x7f1bec099240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40098b0 tx=0x7f1bf4005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:50.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:50.942+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f1bf808cf00 con 0x7f1c040fd270 2022-03-21T14:06:51.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.151+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c04102f70 con 0x7f1c040fd270 2022-03-21T14:06:51.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.152+0000 7f1bebfff700 1 -- 172.21.15.2:0/2189909188 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1bf8090520 con 0x7f1c040fd270 2022-03-21T14:06:51.153 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:06:51.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 msgr2=0x7f1bec099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 0x7f1bec099240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40098b0 tx=0x7f1bf4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:51.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 msgr2=0x7f1c0410d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800fa90 tx=0x7f1bf8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 shutdown_connections 2022-03-21T14:06:51.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c040fbfd0 0x7f1c0410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1bec096d90 0x7f1bec099240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c040fdc00 0x7f1c04114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 --2- 172.21.15.2:0/2189909188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c040fd270 0x7f1c0410d2d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 >> 172.21.15.2:0/2189909188 conn(0x7f1c040f7780 msgr2=0x7f1c040f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:51.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 shutdown_connections 2022-03-21T14:06:51.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.155+0000 7f1c08afa700 1 -- 172.21.15.2:0/2189909188 wait complete. 2022-03-21T14:06:51.183 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-03-21T14:06:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:51 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3010361668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:51 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/932972268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 -- 172.21.15.2:0/2112982014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c065200 msgr2=0x7f1c4c0feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 --2- 172.21.15.2:0/2112982014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c065200 0x7f1c4c0feb70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1c480098e0 tx=0x7f1c48005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:51.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 -- 172.21.15.2:0/2112982014 shutdown_connections 2022-03-21T14:06:51.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 --2- 172.21.15.2:0/2112982014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c101c10 0x7f1c4c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 --2- 172.21.15.2:0/2112982014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c065200 0x7f1c4c0feb70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 --2- 172.21.15.2:0/2112982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 -- 172.21.15.2:0/2112982014 >> 172.21.15.2:0/2112982014 conn(0x7f1c4c0f6640 msgr2=0x7f1c4c0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:51.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 -- 172.21.15.2:0/2112982014 shutdown_connections 2022-03-21T14:06:51.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.391+0000 7f1c525ce700 1 -- 172.21.15.2:0/2112982014 wait complete. 2022-03-21T14:06:51.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 Processor -- start 2022-03-21T14:06:51.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 -- start start 2022-03-21T14:06:51.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c065200 0x7f1c4c10f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c101c10 0x7f1c4c1166a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c116be0 con 0x7f1c4c101c10 2022-03-21T14:06:51.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c116d50 con 0x7f1c4c065200 2022-03-21T14:06:51.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.392+0000 7f1c525ce700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c116f10 con 0x7f1c4c0ff140 2022-03-21T14:06:51.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c515cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c065200 0x7f1c4c10f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39598/0 (socket says 172.21.15.2:39598) 2022-03-21T14:06:51.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 -- 172.21.15.2:0/363186016 learned_addr learned my addr 172.21.15.2:0/363186016 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:51.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c51dcd700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c101c10 0x7f1c4c1166a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 -- 172.21.15.2:0/363186016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c065200 msgr2=0x7f1c4c10f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c065200 0x7f1c4c10f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 -- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c101c10 msgr2=0x7f1c4c1166a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c101c10 0x7f1c4c1166a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 -- 172.21.15.2:0/363186016 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480093f0 con 0x7f1c4c0ff140 2022-03-21T14:06:51.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.393+0000 7f1c50dcb700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00d790 tx=0x7f1c3c00b320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:51.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.394+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c00f6a0 con 0x7f1c4c0ff140 2022-03-21T14:06:51.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.394+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1c3c00b460 con 0x7f1c4c0ff140 2022-03-21T14:06:51.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.394+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c010d60 con 0x7f1c4c0ff140 2022-03-21T14:06:51.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.394+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c1171f0 con 0x7f1c4c0ff140 2022-03-21T14:06:51.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.394+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c117690 con 0x7f1c4c0ff140 2022-03-21T14:06:51.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.395+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c107740 con 0x7f1c4c0ff140 2022-03-21T14:06:51.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.396+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f1c3c00c040 con 0x7f1c4c0ff140 2022-03-21T14:06:51.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.398+0000 7f1c41ffb700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 0x7f1c38099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.398+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1c3c016030 con 0x7f1c4c0ff140 2022-03-21T14:06:51.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.398+0000 7f1c515cc700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 0x7f1c38099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.399+0000 7f1c515cc700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 0x7f1c38099240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800fa40 tx=0x7f1c48005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:51.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.401+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f1c3c08ca80 con 0x7f1c4c0ff140 2022-03-21T14:06:51.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:51 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3010361668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:51 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/932972268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:51 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3010361668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:51 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/932972268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:51.607 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.607+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1c4c0fbf10 con 0x7f1c4c0ff140 2022-03-21T14:06:51.608 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.607+0000 7f1c41ffb700 1 -- 172.21.15.2:0/363186016 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1c3c0900a0 con 0x7f1c4c0ff140 2022-03-21T14:06:51.608 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:51.609 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2022-03-21T14:06:51.609 INFO:teuthology.orchestra.run.smithi002.stdout: { 2022-03-21T14:06:51.609 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2022-03-21T14:06:51.609 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2022-03-21T14:06:51.610 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T14:06:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:] 2022-03-21T14:06:51.612 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 msgr2=0x7f1c38099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 0x7f1c38099240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800fa40 tx=0x7f1c48005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:51.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 msgr2=0x7f1c4c10f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00d790 tx=0x7f1c3c00b320 comp rx=0 tx=0).stop 2022-03-21T14:06:51.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 shutdown_connections 2022-03-21T14:06:51.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c4c065200 0x7f1c4c10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1c38096d90 0x7f1c38099240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c4c101c10 0x7f1c4c1166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 --2- 172.21.15.2:0/363186016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c4c0ff140 0x7f1c4c10f610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.611+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 >> 172.21.15.2:0/363186016 conn(0x7f1c4c0f6640 msgr2=0x7f1c4c0f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:51.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.612+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 shutdown_connections 2022-03-21T14:06:51.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.612+0000 7f1c525ce700 1 -- 172.21.15.2:0/363186016 wait complete. 2022-03-21T14:06:51.638 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-03-21T14:06:51.638 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-03-21T14:06:51.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 -- 172.21.15.2:0/3800105140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fd270 msgr2=0x7fc9d00fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 --2- 172.21.15.2:0/3800105140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fd270 0x7fc9d00fd6c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40098e0 tx=0x7fc9c4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:51.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 -- 172.21.15.2:0/3800105140 shutdown_connections 2022-03-21T14:06:51.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 --2- 172.21.15.2:0/3800105140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fdc00 0x7fc9d01044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 --2- 172.21.15.2:0/3800105140 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d00fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 --2- 172.21.15.2:0/3800105140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fd270 0x7fc9d00fd6c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 -- 172.21.15.2:0/3800105140 >> 172.21.15.2:0/3800105140 conn(0x7fc9d00f7780 msgr2=0x7fc9d00f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:51.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 -- 172.21.15.2:0/3800105140 shutdown_connections 2022-03-21T14:06:51.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.854+0000 7fc9d5059700 1 -- 172.21.15.2:0/3800105140 wait complete. 2022-03-21T14:06:51.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 Processor -- start 2022-03-21T14:06:51.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 -- start start 2022-03-21T14:06:51.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 0x7fc9d006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fdc00 0x7fc9d006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d006ff70 con 0x7fc9d00fd270 2022-03-21T14:06:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00700e0 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.856+0000 7fc9d5059700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d0116f70 con 0x7fc9d00fdc00 2022-03-21T14:06:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48334/0 (socket says 172.21.15.2:48334) 2022-03-21T14:06:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 -- 172.21.15.2:0/849253762 learned_addr learned my addr 172.21.15.2:0/849253762 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 0x7fc9d006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9d4858700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fdc00 0x7fc9d006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 -- 172.21.15.2:0/849253762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fdc00 msgr2=0x7fc9d006f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fdc00 0x7fc9d006f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 -- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 msgr2=0x7fc9d006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:51.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 0x7fc9d006f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 -- 172.21.15.2:0/849253762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c000b360 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cf7fe700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 0x7fc9d006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.857+0000 7fc9cffff700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000ea50 tx=0x7fc9c0005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.858+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000b6e0 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.858+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c40093f0 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.858+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc9c000ce20 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.858+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0018420 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.858+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d01174d0 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.859+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d01077c0 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.861+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc9c000c040 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.862+0000 7fc9ccff9700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 0x7fc9b80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.862+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc9c0014070 con 0x7fc9d00fbfd0 2022-03-21T14:06:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.862+0000 7fc9cf7fe700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 0x7fc9b80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.863+0000 7fc9cf7fe700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 0x7fc9b80991c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009f50 tx=0x7fc9c401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:51.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:51.865+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc9c008d140 con 0x7fc9d00fbfd0 2022-03-21T14:06:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:52.075+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc9d01176f0 con 0x7fc9d00fbfd0 2022-03-21T14:06:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:52.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:52.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2189909188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:52.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:52 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/363186016' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-03-21T14:06:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2189909188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:52 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/363186016' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-03-21T14:06:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:06:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:06:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:06:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2189909188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:06:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:52 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/363186016' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-03-21T14:06:53.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.073+0000 7fc9ccff9700 1 -- 172.21.15.2:0/849253762 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc9c0093e60 con 0x7fc9d00fbfd0 2022-03-21T14:06:53.079 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.078+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 msgr2=0x7fc9b80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:53.079 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.078+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 0x7fc9b80991c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009f50 tx=0x7fc9c401a040 comp rx=0 tx=0).stop 2022-03-21T14:06:53.079 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.078+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 msgr2=0x7fc9d0070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:53.079 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.078+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000ea50 tx=0x7fc9c0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.080 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 shutdown_connections 2022-03-21T14:06:53.080 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9d00fbfd0 0x7fc9d0070cc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.080 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc9b8096d10 0x7fc9b80991c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.081 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9d00fdc00 0x7fc9d006f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.081 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 --2- 172.21.15.2:0/849253762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9d00fd270 0x7fc9d006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.081 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.079+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 >> 172.21.15.2:0/849253762 conn(0x7fc9d00f7780 msgr2=0x7fc9d01004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.080+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 shutdown_connections 2022-03-21T14:06:53.083 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.080+0000 7fc9d5059700 1 -- 172.21.15.2:0/849253762 wait complete. 2022-03-21T14:06:53.083 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2022-03-21T14:06:53.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-03-21T14:06:53.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 -- 172.21.15.2:0/2825565596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fdc00 msgr2=0x7fc4b01044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:53.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 --2- 172.21.15.2:0/2825565596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fdc00 0x7fc4b01044c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009c70 tx=0x7fc4a80094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 -- 172.21.15.2:0/2825565596 shutdown_connections 2022-03-21T14:06:53.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 --2- 172.21.15.2:0/2825565596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fdc00 0x7fc4b01044c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 --2- 172.21.15.2:0/2825565596 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b00fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 --2- 172.21.15.2:0/2825565596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fd270 0x7fc4b00fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 -- 172.21.15.2:0/2825565596 >> 172.21.15.2:0/2825565596 conn(0x7fc4b00f7780 msgr2=0x7fc4b00f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:53.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 -- 172.21.15.2:0/2825565596 shutdown_connections 2022-03-21T14:06:53.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.331+0000 7fc4af59e700 1 -- 172.21.15.2:0/2825565596 wait complete. 2022-03-21T14:06:53.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.332+0000 7fc4af59e700 1 Processor -- start 2022-03-21T14:06:53.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.332+0000 7fc4af59e700 1 -- start start 2022-03-21T14:06:53.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:53.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fd270 0x7fc4b006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:53.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 0x7fc4b006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:53.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b006ff70 con 0x7fc4b00fdc00 2022-03-21T14:06:53.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00700e0 con 0x7fc4b00fbfd0 2022-03-21T14:06:53.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4af59e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b0116c40 con 0x7fc4b00fd270 2022-03-21T14:06:53.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4ae59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:53.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4aed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 0x7fc4b006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:53.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4ae59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48340/0 (socket says 172.21.15.2:48340) 2022-03-21T14:06:53.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4ae59c700 1 -- 172.21.15.2:0/4279350523 learned_addr learned my addr 172.21.15.2:0/4279350523 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:53.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.333+0000 7fc4add9b700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fd270 0x7fc4b006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:53.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 -- 172.21.15.2:0/4279350523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fd270 msgr2=0x7fc4b006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:53.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fd270 0x7fc4b006f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 -- 172.21.15.2:0/4279350523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 msgr2=0x7fc4b0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:53.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:53.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 -- 172.21.15.2:0/4279350523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8009780 con 0x7fc4b00fdc00 2022-03-21T14:06:53.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4ae59c700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:53.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4aed9d700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 0x7fc4b006f860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009b10 tx=0x7fc4a80094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:53.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a801c070 con 0x7fc4b00fdc00 2022-03-21T14:06:53.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b0116de0 con 0x7fc4b00fdc00 2022-03-21T14:06:53.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc4a800f460 con 0x7fc4b00fdc00 2022-03-21T14:06:53.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8003bd0 con 0x7fc4b00fdc00 2022-03-21T14:06:53.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.334+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b0117210 con 0x7fc4b00fdc00 2022-03-21T14:06:53.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.335+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b0107830 con 0x7fc4b00fdc00 2022-03-21T14:06:53.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.337+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc4a800c040 con 0x7fc4b00fdc00 2022-03-21T14:06:53.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.338+0000 7fc49effd700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 0x7fc4980991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:53.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.339+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fc4a80c0e50 con 0x7fc4b00fdc00 2022-03-21T14:06:53.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.339+0000 7fc4ae59c700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 0x7fc4980991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:53.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.339+0000 7fc4ae59c700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 0x7fc4980991c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000bfd0 tx=0x7fc4a0005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:53.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.342+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc4a808e360 con 0x7fc4b00fdc00 2022-03-21T14:06:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:53 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/849253762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:53 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:53 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/849253762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:53 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:53 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/849253762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:53 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-03-21T14:06:53.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:53.537+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc4b002ab20 con 0x7fc4b00fdc00 2022-03-21T14:06:54.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.300+0000 7fc49effd700 1 -- 172.21.15.2:0/4279350523 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc4a800f940 con 0x7fc4b00fdc00 2022-03-21T14:06:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 msgr2=0x7fc4980991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 0x7fc4980991c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000bfd0 tx=0x7fc4a0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 msgr2=0x7fc4b006f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 0x7fc4b006f860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009b10 tx=0x7fc4a80094e0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 shutdown_connections 2022-03-21T14:06:54.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b00fbfd0 0x7fc4b0070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc498096d10 0x7fc4980991c0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000bfd0 tx=0x7fc4a0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4b00fdc00 0x7fc4b006f860 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 --2- 172.21.15.2:0/4279350523 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4b00fd270 0x7fc4b006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.307+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 >> 172.21.15.2:0/4279350523 conn(0x7fc4b00f7780 msgr2=0x7fc4b00f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.308+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 shutdown_connections 2022-03-21T14:06:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.309+0000 7fc4af59e700 1 -- 172.21.15.2:0/4279350523 wait complete. 2022-03-21T14:06:54.318 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2022-03-21T14:06:54.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-03-21T14:06:54.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 -- 172.21.15.2:0/2860699680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc978103f80 msgr2=0x7fc978106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 --2- 172.21.15.2:0/2860699680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc978103f80 0x7fc978106360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc970009ba0 tx=0x7fc970009430 comp rx=0 tx=0).stop 2022-03-21T14:06:54.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 -- 172.21.15.2:0/2860699680 shutdown_connections 2022-03-21T14:06:54.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 --2- 172.21.15.2:0/2860699680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978070e40 0x7fc97806f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 --2- 172.21.15.2:0/2860699680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc978103f80 0x7fc978106360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 --2- 172.21.15.2:0/2860699680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc97806f880 0x7fc97806fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 -- 172.21.15.2:0/2860699680 >> 172.21.15.2:0/2860699680 conn(0x7fc9780f7780 msgr2=0x7fc9780f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:54.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.570+0000 7fc97d40e700 1 -- 172.21.15.2:0/2860699680 shutdown_connections 2022-03-21T14:06:54.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.571+0000 7fc97d40e700 1 -- 172.21.15.2:0/2860699680 wait complete. 2022-03-21T14:06:54.576 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 Processor -- start 2022-03-21T14:06:54.576 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 -- start start 2022-03-21T14:06:54.577 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 0x7fc97810ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:54.577 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:54.577 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978103f80 0x7fc978116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:54.577 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9781167e0 con 0x7fc97806f880 2022-03-21T14:06:54.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc978116950 con 0x7fc978103f80 2022-03-21T14:06:54.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.572+0000 7fc97d40e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc978116b10 con 0x7fc978070e40 2022-03-21T14:06:54.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc977fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 0x7fc97810ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:54.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:54.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc977fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 0x7fc97810ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43758/0 (socket says 172.21.15.2:43758) 2022-03-21T14:06:54.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc977fff700 1 -- 172.21.15.2:0/1784889509 learned_addr learned my addr 172.21.15.2:0/1784889509 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:54.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39614/0 (socket says 172.21.15.2:39614) 2022-03-21T14:06:54.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc97cc0d700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978103f80 0x7fc978116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:54.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 -- 172.21.15.2:0/1784889509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978103f80 msgr2=0x7fc978116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978103f80 0x7fc978116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 -- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 msgr2=0x7fc97810ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 0x7fc97810ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 -- 172.21.15.2:0/1784889509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9700096b0 con 0x7fc978070e40 2022-03-21T14:06:54.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.573+0000 7fc9777fe700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00f830 tx=0x7fc96c005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:54.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.574+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c011720 con 0x7fc978070e40 2022-03-21T14:06:54.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.574+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc96c011d30 con 0x7fc978070e40 2022-03-21T14:06:54.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.574+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc978116df0 con 0x7fc978070e40 2022-03-21T14:06:54.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.574+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c010e10 con 0x7fc978070e40 2022-03-21T14:06:54.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.574+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9781172e0 con 0x7fc978070e40 2022-03-21T14:06:54.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.575+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc978109a40 con 0x7fc978070e40 2022-03-21T14:06:54.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.577+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc96c018020 con 0x7fc978070e40 2022-03-21T14:06:54.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.578+0000 7fc974ff9700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 0x7fc960099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:54.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.578+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7875+0+0 (secure 0 0 0) 0x7fc96c0bf850 con 0x7fc978070e40 2022-03-21T14:06:54.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.578+0000 7fc977fff700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 0x7fc960099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:54.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.579+0000 7fc977fff700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 0x7fc960099240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc97810fb60 tx=0x7fc968005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:54.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.580+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc96c06cf40 con 0x7fc978070e40 2022-03-21T14:06:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:54 smithi002 ceph-mon[17167]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:54 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-03-21T14:06:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:54 smithi002 ceph-mon[17167]: osdmap e79: 12 total, 12 up, 12 in 2022-03-21T14:06:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:54 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-03-21T14:06:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:54 smithi016 ceph-mon[22010]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:54 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-03-21T14:06:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:54 smithi016 ceph-mon[22010]: osdmap e79: 12 total, 12 up, 12 in 2022-03-21T14:06:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:54 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-03-21T14:06:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:54 smithi079 ceph-mon[23158]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:54 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-03-21T14:06:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:54 smithi079 ceph-mon[23158]: osdmap e79: 12 total, 12 up, 12 in 2022-03-21T14:06:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:54 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-03-21T14:06:54.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.835+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc978102550 con 0x7fc978070e40 2022-03-21T14:06:54.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.845+0000 7fc974ff9700 1 -- 172.21.15.2:0/1784889509 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc96c090670 con 0x7fc978070e40 2022-03-21T14:06:54.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.862+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 msgr2=0x7fc960099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.862+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 0x7fc960099240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc97810fb60 tx=0x7fc968005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:54.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.862+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 msgr2=0x7fc97810f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:54.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.862+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00f830 tx=0x7fc96c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.863+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 shutdown_connections 2022-03-21T14:06:54.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.863+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978103f80 0x7fc978116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.863+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc960096d90 0x7fc960099240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.863+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc97806f880 0x7fc97810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.863+0000 7fc97d40e700 1 --2- 172.21.15.2:0/1784889509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc978070e40 0x7fc97810f1b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:54.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.864+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 >> 172.21.15.2:0/1784889509 conn(0x7fc9780f7780 msgr2=0x7fc978106100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:54.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.864+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 shutdown_connections 2022-03-21T14:06:54.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:54.869+0000 7fc97d40e700 1 -- 172.21.15.2:0/1784889509 wait complete. 2022-03-21T14:06:54.887 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2022-03-21T14:06:54.913 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:06:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 -- 172.21.15.2:0/813907782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fbfb0 msgr2=0x7fc50c0fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 --2- 172.21.15.2:0/813907782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c0fc3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0098e0 tx=0x7fc4fc005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:55.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 -- 172.21.15.2:0/813907782 shutdown_connections 2022-03-21T14:06:55.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 --2- 172.21.15.2:0/813907782 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 --2- 172.21.15.2:0/813907782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c0fc3c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 --2- 172.21.15.2:0/813907782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fd250 0x7fc50c0fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 -- 172.21.15.2:0/813907782 >> 172.21.15.2:0/813907782 conn(0x7fc50c0f7760 msgr2=0x7fc50c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:55.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 -- 172.21.15.2:0/813907782 shutdown_connections 2022-03-21T14:06:55.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.134+0000 7fc51085b700 1 -- 172.21.15.2:0/813907782 wait complete. 2022-03-21T14:06:55.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 Processor -- start 2022-03-21T14:06:55.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 -- start start 2022-03-21T14:06:55.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fd250 0x7fc50c10d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c114a40 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c114bb0 con 0x7fc50c0fbfb0 2022-03-21T14:06:55.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc51085b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c114d70 con 0x7fc50c0fd250 2022-03-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.135+0000 7fc50bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43762/0 (socket says 172.21.15.2:43762) 2022-03-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 -- 172.21.15.2:0/3314633116 learned_addr learned my addr 172.21.15.2:0/3314633116 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48350/0 (socket says 172.21.15.2:48350) 2022-03-21T14:06:55.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50affd700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fd250 0x7fc50c10d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 -- 172.21.15.2:0/3314633116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fd250 msgr2=0x7fc50c10d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fd250 0x7fc50c10d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 -- 172.21.15.2:0/3314633116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 msgr2=0x7fc50c10cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 -- 172.21.15.2:0/3314633116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0093f0 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50b7fe700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:55.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc50bfff700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc50400ec50 tx=0x7fc504009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:55.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.136+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50400caa0 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.137+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc50400cc00 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.137+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504018690 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.137+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c114ff0 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.137+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c115400 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.138+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50c107790 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.140+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc50400c040 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.142+0000 7fc4f3fff700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 0x7fc4f40991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.142+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fc504014070 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.142+0000 7fc50b7fe700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 0x7fc4f40991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.143+0000 7fc50b7fe700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 0x7fc4f40991c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00f8e0 tx=0x7fc4fc01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:55.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.145+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc50408d500 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.336+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc50c1156e0 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.337+0000 7fc4f3fff700 1 -- 172.21.15.2:0/3314633116 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+21223 (secure 0 0 0) 0x7fc504090b20 con 0x7fc50c0fdbe0 2022-03-21T14:06:55.338 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:55.338 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:54.837312+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:06:55.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 msgr2=0x7fc4f40991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 0x7fc4f40991c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00f8e0 tx=0x7fc4fc01a040 comp rx=0 tx=0).stop 2022-03-21T14:06:55.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 msgr2=0x7fc50c114500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc50400ec50 tx=0x7fc504009430 comp rx=0 tx=0).stop 2022-03-21T14:06:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 shutdown_connections 2022-03-21T14:06:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c0fbfb0 0x7fc50c10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc4f4096d10 0x7fc4f40991c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc50c0fdbe0 0x7fc50c114500 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 --2- 172.21.15.2:0/3314633116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc50c0fd250 0x7fc50c10d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 >> 172.21.15.2:0/3314633116 conn(0x7fc50c0f7760 msgr2=0x7fc50c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 shutdown_connections 2022-03-21T14:06:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.339+0000 7fc51085b700 1 -- 172.21.15.2:0/3314633116 wait complete. 2022-03-21T14:06:55.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:06:55.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 -- 172.21.15.2:0/1425562314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fd250 msgr2=0x7fe6340fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 --2- 172.21.15.2:0/1425562314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fd250 0x7fe6340fd6a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6200098e0 tx=0x7fe620005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:55.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 -- 172.21.15.2:0/1425562314 shutdown_connections 2022-03-21T14:06:55.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 --2- 172.21.15.2:0/1425562314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe6340fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 --2- 172.21.15.2:0/1425562314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fdbe0 0x7fe6341044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 --2- 172.21.15.2:0/1425562314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fd250 0x7fe6340fd6a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 -- 172.21.15.2:0/1425562314 >> 172.21.15.2:0/1425562314 conn(0x7fe6340f7760 msgr2=0x7fe6340f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:55.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.570+0000 7fe639968700 1 -- 172.21.15.2:0/1425562314 shutdown_connections 2022-03-21T14:06:55.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.571+0000 7fe639968700 1 -- 172.21.15.2:0/1425562314 wait complete. 2022-03-21T14:06:55.575 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.571+0000 7fe639968700 1 Processor -- start 2022-03-21T14:06:55.575 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.571+0000 7fe639968700 1 -- start start 2022-03-21T14:06:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe63410ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 0x7fe634116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634116a70 con 0x7fe6340fdbe0 2022-03-21T14:06:55.578 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634116be0 con 0x7fe6340fbfb0 2022-03-21T14:06:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639968700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634116da0 con 0x7fe6340fd250 2022-03-21T14:06:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe633fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe639167700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 0x7fe634116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe633fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39622/0 (socket says 172.21.15.2:39622) 2022-03-21T14:06:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe633fff700 1 -- 172.21.15.2:0/1272646 learned_addr learned my addr 172.21.15.2:0/1272646 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.572+0000 7fe638966700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe63410ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 -- 172.21.15.2:0/1272646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 msgr2=0x7fe63410f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 -- 172.21.15.2:0/1272646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 msgr2=0x7fe63410ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe63410ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 -- 172.21.15.2:0/1272646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6200093f0 con 0x7fe6340fdbe0 2022-03-21T14:06:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe633fff700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe638966700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe63410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:55.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639167700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 0x7fe634116530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00eb50 tx=0x7fe62c009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:55.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c00c9d0 con 0x7fe6340fdbe0 2022-03-21T14:06:55.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe62c003840 con 0x7fe6340fdbe0 2022-03-21T14:06:55.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe634117080 con 0x7fe6340fdbe0 2022-03-21T14:06:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0044d0 con 0x7fe6340fdbe0 2022-03-21T14:06:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.573+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6341175f0 con 0x7fe6340fdbe0 2022-03-21T14:06:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.574+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6341077a0 con 0x7fe6340fdbe0 2022-03-21T14:06:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.576+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fe62c00c040 con 0x7fe6340fdbe0 2022-03-21T14:06:55.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.577+0000 7fe6317fa700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 0x7fe624099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:55.585 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.578+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fe62c014070 con 0x7fe6340fdbe0 2022-03-21T14:06:55.585 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.578+0000 7fe638966700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 0x7fe624099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:55.585 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.579+0000 7fe638966700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 0x7fe624099240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6340fd0b0 tx=0x7fe628005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:55.585 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.582+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe62c08d040 con 0x7fe6340fdbe0 2022-03-21T14:06:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-03-21T14:06:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: osdmap e80: 12 total, 12 up, 12 in 2022-03-21T14:06:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: pgmap v267: 97 pgs: 11 creating+peering, 1 active+clean, 85 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1784889509' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-03-21T14:06:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: osdmap e81: 12 total, 12 up, 12 in 2022-03-21T14:06:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-03-21T14:06:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-03-21T14:06:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-03-21T14:06:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-03-21T14:06:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: osdmap e80: 12 total, 12 up, 12 in 2022-03-21T14:06:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: pgmap v267: 97 pgs: 11 creating+peering, 1 active+clean, 85 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1784889509' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-03-21T14:06:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: osdmap e81: 12 total, 12 up, 12 in 2022-03-21T14:06:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-03-21T14:06:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-03-21T14:06:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-03-21T14:06:55.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4279350523' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-03-21T14:06:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: osdmap e80: 12 total, 12 up, 12 in 2022-03-21T14:06:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: pgmap v267: 97 pgs: 11 creating+peering, 1 active+clean, 85 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:06:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1784889509' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-03-21T14:06:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-03-21T14:06:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: osdmap e81: 12 total, 12 up, 12 in 2022-03-21T14:06:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-03-21T14:06:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-03-21T14:06:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-03-21T14:06:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:06:55.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.769+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe634107930 con 0x7fe6340fdbe0 2022-03-21T14:06:55.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.771+0000 7fe6317fa700 1 -- 172.21.15.2:0/1272646 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+21223 (secure 0 0 0) 0x7fe634107930 con 0x7fe6340fdbe0 2022-03-21T14:06:55.772 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:54.837312+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:06:55.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 msgr2=0x7fe624099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 0x7fe624099240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6340fd0b0 tx=0x7fe628005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 msgr2=0x7fe634116530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:55.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 0x7fe634116530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00eb50 tx=0x7fe62c009430 comp rx=0 tx=0).stop 2022-03-21T14:06:55.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 shutdown_connections 2022-03-21T14:06:55.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6340fbfb0 0x7fe63410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe624096d90 0x7fe624099240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6340fdbe0 0x7fe634116530 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 --2- 172.21.15.2:0/1272646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6340fd250 0x7fe63410f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:55.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.773+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 >> 172.21.15.2:0/1272646 conn(0x7fe6340f7760 msgr2=0x7fe6340f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:55.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.774+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 shutdown_connections 2022-03-21T14:06:55.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:55.774+0000 7fe639968700 1 -- 172.21.15.2:0/1272646 wait complete. 2022-03-21T14:06:55.800 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-03-21T14:06:56.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 -- 172.21.15.2:0/1814623490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fbfd0 msgr2=0x7f86e40fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:56.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 --2- 172.21.15.2:0/1814623490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fbfd0 0x7f86e40fc3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098e0 tx=0x7f86d4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:56.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 -- 172.21.15.2:0/1814623490 shutdown_connections 2022-03-21T14:06:56.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 --2- 172.21.15.2:0/1814623490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fbfd0 0x7f86e40fc3e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 --2- 172.21.15.2:0/1814623490 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e41044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 --2- 172.21.15.2:0/1814623490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fd270 0x7f86e40fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 -- 172.21.15.2:0/1814623490 >> 172.21.15.2:0/1814623490 conn(0x7f86e40f7780 msgr2=0x7f86e40f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:56.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.012+0000 7f86e90a7700 1 -- 172.21.15.2:0/1814623490 shutdown_connections 2022-03-21T14:06:56.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.013+0000 7f86e90a7700 1 -- 172.21.15.2:0/1814623490 wait complete. 2022-03-21T14:06:56.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.013+0000 7f86e90a7700 1 Processor -- start 2022-03-21T14:06:56.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.013+0000 7f86e90a7700 1 -- start start 2022-03-21T14:06:56.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:56.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fd270 0x7f86e406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:56.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:56.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e406ff70 con 0x7f86e40fbfd0 2022-03-21T14:06:56.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40700e0 con 0x7f86e40fd270 2022-03-21T14:06:56.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e90a7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e4117010 con 0x7f86e40fdc00 2022-03-21T14:06:56.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:56.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e88a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:56.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43770/0 (socket says 172.21.15.2:43770) 2022-03-21T14:06:56.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e3fff700 1 -- 172.21.15.2:0/671979875 learned_addr learned my addr 172.21.15.2:0/671979875 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:56.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e88a6700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39626/0 (socket says 172.21.15.2:39626) 2022-03-21T14:06:56.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.014+0000 7f86e37fe700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fd270 0x7f86e406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:56.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 -- 172.21.15.2:0/671979875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 msgr2=0x7f86e406f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:56.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 -- 172.21.15.2:0/671979875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fd270 msgr2=0x7f86e406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:56.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fd270 0x7f86e406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 -- 172.21.15.2:0/671979875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40093f0 con 0x7f86e40fbfd0 2022-03-21T14:06:56.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e88a6700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:06:56.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e3fff700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f86d400f790 tx=0x7f86d4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:56.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d401d070 con 0x7f86e40fbfd0 2022-03-21T14:06:56.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f86d4005740 con 0x7f86e40fbfd0 2022-03-21T14:06:56.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e41171b0 con 0x7f86e40fbfd0 2022-03-21T14:06:56.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.015+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4017a80 con 0x7f86e40fbfd0 2022-03-21T14:06:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.016+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e4117550 con 0x7f86e40fbfd0 2022-03-21T14:06:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.016+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e41077e0 con 0x7f86e40fbfd0 2022-03-21T14:06:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.018+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f86d4029030 con 0x7f86e40fbfd0 2022-03-21T14:06:56.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.019+0000 7f86e0ff9700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 0x7f86cc0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:56.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.019+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f86d40c0d10 con 0x7f86e40fbfd0 2022-03-21T14:06:56.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.020+0000 7f86e37fe700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 0x7f86cc0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:56.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.020+0000 7f86e37fe700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 0x7f86cc0991c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f86d8005fd0 tx=0x7f86d8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:56.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.024+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f86d40c6020 con 0x7f86e40fbfd0 2022-03-21T14:06:56.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.259+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f86e4117770 con 0x7f86e40fbfd0 2022-03-21T14:06:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3314633116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1272646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: osdmap e82: 12 total, 12 up, 12 in 2022-03-21T14:06:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:06:56.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:56 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-03-21T14:06:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3314633116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1272646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: osdmap e82: 12 total, 12 up, 12 in 2022-03-21T14:06:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:06:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:56 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-03-21T14:06:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3314633116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1272646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:06:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: osdmap e82: 12 total, 12 up, 12 in 2022-03-21T14:06:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:06:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:56 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-03-21T14:06:56.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.853+0000 7f86e0ff9700 1 -- 172.21.15.2:0/671979875 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f86e4117770 con 0x7f86e40fbfd0 2022-03-21T14:06:56.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.855+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 msgr2=0x7f86cc0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:56.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.855+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 0x7f86cc0991c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f86d8005fd0 tx=0x7f86d8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.855+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 msgr2=0x7f86e4070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:56.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.855+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f86d400f790 tx=0x7f86d4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:56.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 shutdown_connections 2022-03-21T14:06:56.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f86cc096d10 0x7f86cc0991c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86e40fdc00 0x7f86e406f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86e40fbfd0 0x7f86e4070cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 --2- 172.21.15.2:0/671979875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86e40fd270 0x7f86e406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 >> 172.21.15.2:0/671979875 conn(0x7f86e40f7780 msgr2=0x7f86e41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:56.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 shutdown_connections 2022-03-21T14:06:56.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:56.856+0000 7f86e90a7700 1 -- 172.21.15.2:0/671979875 wait complete. 2022-03-21T14:06:56.884 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2022-03-21T14:06:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.094+0000 7fa58993b700 1 -- 172.21.15.2:0/1213416058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa584070e60 msgr2=0x7fa58406f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.094+0000 7fa58993b700 1 --2- 172.21.15.2:0/1213416058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa584070e60 0x7fa58406f360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa57800b820 tx=0x7fa578005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 -- 172.21.15.2:0/1213416058 shutdown_connections 2022-03-21T14:06:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 --2- 172.21.15.2:0/1213416058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa584070e60 0x7fa58406f360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 --2- 172.21.15.2:0/1213416058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584103fb0 0x7fa584106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 --2- 172.21.15.2:0/1213416058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa58406f8a0 0x7fa58406fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 -- 172.21.15.2:0/1213416058 >> 172.21.15.2:0/1213416058 conn(0x7fa5840f77f0 msgr2=0x7fa5840f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 -- 172.21.15.2:0/1213416058 shutdown_connections 2022-03-21T14:06:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 -- 172.21.15.2:0/1213416058 wait complete. 2022-03-21T14:06:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.095+0000 7fa58993b700 1 Processor -- start 2022-03-21T14:06:57.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 -- start start 2022-03-21T14:06:57.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58406f8a0 0x7fa58410ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:57.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:57.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa584103fb0 0x7fa584116550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:57.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584116a90 con 0x7fa584103fb0 2022-03-21T14:06:57.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584116c00 con 0x7fa58406f8a0 2022-03-21T14:06:57.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58993b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584116dc0 con 0x7fa584070e60 2022-03-21T14:06:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa583fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa58913a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa584103fb0 0x7fa584116550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa583fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39632/0 (socket says 172.21.15.2:39632) 2022-03-21T14:06:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa583fff700 1 -- 172.21.15.2:0/322248359 learned_addr learned my addr 172.21.15.2:0/322248359 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.096+0000 7fa588939700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58406f8a0 0x7fa58410ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 -- 172.21.15.2:0/322248359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58406f8a0 msgr2=0x7fa58410ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58406f8a0 0x7fa58410ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 -- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa584103fb0 msgr2=0x7fa584116550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa584103fb0 0x7fa584116550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 -- 172.21.15.2:0/322248359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57800b360 con 0x7fa584070e60 2022-03-21T14:06:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa583fff700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa57000ea80 tx=0x7fa570005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57000c7c0 con 0x7fa584070e60 2022-03-21T14:06:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa57000cdd0 con 0x7fa584070e60 2022-03-21T14:06:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570018420 con 0x7fa584070e60 2022-03-21T14:06:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5841170a0 con 0x7fa584070e60 2022-03-21T14:06:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.097+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5841175c0 con 0x7fa584070e60 2022-03-21T14:06:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.098+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa584109a50 con 0x7fa584070e60 2022-03-21T14:06:57.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.100+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa57000c040 con 0x7fa584070e60 2022-03-21T14:06:57.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.101+0000 7fa5817fa700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 0x7fa5740991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:57.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.101+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa570014070 con 0x7fa584070e60 2022-03-21T14:06:57.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.101+0000 7fa588939700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 0x7fa5740991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:57.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.102+0000 7fa588939700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 0x7fa5740991c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa57800be90 tx=0x7fa578005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:57.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.104+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fa57008cdd0 con 0x7fa584070e60 2022-03-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:57.339+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fa584102580 con 0x7fa584070e60 2022-03-21T14:06:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:57 smithi002 ceph-mon[17167]: pgmap v270: 97 pgs: 19 creating+peering, 36 active+clean, 42 unknown; 578 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2022-03-21T14:06:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:57 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-03-21T14:06:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:57 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:57 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/322248359' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:57 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:57 smithi016 ceph-mon[22010]: pgmap v270: 97 pgs: 19 creating+peering, 36 active+clean, 42 unknown; 578 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2022-03-21T14:06:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:57 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-03-21T14:06:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:57 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/322248359' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:57 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.224 INFO:journalctl@ceph.mds.l.smithi079.stdout:Mar 21 14:06:58 smithi079 conmon[50193]: 2022-03-21T14:06:58.053+0000 7f1060cfe700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-03-21T14:06:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:57 smithi079 ceph-mon[23158]: pgmap v270: 97 pgs: 19 creating+peering, 36 active+clean, 42 unknown; 578 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2022-03-21T14:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:57 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/671979875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-03-21T14:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:57 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/322248359' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:57 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-03-21T14:06:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.336+0000 7fa5817fa700 1 -- 172.21.15.2:0/322248359 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7fa5700903f0 con 0x7fa584070e60 2022-03-21T14:06:58.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 msgr2=0x7fa5740991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:58.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 0x7fa5740991c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa57800be90 tx=0x7fa578005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 msgr2=0x7fa58410f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:58.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa57000ea80 tx=0x7fa570005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:58.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 shutdown_connections 2022-03-21T14:06:58.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58406f8a0 0x7fa58410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa574096d10 0x7fa5740991c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa584103fb0 0x7fa584116550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 --2- 172.21.15.2:0/322248359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa584070e60 0x7fa58410f4c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 >> 172.21.15.2:0/322248359 conn(0x7fa5840f77f0 msgr2=0x7fa5840f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:58.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 shutdown_connections 2022-03-21T14:06:58.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.340+0000 7fa58993b700 1 -- 172.21.15.2:0/322248359 wait complete. 2022-03-21T14:06:58.369 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-03-21T14:06:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 -- 172.21.15.2:0/1960312416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8065160 msgr2=0x7fdef80fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 --2- 172.21.15.2:0/1960312416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8065160 0x7fdef80fec30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdef40098e0 tx=0x7fdef4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 -- 172.21.15.2:0/1960312416 shutdown_connections 2022-03-21T14:06:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 --2- 172.21.15.2:0/1960312416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8065160 0x7fdef80fec30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 --2- 172.21.15.2:0/1960312416 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8101c20 0x7fdef81040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 --2- 172.21.15.2:0/1960312416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 0x7fdef8101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 -- 172.21.15.2:0/1960312416 >> 172.21.15.2:0/1960312416 conn(0x7fdef80f66a0 msgr2=0x7fdef80f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 -- 172.21.15.2:0/1960312416 shutdown_connections 2022-03-21T14:06:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.581+0000 7fdefe510700 1 -- 172.21.15.2:0/1960312416 wait complete. 2022-03-21T14:06:58.587 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.582+0000 7fdefe510700 1 Processor -- start 2022-03-21T14:06:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.582+0000 7fdefe510700 1 -- start start 2022-03-21T14:06:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 0x7fdef810f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 0x7fdef8116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8116a60 con 0x7fdef80ff170 2022-03-21T14:06:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8116bd0 con 0x7fdef8101c20 2022-03-21T14:06:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefe510700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8116d90 con 0x7fdef8065160 2022-03-21T14:06:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:58.590 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39636/0 (socket says 172.21.15.2:39636) 2022-03-21T14:06:58.590 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 -- 172.21.15.2:0/3018002131 learned_addr learned my addr 172.21.15.2:0/3018002131 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:58.590 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefcd0d700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 0x7fdef810f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:58.590 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefdd0f700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 0x7fdef8116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:58.591 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 -- 172.21.15.2:0/3018002131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 msgr2=0x7fdef8116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:58.591 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 0x7fdef8116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.591 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 -- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 msgr2=0x7fdef810f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:58.591 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 0x7fdef810f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:58.591 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefd50e700 1 -- 172.21.15.2:0/3018002131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40093f0 con 0x7fdef8065160 2022-03-21T14:06:58.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.583+0000 7fdefdd0f700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 0x7fdef8116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:06:58.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdefd50e700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdef40093c0 tx=0x7fdef4005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:58.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef401d070 con 0x7fdef8065160 2022-03-21T14:06:58.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fdef4005040 con 0x7fdef8065160 2022-03-21T14:06:58.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef8117010 con 0x7fdef8065160 2022-03-21T14:06:58.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4017a20 con 0x7fdef8065160 2022-03-21T14:06:58.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.584+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81174f0 con 0x7fdef8065160 2022-03-21T14:06:58.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.585+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef81077b0 con 0x7fdef8065160 2022-03-21T14:06:58.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.587+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fdef4029030 con 0x7fdef8065160 2022-03-21T14:06:58.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.588+0000 7fdeedffb700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 0x7fdee4099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:58.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.588+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fdef40c09f0 con 0x7fdef8065160 2022-03-21T14:06:58.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.588+0000 7fdefcd0d700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 0x7fdee4099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:58.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.589+0000 7fdefcd0d700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 0x7fdee4099240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdef8064f30 tx=0x7fdee8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:58.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.590+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fdef408ddb0 con 0x7fdef8065160 2022-03-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:58.824+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fdef80fbfd0 con 0x7fdef8065160 2022-03-21T14:06:59.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.492+0000 7fdeedffb700 1 -- 172.21.15.2:0/3018002131 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7fdef40913d0 con 0x7fdef8065160 2022-03-21T14:06:59.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 msgr2=0x7fdee4099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 0x7fdee4099240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdef8064f30 tx=0x7fdee8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 msgr2=0x7fdef810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdef40093c0 tx=0x7fdef4005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:59.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 shutdown_connections 2022-03-21T14:06:59.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef8101c20 0x7fdef8116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fdee4096d90 0x7fdee4099240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdef8065160 0x7fdef810ef50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 --2- 172.21.15.2:0/3018002131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef80ff170 0x7fdef810f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 >> 172.21.15.2:0/3018002131 conn(0x7fdef80f66a0 msgr2=0x7fdef80f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:59.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 shutdown_connections 2022-03-21T14:06:59.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.495+0000 7fdefe510700 1 -- 172.21.15.2:0/3018002131 wait complete. 2022-03-21T14:06:59.525 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:06:59.588 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-03-21T14:06:59.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:59.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:06:59.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: pgmap v271: 97 pgs: 11 creating+peering, 73 active+clean, 13 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s rd, 2.6 KiB/s wr, 8 op/s 2022-03-21T14:06:59.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3018002131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:06:59 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-03-21T14:06:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:06:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: pgmap v271: 97 pgs: 11 creating+peering, 73 active+clean, 13 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s rd, 2.6 KiB/s wr, 8 op/s 2022-03-21T14:06:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3018002131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:06:59 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-03-21T14:06:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-03-21T14:06:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:06:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: pgmap v271: 97 pgs: 11 creating+peering, 73 active+clean, 13 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 192 B/s rd, 2.6 KiB/s wr, 8 op/s 2022-03-21T14:06:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3018002131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:06:59 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-03-21T14:06:59.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 -- 172.21.15.2:0/2897302193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 msgr2=0x7f8d300feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/2897302193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300feb70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8d200098e0 tx=0x7f8d20005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:59.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 -- 172.21.15.2:0/2897302193 shutdown_connections 2022-03-21T14:06:59.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/2897302193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 0x7f8d30104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/2897302193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300feb70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/2897302193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 0x7f8d30101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 -- 172.21.15.2:0/2897302193 >> 172.21.15.2:0/2897302193 conn(0x7f8d300f6640 msgr2=0x7f8d300f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:59.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 -- 172.21.15.2:0/2897302193 shutdown_connections 2022-03-21T14:06:59.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.732+0000 7f8d34d6f700 1 -- 172.21.15.2:0/2897302193 wait complete. 2022-03-21T14:06:59.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.733+0000 7f8d34d6f700 1 Processor -- start 2022-03-21T14:06:59.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.733+0000 7f8d34d6f700 1 -- start start 2022-03-21T14:06:59.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.733+0000 7f8d34d6f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:59.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.733+0000 7f8d34d6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 0x7f8d300faa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:59.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d34d6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 0x7f8d300fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:59.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d34d6f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300fb6c0 con 0x7f8d300ff140 2022-03-21T14:06:59.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d34d6f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300fb830 con 0x7f8d30101c10 2022-03-21T14:06:59.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d34d6f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30117030 con 0x7f8d30065200 2022-03-21T14:06:59.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:59.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39640/0 (socket says 172.21.15.2:39640) 2022-03-21T14:06:59.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 -- 172.21.15.2:0/3840889897 learned_addr learned my addr 172.21.15.2:0/3840889897 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:06:59.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2effd700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 0x7f8d300faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:59.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2ffff700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 0x7f8d300fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:59.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 -- 172.21.15.2:0/3840889897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 msgr2=0x7f8d300fafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 0x7f8d300fafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 -- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 msgr2=0x7f8d300faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 0x7f8d300faa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 -- 172.21.15.2:0/3840889897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200093f0 con 0x7f8d30065200 2022-03-21T14:06:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.734+0000 7f8d2f7fe700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8d200093c0 tx=0x7f8d20005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.735+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2001d070 con 0x7f8d30065200 2022-03-21T14:06:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.735+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8d20000c90 con 0x7f8d30065200 2022-03-21T14:06:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.735+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d301171d0 con 0x7f8d30065200 2022-03-21T14:06:59.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.735+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20017990 con 0x7f8d30065200 2022-03-21T14:06:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.735+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30117570 con 0x7f8d30065200 2022-03-21T14:06:59.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.736+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d30107790 con 0x7f8d30065200 2022-03-21T14:06:59.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.738+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f8d20028030 con 0x7f8d30065200 2022-03-21T14:06:59.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.739+0000 7f8d17fff700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 0x7f8d180991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:06:59.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.739+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f8d200c0970 con 0x7f8d30065200 2022-03-21T14:06:59.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.739+0000 7f8d2effd700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 0x7f8d180991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:06:59.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.740+0000 7f8d2effd700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 0x7f8d180991c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d30064fd0 tx=0x7f8d24005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:06:59.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.741+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f8d2008dca0 con 0x7f8d30065200 2022-03-21T14:06:59.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.968+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d300f7ff0 con 0x7f8d30065200 2022-03-21T14:06:59.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.969+0000 7f8d17fff700 1 -- 172.21.15.2:0/3840889897 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10650 (secure 0 0 0) 0x7f8d200912c0 con 0x7f8d30065200 2022-03-21T14:06:59.970 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:06:59.970 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-03-21T14:06:54.837756+0000","modified":"2022-03-21T14:06:59.487834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/1907413140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1907413140},{"type":"v1","addr":"172.21.15.79:6839","nonce":1907413140}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-03-21T14:06:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 msgr2=0x7f8d180991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 0x7f8d180991c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d30064fd0 tx=0x7f8d24005eb0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 msgr2=0x7f8d300fc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:06:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8d200093c0 tx=0x7f8d20005f70 comp rx=0 tx=0).stop 2022-03-21T14:06:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 shutdown_connections 2022-03-21T14:06:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d30101c10 0x7f8d300fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8d18096d10 0x7f8d180991c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d30065200 0x7f8d300fc410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 --2- 172.21.15.2:0/3840889897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d300ff140 0x7f8d300faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:06:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.971+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 >> 172.21.15.2:0/3840889897 conn(0x7f8d300f6640 msgr2=0x7f8d300fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:06:59.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.972+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 shutdown_connections 2022-03-21T14:06:59.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:06:59.972+0000 7f8d34d6f700 1 -- 172.21.15.2:0/3840889897 wait complete. 2022-03-21T14:06:59.975 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 20 2022-03-21T14:06:59.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:07:00.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.201+0000 7fa6acd14700 1 -- 172.21.15.2:0/2432847367 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fd250 msgr2=0x7fa6a80fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.201+0000 7fa6acd14700 1 --2- 172.21.15.2:0/2432847367 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fd250 0x7fa6a80fd6a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0098e0 tx=0x7fa69c005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:00.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.201+0000 7fa6acd14700 1 -- 172.21.15.2:0/2432847367 shutdown_connections 2022-03-21T14:07:00.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.201+0000 7fa6acd14700 1 --2- 172.21.15.2:0/2432847367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a80fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.201+0000 7fa6acd14700 1 --2- 172.21.15.2:0/2432847367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.202+0000 7fa6acd14700 1 --2- 172.21.15.2:0/2432847367 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fd250 0x7fa6a80fd6a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.202+0000 7fa6acd14700 1 -- 172.21.15.2:0/2432847367 >> 172.21.15.2:0/2432847367 conn(0x7fa6a80f7760 msgr2=0x7fa6a80f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:00.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.202+0000 7fa6acd14700 1 -- 172.21.15.2:0/2432847367 shutdown_connections 2022-03-21T14:07:00.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.202+0000 7fa6acd14700 1 -- 172.21.15.2:0/2432847367 wait complete. 2022-03-21T14:07:00.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.202+0000 7fa6acd14700 1 Processor -- start 2022-03-21T14:07:00.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 -- start start 2022-03-21T14:07:00.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fd250 0x7fa6a810d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81142b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8114830 con 0x7fa6a80fd250 2022-03-21T14:07:00.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a81149a0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6acd14700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8114b60 con 0x7fa6a80fdbe0 2022-03-21T14:07:00.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6a6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fd250 0x7fa6a810d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48374/0 (socket says 172.21.15.2:48374) 2022-03-21T14:07:00.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6a77fe700 1 -- 172.21.15.2:0/70199353 learned_addr learned my addr 172.21.15.2:0/70199353 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:00.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.203+0000 7fa6a7fff700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 -- 172.21.15.2:0/70199353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 msgr2=0x7fa6a81142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 -- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fd250 msgr2=0x7fa6a810d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fd250 0x7fa6a810d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 -- 172.21.15.2:0/70199353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0093f0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a7fff700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:00.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa6a77fe700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa69800f860 tx=0x7fa698005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:00.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.204+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698004020 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.205+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa698004180 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.205+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a8114e40 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.205+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980039f0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.205+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a81152d0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.206+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a81077a0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.208+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa698022060 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.209+0000 7fa68ffff700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 0x7fa690099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.209+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa6980bf500 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.209+0000 7fa6a6ffd700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 0x7fa690099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.210+0000 7fa6a6ffd700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 0x7fa690099240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009f50 tx=0x7fa69c005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:00.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.211+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fa69808c920 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.402+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6a802acf0 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.403+0000 7fa68ffff700 1 -- 172.21.15.2:0/70199353 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21210 (secure 0 0 0) 0x7fa69808ff40 con 0x7fa6a80fbfb0 2022-03-21T14:07:00.404 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:00.404 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:55.839803+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:07:00.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.405+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 msgr2=0x7fa690099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.405+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 0x7fa690099240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009f50 tx=0x7fa69c005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:00.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 msgr2=0x7fa6a810cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa69800f860 tx=0x7fa698005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 shutdown_connections 2022-03-21T14:07:00.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a80fbfb0 0x7fa6a810cd20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa690096d90 0x7fa690099240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6a80fdbe0 0x7fa6a81142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 --2- 172.21.15.2:0/70199353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6a80fd250 0x7fa6a810d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 >> 172.21.15.2:0/70199353 conn(0x7fa6a80f7760 msgr2=0x7fa6a8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:00.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 shutdown_connections 2022-03-21T14:07:00.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.406+0000 7fa6acd14700 1 -- 172.21.15.2:0/70199353 wait complete. 2022-03-21T14:07:00.432 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-03-21T14:07:00.432 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-03-21T14:07:00.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 -- 172.21.15.2:0/3159236424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc8066100 msgr2=0x7f3fc80febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/3159236424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc8066100 0x7f3fc80febb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0098e0 tx=0x7f3fbc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:00.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 -- 172.21.15.2:0/3159236424 shutdown_connections 2022-03-21T14:07:00.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/3159236424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8101c30 0x7f3fc81040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/3159236424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc8066100 0x7f3fc80febb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/3159236424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc80ff180 0x7f3fc81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 -- 172.21.15.2:0/3159236424 >> 172.21.15.2:0/3159236424 conn(0x7f3fc80f66a0 msgr2=0x7f3fc80f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:00.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 -- 172.21.15.2:0/3159236424 shutdown_connections 2022-03-21T14:07:00.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.638+0000 7f3fcdc31700 1 -- 172.21.15.2:0/3159236424 wait complete. 2022-03-21T14:07:00.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 Processor -- start 2022-03-21T14:07:00.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 -- start start 2022-03-21T14:07:00.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc80ff180 0x7f3fc80faab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 0x7f3fc80faff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc80fb700 con 0x7f3fc8066100 2022-03-21T14:07:00.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc80fb870 con 0x7f3fc8101c30 2022-03-21T14:07:00.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.639+0000 7f3fcdc31700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8116e70 con 0x7f3fc80ff180 2022-03-21T14:07:00.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fcd430700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 0x7f3fc80faff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43792/0 (socket says 172.21.15.2:43792) 2022-03-21T14:07:00.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 -- 172.21.15.2:0/4251031376 learned_addr learned my addr 172.21.15.2:0/4251031376 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:00.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fc7fff700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc80ff180 0x7f3fc80faab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 -- 172.21.15.2:0/4251031376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc80ff180 msgr2=0x7f3fc80faab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc80ff180 0x7f3fc80faab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 -- 172.21.15.2:0/4251031376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 msgr2=0x7f3fc80faff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 0x7f3fc80faff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 -- 172.21.15.2:0/4251031376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0093f0 con 0x7f3fc8066100 2022-03-21T14:07:00.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fcd430700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 0x7f3fc80faff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:00.649 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.640+0000 7f3fccc2f700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0093c0 tx=0x7f3fbc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:00.649 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.641+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc01d070 con 0x7f3fc8066100 2022-03-21T14:07:00.649 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.641+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3fbc000c90 con 0x7f3fc8066100 2022-03-21T14:07:00.649 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.641+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc017990 con 0x7f3fc8066100 2022-03-21T14:07:00.650 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.641+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc8117010 con 0x7f3fc8066100 2022-03-21T14:07:00.650 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.641+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc8117430 con 0x7f3fc8066100 2022-03-21T14:07:00.650 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.642+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc81077c0 con 0x7f3fc8066100 2022-03-21T14:07:00.650 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.644+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f3fbc028030 con 0x7f3fc8066100 2022-03-21T14:07:00.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.645+0000 7f3fc57fa700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 0x7f3fb8099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:00.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.645+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f3fbc0c0c20 con 0x7f3fc8066100 2022-03-21T14:07:00.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.645+0000 7f3fc7fff700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 0x7f3fb8099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:00.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.646+0000 7f3fc7fff700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 0x7f3fb8099240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8065ed0 tx=0x7f3fb4005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:00.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.648+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f3fbc08dfe0 con 0x7f3fc8066100 2022-03-21T14:07:00.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.831+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fc802ac30 con 0x7f3fb8096d90 2022-03-21T14:07:00.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.845+0000 7f3fc57fa700 1 -- 172.21.15.2:0/4251031376 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+333193 (secure 0 0 0) 0x7f3fc802ac30 con 0x7f3fb8096d90 2022-03-21T14:07:00.851 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:00.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 msgr2=0x7f3fb8099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 0x7f3fb8099240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8065ed0 tx=0x7f3fb4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 msgr2=0x7f3fc80fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:00.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0093c0 tx=0x7f3fbc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:00.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 shutdown_connections 2022-03-21T14:07:00.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8101c30 0x7f3fc80faff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3fb8096d90 0x7f3fb8099240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fc8066100 0x7f3fc80fc450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 --2- 172.21.15.2:0/4251031376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fc80ff180 0x7f3fc80faab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:00.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 >> 172.21.15.2:0/4251031376 conn(0x7f3fc80f66a0 msgr2=0x7f3fc80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:00.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 shutdown_connections 2022-03-21T14:07:00.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:00.850+0000 7f3fcdc31700 1 -- 172.21.15.2:0/4251031376 wait complete. 2022-03-21T14:07:00.858 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-03-21T14:07:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:00 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-03-21T14:07:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:00 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:07:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:00 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3840889897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:00 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/70199353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:00.876 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2022-03-21T14:07:00.392583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77944,"kb_used_data":11928,"kb_used_omap":0,"kb_used_meta":65920,"kb_avail":1124863880,"statfs":{"total":1151940427776,"available":1151860613120,"internally_reserved":0,"allocated":12214272,"data_stored":6618615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67502080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.096556"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872656+0000","last_change":"2022-03-21T14:06:54.869479+0000","last_active":"2022-03-21T14:06:54.872656+0000","last_peered":"2022-03-21T14:06:54.872656+0000","last_clean":"2022-03-21T14:06:54.872656+0000","last_became_active":"2022-03-21T14:06:54.869079+0000","last_became_peered":"2022-03-21T14:06:54.869079+0000","last_unstale":"2022-03-21T14:06:54.872656+0000","last_undegraded":"2022-03-21T14:06:54.872656+0000","last_fullsized":"2022-03-21T14:06:54.872656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:09:19.289483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885992+0000","last_change":"2022-03-21T14:06:54.884544+0000","last_active":"2022-03-21T14:06:54.885992+0000","last_peered":"2022-03-21T14:06:54.885992+0000","last_clean":"2022-03-21T14:06:54.885992+0000","last_became_active":"2022-03-21T14:06:54.884210+0000","last_became_peered":"2022-03-21T14:06:54.884210+0000","last_unstale":"2022-03-21T14:06:54.885992+0000","last_undegraded":"2022-03-21T14:06:54.885992+0000","last_fullsized":"2022-03-21T14:06:54.885992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:48:52.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885007+0000","last_change":"2022-03-21T14:06:54.880620+0000","last_active":"2022-03-21T14:06:54.885007+0000","last_peered":"2022-03-21T14:06:54.885007+0000","last_clean":"2022-03-21T14:06:54.885007+0000","last_became_active":"2022-03-21T14:06:54.880297+0000","last_became_peered":"2022-03-21T14:06:54.880297+0000","last_unstale":"2022-03-21T14:06:54.885007+0000","last_undegraded":"2022-03-21T14:06:54.885007+0000","last_fullsized":"2022-03-21T14:06:54.885007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:43:49.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863909+0000","last_change":"2022-03-21T14:06:54.861753+0000","last_active":"2022-03-21T14:06:54.863909+0000","last_peered":"2022-03-21T14:06:54.863909+0000","last_clean":"2022-03-21T14:06:54.863909+0000","last_became_active":"2022-03-21T14:06:54.860649+0000","last_became_peered":"2022-03-21T14:06:54.860649+0000","last_unstale":"2022-03-21T14:06:54.863909+0000","last_undegraded":"2022-03-21T14:06:54.863909+0000","last_fullsized":"2022-03-21T14:06:54.863909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:22:24.251084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876192+0000","last_change":"2022-03-21T14:06:54.875705+0000","last_active":"2022-03-21T14:06:54.876192+0000","last_peered":"2022-03-21T14:06:54.876192+0000","last_clean":"2022-03-21T14:06:54.876192+0000","last_became_active":"2022-03-21T14:06:54.875171+0000","last_became_peered":"2022-03-21T14:06:54.875171+0000","last_unstale":"2022-03-21T14:06:54.876192+0000","last_undegraded":"2022-03-21T14:06:54.876192+0000","last_fullsized":"2022-03-21T14:06:54.876192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:52:39.257984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.858664+0000","last_change":"2022-03-21T14:06:54.858664+0000","last_active":"2022-03-21T14:06:54.858664+0000","last_peered":"2022-03-21T14:06:54.858664+0000","last_clean":"2022-03-21T14:06:54.858664+0000","last_became_active":"2022-03-21T14:06:54.858283+0000","last_became_peered":"2022-03-21T14:06:54.858283+0000","last_unstale":"2022-03-21T14:06:54.858664+0000","last_undegraded":"2022-03-21T14:06:54.858664+0000","last_fullsized":"2022-03-21T14:06:54.858664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:58:34.398164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.857542+0000","last_change":"2022-03-21T14:06:54.856225+0000","last_active":"2022-03-21T14:06:54.857542+0000","last_peered":"2022-03-21T14:06:54.857542+0000","last_clean":"2022-03-21T14:06:54.857542+0000","last_became_active":"2022-03-21T14:06:54.855801+0000","last_became_peered":"2022-03-21T14:06:54.855801+0000","last_unstale":"2022-03-21T14:06:54.857542+0000","last_undegraded":"2022-03-21T14:06:54.857542+0000","last_fullsized":"2022-03-21T14:06:54.857542+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:26:47.895423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876168+0000","last_change":"2022-03-21T14:06:54.873142+0000","last_active":"2022-03-21T14:06:54.876168+0000","last_peered":"2022-03-21T14:06:54.876168+0000","last_clean":"2022-03-21T14:06:54.876168+0000","last_became_active":"2022-03-21T14:06:54.872788+0000","last_became_peered":"2022-03-21T14:06:54.872788+0000","last_unstale":"2022-03-21T14:06:54.876168+0000","last_undegraded":"2022-03-21T14:06:54.876168+0000","last_fullsized":"2022-03-21T14:06:54.876168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:53:53.652396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860355+0000","last_change":"2022-03-21T14:06:54.860355+0000","last_active":"2022-03-21T14:06:54.860355+0000","last_peered":"2022-03-21T14:06:54.860355+0000","last_clean":"2022-03-21T14:06:54.860355+0000","last_became_active":"2022-03-21T14:06:54.858723+0000","last_became_peered":"2022-03-21T14:06:54.858723+0000","last_unstale":"2022-03-21T14:06:54.860355+0000","last_undegraded":"2022-03-21T14:06:54.860355+0000","last_fullsized":"2022-03-21T14:06:54.860355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:34.198090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.855421+0000","last_change":"2022-03-21T14:06:54.855421+0000","last_active":"2022-03-21T14:06:54.855421+0000","last_peered":"2022-03-21T14:06:54.855421+0000","last_clean":"2022-03-21T14:06:54.855421+0000","last_became_active":"2022-03-21T14:06:54.855027+0000","last_became_peered":"2022-03-21T14:06:54.855027+0000","last_unstale":"2022-03-21T14:06:54.855421+0000","last_undegraded":"2022-03-21T14:06:54.855421+0000","last_fullsized":"2022-03-21T14:06:54.855421+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:36:14.968816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871860+0000","last_change":"2022-03-21T14:06:54.869318+0000","last_active":"2022-03-21T14:06:54.871860+0000","last_peered":"2022-03-21T14:06:54.871860+0000","last_clean":"2022-03-21T14:06:54.871860+0000","last_became_active":"2022-03-21T14:06:54.868664+0000","last_became_peered":"2022-03-21T14:06:54.868664+0000","last_unstale":"2022-03-21T14:06:54.871860+0000","last_undegraded":"2022-03-21T14:06:54.871860+0000","last_fullsized":"2022-03-21T14:06:54.871860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:29.775263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.864680+0000","last_change":"2022-03-21T14:06:54.862293+0000","last_active":"2022-03-21T14:06:54.864680+0000","last_peered":"2022-03-21T14:06:54.864680+0000","last_clean":"2022-03-21T14:06:54.864680+0000","last_became_active":"2022-03-21T14:06:54.861926+0000","last_became_peered":"2022-03-21T14:06:54.861926+0000","last_unstale":"2022-03-21T14:06:54.864680+0000","last_undegraded":"2022-03-21T14:06:54.864680+0000","last_fullsized":"2022-03-21T14:06:54.864680+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:16:27.088619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861063+0000","last_change":"2022-03-21T14:06:54.859631+0000","last_active":"2022-03-21T14:06:54.861063+0000","last_peered":"2022-03-21T14:06:54.861063+0000","last_clean":"2022-03-21T14:06:54.861063+0000","last_became_active":"2022-03-21T14:06:54.858657+0000","last_became_peered":"2022-03-21T14:06:54.858657+0000","last_unstale":"2022-03-21T14:06:54.861063+0000","last_undegraded":"2022-03-21T14:06:54.861063+0000","last_fullsized":"2022-03-21T14:06:54.861063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:03.821600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871061+0000","last_change":"2022-03-21T14:06:54.866492+0000","last_active":"2022-03-21T14:06:54.871061+0000","last_peered":"2022-03-21T14:06:54.871061+0000","last_clean":"2022-03-21T14:06:54.871061+0000","last_became_active":"2022-03-21T14:06:54.866091+0000","last_became_peered":"2022-03-21T14:06:54.866091+0000","last_unstale":"2022-03-21T14:06:54.871061+0000","last_undegraded":"2022-03-21T14:06:54.871061+0000","last_fullsized":"2022-03-21T14:06:54.871061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:31:33.741891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.884962+0000","last_change":"2022-03-21T14:06:54.879848+0000","last_active":"2022-03-21T14:06:54.884962+0000","last_peered":"2022-03-21T14:06:54.884962+0000","last_clean":"2022-03-21T14:06:54.884962+0000","last_became_active":"2022-03-21T14:06:54.879496+0000","last_became_peered":"2022-03-21T14:06:54.879496+0000","last_unstale":"2022-03-21T14:06:54.884962+0000","last_undegraded":"2022-03-21T14:06:54.884962+0000","last_fullsized":"2022-03-21T14:06:54.884962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:18:13.350002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.877838+0000","last_change":"2022-03-21T14:06:54.877838+0000","last_active":"2022-03-21T14:06:54.877838+0000","last_peered":"2022-03-21T14:06:54.877838+0000","last_clean":"2022-03-21T14:06:54.877838+0000","last_became_active":"2022-03-21T14:06:54.877444+0000","last_became_peered":"2022-03-21T14:06:54.877444+0000","last_unstale":"2022-03-21T14:06:54.877838+0000","last_undegraded":"2022-03-21T14:06:54.877838+0000","last_fullsized":"2022-03-21T14:06:54.877838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:32.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.886488+0000","last_change":"2022-03-21T14:06:54.885978+0000","last_active":"2022-03-21T14:06:54.886488+0000","last_peered":"2022-03-21T14:06:54.886488+0000","last_clean":"2022-03-21T14:06:54.886488+0000","last_became_active":"2022-03-21T14:06:54.885645+0000","last_became_peered":"2022-03-21T14:06:54.885645+0000","last_unstale":"2022-03-21T14:06:54.886488+0000","last_undegraded":"2022-03-21T14:06:54.886488+0000","last_fullsized":"2022-03-21T14:06:54.886488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:51.003441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863262+0000","last_change":"2022-03-21T14:06:54.860687+0000","last_active":"2022-03-21T14:06:54.863262+0000","last_peered":"2022-03-21T14:06:54.863262+0000","last_clean":"2022-03-21T14:06:54.863262+0000","last_became_active":"2022-03-21T14:06:54.860315+0000","last_became_peered":"2022-03-21T14:06:54.860315+0000","last_unstale":"2022-03-21T14:06:54.863262+0000","last_undegraded":"2022-03-21T14:06:54.863262+0000","last_fullsized":"2022-03-21T14:06:54.863262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:27:56.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.869847+0000","last_change":"2022-03-21T14:06:54.867688+0000","last_active":"2022-03-21T14:06:54.869847+0000","last_peered":"2022-03-21T14:06:54.869847+0000","last_clean":"2022-03-21T14:06:54.869847+0000","last_became_active":"2022-03-21T14:06:54.866998+0000","last_became_peered":"2022-03-21T14:06:54.866998+0000","last_unstale":"2022-03-21T14:06:54.869847+0000","last_undegraded":"2022-03-21T14:06:54.869847+0000","last_fullsized":"2022-03-21T14:06:54.869847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:17:24.979917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-03-21T14:06:55.844294+0000","last_change":"2022-03-21T14:06:54.326883+0000","last_active":"2022-03-21T14:06:55.844294+0000","last_peered":"2022-03-21T14:06:55.844294+0000","last_clean":"2022-03-21T14:06:55.844294+0000","last_became_active":"2022-03-21T14:06:54.326424+0000","last_became_peered":"2022-03-21T14:06:54.326424+0000","last_unstale":"2022-03-21T14:06:55.844294+0000","last_undegraded":"2022-03-21T14:06:55.844294+0000","last_fullsized":"2022-03-21T14:06:55.844294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:39.818504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.877311+0000","last_change":"2022-03-21T14:06:54.875068+0000","last_active":"2022-03-21T14:06:54.877311+0000","last_peered":"2022-03-21T14:06:54.877311+0000","last_clean":"2022-03-21T14:06:54.877311+0000","last_became_active":"2022-03-21T14:06:54.874715+0000","last_became_peered":"2022-03-21T14:06:54.874715+0000","last_unstale":"2022-03-21T14:06:54.877311+0000","last_undegraded":"2022-03-21T14:06:54.877311+0000","last_fullsized":"2022-03-21T14:06:54.877311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:33.864263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882656+0000","last_change":"2022-03-21T14:06:54.335459+0000","last_active":"2022-03-21T14:06:54.882656+0000","last_peered":"2022-03-21T14:06:54.882656+0000","last_clean":"2022-03-21T14:06:54.882656+0000","last_became_active":"2022-03-21T14:06:54.335109+0000","last_became_peered":"2022-03-21T14:06:54.335109+0000","last_unstale":"2022-03-21T14:06:54.882656+0000","last_undegraded":"2022-03-21T14:06:54.882656+0000","last_fullsized":"2022-03-21T14:06:54.882656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:06:56.531385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885999+0000","last_change":"2022-03-21T14:06:54.884043+0000","last_active":"2022-03-21T14:06:54.885999+0000","last_peered":"2022-03-21T14:06:54.885999+0000","last_clean":"2022-03-21T14:06:54.885999+0000","last_became_active":"2022-03-21T14:06:54.883647+0000","last_became_peered":"2022-03-21T14:06:54.883647+0000","last_unstale":"2022-03-21T14:06:54.885999+0000","last_undegraded":"2022-03-21T14:06:54.885999+0000","last_fullsized":"2022-03-21T14:06:54.885999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:32:17.243588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872866+0000","last_change":"2022-03-21T14:06:54.314357+0000","last_active":"2022-03-21T14:06:54.872866+0000","last_peered":"2022-03-21T14:06:54.872866+0000","last_clean":"2022-03-21T14:06:54.872866+0000","last_became_active":"2022-03-21T14:06:54.313855+0000","last_became_peered":"2022-03-21T14:06:54.313855+0000","last_unstale":"2022-03-21T14:06:54.872866+0000","last_undegraded":"2022-03-21T14:06:54.872866+0000","last_fullsized":"2022-03-21T14:06:54.872866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:58:09.686843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861824+0000","last_change":"2022-03-21T14:06:54.861824+0000","last_active":"2022-03-21T14:06:54.861824+0000","last_peered":"2022-03-21T14:06:54.861824+0000","last_clean":"2022-03-21T14:06:54.861824+0000","last_became_active":"2022-03-21T14:06:54.859283+0000","last_became_peered":"2022-03-21T14:06:54.859283+0000","last_unstale":"2022-03-21T14:06:54.861824+0000","last_undegraded":"2022-03-21T14:06:54.861824+0000","last_fullsized":"2022-03-21T14:06:54.861824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:00:42.993237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871575+0000","last_change":"2022-03-21T14:06:54.312853+0000","last_active":"2022-03-21T14:06:54.871575+0000","last_peered":"2022-03-21T14:06:54.871575+0000","last_clean":"2022-03-21T14:06:54.871575+0000","last_became_active":"2022-03-21T14:06:54.312474+0000","last_became_peered":"2022-03-21T14:06:54.312474+0000","last_unstale":"2022-03-21T14:06:54.871575+0000","last_undegraded":"2022-03-21T14:06:54.871575+0000","last_fullsized":"2022-03-21T14:06:54.871575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:53:19.818645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863910+0000","last_change":"2022-03-21T14:06:54.859678+0000","last_active":"2022-03-21T14:06:54.863910+0000","last_peered":"2022-03-21T14:06:54.863910+0000","last_clean":"2022-03-21T14:06:54.863910+0000","last_became_active":"2022-03-21T14:06:54.858928+0000","last_became_peered":"2022-03-21T14:06:54.858928+0000","last_unstale":"2022-03-21T14:06:54.863910+0000","last_undegraded":"2022-03-21T14:06:54.863910+0000","last_fullsized":"2022-03-21T14:06:54.863910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:10.326574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872009+0000","last_change":"2022-03-21T14:06:54.314216+0000","last_active":"2022-03-21T14:06:54.872009+0000","last_peered":"2022-03-21T14:06:54.872009+0000","last_clean":"2022-03-21T14:06:54.872009+0000","last_became_active":"2022-03-21T14:06:54.313808+0000","last_became_peered":"2022-03-21T14:06:54.313808+0000","last_unstale":"2022-03-21T14:06:54.872009+0000","last_undegraded":"2022-03-21T14:06:54.872009+0000","last_fullsized":"2022-03-21T14:06:54.872009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:52:09.937102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.869625+0000","last_change":"2022-03-21T14:06:54.868066+0000","last_active":"2022-03-21T14:06:54.869625+0000","last_peered":"2022-03-21T14:06:54.869625+0000","last_clean":"2022-03-21T14:06:54.869625+0000","last_became_active":"2022-03-21T14:06:54.867711+0000","last_became_peered":"2022-03-21T14:06:54.867711+0000","last_unstale":"2022-03-21T14:06:54.869625+0000","last_undegraded":"2022-03-21T14:06:54.869625+0000","last_fullsized":"2022-03-21T14:06:54.869625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:22:29.470011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843138+0000","last_change":"2022-03-21T14:06:54.322793+0000","last_active":"2022-03-21T14:06:54.843138+0000","last_peered":"2022-03-21T14:06:54.843138+0000","last_clean":"2022-03-21T14:06:54.843138+0000","last_became_active":"2022-03-21T14:06:54.322360+0000","last_became_peered":"2022-03-21T14:06:54.322360+0000","last_unstale":"2022-03-21T14:06:54.843138+0000","last_undegraded":"2022-03-21T14:06:54.843138+0000","last_fullsized":"2022-03-21T14:06:54.843138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:15:12.220667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875260+0000","last_change":"2022-03-21T14:06:54.874475+0000","last_active":"2022-03-21T14:06:54.875260+0000","last_peered":"2022-03-21T14:06:54.875260+0000","last_clean":"2022-03-21T14:06:54.875260+0000","last_became_active":"2022-03-21T14:06:54.874113+0000","last_became_peered":"2022-03-21T14:06:54.874113+0000","last_unstale":"2022-03-21T14:06:54.875260+0000","last_undegraded":"2022-03-21T14:06:54.875260+0000","last_fullsized":"2022-03-21T14:06:54.875260+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:25.860831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847301+0000","last_change":"2022-03-21T14:06:54.331708+0000","last_active":"2022-03-21T14:06:54.847301+0000","last_peered":"2022-03-21T14:06:54.847301+0000","last_clean":"2022-03-21T14:06:54.847301+0000","last_became_active":"2022-03-21T14:06:54.331347+0000","last_became_peered":"2022-03-21T14:06:54.331347+0000","last_unstale":"2022-03-21T14:06:54.847301+0000","last_undegraded":"2022-03-21T14:06:54.847301+0000","last_fullsized":"2022-03-21T14:06:54.847301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:59:42.282136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865062+0000","last_change":"2022-03-21T14:06:54.864160+0000","last_active":"2022-03-21T14:06:54.865062+0000","last_peered":"2022-03-21T14:06:54.865062+0000","last_clean":"2022-03-21T14:06:54.865062+0000","last_became_active":"2022-03-21T14:06:54.863699+0000","last_became_peered":"2022-03-21T14:06:54.863699+0000","last_unstale":"2022-03-21T14:06:54.865062+0000","last_undegraded":"2022-03-21T14:06:54.865062+0000","last_fullsized":"2022-03-21T14:06:54.865062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:40:53.670759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843617+0000","last_change":"2022-03-21T14:06:54.314228+0000","last_active":"2022-03-21T14:06:54.843617+0000","last_peered":"2022-03-21T14:06:54.843617+0000","last_clean":"2022-03-21T14:06:54.843617+0000","last_became_active":"2022-03-21T14:06:54.313667+0000","last_became_peered":"2022-03-21T14:06:54.313667+0000","last_unstale":"2022-03-21T14:06:54.843617+0000","last_undegraded":"2022-03-21T14:06:54.843617+0000","last_fullsized":"2022-03-21T14:06:54.843617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:20:10.791422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.867382+0000","last_change":"2022-03-21T14:06:54.866898+0000","last_active":"2022-03-21T14:06:54.867382+0000","last_peered":"2022-03-21T14:06:54.867382+0000","last_clean":"2022-03-21T14:06:54.867382+0000","last_became_active":"2022-03-21T14:06:54.865450+0000","last_became_peered":"2022-03-21T14:06:54.865450+0000","last_unstale":"2022-03-21T14:06:54.867382+0000","last_undegraded":"2022-03-21T14:06:54.867382+0000","last_fullsized":"2022-03-21T14:06:54.867382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:39:42.699187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.328729+0000","last_change":"2022-03-21T14:06:54.328313+0000","last_active":"2022-03-21T14:06:54.328729+0000","last_peered":"2022-03-21T14:06:54.328729+0000","last_clean":"2022-03-21T14:06:54.328729+0000","last_became_active":"2022-03-21T14:06:54.327989+0000","last_became_peered":"2022-03-21T14:06:54.327989+0000","last_unstale":"2022-03-21T14:06:54.328729+0000","last_undegraded":"2022-03-21T14:06:54.328729+0000","last_fullsized":"2022-03-21T14:06:54.328729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:59:23.877877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861081+0000","last_change":"2022-03-21T14:06:54.861081+0000","last_active":"2022-03-21T14:06:54.861081+0000","last_peered":"2022-03-21T14:06:54.861081+0000","last_clean":"2022-03-21T14:06:54.861081+0000","last_became_active":"2022-03-21T14:06:54.860526+0000","last_became_peered":"2022-03-21T14:06:54.860526+0000","last_unstale":"2022-03-21T14:06:54.861081+0000","last_undegraded":"2022-03-21T14:06:54.861081+0000","last_fullsized":"2022-03-21T14:06:54.861081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:17:37.984659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.846891+0000","last_change":"2022-03-21T14:06:54.317279+0000","last_active":"2022-03-21T14:06:54.846891+0000","last_peered":"2022-03-21T14:06:54.846891+0000","last_clean":"2022-03-21T14:06:54.846891+0000","last_became_active":"2022-03-21T14:06:54.316863+0000","last_became_peered":"2022-03-21T14:06:54.316863+0000","last_unstale":"2022-03-21T14:06:54.846891+0000","last_undegraded":"2022-03-21T14:06:54.846891+0000","last_fullsized":"2022-03-21T14:06:54.846891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:17.370559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881566+0000","last_change":"2022-03-21T14:06:54.878112+0000","last_active":"2022-03-21T14:06:54.881566+0000","last_peered":"2022-03-21T14:06:54.881566+0000","last_clean":"2022-03-21T14:06:54.881566+0000","last_became_active":"2022-03-21T14:06:54.877763+0000","last_became_peered":"2022-03-21T14:06:54.877763+0000","last_unstale":"2022-03-21T14:06:54.881566+0000","last_undegraded":"2022-03-21T14:06:54.881566+0000","last_fullsized":"2022-03-21T14:06:54.881566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:51:50.763491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'3","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885989+0000","last_change":"2022-03-21T14:06:54.333183+0000","last_active":"2022-03-21T14:06:54.885989+0000","last_peered":"2022-03-21T14:06:54.885989+0000","last_clean":"2022-03-21T14:06:54.885989+0000","last_became_active":"2022-03-21T14:06:54.332795+0000","last_became_peered":"2022-03-21T14:06:54.332795+0000","last_unstale":"2022-03-21T14:06:54.885989+0000","last_undegraded":"2022-03-21T14:06:54.885989+0000","last_fullsized":"2022-03-21T14:06:54.885989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:29.651861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.862390+0000","last_change":"2022-03-21T14:06:54.862390+0000","last_active":"2022-03-21T14:06:54.862390+0000","last_peered":"2022-03-21T14:06:54.862390+0000","last_clean":"2022-03-21T14:06:54.862390+0000","last_became_active":"2022-03-21T14:06:54.862008+0000","last_became_peered":"2022-03-21T14:06:54.862008+0000","last_unstale":"2022-03-21T14:06:54.862390+0000","last_undegraded":"2022-03-21T14:06:54.862390+0000","last_fullsized":"2022-03-21T14:06:54.862390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:53:50.255768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'6","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875617+0000","last_change":"2022-03-21T14:06:54.315167+0000","last_active":"2022-03-21T14:06:54.875617+0000","last_peered":"2022-03-21T14:06:54.875617+0000","last_clean":"2022-03-21T14:06:54.875617+0000","last_became_active":"2022-03-21T14:06:54.314432+0000","last_became_peered":"2022-03-21T14:06:54.314432+0000","last_unstale":"2022-03-21T14:06:54.875617+0000","last_undegraded":"2022-03-21T14:06:54.875617+0000","last_fullsized":"2022-03-21T14:06:54.875617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:48:27.504920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860333+0000","last_change":"2022-03-21T14:06:54.859057+0000","last_active":"2022-03-21T14:06:54.860333+0000","last_peered":"2022-03-21T14:06:54.860333+0000","last_clean":"2022-03-21T14:06:54.860333+0000","last_became_active":"2022-03-21T14:06:54.858674+0000","last_became_peered":"2022-03-21T14:06:54.858674+0000","last_unstale":"2022-03-21T14:06:54.860333+0000","last_undegraded":"2022-03-21T14:06:54.860333+0000","last_fullsized":"2022-03-21T14:06:54.860333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:38:29.386832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881770+0000","last_change":"2022-03-21T14:06:54.323631+0000","last_active":"2022-03-21T14:06:54.881770+0000","last_peered":"2022-03-21T14:06:54.881770+0000","last_clean":"2022-03-21T14:06:54.881770+0000","last_became_active":"2022-03-21T14:06:54.323179+0000","last_became_peered":"2022-03-21T14:06:54.323179+0000","last_unstale":"2022-03-21T14:06:54.881770+0000","last_undegraded":"2022-03-21T14:06:54.881770+0000","last_fullsized":"2022-03-21T14:06:54.881770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:35:04.480559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.879029+0000","last_change":"2022-03-21T14:06:54.878235+0000","last_active":"2022-03-21T14:06:54.879029+0000","last_peered":"2022-03-21T14:06:54.879029+0000","last_clean":"2022-03-21T14:06:54.879029+0000","last_became_active":"2022-03-21T14:06:54.877650+0000","last_became_peered":"2022-03-21T14:06:54.877650+0000","last_unstale":"2022-03-21T14:06:54.879029+0000","last_undegraded":"2022-03-21T14:06:54.879029+0000","last_fullsized":"2022-03-21T14:06:54.879029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:06:58.716437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843743+0000","last_change":"2022-03-21T14:06:54.316053+0000","last_active":"2022-03-21T14:06:54.843743+0000","last_peered":"2022-03-21T14:06:54.843743+0000","last_clean":"2022-03-21T14:06:54.843743+0000","last_became_active":"2022-03-21T14:06:54.315685+0000","last_became_peered":"2022-03-21T14:06:54.315685+0000","last_unstale":"2022-03-21T14:06:54.843743+0000","last_undegraded":"2022-03-21T14:06:54.843743+0000","last_fullsized":"2022-03-21T14:06:54.843743+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:16:47.746232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.879582+0000","last_change":"2022-03-21T14:06:54.878905+0000","last_active":"2022-03-21T14:06:54.879582+0000","last_peered":"2022-03-21T14:06:54.879582+0000","last_clean":"2022-03-21T14:06:54.879582+0000","last_became_active":"2022-03-21T14:06:54.878341+0000","last_became_peered":"2022-03-21T14:06:54.878341+0000","last_unstale":"2022-03-21T14:06:54.879582+0000","last_undegraded":"2022-03-21T14:06:54.879582+0000","last_fullsized":"2022-03-21T14:06:54.879582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:05:50.071721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.855232+0000","last_change":"2022-03-21T14:06:54.313255+0000","last_active":"2022-03-21T14:06:54.855232+0000","last_peered":"2022-03-21T14:06:54.855232+0000","last_clean":"2022-03-21T14:06:54.855232+0000","last_became_active":"2022-03-21T14:06:54.312849+0000","last_became_peered":"2022-03-21T14:06:54.312849+0000","last_unstale":"2022-03-21T14:06:54.855232+0000","last_undegraded":"2022-03-21T14:06:54.855232+0000","last_fullsized":"2022-03-21T14:06:54.855232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:19:59.556267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860448+0000","last_change":"2022-03-21T14:06:54.857792+0000","last_active":"2022-03-21T14:06:54.860448+0000","last_peered":"2022-03-21T14:06:54.860448+0000","last_clean":"2022-03-21T14:06:54.860448+0000","last_became_active":"2022-03-21T14:06:54.857425+0000","last_became_peered":"2022-03-21T14:06:54.857425+0000","last_unstale":"2022-03-21T14:06:54.860448+0000","last_undegraded":"2022-03-21T14:06:54.860448+0000","last_fullsized":"2022-03-21T14:06:54.860448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:51:33.943688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.887429+0000","last_change":"2022-03-21T14:06:54.335082+0000","last_active":"2022-03-21T14:06:54.887429+0000","last_peered":"2022-03-21T14:06:54.887429+0000","last_clean":"2022-03-21T14:06:54.887429+0000","last_became_active":"2022-03-21T14:06:54.334703+0000","last_became_peered":"2022-03-21T14:06:54.334703+0000","last_unstale":"2022-03-21T14:06:54.887429+0000","last_undegraded":"2022-03-21T14:06:54.887429+0000","last_fullsized":"2022-03-21T14:06:54.887429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:12.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859124+0000","last_change":"2022-03-21T14:06:54.858370+0000","last_active":"2022-03-21T14:06:54.859124+0000","last_peered":"2022-03-21T14:06:54.859124+0000","last_clean":"2022-03-21T14:06:54.859124+0000","last_became_active":"2022-03-21T14:06:54.858074+0000","last_became_peered":"2022-03-21T14:06:54.858074+0000","last_unstale":"2022-03-21T14:06:54.859124+0000","last_undegraded":"2022-03-21T14:06:54.859124+0000","last_fullsized":"2022-03-21T14:06:54.859124+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:55:49.919857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861944+0000","last_change":"2022-03-21T14:06:54.315652+0000","last_active":"2022-03-21T14:06:54.861944+0000","last_peered":"2022-03-21T14:06:54.861944+0000","last_clean":"2022-03-21T14:06:54.861944+0000","last_became_active":"2022-03-21T14:06:54.315317+0000","last_became_peered":"2022-03-21T14:06:54.315317+0000","last_unstale":"2022-03-21T14:06:54.861944+0000","last_undegraded":"2022-03-21T14:06:54.861944+0000","last_fullsized":"2022-03-21T14:06:54.861944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:55:42.394760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859484+0000","last_change":"2022-03-21T14:06:54.856509+0000","last_active":"2022-03-21T14:06:54.859484+0000","last_peered":"2022-03-21T14:06:54.859484+0000","last_clean":"2022-03-21T14:06:54.859484+0000","last_became_active":"2022-03-21T14:06:54.855995+0000","last_became_peered":"2022-03-21T14:06:54.855995+0000","last_unstale":"2022-03-21T14:06:54.859484+0000","last_undegraded":"2022-03-21T14:06:54.859484+0000","last_fullsized":"2022-03-21T14:06:54.859484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:02:52.429679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.849905+0000","last_change":"2022-03-21T14:06:54.336921+0000","last_active":"2022-03-21T14:06:54.849905+0000","last_peered":"2022-03-21T14:06:54.849905+0000","last_clean":"2022-03-21T14:06:54.849905+0000","last_became_active":"2022-03-21T14:06:54.336560+0000","last_became_peered":"2022-03-21T14:06:54.336560+0000","last_unstale":"2022-03-21T14:06:54.849905+0000","last_undegraded":"2022-03-21T14:06:54.849905+0000","last_fullsized":"2022-03-21T14:06:54.849905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:26:06.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865783+0000","last_change":"2022-03-21T14:06:54.865783+0000","last_active":"2022-03-21T14:06:54.865783+0000","last_peered":"2022-03-21T14:06:54.865783+0000","last_clean":"2022-03-21T14:06:54.865783+0000","last_became_active":"2022-03-21T14:06:54.865399+0000","last_became_peered":"2022-03-21T14:06:54.865399+0000","last_unstale":"2022-03-21T14:06:54.865783+0000","last_undegraded":"2022-03-21T14:06:54.865783+0000","last_fullsized":"2022-03-21T14:06:54.865783+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:15:19.155641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847285+0000","last_change":"2022-03-21T14:06:54.321852+0000","last_active":"2022-03-21T14:06:54.847285+0000","last_peered":"2022-03-21T14:06:54.847285+0000","last_clean":"2022-03-21T14:06:54.847285+0000","last_became_active":"2022-03-21T14:06:54.321388+0000","last_became_peered":"2022-03-21T14:06:54.321388+0000","last_unstale":"2022-03-21T14:06:54.847285+0000","last_undegraded":"2022-03-21T14:06:54.847285+0000","last_fullsized":"2022-03-21T14:06:54.847285+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:58:18.274331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871690+0000","last_change":"2022-03-21T14:06:54.871690+0000","last_active":"2022-03-21T14:06:54.871690+0000","last_peered":"2022-03-21T14:06:54.871690+0000","last_clean":"2022-03-21T14:06:54.871690+0000","last_became_active":"2022-03-21T14:06:54.871272+0000","last_became_peered":"2022-03-21T14:06:54.871272+0000","last_unstale":"2022-03-21T14:06:54.871690+0000","last_undegraded":"2022-03-21T14:06:54.871690+0000","last_fullsized":"2022-03-21T14:06:54.871690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:17.999041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863960+0000","last_change":"2022-03-21T14:06:54.860140+0000","last_active":"2022-03-21T14:06:54.863960+0000","last_peered":"2022-03-21T14:06:54.863960+0000","last_clean":"2022-03-21T14:06:54.863960+0000","last_became_active":"2022-03-21T14:06:54.859700+0000","last_became_peered":"2022-03-21T14:06:54.859700+0000","last_unstale":"2022-03-21T14:06:54.863960+0000","last_undegraded":"2022-03-21T14:06:54.863960+0000","last_fullsized":"2022-03-21T14:06:54.863960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:57:46.264617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847150+0000","last_change":"2022-03-21T14:06:54.334370+0000","last_active":"2022-03-21T14:06:54.847150+0000","last_peered":"2022-03-21T14:06:54.847150+0000","last_clean":"2022-03-21T14:06:54.847150+0000","last_became_active":"2022-03-21T14:06:54.333957+0000","last_became_peered":"2022-03-21T14:06:54.333957+0000","last_unstale":"2022-03-21T14:06:54.847150+0000","last_undegraded":"2022-03-21T14:06:54.847150+0000","last_fullsized":"2022-03-21T14:06:54.847150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:29.048808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876136+0000","last_change":"2022-03-21T14:06:54.873196+0000","last_active":"2022-03-21T14:06:54.876136+0000","last_peered":"2022-03-21T14:06:54.876136+0000","last_clean":"2022-03-21T14:06:54.876136+0000","last_became_active":"2022-03-21T14:06:54.872873+0000","last_became_peered":"2022-03-21T14:06:54.872873+0000","last_unstale":"2022-03-21T14:06:54.876136+0000","last_undegraded":"2022-03-21T14:06:54.876136+0000","last_fullsized":"2022-03-21T14:06:54.876136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:01:38.878919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.857028+0000","last_change":"2022-03-21T14:06:54.315511+0000","last_active":"2022-03-21T14:06:54.857028+0000","last_peered":"2022-03-21T14:06:54.857028+0000","last_clean":"2022-03-21T14:06:54.857028+0000","last_became_active":"2022-03-21T14:06:54.315144+0000","last_became_peered":"2022-03-21T14:06:54.315144+0000","last_unstale":"2022-03-21T14:06:54.857028+0000","last_undegraded":"2022-03-21T14:06:54.857028+0000","last_fullsized":"2022-03-21T14:06:54.857028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:30:11.627760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850861+0000","last_change":"2022-03-21T14:05:44.276967+0000","last_active":"2022-03-21T14:06:54.850861+0000","last_peered":"2022-03-21T14:06:54.850861+0000","last_clean":"2022-03-21T14:06:54.850861+0000","last_became_active":"2022-03-21T14:05:44.276297+0000","last_became_peered":"2022-03-21T14:05:44.276297+0000","last_unstale":"2022-03-21T14:06:54.850861+0000","last_undegraded":"2022-03-21T14:06:54.850861+0000","last_fullsized":"2022-03-21T14:06:54.850861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_clean_scrub_stamp":"2022-03-21T14:02:31.386985+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:07:49.689146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861226+0000","last_change":"2022-03-21T14:06:54.860044+0000","last_active":"2022-03-21T14:06:54.861226+0000","last_peered":"2022-03-21T14:06:54.861226+0000","last_clean":"2022-03-21T14:06:54.861226+0000","last_became_active":"2022-03-21T14:06:54.859511+0000","last_became_peered":"2022-03-21T14:06:54.859511+0000","last_unstale":"2022-03-21T14:06:54.861226+0000","last_undegraded":"2022-03-21T14:06:54.861226+0000","last_fullsized":"2022-03-21T14:06:54.861226+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:50:01.100612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850971+0000","last_change":"2022-03-21T14:06:54.313788+0000","last_active":"2022-03-21T14:06:54.850971+0000","last_peered":"2022-03-21T14:06:54.850971+0000","last_clean":"2022-03-21T14:06:54.850971+0000","last_became_active":"2022-03-21T14:06:54.313328+0000","last_became_peered":"2022-03-21T14:06:54.313328+0000","last_unstale":"2022-03-21T14:06:54.850971+0000","last_undegraded":"2022-03-21T14:06:54.850971+0000","last_fullsized":"2022-03-21T14:06:54.850971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:24:17.930196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.849363+0000","last_change":"2022-03-21T14:06:54.334716+0000","last_active":"2022-03-21T14:06:54.849363+0000","last_peered":"2022-03-21T14:06:54.849363+0000","last_clean":"2022-03-21T14:06:54.849363+0000","last_became_active":"2022-03-21T14:06:54.334384+0000","last_became_peered":"2022-03-21T14:06:54.334384+0000","last_unstale":"2022-03-21T14:06:54.849363+0000","last_undegraded":"2022-03-21T14:06:54.849363+0000","last_fullsized":"2022-03-21T14:06:54.849363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:29.598258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878630+0000","last_change":"2022-03-21T14:06:54.872297+0000","last_active":"2022-03-21T14:06:54.878630+0000","last_peered":"2022-03-21T14:06:54.878630+0000","last_clean":"2022-03-21T14:06:54.878630+0000","last_became_active":"2022-03-21T14:06:54.871557+0000","last_became_peered":"2022-03-21T14:06:54.871557+0000","last_unstale":"2022-03-21T14:06:54.878630+0000","last_undegraded":"2022-03-21T14:06:54.878630+0000","last_fullsized":"2022-03-21T14:06:54.878630+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:21:13.076650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-03-21T14:06:55.844414+0000","last_change":"2022-03-21T14:06:54.327446+0000","last_active":"2022-03-21T14:06:55.844414+0000","last_peered":"2022-03-21T14:06:55.844414+0000","last_clean":"2022-03-21T14:06:55.844414+0000","last_became_active":"2022-03-21T14:06:54.327102+0000","last_became_peered":"2022-03-21T14:06:54.327102+0000","last_unstale":"2022-03-21T14:06:55.844414+0000","last_undegraded":"2022-03-21T14:06:55.844414+0000","last_fullsized":"2022-03-21T14:06:55.844414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:28:18.309364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874840+0000","last_change":"2022-03-21T14:06:54.869348+0000","last_active":"2022-03-21T14:06:54.874840+0000","last_peered":"2022-03-21T14:06:54.874840+0000","last_clean":"2022-03-21T14:06:54.874840+0000","last_became_active":"2022-03-21T14:06:54.868917+0000","last_became_peered":"2022-03-21T14:06:54.868917+0000","last_unstale":"2022-03-21T14:06:54.874840+0000","last_undegraded":"2022-03-21T14:06:54.874840+0000","last_fullsized":"2022-03-21T14:06:54.874840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:38:56.199987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878893+0000","last_change":"2022-03-21T14:06:54.321746+0000","last_active":"2022-03-21T14:06:54.878893+0000","last_peered":"2022-03-21T14:06:54.878893+0000","last_clean":"2022-03-21T14:06:54.878893+0000","last_became_active":"2022-03-21T14:06:54.321294+0000","last_became_peered":"2022-03-21T14:06:54.321294+0000","last_unstale":"2022-03-21T14:06:54.878893+0000","last_undegraded":"2022-03-21T14:06:54.878893+0000","last_fullsized":"2022-03-21T14:06:54.878893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:06:26.961696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875678+0000","last_change":"2022-03-21T14:06:54.874949+0000","last_active":"2022-03-21T14:06:54.875678+0000","last_peered":"2022-03-21T14:06:54.875678+0000","last_clean":"2022-03-21T14:06:54.875678+0000","last_became_active":"2022-03-21T14:06:54.874429+0000","last_became_peered":"2022-03-21T14:06:54.874429+0000","last_unstale":"2022-03-21T14:06:54.875678+0000","last_undegraded":"2022-03-21T14:06:54.875678+0000","last_fullsized":"2022-03-21T14:06:54.875678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:43:38.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.327133+0000","last_change":"2022-03-21T14:06:54.326806+0000","last_active":"2022-03-21T14:06:54.327133+0000","last_peered":"2022-03-21T14:06:54.327133+0000","last_clean":"2022-03-21T14:06:54.327133+0000","last_became_active":"2022-03-21T14:06:54.326405+0000","last_became_peered":"2022-03-21T14:06:54.326405+0000","last_unstale":"2022-03-21T14:06:54.327133+0000","last_undegraded":"2022-03-21T14:06:54.327133+0000","last_fullsized":"2022-03-21T14:06:54.327133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:29.562293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.864674+0000","last_change":"2022-03-21T14:06:54.860972+0000","last_active":"2022-03-21T14:06:54.864674+0000","last_peered":"2022-03-21T14:06:54.864674+0000","last_clean":"2022-03-21T14:06:54.864674+0000","last_became_active":"2022-03-21T14:06:54.860557+0000","last_became_peered":"2022-03-21T14:06:54.860557+0000","last_unstale":"2022-03-21T14:06:54.864674+0000","last_undegraded":"2022-03-21T14:06:54.864674+0000","last_fullsized":"2022-03-21T14:06:54.864674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:28:45.487047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.858436+0000","last_change":"2022-03-21T14:06:54.320161+0000","last_active":"2022-03-21T14:06:54.858436+0000","last_peered":"2022-03-21T14:06:54.858436+0000","last_clean":"2022-03-21T14:06:54.858436+0000","last_became_active":"2022-03-21T14:06:54.318104+0000","last_became_peered":"2022-03-21T14:06:54.318104+0000","last_unstale":"2022-03-21T14:06:54.858436+0000","last_undegraded":"2022-03-21T14:06:54.858436+0000","last_fullsized":"2022-03-21T14:06:54.858436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:12.154964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872757+0000","last_change":"2022-03-21T14:06:54.870272+0000","last_active":"2022-03-21T14:06:54.872757+0000","last_peered":"2022-03-21T14:06:54.872757+0000","last_clean":"2022-03-21T14:06:54.872757+0000","last_became_active":"2022-03-21T14:06:54.869649+0000","last_became_peered":"2022-03-21T14:06:54.869649+0000","last_unstale":"2022-03-21T14:06:54.872757+0000","last_undegraded":"2022-03-21T14:06:54.872757+0000","last_fullsized":"2022-03-21T14:06:54.872757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:09:57.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850547+0000","last_change":"2022-03-21T14:06:54.323589+0000","last_active":"2022-03-21T14:06:54.850547+0000","last_peered":"2022-03-21T14:06:54.850547+0000","last_clean":"2022-03-21T14:06:54.850547+0000","last_became_active":"2022-03-21T14:06:54.323187+0000","last_became_peered":"2022-03-21T14:06:54.323187+0000","last_unstale":"2022-03-21T14:06:54.850547+0000","last_undegraded":"2022-03-21T14:06:54.850547+0000","last_fullsized":"2022-03-21T14:06:54.850547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:05:33.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881436+0000","last_change":"2022-03-21T14:06:54.880453+0000","last_active":"2022-03-21T14:06:54.881436+0000","last_peered":"2022-03-21T14:06:54.881436+0000","last_clean":"2022-03-21T14:06:54.881436+0000","last_became_active":"2022-03-21T14:06:54.880078+0000","last_became_peered":"2022-03-21T14:06:54.880078+0000","last_unstale":"2022-03-21T14:06:54.881436+0000","last_undegraded":"2022-03-21T14:06:54.881436+0000","last_fullsized":"2022-03-21T14:06:54.881436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:37:36.812969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872269+0000","last_change":"2022-03-21T14:06:54.314137+0000","last_active":"2022-03-21T14:06:54.872269+0000","last_peered":"2022-03-21T14:06:54.872269+0000","last_clean":"2022-03-21T14:06:54.872269+0000","last_became_active":"2022-03-21T14:06:54.313783+0000","last_became_peered":"2022-03-21T14:06:54.313783+0000","last_unstale":"2022-03-21T14:06:54.872269+0000","last_undegraded":"2022-03-21T14:06:54.872269+0000","last_fullsized":"2022-03-21T14:06:54.872269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:58.358466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.868862+0000","last_change":"2022-03-21T14:06:54.867675+0000","last_active":"2022-03-21T14:06:54.868862+0000","last_peered":"2022-03-21T14:06:54.868862+0000","last_clean":"2022-03-21T14:06:54.868862+0000","last_became_active":"2022-03-21T14:06:54.867097+0000","last_became_peered":"2022-03-21T14:06:54.867097+0000","last_unstale":"2022-03-21T14:06:54.868862+0000","last_undegraded":"2022-03-21T14:06:54.868862+0000","last_fullsized":"2022-03-21T14:06:54.868862+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:07:22.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.852660+0000","last_change":"2022-03-21T14:06:54.333269+0000","last_active":"2022-03-21T14:06:54.852660+0000","last_peered":"2022-03-21T14:06:54.852660+0000","last_clean":"2022-03-21T14:06:54.852660+0000","last_became_active":"2022-03-21T14:06:54.332844+0000","last_became_peered":"2022-03-21T14:06:54.332844+0000","last_unstale":"2022-03-21T14:06:54.852660+0000","last_undegraded":"2022-03-21T14:06:54.852660+0000","last_fullsized":"2022-03-21T14:06:54.852660+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:45:33.278652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871838+0000","last_change":"2022-03-21T14:06:54.869222+0000","last_active":"2022-03-21T14:06:54.871838+0000","last_peered":"2022-03-21T14:06:54.871838+0000","last_clean":"2022-03-21T14:06:54.871838+0000","last_became_active":"2022-03-21T14:06:54.868242+0000","last_became_peered":"2022-03-21T14:06:54.868242+0000","last_unstale":"2022-03-21T14:06:54.871838+0000","last_undegraded":"2022-03-21T14:06:54.871838+0000","last_fullsized":"2022-03-21T14:06:54.871838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:28:30.012341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882343+0000","last_change":"2022-03-21T14:06:54.317425+0000","last_active":"2022-03-21T14:06:54.882343+0000","last_peered":"2022-03-21T14:06:54.882343+0000","last_clean":"2022-03-21T14:06:54.882343+0000","last_became_active":"2022-03-21T14:06:54.317035+0000","last_became_peered":"2022-03-21T14:06:54.317035+0000","last_unstale":"2022-03-21T14:06:54.882343+0000","last_undegraded":"2022-03-21T14:06:54.882343+0000","last_fullsized":"2022-03-21T14:06:54.882343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:24:33.693292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.868940+0000","last_change":"2022-03-21T14:06:54.867673+0000","last_active":"2022-03-21T14:06:54.868940+0000","last_peered":"2022-03-21T14:06:54.868940+0000","last_clean":"2022-03-21T14:06:54.868940+0000","last_became_active":"2022-03-21T14:06:54.867135+0000","last_became_peered":"2022-03-21T14:06:54.867135+0000","last_unstale":"2022-03-21T14:06:54.868940+0000","last_undegraded":"2022-03-21T14:06:54.868940+0000","last_fullsized":"2022-03-21T14:06:54.868940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:27:26.251570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882196+0000","last_change":"2022-03-21T14:06:54.326963+0000","last_active":"2022-03-21T14:06:54.882196+0000","last_peered":"2022-03-21T14:06:54.882196+0000","last_clean":"2022-03-21T14:06:54.882196+0000","last_became_active":"2022-03-21T14:06:54.326592+0000","last_became_peered":"2022-03-21T14:06:54.326592+0000","last_unstale":"2022-03-21T14:06:54.882196+0000","last_undegraded":"2022-03-21T14:06:54.882196+0000","last_fullsized":"2022-03-21T14:06:54.882196+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:11:23.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.866815+0000","last_change":"2022-03-21T14:06:54.865257+0000","last_active":"2022-03-21T14:06:54.866815+0000","last_peered":"2022-03-21T14:06:54.866815+0000","last_clean":"2022-03-21T14:06:54.866815+0000","last_became_active":"2022-03-21T14:06:54.864119+0000","last_became_peered":"2022-03-21T14:06:54.864119+0000","last_unstale":"2022-03-21T14:06:54.866815+0000","last_undegraded":"2022-03-21T14:06:54.866815+0000","last_fullsized":"2022-03-21T14:06:54.866815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:45.228838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865661+0000","last_change":"2022-03-21T14:06:54.864027+0000","last_active":"2022-03-21T14:06:54.865661+0000","last_peered":"2022-03-21T14:06:54.865661+0000","last_clean":"2022-03-21T14:06:54.865661+0000","last_became_active":"2022-03-21T14:06:54.863684+0000","last_became_peered":"2022-03-21T14:06:54.863684+0000","last_unstale":"2022-03-21T14:06:54.865661+0000","last_undegraded":"2022-03-21T14:06:54.865661+0000","last_fullsized":"2022-03-21T14:06:54.865661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:11.453033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860827+0000","last_change":"2022-03-21T14:06:54.860266+0000","last_active":"2022-03-21T14:06:54.860827+0000","last_peered":"2022-03-21T14:06:54.860827+0000","last_clean":"2022-03-21T14:06:54.860827+0000","last_became_active":"2022-03-21T14:06:54.859922+0000","last_became_peered":"2022-03-21T14:06:54.859922+0000","last_unstale":"2022-03-21T14:06:54.860827+0000","last_undegraded":"2022-03-21T14:06:54.860827+0000","last_fullsized":"2022-03-21T14:06:54.860827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:23.689919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875670+0000","last_change":"2022-03-21T14:06:54.875261+0000","last_active":"2022-03-21T14:06:54.875670+0000","last_peered":"2022-03-21T14:06:54.875670+0000","last_clean":"2022-03-21T14:06:54.875670+0000","last_became_active":"2022-03-21T14:06:54.874826+0000","last_became_peered":"2022-03-21T14:06:54.874826+0000","last_unstale":"2022-03-21T14:06:54.875670+0000","last_undegraded":"2022-03-21T14:06:54.875670+0000","last_fullsized":"2022-03-21T14:06:54.875670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:02.673575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874016+0000","last_change":"2022-03-21T14:06:54.867997+0000","last_active":"2022-03-21T14:06:54.874016+0000","last_peered":"2022-03-21T14:06:54.874016+0000","last_clean":"2022-03-21T14:06:54.874016+0000","last_became_active":"2022-03-21T14:06:54.867624+0000","last_became_peered":"2022-03-21T14:06:54.867624+0000","last_unstale":"2022-03-21T14:06:54.874016+0000","last_undegraded":"2022-03-21T14:06:54.874016+0000","last_fullsized":"2022-03-21T14:06:54.874016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:17:37.552083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878655+0000","last_change":"2022-03-21T14:06:54.876868+0000","last_active":"2022-03-21T14:06:54.878655+0000","last_peered":"2022-03-21T14:06:54.878655+0000","last_clean":"2022-03-21T14:06:54.878655+0000","last_became_active":"2022-03-21T14:06:54.876518+0000","last_became_peered":"2022-03-21T14:06:54.876518+0000","last_unstale":"2022-03-21T14:06:54.878655+0000","last_undegraded":"2022-03-21T14:06:54.878655+0000","last_fullsized":"2022-03-21T14:06:54.878655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:50:48.947076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860291+0000","last_change":"2022-03-21T14:06:54.858384+0000","last_active":"2022-03-21T14:06:54.860291+0000","last_peered":"2022-03-21T14:06:54.860291+0000","last_clean":"2022-03-21T14:06:54.860291+0000","last_became_active":"2022-03-21T14:06:54.857183+0000","last_became_peered":"2022-03-21T14:06:54.857183+0000","last_unstale":"2022-03-21T14:06:54.860291+0000","last_undegraded":"2022-03-21T14:06:54.860291+0000","last_fullsized":"2022-03-21T14:06:54.860291+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:03:58.964777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859536+0000","last_change":"2022-03-21T14:06:54.859071+0000","last_active":"2022-03-21T14:06:54.859536+0000","last_peered":"2022-03-21T14:06:54.859536+0000","last_clean":"2022-03-21T14:06:54.859536+0000","last_became_active":"2022-03-21T14:06:54.858740+0000","last_became_peered":"2022-03-21T14:06:54.858740+0000","last_unstale":"2022-03-21T14:06:54.859536+0000","last_undegraded":"2022-03-21T14:06:54.859536+0000","last_fullsized":"2022-03-21T14:06:54.859536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:12.263178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878527+0000","last_change":"2022-03-21T14:06:54.877506+0000","last_active":"2022-03-21T14:06:54.878527+0000","last_peered":"2022-03-21T14:06:54.878527+0000","last_clean":"2022-03-21T14:06:54.878527+0000","last_became_active":"2022-03-21T14:06:54.877201+0000","last_became_peered":"2022-03-21T14:06:54.877201+0000","last_unstale":"2022-03-21T14:06:54.878527+0000","last_undegraded":"2022-03-21T14:06:54.878527+0000","last_fullsized":"2022-03-21T14:06:54.878527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:27:57.264889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875620+0000","last_change":"2022-03-21T14:06:54.875160+0000","last_active":"2022-03-21T14:06:54.875620+0000","last_peered":"2022-03-21T14:06:54.875620+0000","last_clean":"2022-03-21T14:06:54.875620+0000","last_became_active":"2022-03-21T14:06:54.874453+0000","last_became_peered":"2022-03-21T14:06:54.874453+0000","last_unstale":"2022-03-21T14:06:54.875620+0000","last_undegraded":"2022-03-21T14:06:54.875620+0000","last_fullsized":"2022-03-21T14:06:54.875620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:53.657732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878457+0000","last_change":"2022-03-21T14:06:54.876605+0000","last_active":"2022-03-21T14:06:54.878457+0000","last_peered":"2022-03-21T14:06:54.878457+0000","last_clean":"2022-03-21T14:06:54.878457+0000","last_became_active":"2022-03-21T14:06:54.876241+0000","last_became_peered":"2022-03-21T14:06:54.876241+0000","last_unstale":"2022-03-21T14:06:54.878457+0000","last_undegraded":"2022-03-21T14:06:54.878457+0000","last_fullsized":"2022-03-21T14:06:54.878457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:38.713657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874849+0000","last_change":"2022-03-21T14:06:54.874849+0000","last_active":"2022-03-21T14:06:54.874849+0000","last_peered":"2022-03-21T14:06:54.874849+0000","last_clean":"2022-03-21T14:06:54.874849+0000","last_became_active":"2022-03-21T14:06:54.874203+0000","last_became_peered":"2022-03-21T14:06:54.874203+0000","last_unstale":"2022-03-21T14:06:54.874849+0000","last_undegraded":"2022-03-21T14:06:54.874849+0000","last_fullsized":"2022-03-21T14:06:54.874849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:17:32.906357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.870601+0000","last_change":"2022-03-21T14:06:54.869399+0000","last_active":"2022-03-21T14:06:54.870601+0000","last_peered":"2022-03-21T14:06:54.870601+0000","last_clean":"2022-03-21T14:06:54.870601+0000","last_became_active":"2022-03-21T14:06:54.869078+0000","last_became_peered":"2022-03-21T14:06:54.869078+0000","last_unstale":"2022-03-21T14:06:54.870601+0000","last_undegraded":"2022-03-21T14:06:54.870601+0000","last_fullsized":"2022-03-21T14:06:54.870601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:39:49.492701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860566+0000","last_change":"2022-03-21T14:06:54.858492+0000","last_active":"2022-03-21T14:06:54.860566+0000","last_peered":"2022-03-21T14:06:54.860566+0000","last_clean":"2022-03-21T14:06:54.860566+0000","last_became_active":"2022-03-21T14:06:54.858113+0000","last_became_peered":"2022-03-21T14:06:54.858113+0000","last_unstale":"2022-03-21T14:06:54.860566+0000","last_undegraded":"2022-03-21T14:06:54.860566+0000","last_fullsized":"2022-03-21T14:06:54.860566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:41:03.763238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481806,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":880640,"data_stored":420957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":10,"up_from":70,"seq":300647710737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":1388,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":1421312,"data_stored":942746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":1,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35999999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.433}]},{"osd":6,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.58099999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59399999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":9,"up_from":65,"seq":279172874262,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":421523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.63900000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.012}]},{"osd":3,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.98699999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.84699999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.0249999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":1.093},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.93899999999999995}]}]},{"osd":8,"up_from":57,"seq":244813135898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":872448,"data_stored":412664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.39500000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.437},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.501},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53100000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:06:20 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57399999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56299999999999994}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6704,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738448,"statfs":{"total":95995035648,"available":95988170752,"internally_reserved":0,"allocated":1417216,"data_stored":950451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84199999999999997,"5min":0.88,"15min":0.88},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68400000000000005}]},{"osd":3,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66800000000000004,"15min":0.66400000000000003},"min":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73499999999999999,"15min":0.73399999999999999},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.98099999999999998,"5min":1.103,"15min":1.103},"last":0.58199999999999996}]},{"osd":4,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.58799999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.627},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.78800000000000003,"15min":0.77700000000000002},"min":{"1min":0.69899999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.77800000000000002,"15min":0.77100000000000002},"min":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.66300000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.73799999999999999},"min":{"1min":0.57899999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73699999999999999,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71599999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:23 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.753}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.745},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73699999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6116,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739036,"statfs":{"total":95995035648,"available":95988772864,"internally_reserved":0,"allocated":880640,"data_stored":420957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.44400000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.63},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.71399999999999997,"15min":0.69999999999999996},"min":{"1min":0.622,"5min":0.495,"15min":0.495},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.70199999999999996,"15min":0.69199999999999995},"min":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.048,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.77900000000000003,"15min":0.77800000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1679999999999999,"5min":1.208,"15min":1.208},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73799999999999999,"15min":0.72999999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76600000000000001,"15min":0.76000000000000001},"min":{"1min":0.55500000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.996}]},{"osd":4,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.63800000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.099,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.86299999999999999,"15min":0.86099999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.137,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:31 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81299999999999994,"15min":0.81000000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.103},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89500000000000002,"15min":0.88700000000000001},"min":{"1min":0.70899999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.3420000000000001}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.91600000000000004},"min":{"1min":0.70399999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.306},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.163}]},{"osd":9,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.413},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.2529999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:06:46 2022","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":1.071}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7012,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738140,"statfs":{"total":95995035648,"available":95987855360,"internally_reserved":0,"allocated":880640,"data_stored":413947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:15 2022","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53700000000000003,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78000000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.374}]},{"osd":3,"last update":"Mon Mar 21 14:06:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89100000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.67400000000000004},"min":{"1min":0.47499999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Mar 21 14:06:30 2022","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.29699999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:50 2022","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.45100000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:06:15 2022","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.66500000000000004,"15min":0.66900000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:35 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83799999999999997,"5min":0.871,"15min":0.871},"last":0.433}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64600000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:06:20 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Mon Mar 21 14:06:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.53400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7016,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738136,"statfs":{"total":95995035648,"available":95987851264,"internally_reserved":0,"allocated":884736,"data_stored":421047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 21 14:06:17 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.60999999999999999,"15min":0.57799999999999996},"min":{"1min":0.52200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.55700000000000005},"min":{"1min":0.504,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon Mar 21 14:06:36 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84999999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.71499999999999997,"15min":0.68999999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:05:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.58599999999999997,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.72799999999999998,"15min":0.67600000000000005},"min":{"1min":0.58799999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.161,"5min":1.173,"15min":1.173},"last":0.871},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.60799999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.89400000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78400000000000003,"15min":0.77200000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.78700000000000003,"15min":0.77700000000000002},"min":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0609999999999999,"5min":1.153,"15min":1.153},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.77500000000000002,"15min":0.749},"min":{"1min":0.57699999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.83699999999999997,"15min":0.80500000000000005},"min":{"1min":0.56100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.93799999999999994}]},{"osd":7,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92100000000000004,"15min":0.88500000000000001},"min":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.766,"5min":5.766,"15min":5.766},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.1890000000000001,"5min":1.224,"15min":1.224},"last":0.70199999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:57 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.96599999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.27,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.997,"15min":1.0009999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.3500000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.90900000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.96199999999999997}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7092,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738060,"statfs":{"total":95995035648,"available":95987773440,"internally_reserved":0,"allocated":897024,"data_stored":422539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:38 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:06:38 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:06:05 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.623},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Mar 21 14:06:28 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67100000000000004,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 21 14:06:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.622},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:54 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.59899999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.51300000000000001}]},{"osd":9,"last update":"Mon Mar 21 14:06:23 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.68300000000000005}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.69299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182450,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":888832,"data_stored":421061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.48099999999999998,"15min":0.45000000000000001},"min":{"1min":0.39600000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80500000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.58899999999999997,"15min":0.55500000000000005},"min":{"1min":0.48499999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.504}]},{"osd":1,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.628,"15min":0.58399999999999996},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.64000000000000001,"15min":0.61199999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.69399999999999995}]},{"osd":2,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.69999999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.58699999999999997,"5min":0.504,"15min":0.504},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.76800000000000002}]},{"osd":4,"last update":"Mon Mar 21 14:06:24 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.70699999999999996,"15min":0.66500000000000004},"min":{"1min":0.58199999999999996,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.70699999999999996,"15min":0.67300000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.76300000000000001}]},{"osd":5,"last update":"Mon Mar 21 14:06:49 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81100000000000005,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.77500000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.119,"15min":1.119},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.876,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.498,"15min":0.498},"max":{"1min":1.2849999999999999,"5min":1.345,"15min":1.345},"last":0.96399999999999997}]},{"osd":7,"last update":"Mon Mar 21 14:06:33 2022","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.93500000000000005,"15min":0.90300000000000002},"min":{"1min":0.68700000000000006,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.84899999999999998,"15min":0.83699999999999997},"min":{"1min":0.73099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.78500000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.96199999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:18 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.995}]},{"osd":10,"last update":"Mon Mar 21 14:06:41 2022","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.02},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.82399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":420979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.58199999999999996,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.745},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.46999999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.89800000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.46400000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76800000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.65600000000000003},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.81299999999999994}]},{"osd":7,"last update":"Mon Mar 21 14:06:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.56799999999999995,"15min":0.54900000000000004},"min":{"1min":0.32300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.374,"15min":0.374},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.98199999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88500000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0469999999999999,"5min":1.05,"15min":1.05},"last":1.0349999999999999}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":1.05}]},{"osd":10,"last update":"Mon Mar 21 14:06:41 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.095}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":1388,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":1421312,"data_stored":949744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.39500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.44},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.80200000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Mar 21 14:06:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48999999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.72199999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.502},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64500000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.78500000000000003}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:07:00.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-03-21T14:07:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:00 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-03-21T14:07:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:00 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3840889897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:00 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/70199353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:00 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-03-21T14:07:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:00 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3840889897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:00 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/70199353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:01.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 -- 172.21.15.2:0/128654584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c073590 msgr2=0x7f9b1c073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:01.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 --2- 172.21.15.2:0/128654584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c073590 0x7f9b1c073e20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009ba0 tx=0x7f9b14009430 comp rx=0 tx=0).stop 2022-03-21T14:07:01.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 -- 172.21.15.2:0/128654584 shutdown_connections 2022-03-21T14:07:01.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 --2- 172.21.15.2:0/128654584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c073590 0x7f9b1c073e20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 --2- 172.21.15.2:0/128654584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 --2- 172.21.15.2:0/128654584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c072be0 0x7f9b1c073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 -- 172.21.15.2:0/128654584 >> 172.21.15.2:0/128654584 conn(0x7f9b1c0f9990 msgr2=0x7f9b1c0fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:01.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 -- 172.21.15.2:0/128654584 shutdown_connections 2022-03-21T14:07:01.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.121+0000 7f9b22e42700 1 -- 172.21.15.2:0/128654584 wait complete. 2022-03-21T14:07:01.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.122+0000 7f9b22e42700 1 Processor -- start 2022-03-21T14:07:01.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.122+0000 7f9b22e42700 1 -- start start 2022-03-21T14:07:01.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:01.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c072be0 0x7f9b1c113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:01.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 0x7f9b1c11a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:01.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c11abd0 con 0x7f9b1c0718f0 2022-03-21T14:07:01.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c11ad40 con 0x7f9b1c072be0 2022-03-21T14:07:01.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22e42700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c11af00 con 0x7f9b1c073590 2022-03-21T14:07:01.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:01.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43796/0 (socket says 172.21.15.2:43796) 2022-03-21T14:07:01.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 -- 172.21.15.2:0/4045810494 learned_addr learned my addr 172.21.15.2:0/4045810494 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:01.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b2163f700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c072be0 0x7f9b1c113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:01.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22641700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 0x7f9b1c11a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:01.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 -- 172.21.15.2:0/4045810494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 msgr2=0x7f9b1c11a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:01.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 0x7f9b1c11a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 -- 172.21.15.2:0/4045810494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c072be0 msgr2=0x7f9b1c113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:01.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c072be0 0x7f9b1c113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b21e40700 1 -- 172.21.15.2:0/4045810494 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0093f0 con 0x7f9b1c0718f0 2022-03-21T14:07:01.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.123+0000 7f9b22641700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 0x7f9b1c11a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:01.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b21e40700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800b730 tx=0x7f9b18005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:01.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18004170 con 0x7f9b1c0718f0 2022-03-21T14:07:01.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b140096b0 con 0x7f9b1c0718f0 2022-03-21T14:07:01.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9b18010af0 con 0x7f9b1c0718f0 2022-03-21T14:07:01.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18018d90 con 0x7f9b1c0718f0 2022-03-21T14:07:01.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.124+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c11b5c0 con 0x7f9b1c0718f0 2022-03-21T14:07:01.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.125+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1c10df40 con 0x7f9b1c0718f0 2022-03-21T14:07:01.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.127+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f9b1801f060 con 0x7f9b1c0718f0 2022-03-21T14:07:01.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.128+0000 7f9b127fc700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 0x7f9b080991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:01.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.128+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f9b180c0120 con 0x7f9b1c0718f0 2022-03-21T14:07:01.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.128+0000 7f9b2163f700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 0x7f9b080991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:01.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.129+0000 7f9b2163f700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 0x7f9b080991c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c009e50 tx=0x7f9b0c005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:01.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.130+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f9b1808d450 con 0x7f9b1c0718f0 2022-03-21T14:07:01.313 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.313+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b1c02ab90 con 0x7f9b08096d10 2022-03-21T14:07:01.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.324+0000 7f9b127fc700 1 -- 172.21.15.2:0/4045810494 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+333193 (secure 0 0 0) 0x7f9b1c02ab90 con 0x7f9b08096d10 2022-03-21T14:07:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:01.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 msgr2=0x7f9b080991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:01.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 0x7f9b080991c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c009e50 tx=0x7f9b0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:01.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 msgr2=0x7f9b1c1130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:01.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800b730 tx=0x7f9b18005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 shutdown_connections 2022-03-21T14:07:01.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f9b08096d10 0x7f9b080991c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b1c073590 0x7f9b1c11a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b1c0718f0 0x7f9b1c1130c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 --2- 172.21.15.2:0/4045810494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b1c072be0 0x7f9b1c113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 >> 172.21.15.2:0/4045810494 conn(0x7f9b1c0f9990 msgr2=0x7f9b1c0fb560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 shutdown_connections 2022-03-21T14:07:01.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:01.329+0000 7f9b22e42700 1 -- 172.21.15.2:0/4045810494 wait complete. 2022-03-21T14:07:01.335 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-03-21T14:07:01.355 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2022-03-21T14:07:00.392583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77944,"kb_used_data":11928,"kb_used_omap":0,"kb_used_meta":65920,"kb_avail":1124863880,"statfs":{"total":1151940427776,"available":1151860613120,"internally_reserved":0,"allocated":12214272,"data_stored":6618615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67502080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.096556"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872656+0000","last_change":"2022-03-21T14:06:54.869479+0000","last_active":"2022-03-21T14:06:54.872656+0000","last_peered":"2022-03-21T14:06:54.872656+0000","last_clean":"2022-03-21T14:06:54.872656+0000","last_became_active":"2022-03-21T14:06:54.869079+0000","last_became_peered":"2022-03-21T14:06:54.869079+0000","last_unstale":"2022-03-21T14:06:54.872656+0000","last_undegraded":"2022-03-21T14:06:54.872656+0000","last_fullsized":"2022-03-21T14:06:54.872656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:09:19.289483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885992+0000","last_change":"2022-03-21T14:06:54.884544+0000","last_active":"2022-03-21T14:06:54.885992+0000","last_peered":"2022-03-21T14:06:54.885992+0000","last_clean":"2022-03-21T14:06:54.885992+0000","last_became_active":"2022-03-21T14:06:54.884210+0000","last_became_peered":"2022-03-21T14:06:54.884210+0000","last_unstale":"2022-03-21T14:06:54.885992+0000","last_undegraded":"2022-03-21T14:06:54.885992+0000","last_fullsized":"2022-03-21T14:06:54.885992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:48:52.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885007+0000","last_change":"2022-03-21T14:06:54.880620+0000","last_active":"2022-03-21T14:06:54.885007+0000","last_peered":"2022-03-21T14:06:54.885007+0000","last_clean":"2022-03-21T14:06:54.885007+0000","last_became_active":"2022-03-21T14:06:54.880297+0000","last_became_peered":"2022-03-21T14:06:54.880297+0000","last_unstale":"2022-03-21T14:06:54.885007+0000","last_undegraded":"2022-03-21T14:06:54.885007+0000","last_fullsized":"2022-03-21T14:06:54.885007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:43:49.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863909+0000","last_change":"2022-03-21T14:06:54.861753+0000","last_active":"2022-03-21T14:06:54.863909+0000","last_peered":"2022-03-21T14:06:54.863909+0000","last_clean":"2022-03-21T14:06:54.863909+0000","last_became_active":"2022-03-21T14:06:54.860649+0000","last_became_peered":"2022-03-21T14:06:54.860649+0000","last_unstale":"2022-03-21T14:06:54.863909+0000","last_undegraded":"2022-03-21T14:06:54.863909+0000","last_fullsized":"2022-03-21T14:06:54.863909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:22:24.251084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876192+0000","last_change":"2022-03-21T14:06:54.875705+0000","last_active":"2022-03-21T14:06:54.876192+0000","last_peered":"2022-03-21T14:06:54.876192+0000","last_clean":"2022-03-21T14:06:54.876192+0000","last_became_active":"2022-03-21T14:06:54.875171+0000","last_became_peered":"2022-03-21T14:06:54.875171+0000","last_unstale":"2022-03-21T14:06:54.876192+0000","last_undegraded":"2022-03-21T14:06:54.876192+0000","last_fullsized":"2022-03-21T14:06:54.876192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:52:39.257984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.858664+0000","last_change":"2022-03-21T14:06:54.858664+0000","last_active":"2022-03-21T14:06:54.858664+0000","last_peered":"2022-03-21T14:06:54.858664+0000","last_clean":"2022-03-21T14:06:54.858664+0000","last_became_active":"2022-03-21T14:06:54.858283+0000","last_became_peered":"2022-03-21T14:06:54.858283+0000","last_unstale":"2022-03-21T14:06:54.858664+0000","last_undegraded":"2022-03-21T14:06:54.858664+0000","last_fullsized":"2022-03-21T14:06:54.858664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:58:34.398164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.857542+0000","last_change":"2022-03-21T14:06:54.856225+0000","last_active":"2022-03-21T14:06:54.857542+0000","last_peered":"2022-03-21T14:06:54.857542+0000","last_clean":"2022-03-21T14:06:54.857542+0000","last_became_active":"2022-03-21T14:06:54.855801+0000","last_became_peered":"2022-03-21T14:06:54.855801+0000","last_unstale":"2022-03-21T14:06:54.857542+0000","last_undegraded":"2022-03-21T14:06:54.857542+0000","last_fullsized":"2022-03-21T14:06:54.857542+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:26:47.895423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876168+0000","last_change":"2022-03-21T14:06:54.873142+0000","last_active":"2022-03-21T14:06:54.876168+0000","last_peered":"2022-03-21T14:06:54.876168+0000","last_clean":"2022-03-21T14:06:54.876168+0000","last_became_active":"2022-03-21T14:06:54.872788+0000","last_became_peered":"2022-03-21T14:06:54.872788+0000","last_unstale":"2022-03-21T14:06:54.876168+0000","last_undegraded":"2022-03-21T14:06:54.876168+0000","last_fullsized":"2022-03-21T14:06:54.876168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:53:53.652396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860355+0000","last_change":"2022-03-21T14:06:54.860355+0000","last_active":"2022-03-21T14:06:54.860355+0000","last_peered":"2022-03-21T14:06:54.860355+0000","last_clean":"2022-03-21T14:06:54.860355+0000","last_became_active":"2022-03-21T14:06:54.858723+0000","last_became_peered":"2022-03-21T14:06:54.858723+0000","last_unstale":"2022-03-21T14:06:54.860355+0000","last_undegraded":"2022-03-21T14:06:54.860355+0000","last_fullsized":"2022-03-21T14:06:54.860355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:34.198090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.855421+0000","last_change":"2022-03-21T14:06:54.855421+0000","last_active":"2022-03-21T14:06:54.855421+0000","last_peered":"2022-03-21T14:06:54.855421+0000","last_clean":"2022-03-21T14:06:54.855421+0000","last_became_active":"2022-03-21T14:06:54.855027+0000","last_became_peered":"2022-03-21T14:06:54.855027+0000","last_unstale":"2022-03-21T14:06:54.855421+0000","last_undegraded":"2022-03-21T14:06:54.855421+0000","last_fullsized":"2022-03-21T14:06:54.855421+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:36:14.968816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871860+0000","last_change":"2022-03-21T14:06:54.869318+0000","last_active":"2022-03-21T14:06:54.871860+0000","last_peered":"2022-03-21T14:06:54.871860+0000","last_clean":"2022-03-21T14:06:54.871860+0000","last_became_active":"2022-03-21T14:06:54.868664+0000","last_became_peered":"2022-03-21T14:06:54.868664+0000","last_unstale":"2022-03-21T14:06:54.871860+0000","last_undegraded":"2022-03-21T14:06:54.871860+0000","last_fullsized":"2022-03-21T14:06:54.871860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:29.775263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.864680+0000","last_change":"2022-03-21T14:06:54.862293+0000","last_active":"2022-03-21T14:06:54.864680+0000","last_peered":"2022-03-21T14:06:54.864680+0000","last_clean":"2022-03-21T14:06:54.864680+0000","last_became_active":"2022-03-21T14:06:54.861926+0000","last_became_peered":"2022-03-21T14:06:54.861926+0000","last_unstale":"2022-03-21T14:06:54.864680+0000","last_undegraded":"2022-03-21T14:06:54.864680+0000","last_fullsized":"2022-03-21T14:06:54.864680+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:16:27.088619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861063+0000","last_change":"2022-03-21T14:06:54.859631+0000","last_active":"2022-03-21T14:06:54.861063+0000","last_peered":"2022-03-21T14:06:54.861063+0000","last_clean":"2022-03-21T14:06:54.861063+0000","last_became_active":"2022-03-21T14:06:54.858657+0000","last_became_peered":"2022-03-21T14:06:54.858657+0000","last_unstale":"2022-03-21T14:06:54.861063+0000","last_undegraded":"2022-03-21T14:06:54.861063+0000","last_fullsized":"2022-03-21T14:06:54.861063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:03.821600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871061+0000","last_change":"2022-03-21T14:06:54.866492+0000","last_active":"2022-03-21T14:06:54.871061+0000","last_peered":"2022-03-21T14:06:54.871061+0000","last_clean":"2022-03-21T14:06:54.871061+0000","last_became_active":"2022-03-21T14:06:54.866091+0000","last_became_peered":"2022-03-21T14:06:54.866091+0000","last_unstale":"2022-03-21T14:06:54.871061+0000","last_undegraded":"2022-03-21T14:06:54.871061+0000","last_fullsized":"2022-03-21T14:06:54.871061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:31:33.741891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.884962+0000","last_change":"2022-03-21T14:06:54.879848+0000","last_active":"2022-03-21T14:06:54.884962+0000","last_peered":"2022-03-21T14:06:54.884962+0000","last_clean":"2022-03-21T14:06:54.884962+0000","last_became_active":"2022-03-21T14:06:54.879496+0000","last_became_peered":"2022-03-21T14:06:54.879496+0000","last_unstale":"2022-03-21T14:06:54.884962+0000","last_undegraded":"2022-03-21T14:06:54.884962+0000","last_fullsized":"2022-03-21T14:06:54.884962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:18:13.350002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.877838+0000","last_change":"2022-03-21T14:06:54.877838+0000","last_active":"2022-03-21T14:06:54.877838+0000","last_peered":"2022-03-21T14:06:54.877838+0000","last_clean":"2022-03-21T14:06:54.877838+0000","last_became_active":"2022-03-21T14:06:54.877444+0000","last_became_peered":"2022-03-21T14:06:54.877444+0000","last_unstale":"2022-03-21T14:06:54.877838+0000","last_undegraded":"2022-03-21T14:06:54.877838+0000","last_fullsized":"2022-03-21T14:06:54.877838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:32.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.886488+0000","last_change":"2022-03-21T14:06:54.885978+0000","last_active":"2022-03-21T14:06:54.886488+0000","last_peered":"2022-03-21T14:06:54.886488+0000","last_clean":"2022-03-21T14:06:54.886488+0000","last_became_active":"2022-03-21T14:06:54.885645+0000","last_became_peered":"2022-03-21T14:06:54.885645+0000","last_unstale":"2022-03-21T14:06:54.886488+0000","last_undegraded":"2022-03-21T14:06:54.886488+0000","last_fullsized":"2022-03-21T14:06:54.886488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:51.003441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863262+0000","last_change":"2022-03-21T14:06:54.860687+0000","last_active":"2022-03-21T14:06:54.863262+0000","last_peered":"2022-03-21T14:06:54.863262+0000","last_clean":"2022-03-21T14:06:54.863262+0000","last_became_active":"2022-03-21T14:06:54.860315+0000","last_became_peered":"2022-03-21T14:06:54.860315+0000","last_unstale":"2022-03-21T14:06:54.863262+0000","last_undegraded":"2022-03-21T14:06:54.863262+0000","last_fullsized":"2022-03-21T14:06:54.863262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:27:56.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.869847+0000","last_change":"2022-03-21T14:06:54.867688+0000","last_active":"2022-03-21T14:06:54.869847+0000","last_peered":"2022-03-21T14:06:54.869847+0000","last_clean":"2022-03-21T14:06:54.869847+0000","last_became_active":"2022-03-21T14:06:54.866998+0000","last_became_peered":"2022-03-21T14:06:54.866998+0000","last_unstale":"2022-03-21T14:06:54.869847+0000","last_undegraded":"2022-03-21T14:06:54.869847+0000","last_fullsized":"2022-03-21T14:06:54.869847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:17:24.979917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-03-21T14:06:55.844294+0000","last_change":"2022-03-21T14:06:54.326883+0000","last_active":"2022-03-21T14:06:55.844294+0000","last_peered":"2022-03-21T14:06:55.844294+0000","last_clean":"2022-03-21T14:06:55.844294+0000","last_became_active":"2022-03-21T14:06:54.326424+0000","last_became_peered":"2022-03-21T14:06:54.326424+0000","last_unstale":"2022-03-21T14:06:55.844294+0000","last_undegraded":"2022-03-21T14:06:55.844294+0000","last_fullsized":"2022-03-21T14:06:55.844294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:39.818504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.877311+0000","last_change":"2022-03-21T14:06:54.875068+0000","last_active":"2022-03-21T14:06:54.877311+0000","last_peered":"2022-03-21T14:06:54.877311+0000","last_clean":"2022-03-21T14:06:54.877311+0000","last_became_active":"2022-03-21T14:06:54.874715+0000","last_became_peered":"2022-03-21T14:06:54.874715+0000","last_unstale":"2022-03-21T14:06:54.877311+0000","last_undegraded":"2022-03-21T14:06:54.877311+0000","last_fullsized":"2022-03-21T14:06:54.877311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:33.864263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882656+0000","last_change":"2022-03-21T14:06:54.335459+0000","last_active":"2022-03-21T14:06:54.882656+0000","last_peered":"2022-03-21T14:06:54.882656+0000","last_clean":"2022-03-21T14:06:54.882656+0000","last_became_active":"2022-03-21T14:06:54.335109+0000","last_became_peered":"2022-03-21T14:06:54.335109+0000","last_unstale":"2022-03-21T14:06:54.882656+0000","last_undegraded":"2022-03-21T14:06:54.882656+0000","last_fullsized":"2022-03-21T14:06:54.882656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:06:56.531385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885999+0000","last_change":"2022-03-21T14:06:54.884043+0000","last_active":"2022-03-21T14:06:54.885999+0000","last_peered":"2022-03-21T14:06:54.885999+0000","last_clean":"2022-03-21T14:06:54.885999+0000","last_became_active":"2022-03-21T14:06:54.883647+0000","last_became_peered":"2022-03-21T14:06:54.883647+0000","last_unstale":"2022-03-21T14:06:54.885999+0000","last_undegraded":"2022-03-21T14:06:54.885999+0000","last_fullsized":"2022-03-21T14:06:54.885999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:32:17.243588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872866+0000","last_change":"2022-03-21T14:06:54.314357+0000","last_active":"2022-03-21T14:06:54.872866+0000","last_peered":"2022-03-21T14:06:54.872866+0000","last_clean":"2022-03-21T14:06:54.872866+0000","last_became_active":"2022-03-21T14:06:54.313855+0000","last_became_peered":"2022-03-21T14:06:54.313855+0000","last_unstale":"2022-03-21T14:06:54.872866+0000","last_undegraded":"2022-03-21T14:06:54.872866+0000","last_fullsized":"2022-03-21T14:06:54.872866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:58:09.686843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861824+0000","last_change":"2022-03-21T14:06:54.861824+0000","last_active":"2022-03-21T14:06:54.861824+0000","last_peered":"2022-03-21T14:06:54.861824+0000","last_clean":"2022-03-21T14:06:54.861824+0000","last_became_active":"2022-03-21T14:06:54.859283+0000","last_became_peered":"2022-03-21T14:06:54.859283+0000","last_unstale":"2022-03-21T14:06:54.861824+0000","last_undegraded":"2022-03-21T14:06:54.861824+0000","last_fullsized":"2022-03-21T14:06:54.861824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:00:42.993237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871575+0000","last_change":"2022-03-21T14:06:54.312853+0000","last_active":"2022-03-21T14:06:54.871575+0000","last_peered":"2022-03-21T14:06:54.871575+0000","last_clean":"2022-03-21T14:06:54.871575+0000","last_became_active":"2022-03-21T14:06:54.312474+0000","last_became_peered":"2022-03-21T14:06:54.312474+0000","last_unstale":"2022-03-21T14:06:54.871575+0000","last_undegraded":"2022-03-21T14:06:54.871575+0000","last_fullsized":"2022-03-21T14:06:54.871575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:53:19.818645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863910+0000","last_change":"2022-03-21T14:06:54.859678+0000","last_active":"2022-03-21T14:06:54.863910+0000","last_peered":"2022-03-21T14:06:54.863910+0000","last_clean":"2022-03-21T14:06:54.863910+0000","last_became_active":"2022-03-21T14:06:54.858928+0000","last_became_peered":"2022-03-21T14:06:54.858928+0000","last_unstale":"2022-03-21T14:06:54.863910+0000","last_undegraded":"2022-03-21T14:06:54.863910+0000","last_fullsized":"2022-03-21T14:06:54.863910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:10.326574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872009+0000","last_change":"2022-03-21T14:06:54.314216+0000","last_active":"2022-03-21T14:06:54.872009+0000","last_peered":"2022-03-21T14:06:54.872009+0000","last_clean":"2022-03-21T14:06:54.872009+0000","last_became_active":"2022-03-21T14:06:54.313808+0000","last_became_peered":"2022-03-21T14:06:54.313808+0000","last_unstale":"2022-03-21T14:06:54.872009+0000","last_undegraded":"2022-03-21T14:06:54.872009+0000","last_fullsized":"2022-03-21T14:06:54.872009+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:52:09.937102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.869625+0000","last_change":"2022-03-21T14:06:54.868066+0000","last_active":"2022-03-21T14:06:54.869625+0000","last_peered":"2022-03-21T14:06:54.869625+0000","last_clean":"2022-03-21T14:06:54.869625+0000","last_became_active":"2022-03-21T14:06:54.867711+0000","last_became_peered":"2022-03-21T14:06:54.867711+0000","last_unstale":"2022-03-21T14:06:54.869625+0000","last_undegraded":"2022-03-21T14:06:54.869625+0000","last_fullsized":"2022-03-21T14:06:54.869625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:22:29.470011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843138+0000","last_change":"2022-03-21T14:06:54.322793+0000","last_active":"2022-03-21T14:06:54.843138+0000","last_peered":"2022-03-21T14:06:54.843138+0000","last_clean":"2022-03-21T14:06:54.843138+0000","last_became_active":"2022-03-21T14:06:54.322360+0000","last_became_peered":"2022-03-21T14:06:54.322360+0000","last_unstale":"2022-03-21T14:06:54.843138+0000","last_undegraded":"2022-03-21T14:06:54.843138+0000","last_fullsized":"2022-03-21T14:06:54.843138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:15:12.220667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875260+0000","last_change":"2022-03-21T14:06:54.874475+0000","last_active":"2022-03-21T14:06:54.875260+0000","last_peered":"2022-03-21T14:06:54.875260+0000","last_clean":"2022-03-21T14:06:54.875260+0000","last_became_active":"2022-03-21T14:06:54.874113+0000","last_became_peered":"2022-03-21T14:06:54.874113+0000","last_unstale":"2022-03-21T14:06:54.875260+0000","last_undegraded":"2022-03-21T14:06:54.875260+0000","last_fullsized":"2022-03-21T14:06:54.875260+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:25.860831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847301+0000","last_change":"2022-03-21T14:06:54.331708+0000","last_active":"2022-03-21T14:06:54.847301+0000","last_peered":"2022-03-21T14:06:54.847301+0000","last_clean":"2022-03-21T14:06:54.847301+0000","last_became_active":"2022-03-21T14:06:54.331347+0000","last_became_peered":"2022-03-21T14:06:54.331347+0000","last_unstale":"2022-03-21T14:06:54.847301+0000","last_undegraded":"2022-03-21T14:06:54.847301+0000","last_fullsized":"2022-03-21T14:06:54.847301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:59:42.282136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865062+0000","last_change":"2022-03-21T14:06:54.864160+0000","last_active":"2022-03-21T14:06:54.865062+0000","last_peered":"2022-03-21T14:06:54.865062+0000","last_clean":"2022-03-21T14:06:54.865062+0000","last_became_active":"2022-03-21T14:06:54.863699+0000","last_became_peered":"2022-03-21T14:06:54.863699+0000","last_unstale":"2022-03-21T14:06:54.865062+0000","last_undegraded":"2022-03-21T14:06:54.865062+0000","last_fullsized":"2022-03-21T14:06:54.865062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:40:53.670759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843617+0000","last_change":"2022-03-21T14:06:54.314228+0000","last_active":"2022-03-21T14:06:54.843617+0000","last_peered":"2022-03-21T14:06:54.843617+0000","last_clean":"2022-03-21T14:06:54.843617+0000","last_became_active":"2022-03-21T14:06:54.313667+0000","last_became_peered":"2022-03-21T14:06:54.313667+0000","last_unstale":"2022-03-21T14:06:54.843617+0000","last_undegraded":"2022-03-21T14:06:54.843617+0000","last_fullsized":"2022-03-21T14:06:54.843617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:20:10.791422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.867382+0000","last_change":"2022-03-21T14:06:54.866898+0000","last_active":"2022-03-21T14:06:54.867382+0000","last_peered":"2022-03-21T14:06:54.867382+0000","last_clean":"2022-03-21T14:06:54.867382+0000","last_became_active":"2022-03-21T14:06:54.865450+0000","last_became_peered":"2022-03-21T14:06:54.865450+0000","last_unstale":"2022-03-21T14:06:54.867382+0000","last_undegraded":"2022-03-21T14:06:54.867382+0000","last_fullsized":"2022-03-21T14:06:54.867382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:39:42.699187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.328729+0000","last_change":"2022-03-21T14:06:54.328313+0000","last_active":"2022-03-21T14:06:54.328729+0000","last_peered":"2022-03-21T14:06:54.328729+0000","last_clean":"2022-03-21T14:06:54.328729+0000","last_became_active":"2022-03-21T14:06:54.327989+0000","last_became_peered":"2022-03-21T14:06:54.327989+0000","last_unstale":"2022-03-21T14:06:54.328729+0000","last_undegraded":"2022-03-21T14:06:54.328729+0000","last_fullsized":"2022-03-21T14:06:54.328729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:59:23.877877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861081+0000","last_change":"2022-03-21T14:06:54.861081+0000","last_active":"2022-03-21T14:06:54.861081+0000","last_peered":"2022-03-21T14:06:54.861081+0000","last_clean":"2022-03-21T14:06:54.861081+0000","last_became_active":"2022-03-21T14:06:54.860526+0000","last_became_peered":"2022-03-21T14:06:54.860526+0000","last_unstale":"2022-03-21T14:06:54.861081+0000","last_undegraded":"2022-03-21T14:06:54.861081+0000","last_fullsized":"2022-03-21T14:06:54.861081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:17:37.984659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.846891+0000","last_change":"2022-03-21T14:06:54.317279+0000","last_active":"2022-03-21T14:06:54.846891+0000","last_peered":"2022-03-21T14:06:54.846891+0000","last_clean":"2022-03-21T14:06:54.846891+0000","last_became_active":"2022-03-21T14:06:54.316863+0000","last_became_peered":"2022-03-21T14:06:54.316863+0000","last_unstale":"2022-03-21T14:06:54.846891+0000","last_undegraded":"2022-03-21T14:06:54.846891+0000","last_fullsized":"2022-03-21T14:06:54.846891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:17.370559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881566+0000","last_change":"2022-03-21T14:06:54.878112+0000","last_active":"2022-03-21T14:06:54.881566+0000","last_peered":"2022-03-21T14:06:54.881566+0000","last_clean":"2022-03-21T14:06:54.881566+0000","last_became_active":"2022-03-21T14:06:54.877763+0000","last_became_peered":"2022-03-21T14:06:54.877763+0000","last_unstale":"2022-03-21T14:06:54.881566+0000","last_undegraded":"2022-03-21T14:06:54.881566+0000","last_fullsized":"2022-03-21T14:06:54.881566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:51:50.763491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'3","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.885989+0000","last_change":"2022-03-21T14:06:54.333183+0000","last_active":"2022-03-21T14:06:54.885989+0000","last_peered":"2022-03-21T14:06:54.885989+0000","last_clean":"2022-03-21T14:06:54.885989+0000","last_became_active":"2022-03-21T14:06:54.332795+0000","last_became_peered":"2022-03-21T14:06:54.332795+0000","last_unstale":"2022-03-21T14:06:54.885989+0000","last_undegraded":"2022-03-21T14:06:54.885989+0000","last_fullsized":"2022-03-21T14:06:54.885989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:29.651861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.862390+0000","last_change":"2022-03-21T14:06:54.862390+0000","last_active":"2022-03-21T14:06:54.862390+0000","last_peered":"2022-03-21T14:06:54.862390+0000","last_clean":"2022-03-21T14:06:54.862390+0000","last_became_active":"2022-03-21T14:06:54.862008+0000","last_became_peered":"2022-03-21T14:06:54.862008+0000","last_unstale":"2022-03-21T14:06:54.862390+0000","last_undegraded":"2022-03-21T14:06:54.862390+0000","last_fullsized":"2022-03-21T14:06:54.862390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:53:50.255768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'6","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875617+0000","last_change":"2022-03-21T14:06:54.315167+0000","last_active":"2022-03-21T14:06:54.875617+0000","last_peered":"2022-03-21T14:06:54.875617+0000","last_clean":"2022-03-21T14:06:54.875617+0000","last_became_active":"2022-03-21T14:06:54.314432+0000","last_became_peered":"2022-03-21T14:06:54.314432+0000","last_unstale":"2022-03-21T14:06:54.875617+0000","last_undegraded":"2022-03-21T14:06:54.875617+0000","last_fullsized":"2022-03-21T14:06:54.875617+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:48:27.504920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860333+0000","last_change":"2022-03-21T14:06:54.859057+0000","last_active":"2022-03-21T14:06:54.860333+0000","last_peered":"2022-03-21T14:06:54.860333+0000","last_clean":"2022-03-21T14:06:54.860333+0000","last_became_active":"2022-03-21T14:06:54.858674+0000","last_became_peered":"2022-03-21T14:06:54.858674+0000","last_unstale":"2022-03-21T14:06:54.860333+0000","last_undegraded":"2022-03-21T14:06:54.860333+0000","last_fullsized":"2022-03-21T14:06:54.860333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:38:29.386832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881770+0000","last_change":"2022-03-21T14:06:54.323631+0000","last_active":"2022-03-21T14:06:54.881770+0000","last_peered":"2022-03-21T14:06:54.881770+0000","last_clean":"2022-03-21T14:06:54.881770+0000","last_became_active":"2022-03-21T14:06:54.323179+0000","last_became_peered":"2022-03-21T14:06:54.323179+0000","last_unstale":"2022-03-21T14:06:54.881770+0000","last_undegraded":"2022-03-21T14:06:54.881770+0000","last_fullsized":"2022-03-21T14:06:54.881770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:35:04.480559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.879029+0000","last_change":"2022-03-21T14:06:54.878235+0000","last_active":"2022-03-21T14:06:54.879029+0000","last_peered":"2022-03-21T14:06:54.879029+0000","last_clean":"2022-03-21T14:06:54.879029+0000","last_became_active":"2022-03-21T14:06:54.877650+0000","last_became_peered":"2022-03-21T14:06:54.877650+0000","last_unstale":"2022-03-21T14:06:54.879029+0000","last_undegraded":"2022-03-21T14:06:54.879029+0000","last_fullsized":"2022-03-21T14:06:54.879029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:06:58.716437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.843743+0000","last_change":"2022-03-21T14:06:54.316053+0000","last_active":"2022-03-21T14:06:54.843743+0000","last_peered":"2022-03-21T14:06:54.843743+0000","last_clean":"2022-03-21T14:06:54.843743+0000","last_became_active":"2022-03-21T14:06:54.315685+0000","last_became_peered":"2022-03-21T14:06:54.315685+0000","last_unstale":"2022-03-21T14:06:54.843743+0000","last_undegraded":"2022-03-21T14:06:54.843743+0000","last_fullsized":"2022-03-21T14:06:54.843743+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:16:47.746232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.879582+0000","last_change":"2022-03-21T14:06:54.878905+0000","last_active":"2022-03-21T14:06:54.879582+0000","last_peered":"2022-03-21T14:06:54.879582+0000","last_clean":"2022-03-21T14:06:54.879582+0000","last_became_active":"2022-03-21T14:06:54.878341+0000","last_became_peered":"2022-03-21T14:06:54.878341+0000","last_unstale":"2022-03-21T14:06:54.879582+0000","last_undegraded":"2022-03-21T14:06:54.879582+0000","last_fullsized":"2022-03-21T14:06:54.879582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:05:50.071721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.855232+0000","last_change":"2022-03-21T14:06:54.313255+0000","last_active":"2022-03-21T14:06:54.855232+0000","last_peered":"2022-03-21T14:06:54.855232+0000","last_clean":"2022-03-21T14:06:54.855232+0000","last_became_active":"2022-03-21T14:06:54.312849+0000","last_became_peered":"2022-03-21T14:06:54.312849+0000","last_unstale":"2022-03-21T14:06:54.855232+0000","last_undegraded":"2022-03-21T14:06:54.855232+0000","last_fullsized":"2022-03-21T14:06:54.855232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:19:59.556267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860448+0000","last_change":"2022-03-21T14:06:54.857792+0000","last_active":"2022-03-21T14:06:54.860448+0000","last_peered":"2022-03-21T14:06:54.860448+0000","last_clean":"2022-03-21T14:06:54.860448+0000","last_became_active":"2022-03-21T14:06:54.857425+0000","last_became_peered":"2022-03-21T14:06:54.857425+0000","last_unstale":"2022-03-21T14:06:54.860448+0000","last_undegraded":"2022-03-21T14:06:54.860448+0000","last_fullsized":"2022-03-21T14:06:54.860448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:51:33.943688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'8","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.887429+0000","last_change":"2022-03-21T14:06:54.335082+0000","last_active":"2022-03-21T14:06:54.887429+0000","last_peered":"2022-03-21T14:06:54.887429+0000","last_clean":"2022-03-21T14:06:54.887429+0000","last_became_active":"2022-03-21T14:06:54.334703+0000","last_became_peered":"2022-03-21T14:06:54.334703+0000","last_unstale":"2022-03-21T14:06:54.887429+0000","last_undegraded":"2022-03-21T14:06:54.887429+0000","last_fullsized":"2022-03-21T14:06:54.887429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:12.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859124+0000","last_change":"2022-03-21T14:06:54.858370+0000","last_active":"2022-03-21T14:06:54.859124+0000","last_peered":"2022-03-21T14:06:54.859124+0000","last_clean":"2022-03-21T14:06:54.859124+0000","last_became_active":"2022-03-21T14:06:54.858074+0000","last_became_peered":"2022-03-21T14:06:54.858074+0000","last_unstale":"2022-03-21T14:06:54.859124+0000","last_undegraded":"2022-03-21T14:06:54.859124+0000","last_fullsized":"2022-03-21T14:06:54.859124+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:55:49.919857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861944+0000","last_change":"2022-03-21T14:06:54.315652+0000","last_active":"2022-03-21T14:06:54.861944+0000","last_peered":"2022-03-21T14:06:54.861944+0000","last_clean":"2022-03-21T14:06:54.861944+0000","last_became_active":"2022-03-21T14:06:54.315317+0000","last_became_peered":"2022-03-21T14:06:54.315317+0000","last_unstale":"2022-03-21T14:06:54.861944+0000","last_undegraded":"2022-03-21T14:06:54.861944+0000","last_fullsized":"2022-03-21T14:06:54.861944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:55:42.394760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859484+0000","last_change":"2022-03-21T14:06:54.856509+0000","last_active":"2022-03-21T14:06:54.859484+0000","last_peered":"2022-03-21T14:06:54.859484+0000","last_clean":"2022-03-21T14:06:54.859484+0000","last_became_active":"2022-03-21T14:06:54.855995+0000","last_became_peered":"2022-03-21T14:06:54.855995+0000","last_unstale":"2022-03-21T14:06:54.859484+0000","last_undegraded":"2022-03-21T14:06:54.859484+0000","last_fullsized":"2022-03-21T14:06:54.859484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:02:52.429679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.849905+0000","last_change":"2022-03-21T14:06:54.336921+0000","last_active":"2022-03-21T14:06:54.849905+0000","last_peered":"2022-03-21T14:06:54.849905+0000","last_clean":"2022-03-21T14:06:54.849905+0000","last_became_active":"2022-03-21T14:06:54.336560+0000","last_became_peered":"2022-03-21T14:06:54.336560+0000","last_unstale":"2022-03-21T14:06:54.849905+0000","last_undegraded":"2022-03-21T14:06:54.849905+0000","last_fullsized":"2022-03-21T14:06:54.849905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:26:06.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865783+0000","last_change":"2022-03-21T14:06:54.865783+0000","last_active":"2022-03-21T14:06:54.865783+0000","last_peered":"2022-03-21T14:06:54.865783+0000","last_clean":"2022-03-21T14:06:54.865783+0000","last_became_active":"2022-03-21T14:06:54.865399+0000","last_became_peered":"2022-03-21T14:06:54.865399+0000","last_unstale":"2022-03-21T14:06:54.865783+0000","last_undegraded":"2022-03-21T14:06:54.865783+0000","last_fullsized":"2022-03-21T14:06:54.865783+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:15:19.155641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847285+0000","last_change":"2022-03-21T14:06:54.321852+0000","last_active":"2022-03-21T14:06:54.847285+0000","last_peered":"2022-03-21T14:06:54.847285+0000","last_clean":"2022-03-21T14:06:54.847285+0000","last_became_active":"2022-03-21T14:06:54.321388+0000","last_became_peered":"2022-03-21T14:06:54.321388+0000","last_unstale":"2022-03-21T14:06:54.847285+0000","last_undegraded":"2022-03-21T14:06:54.847285+0000","last_fullsized":"2022-03-21T14:06:54.847285+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:58:18.274331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871690+0000","last_change":"2022-03-21T14:06:54.871690+0000","last_active":"2022-03-21T14:06:54.871690+0000","last_peered":"2022-03-21T14:06:54.871690+0000","last_clean":"2022-03-21T14:06:54.871690+0000","last_became_active":"2022-03-21T14:06:54.871272+0000","last_became_peered":"2022-03-21T14:06:54.871272+0000","last_unstale":"2022-03-21T14:06:54.871690+0000","last_undegraded":"2022-03-21T14:06:54.871690+0000","last_fullsized":"2022-03-21T14:06:54.871690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:17.999041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.863960+0000","last_change":"2022-03-21T14:06:54.860140+0000","last_active":"2022-03-21T14:06:54.863960+0000","last_peered":"2022-03-21T14:06:54.863960+0000","last_clean":"2022-03-21T14:06:54.863960+0000","last_became_active":"2022-03-21T14:06:54.859700+0000","last_became_peered":"2022-03-21T14:06:54.859700+0000","last_unstale":"2022-03-21T14:06:54.863960+0000","last_undegraded":"2022-03-21T14:06:54.863960+0000","last_fullsized":"2022-03-21T14:06:54.863960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:57:46.264617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.847150+0000","last_change":"2022-03-21T14:06:54.334370+0000","last_active":"2022-03-21T14:06:54.847150+0000","last_peered":"2022-03-21T14:06:54.847150+0000","last_clean":"2022-03-21T14:06:54.847150+0000","last_became_active":"2022-03-21T14:06:54.333957+0000","last_became_peered":"2022-03-21T14:06:54.333957+0000","last_unstale":"2022-03-21T14:06:54.847150+0000","last_undegraded":"2022-03-21T14:06:54.847150+0000","last_fullsized":"2022-03-21T14:06:54.847150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:29.048808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.876136+0000","last_change":"2022-03-21T14:06:54.873196+0000","last_active":"2022-03-21T14:06:54.876136+0000","last_peered":"2022-03-21T14:06:54.876136+0000","last_clean":"2022-03-21T14:06:54.876136+0000","last_became_active":"2022-03-21T14:06:54.872873+0000","last_became_peered":"2022-03-21T14:06:54.872873+0000","last_unstale":"2022-03-21T14:06:54.876136+0000","last_undegraded":"2022-03-21T14:06:54.876136+0000","last_fullsized":"2022-03-21T14:06:54.876136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:01:38.878919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.857028+0000","last_change":"2022-03-21T14:06:54.315511+0000","last_active":"2022-03-21T14:06:54.857028+0000","last_peered":"2022-03-21T14:06:54.857028+0000","last_clean":"2022-03-21T14:06:54.857028+0000","last_became_active":"2022-03-21T14:06:54.315144+0000","last_became_peered":"2022-03-21T14:06:54.315144+0000","last_unstale":"2022-03-21T14:06:54.857028+0000","last_undegraded":"2022-03-21T14:06:54.857028+0000","last_fullsized":"2022-03-21T14:06:54.857028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:30:11.627760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850861+0000","last_change":"2022-03-21T14:05:44.276967+0000","last_active":"2022-03-21T14:06:54.850861+0000","last_peered":"2022-03-21T14:06:54.850861+0000","last_clean":"2022-03-21T14:06:54.850861+0000","last_became_active":"2022-03-21T14:05:44.276297+0000","last_became_peered":"2022-03-21T14:05:44.276297+0000","last_unstale":"2022-03-21T14:06:54.850861+0000","last_undegraded":"2022-03-21T14:06:54.850861+0000","last_fullsized":"2022-03-21T14:06:54.850861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_clean_scrub_stamp":"2022-03-21T14:02:31.386985+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:07:49.689146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.861226+0000","last_change":"2022-03-21T14:06:54.860044+0000","last_active":"2022-03-21T14:06:54.861226+0000","last_peered":"2022-03-21T14:06:54.861226+0000","last_clean":"2022-03-21T14:06:54.861226+0000","last_became_active":"2022-03-21T14:06:54.859511+0000","last_became_peered":"2022-03-21T14:06:54.859511+0000","last_unstale":"2022-03-21T14:06:54.861226+0000","last_undegraded":"2022-03-21T14:06:54.861226+0000","last_fullsized":"2022-03-21T14:06:54.861226+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:50:01.100612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850971+0000","last_change":"2022-03-21T14:06:54.313788+0000","last_active":"2022-03-21T14:06:54.850971+0000","last_peered":"2022-03-21T14:06:54.850971+0000","last_clean":"2022-03-21T14:06:54.850971+0000","last_became_active":"2022-03-21T14:06:54.313328+0000","last_became_peered":"2022-03-21T14:06:54.313328+0000","last_unstale":"2022-03-21T14:06:54.850971+0000","last_undegraded":"2022-03-21T14:06:54.850971+0000","last_fullsized":"2022-03-21T14:06:54.850971+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:24:17.930196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.849363+0000","last_change":"2022-03-21T14:06:54.334716+0000","last_active":"2022-03-21T14:06:54.849363+0000","last_peered":"2022-03-21T14:06:54.849363+0000","last_clean":"2022-03-21T14:06:54.849363+0000","last_became_active":"2022-03-21T14:06:54.334384+0000","last_became_peered":"2022-03-21T14:06:54.334384+0000","last_unstale":"2022-03-21T14:06:54.849363+0000","last_undegraded":"2022-03-21T14:06:54.849363+0000","last_fullsized":"2022-03-21T14:06:54.849363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:29.598258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878630+0000","last_change":"2022-03-21T14:06:54.872297+0000","last_active":"2022-03-21T14:06:54.878630+0000","last_peered":"2022-03-21T14:06:54.878630+0000","last_clean":"2022-03-21T14:06:54.878630+0000","last_became_active":"2022-03-21T14:06:54.871557+0000","last_became_peered":"2022-03-21T14:06:54.871557+0000","last_unstale":"2022-03-21T14:06:54.878630+0000","last_undegraded":"2022-03-21T14:06:54.878630+0000","last_fullsized":"2022-03-21T14:06:54.878630+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:21:13.076650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-03-21T14:06:55.844414+0000","last_change":"2022-03-21T14:06:54.327446+0000","last_active":"2022-03-21T14:06:55.844414+0000","last_peered":"2022-03-21T14:06:55.844414+0000","last_clean":"2022-03-21T14:06:55.844414+0000","last_became_active":"2022-03-21T14:06:54.327102+0000","last_became_peered":"2022-03-21T14:06:54.327102+0000","last_unstale":"2022-03-21T14:06:55.844414+0000","last_undegraded":"2022-03-21T14:06:55.844414+0000","last_fullsized":"2022-03-21T14:06:55.844414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:28:18.309364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874840+0000","last_change":"2022-03-21T14:06:54.869348+0000","last_active":"2022-03-21T14:06:54.874840+0000","last_peered":"2022-03-21T14:06:54.874840+0000","last_clean":"2022-03-21T14:06:54.874840+0000","last_became_active":"2022-03-21T14:06:54.868917+0000","last_became_peered":"2022-03-21T14:06:54.868917+0000","last_unstale":"2022-03-21T14:06:54.874840+0000","last_undegraded":"2022-03-21T14:06:54.874840+0000","last_fullsized":"2022-03-21T14:06:54.874840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:38:56.199987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878893+0000","last_change":"2022-03-21T14:06:54.321746+0000","last_active":"2022-03-21T14:06:54.878893+0000","last_peered":"2022-03-21T14:06:54.878893+0000","last_clean":"2022-03-21T14:06:54.878893+0000","last_became_active":"2022-03-21T14:06:54.321294+0000","last_became_peered":"2022-03-21T14:06:54.321294+0000","last_unstale":"2022-03-21T14:06:54.878893+0000","last_undegraded":"2022-03-21T14:06:54.878893+0000","last_fullsized":"2022-03-21T14:06:54.878893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:06:26.961696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875678+0000","last_change":"2022-03-21T14:06:54.874949+0000","last_active":"2022-03-21T14:06:54.875678+0000","last_peered":"2022-03-21T14:06:54.875678+0000","last_clean":"2022-03-21T14:06:54.875678+0000","last_became_active":"2022-03-21T14:06:54.874429+0000","last_became_peered":"2022-03-21T14:06:54.874429+0000","last_unstale":"2022-03-21T14:06:54.875678+0000","last_undegraded":"2022-03-21T14:06:54.875678+0000","last_fullsized":"2022-03-21T14:06:54.875678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:43:38.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.327133+0000","last_change":"2022-03-21T14:06:54.326806+0000","last_active":"2022-03-21T14:06:54.327133+0000","last_peered":"2022-03-21T14:06:54.327133+0000","last_clean":"2022-03-21T14:06:54.327133+0000","last_became_active":"2022-03-21T14:06:54.326405+0000","last_became_peered":"2022-03-21T14:06:54.326405+0000","last_unstale":"2022-03-21T14:06:54.327133+0000","last_undegraded":"2022-03-21T14:06:54.327133+0000","last_fullsized":"2022-03-21T14:06:54.327133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:29.562293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.864674+0000","last_change":"2022-03-21T14:06:54.860972+0000","last_active":"2022-03-21T14:06:54.864674+0000","last_peered":"2022-03-21T14:06:54.864674+0000","last_clean":"2022-03-21T14:06:54.864674+0000","last_became_active":"2022-03-21T14:06:54.860557+0000","last_became_peered":"2022-03-21T14:06:54.860557+0000","last_unstale":"2022-03-21T14:06:54.864674+0000","last_undegraded":"2022-03-21T14:06:54.864674+0000","last_fullsized":"2022-03-21T14:06:54.864674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:28:45.487047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.858436+0000","last_change":"2022-03-21T14:06:54.320161+0000","last_active":"2022-03-21T14:06:54.858436+0000","last_peered":"2022-03-21T14:06:54.858436+0000","last_clean":"2022-03-21T14:06:54.858436+0000","last_became_active":"2022-03-21T14:06:54.318104+0000","last_became_peered":"2022-03-21T14:06:54.318104+0000","last_unstale":"2022-03-21T14:06:54.858436+0000","last_undegraded":"2022-03-21T14:06:54.858436+0000","last_fullsized":"2022-03-21T14:06:54.858436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:12.154964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872757+0000","last_change":"2022-03-21T14:06:54.870272+0000","last_active":"2022-03-21T14:06:54.872757+0000","last_peered":"2022-03-21T14:06:54.872757+0000","last_clean":"2022-03-21T14:06:54.872757+0000","last_became_active":"2022-03-21T14:06:54.869649+0000","last_became_peered":"2022-03-21T14:06:54.869649+0000","last_unstale":"2022-03-21T14:06:54.872757+0000","last_undegraded":"2022-03-21T14:06:54.872757+0000","last_fullsized":"2022-03-21T14:06:54.872757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:09:57.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.850547+0000","last_change":"2022-03-21T14:06:54.323589+0000","last_active":"2022-03-21T14:06:54.850547+0000","last_peered":"2022-03-21T14:06:54.850547+0000","last_clean":"2022-03-21T14:06:54.850547+0000","last_became_active":"2022-03-21T14:06:54.323187+0000","last_became_peered":"2022-03-21T14:06:54.323187+0000","last_unstale":"2022-03-21T14:06:54.850547+0000","last_undegraded":"2022-03-21T14:06:54.850547+0000","last_fullsized":"2022-03-21T14:06:54.850547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:05:33.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.881436+0000","last_change":"2022-03-21T14:06:54.880453+0000","last_active":"2022-03-21T14:06:54.881436+0000","last_peered":"2022-03-21T14:06:54.881436+0000","last_clean":"2022-03-21T14:06:54.881436+0000","last_became_active":"2022-03-21T14:06:54.880078+0000","last_became_peered":"2022-03-21T14:06:54.880078+0000","last_unstale":"2022-03-21T14:06:54.881436+0000","last_undegraded":"2022-03-21T14:06:54.881436+0000","last_fullsized":"2022-03-21T14:06:54.881436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:37:36.812969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.872269+0000","last_change":"2022-03-21T14:06:54.314137+0000","last_active":"2022-03-21T14:06:54.872269+0000","last_peered":"2022-03-21T14:06:54.872269+0000","last_clean":"2022-03-21T14:06:54.872269+0000","last_became_active":"2022-03-21T14:06:54.313783+0000","last_became_peered":"2022-03-21T14:06:54.313783+0000","last_unstale":"2022-03-21T14:06:54.872269+0000","last_undegraded":"2022-03-21T14:06:54.872269+0000","last_fullsized":"2022-03-21T14:06:54.872269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:58.358466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.868862+0000","last_change":"2022-03-21T14:06:54.867675+0000","last_active":"2022-03-21T14:06:54.868862+0000","last_peered":"2022-03-21T14:06:54.868862+0000","last_clean":"2022-03-21T14:06:54.868862+0000","last_became_active":"2022-03-21T14:06:54.867097+0000","last_became_peered":"2022-03-21T14:06:54.867097+0000","last_unstale":"2022-03-21T14:06:54.868862+0000","last_undegraded":"2022-03-21T14:06:54.868862+0000","last_fullsized":"2022-03-21T14:06:54.868862+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:07:22.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.852660+0000","last_change":"2022-03-21T14:06:54.333269+0000","last_active":"2022-03-21T14:06:54.852660+0000","last_peered":"2022-03-21T14:06:54.852660+0000","last_clean":"2022-03-21T14:06:54.852660+0000","last_became_active":"2022-03-21T14:06:54.332844+0000","last_became_peered":"2022-03-21T14:06:54.332844+0000","last_unstale":"2022-03-21T14:06:54.852660+0000","last_undegraded":"2022-03-21T14:06:54.852660+0000","last_fullsized":"2022-03-21T14:06:54.852660+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:45:33.278652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.871838+0000","last_change":"2022-03-21T14:06:54.869222+0000","last_active":"2022-03-21T14:06:54.871838+0000","last_peered":"2022-03-21T14:06:54.871838+0000","last_clean":"2022-03-21T14:06:54.871838+0000","last_became_active":"2022-03-21T14:06:54.868242+0000","last_became_peered":"2022-03-21T14:06:54.868242+0000","last_unstale":"2022-03-21T14:06:54.871838+0000","last_undegraded":"2022-03-21T14:06:54.871838+0000","last_fullsized":"2022-03-21T14:06:54.871838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:28:30.012341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882343+0000","last_change":"2022-03-21T14:06:54.317425+0000","last_active":"2022-03-21T14:06:54.882343+0000","last_peered":"2022-03-21T14:06:54.882343+0000","last_clean":"2022-03-21T14:06:54.882343+0000","last_became_active":"2022-03-21T14:06:54.317035+0000","last_became_peered":"2022-03-21T14:06:54.317035+0000","last_unstale":"2022-03-21T14:06:54.882343+0000","last_undegraded":"2022-03-21T14:06:54.882343+0000","last_fullsized":"2022-03-21T14:06:54.882343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:24:33.693292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.868940+0000","last_change":"2022-03-21T14:06:54.867673+0000","last_active":"2022-03-21T14:06:54.868940+0000","last_peered":"2022-03-21T14:06:54.868940+0000","last_clean":"2022-03-21T14:06:54.868940+0000","last_became_active":"2022-03-21T14:06:54.867135+0000","last_became_peered":"2022-03-21T14:06:54.867135+0000","last_unstale":"2022-03-21T14:06:54.868940+0000","last_undegraded":"2022-03-21T14:06:54.868940+0000","last_fullsized":"2022-03-21T14:06:54.868940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:27:26.251570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.882196+0000","last_change":"2022-03-21T14:06:54.326963+0000","last_active":"2022-03-21T14:06:54.882196+0000","last_peered":"2022-03-21T14:06:54.882196+0000","last_clean":"2022-03-21T14:06:54.882196+0000","last_became_active":"2022-03-21T14:06:54.326592+0000","last_became_peered":"2022-03-21T14:06:54.326592+0000","last_unstale":"2022-03-21T14:06:54.882196+0000","last_undegraded":"2022-03-21T14:06:54.882196+0000","last_fullsized":"2022-03-21T14:06:54.882196+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:11:23.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.866815+0000","last_change":"2022-03-21T14:06:54.865257+0000","last_active":"2022-03-21T14:06:54.866815+0000","last_peered":"2022-03-21T14:06:54.866815+0000","last_clean":"2022-03-21T14:06:54.866815+0000","last_became_active":"2022-03-21T14:06:54.864119+0000","last_became_peered":"2022-03-21T14:06:54.864119+0000","last_unstale":"2022-03-21T14:06:54.866815+0000","last_undegraded":"2022-03-21T14:06:54.866815+0000","last_fullsized":"2022-03-21T14:06:54.866815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:45.228838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.865661+0000","last_change":"2022-03-21T14:06:54.864027+0000","last_active":"2022-03-21T14:06:54.865661+0000","last_peered":"2022-03-21T14:06:54.865661+0000","last_clean":"2022-03-21T14:06:54.865661+0000","last_became_active":"2022-03-21T14:06:54.863684+0000","last_became_peered":"2022-03-21T14:06:54.863684+0000","last_unstale":"2022-03-21T14:06:54.865661+0000","last_undegraded":"2022-03-21T14:06:54.865661+0000","last_fullsized":"2022-03-21T14:06:54.865661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:11.453033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860827+0000","last_change":"2022-03-21T14:06:54.860266+0000","last_active":"2022-03-21T14:06:54.860827+0000","last_peered":"2022-03-21T14:06:54.860827+0000","last_clean":"2022-03-21T14:06:54.860827+0000","last_became_active":"2022-03-21T14:06:54.859922+0000","last_became_peered":"2022-03-21T14:06:54.859922+0000","last_unstale":"2022-03-21T14:06:54.860827+0000","last_undegraded":"2022-03-21T14:06:54.860827+0000","last_fullsized":"2022-03-21T14:06:54.860827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:23.689919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875670+0000","last_change":"2022-03-21T14:06:54.875261+0000","last_active":"2022-03-21T14:06:54.875670+0000","last_peered":"2022-03-21T14:06:54.875670+0000","last_clean":"2022-03-21T14:06:54.875670+0000","last_became_active":"2022-03-21T14:06:54.874826+0000","last_became_peered":"2022-03-21T14:06:54.874826+0000","last_unstale":"2022-03-21T14:06:54.875670+0000","last_undegraded":"2022-03-21T14:06:54.875670+0000","last_fullsized":"2022-03-21T14:06:54.875670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:02.673575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874016+0000","last_change":"2022-03-21T14:06:54.867997+0000","last_active":"2022-03-21T14:06:54.874016+0000","last_peered":"2022-03-21T14:06:54.874016+0000","last_clean":"2022-03-21T14:06:54.874016+0000","last_became_active":"2022-03-21T14:06:54.867624+0000","last_became_peered":"2022-03-21T14:06:54.867624+0000","last_unstale":"2022-03-21T14:06:54.874016+0000","last_undegraded":"2022-03-21T14:06:54.874016+0000","last_fullsized":"2022-03-21T14:06:54.874016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:17:37.552083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878655+0000","last_change":"2022-03-21T14:06:54.876868+0000","last_active":"2022-03-21T14:06:54.878655+0000","last_peered":"2022-03-21T14:06:54.878655+0000","last_clean":"2022-03-21T14:06:54.878655+0000","last_became_active":"2022-03-21T14:06:54.876518+0000","last_became_peered":"2022-03-21T14:06:54.876518+0000","last_unstale":"2022-03-21T14:06:54.878655+0000","last_undegraded":"2022-03-21T14:06:54.878655+0000","last_fullsized":"2022-03-21T14:06:54.878655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:50:48.947076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860291+0000","last_change":"2022-03-21T14:06:54.858384+0000","last_active":"2022-03-21T14:06:54.860291+0000","last_peered":"2022-03-21T14:06:54.860291+0000","last_clean":"2022-03-21T14:06:54.860291+0000","last_became_active":"2022-03-21T14:06:54.857183+0000","last_became_peered":"2022-03-21T14:06:54.857183+0000","last_unstale":"2022-03-21T14:06:54.860291+0000","last_undegraded":"2022-03-21T14:06:54.860291+0000","last_fullsized":"2022-03-21T14:06:54.860291+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:03:58.964777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.859536+0000","last_change":"2022-03-21T14:06:54.859071+0000","last_active":"2022-03-21T14:06:54.859536+0000","last_peered":"2022-03-21T14:06:54.859536+0000","last_clean":"2022-03-21T14:06:54.859536+0000","last_became_active":"2022-03-21T14:06:54.858740+0000","last_became_peered":"2022-03-21T14:06:54.858740+0000","last_unstale":"2022-03-21T14:06:54.859536+0000","last_undegraded":"2022-03-21T14:06:54.859536+0000","last_fullsized":"2022-03-21T14:06:54.859536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:12.263178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878527+0000","last_change":"2022-03-21T14:06:54.877506+0000","last_active":"2022-03-21T14:06:54.878527+0000","last_peered":"2022-03-21T14:06:54.878527+0000","last_clean":"2022-03-21T14:06:54.878527+0000","last_became_active":"2022-03-21T14:06:54.877201+0000","last_became_peered":"2022-03-21T14:06:54.877201+0000","last_unstale":"2022-03-21T14:06:54.878527+0000","last_undegraded":"2022-03-21T14:06:54.878527+0000","last_fullsized":"2022-03-21T14:06:54.878527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:27:57.264889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.875620+0000","last_change":"2022-03-21T14:06:54.875160+0000","last_active":"2022-03-21T14:06:54.875620+0000","last_peered":"2022-03-21T14:06:54.875620+0000","last_clean":"2022-03-21T14:06:54.875620+0000","last_became_active":"2022-03-21T14:06:54.874453+0000","last_became_peered":"2022-03-21T14:06:54.874453+0000","last_unstale":"2022-03-21T14:06:54.875620+0000","last_undegraded":"2022-03-21T14:06:54.875620+0000","last_fullsized":"2022-03-21T14:06:54.875620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:53.657732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.878457+0000","last_change":"2022-03-21T14:06:54.876605+0000","last_active":"2022-03-21T14:06:54.878457+0000","last_peered":"2022-03-21T14:06:54.878457+0000","last_clean":"2022-03-21T14:06:54.878457+0000","last_became_active":"2022-03-21T14:06:54.876241+0000","last_became_peered":"2022-03-21T14:06:54.876241+0000","last_unstale":"2022-03-21T14:06:54.878457+0000","last_undegraded":"2022-03-21T14:06:54.878457+0000","last_fullsized":"2022-03-21T14:06:54.878457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:38.713657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.874849+0000","last_change":"2022-03-21T14:06:54.874849+0000","last_active":"2022-03-21T14:06:54.874849+0000","last_peered":"2022-03-21T14:06:54.874849+0000","last_clean":"2022-03-21T14:06:54.874849+0000","last_became_active":"2022-03-21T14:06:54.874203+0000","last_became_peered":"2022-03-21T14:06:54.874203+0000","last_unstale":"2022-03-21T14:06:54.874849+0000","last_undegraded":"2022-03-21T14:06:54.874849+0000","last_fullsized":"2022-03-21T14:06:54.874849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:17:32.906357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.870601+0000","last_change":"2022-03-21T14:06:54.869399+0000","last_active":"2022-03-21T14:06:54.870601+0000","last_peered":"2022-03-21T14:06:54.870601+0000","last_clean":"2022-03-21T14:06:54.870601+0000","last_became_active":"2022-03-21T14:06:54.869078+0000","last_became_peered":"2022-03-21T14:06:54.869078+0000","last_unstale":"2022-03-21T14:06:54.870601+0000","last_undegraded":"2022-03-21T14:06:54.870601+0000","last_fullsized":"2022-03-21T14:06:54.870601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:39:49.492701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-03-21T14:06:54.860566+0000","last_change":"2022-03-21T14:06:54.858492+0000","last_active":"2022-03-21T14:06:54.860566+0000","last_peered":"2022-03-21T14:06:54.860566+0000","last_clean":"2022-03-21T14:06:54.860566+0000","last_became_active":"2022-03-21T14:06:54.858113+0000","last_became_peered":"2022-03-21T14:06:54.858113+0000","last_unstale":"2022-03-21T14:06:54.860566+0000","last_undegraded":"2022-03-21T14:06:54.860566+0000","last_fullsized":"2022-03-21T14:06:54.860566+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:41:03.763238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481806,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":880640,"data_stored":420957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":10,"up_from":70,"seq":300647710737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":1388,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":1421312,"data_stored":942746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":1,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35999999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.433}]},{"osd":6,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.58099999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59399999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":9,"up_from":65,"seq":279172874262,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":421523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.63900000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.012}]},{"osd":3,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.98699999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.84699999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.0249999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":1.093},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.93899999999999995}]}]},{"osd":8,"up_from":57,"seq":244813135898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":872448,"data_stored":412664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.39500000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.437},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.501},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53100000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:06:20 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57399999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56299999999999994}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6704,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738448,"statfs":{"total":95995035648,"available":95988170752,"internally_reserved":0,"allocated":1417216,"data_stored":950451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84199999999999997,"5min":0.88,"15min":0.88},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68400000000000005}]},{"osd":3,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66800000000000004,"15min":0.66400000000000003},"min":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73499999999999999,"15min":0.73399999999999999},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.98099999999999998,"5min":1.103,"15min":1.103},"last":0.58199999999999996}]},{"osd":4,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.58799999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.627},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.78800000000000003,"15min":0.77700000000000002},"min":{"1min":0.69899999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.77800000000000002,"15min":0.77100000000000002},"min":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.66300000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.73799999999999999},"min":{"1min":0.57899999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73699999999999999,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71599999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:23 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.753}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.745},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73699999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6116,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739036,"statfs":{"total":95995035648,"available":95988772864,"internally_reserved":0,"allocated":880640,"data_stored":420957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.44400000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.63},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.70599999999999996,"15min":0.71099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.71399999999999997,"15min":0.69999999999999996},"min":{"1min":0.622,"5min":0.495,"15min":0.495},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.70199999999999996,"15min":0.69199999999999995},"min":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.048,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.77900000000000003,"15min":0.77800000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1679999999999999,"5min":1.208,"15min":1.208},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.73799999999999999,"15min":0.72999999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76600000000000001,"15min":0.76000000000000001},"min":{"1min":0.55500000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.996}]},{"osd":4,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.63800000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.099,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.86299999999999999,"15min":0.86099999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.137,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:31 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81299999999999994,"15min":0.81000000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.103},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89500000000000002,"15min":0.88700000000000001},"min":{"1min":0.70899999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.3420000000000001}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.91600000000000004},"min":{"1min":0.70399999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.306},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.163}]},{"osd":9,"last update":"Mon Mar 21 14:06:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.413},{"interface":"front","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.2529999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:06:46 2022","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":1.071}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7012,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738140,"statfs":{"total":95995035648,"available":95987855360,"internally_reserved":0,"allocated":880640,"data_stored":413947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:15 2022","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53700000000000003,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78000000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.374}]},{"osd":3,"last update":"Mon Mar 21 14:06:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89100000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.67400000000000004},"min":{"1min":0.47499999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Mar 21 14:06:30 2022","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.29699999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:50 2022","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.45100000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:06:15 2022","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.66500000000000004,"15min":0.66900000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:35 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83799999999999997,"5min":0.871,"15min":0.871},"last":0.433}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64600000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:06:20 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Mon Mar 21 14:06:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.53400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7016,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738136,"statfs":{"total":95995035648,"available":95987851264,"internally_reserved":0,"allocated":884736,"data_stored":421047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 21 14:06:17 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.60999999999999999,"15min":0.57799999999999996},"min":{"1min":0.52200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.55700000000000005},"min":{"1min":0.504,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon Mar 21 14:06:36 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84999999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.71499999999999997,"15min":0.68999999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:05:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.58599999999999997,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.72799999999999998,"15min":0.67600000000000005},"min":{"1min":0.58799999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.161,"5min":1.173,"15min":1.173},"last":0.871},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.60799999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.89400000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78400000000000003,"15min":0.77200000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.78700000000000003,"15min":0.77700000000000002},"min":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0609999999999999,"5min":1.153,"15min":1.153},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.77500000000000002,"15min":0.749},"min":{"1min":0.57699999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.83699999999999997,"15min":0.80500000000000005},"min":{"1min":0.56100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.93799999999999994}]},{"osd":7,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92100000000000004,"15min":0.88500000000000001},"min":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.766,"5min":5.766,"15min":5.766},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.1890000000000001,"5min":1.224,"15min":1.224},"last":0.70199999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:57 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.96599999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.27,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.997,"15min":1.0009999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.3500000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.90900000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.96199999999999997}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.94699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7092,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738060,"statfs":{"total":95995035648,"available":95987773440,"internally_reserved":0,"allocated":897024,"data_stored":422539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:38 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:06:38 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:06:05 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.623},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Mar 21 14:06:28 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.67100000000000004,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 21 14:06:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.622},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:54 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.59899999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.51300000000000001}]},{"osd":9,"last update":"Mon Mar 21 14:06:23 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.68300000000000005}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.69299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182450,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":888832,"data_stored":421061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.48099999999999998,"15min":0.45000000000000001},"min":{"1min":0.39600000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80500000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.58899999999999997,"15min":0.55500000000000005},"min":{"1min":0.48499999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.504}]},{"osd":1,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.628,"15min":0.58399999999999996},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.64000000000000001,"15min":0.61199999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.69399999999999995}]},{"osd":2,"last update":"Mon Mar 21 14:06:09 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.72199999999999998,"15min":0.69999999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.58699999999999997,"5min":0.504,"15min":0.504},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.76800000000000002}]},{"osd":4,"last update":"Mon Mar 21 14:06:24 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.70699999999999996,"15min":0.66500000000000004},"min":{"1min":0.58199999999999996,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.70699999999999996,"15min":0.67300000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.76300000000000001}]},{"osd":5,"last update":"Mon Mar 21 14:06:49 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81100000000000005,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.77500000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.119,"15min":1.119},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:06:13 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.876,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.498,"15min":0.498},"max":{"1min":1.2849999999999999,"5min":1.345,"15min":1.345},"last":0.96399999999999997}]},{"osd":7,"last update":"Mon Mar 21 14:06:33 2022","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.93500000000000005,"15min":0.90300000000000002},"min":{"1min":0.68700000000000006,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.84899999999999998,"15min":0.83699999999999997},"min":{"1min":0.73099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.78500000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.96199999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:06:18 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.995}]},{"osd":10,"last update":"Mon Mar 21 14:06:41 2022","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.02},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.82399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":420979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.58199999999999996,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.745},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.46999999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.89800000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.46400000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76800000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.65600000000000003},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.81299999999999994}]},{"osd":7,"last update":"Mon Mar 21 14:06:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.56799999999999995,"15min":0.54900000000000004},"min":{"1min":0.32300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.374,"15min":0.374},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.98199999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88500000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0469999999999999,"5min":1.05,"15min":1.05},"last":1.0349999999999999}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":1.05}]},{"osd":10,"last update":"Mon Mar 21 14:06:41 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.095}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":1388,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":1421312,"data_stored":949744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.39500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.44},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.80200000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Mar 21 14:06:14 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48999999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.72199999999999998}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.502},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64500000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:06:19 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.78500000000000003}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:07:01.363 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-03-21T14:07:01.364 INFO:tasks.cephadm:Setting up client nodes... 2022-03-21T14:07:01.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-21T14:07:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:01 smithi002 ceph-mon[17167]: pgmap v272: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 167 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-03-21T14:07:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:01 smithi002 ceph-mon[17167]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:01 smithi016 ceph-mon[22010]: pgmap v272: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 167 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-03-21T14:07:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:01 smithi016 ceph-mon[22010]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:01 smithi079 ceph-mon[23158]: pgmap v272: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 167 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-03-21T14:07:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:01 smithi079 ceph-mon[23158]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:02 smithi002 ceph-mon[17167]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:02 smithi016 ceph-mon[22010]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:02 smithi079 ceph-mon[23158]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:03.065 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:03.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 -- 172.21.15.2:0/1088887849 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3900ff120 msgr2=0x7fc390101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:03.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 --2- 172.21.15.2:0/1088887849 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3900ff120 0x7fc390101520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc38400b820 tx=0x7fc384005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 -- 172.21.15.2:0/1088887849 shutdown_connections 2022-03-21T14:07:03.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 --2- 172.21.15.2:0/1088887849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc390065fd0 0x7fc3900feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 --2- 172.21.15.2:0/1088887849 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 0x7fc390104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.550+0000 7fc394c4d700 1 --2- 172.21.15.2:0/1088887849 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3900ff120 0x7fc390101520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.551+0000 7fc394c4d700 1 -- 172.21.15.2:0/1088887849 >> 172.21.15.2:0/1088887849 conn(0x7fc3900f6620 msgr2=0x7fc3900f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:03.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.551+0000 7fc394c4d700 1 -- 172.21.15.2:0/1088887849 shutdown_connections 2022-03-21T14:07:03.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.551+0000 7fc394c4d700 1 -- 172.21.15.2:0/1088887849 wait complete. 2022-03-21T14:07:03.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.551+0000 7fc394c4d700 1 Processor -- start 2022-03-21T14:07:03.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 -- start start 2022-03-21T14:07:03.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 0x7fc39010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:03.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 0x7fc39010d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:03.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 0x7fc390114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:03.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc390114950 con 0x7fc390101bb0 2022-03-21T14:07:03.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc390114ac0 con 0x7fc3900ff120 2022-03-21T14:07:03.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc394c4d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc390114c80 con 0x7fc390065fd0 2022-03-21T14:07:03.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc38dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 0x7fc39010d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:03.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc38e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 0x7fc39010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:03.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc38dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 0x7fc39010d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48386/0 (socket says 172.21.15.2:48386) 2022-03-21T14:07:03.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc38ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 0x7fc390114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:03.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.552+0000 7fc38dd9b700 1 -- 172.21.15.2:0/905912699 learned_addr learned my addr 172.21.15.2:0/905912699 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:03.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 -- 172.21.15.2:0/905912699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 msgr2=0x7fc39010d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:03.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 0x7fc39010d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 -- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 msgr2=0x7fc390114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:03.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 0x7fc390114410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 -- 172.21.15.2:0/905912699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38400b360 con 0x7fc390065fd0 2022-03-21T14:07:03.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc38e59c700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 0x7fc39010ce40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e870 tx=0x7fc380005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:03.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380010710 con 0x7fc390065fd0 2022-03-21T14:07:03.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.553+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc380010d20 con 0x7fc390065fd0 2022-03-21T14:07:03.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.554+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc390114f60 con 0x7fc390065fd0 2022-03-21T14:07:03.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.554+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380018e90 con 0x7fc390065fd0 2022-03-21T14:07:03.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.554+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc390115360 con 0x7fc390065fd0 2022-03-21T14:07:03.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.555+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc390107720 con 0x7fc390065fd0 2022-03-21T14:07:03.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.556+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc38000c040 con 0x7fc390065fd0 2022-03-21T14:07:03.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.558+0000 7fc37effd700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 0x7fc3780991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:03.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.558+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fc380014070 con 0x7fc390065fd0 2022-03-21T14:07:03.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.558+0000 7fc38dd9b700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 0x7fc3780991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:03.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.559+0000 7fc38dd9b700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 0x7fc3780991c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc38400b6f0 tx=0x7fc384005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:03.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.562+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc38004df40 con 0x7fc390065fd0 2022-03-21T14:07:03.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.823+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc3900f7fd0 con 0x7fc390065fd0 2022-03-21T14:07:03.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.829+0000 7fc37effd700 1 -- 172.21.15.2:0/905912699 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc38008cb80 con 0x7fc390065fd0 2022-03-21T14:07:03.830 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2022-03-21T14:07:03.831 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQCHhjhiPfUvMRAA24kVtn8eahJa9JgKAIPVnQ== 2022-03-21T14:07:03.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 msgr2=0x7fc3780991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:03.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 0x7fc3780991c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc38400b6f0 tx=0x7fc384005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 msgr2=0x7fc39010ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:03.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 0x7fc39010ce40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e870 tx=0x7fc380005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:03.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 shutdown_connections 2022-03-21T14:07:03.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc378096d10 0x7fc3780991c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc390101bb0 0x7fc390114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc390065fd0 0x7fc39010ce40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 --2- 172.21.15.2:0/905912699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3900ff120 0x7fc39010d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:03.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 >> 172.21.15.2:0/905912699 conn(0x7fc3900f6620 msgr2=0x7fc3900fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:03.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 shutdown_connections 2022-03-21T14:07:03.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:03.832+0000 7fc394c4d700 1 -- 172.21.15.2:0/905912699 wait complete. 2022-03-21T14:07:03.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:03 smithi002 ceph-mon[17167]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2022-03-21T14:07:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:03 smithi016 ceph-mon[22010]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2022-03-21T14:07:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:03 smithi079 ceph-mon[23158]: pgmap v273: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2022-03-21T14:07:04.370 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:07:04.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-03-21T14:07:04.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-03-21T14:07:04.477 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-21T14:07:04.478 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-03-21T14:07:04.478 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph mgr dump --format=json 2022-03-21T14:07:04.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:04 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/905912699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.579 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:04 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:04 smithi002 ceph-mon[17167]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-03-21T14:07:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:04 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/905912699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:04 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:04 smithi016 ceph-mon[22010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-03-21T14:07:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:04 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/905912699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:04 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-03-21T14:07:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:04 smithi079 ceph-mon[23158]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-03-21T14:07:05.167 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:05 smithi002 ceph-mon[17167]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2022-03-21T14:07:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:05 smithi016 ceph-mon[22010]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2022-03-21T14:07:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:05 smithi079 ceph-mon[23158]: pgmap v274: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2022-03-21T14:07:06.407 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:06.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 -- 172.21.15.2:0/1089585569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48700660a0 msgr2=0x7f48701010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:06.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 --2- 172.21.15.2:0/1089585569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48700660a0 0x7f48701010b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48600098e0 tx=0x7f4860005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:06.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 -- 172.21.15.2:0/1089585569 shutdown_connections 2022-03-21T14:07:06.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 --2- 172.21.15.2:0/1089585569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870103f10 0x7f4870106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:06.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 --2- 172.21.15.2:0/1089585569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48700660a0 0x7f48701010b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:06.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 --2- 172.21.15.2:0/1089585569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48701015f0 0x7f48701039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:06.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 -- 172.21.15.2:0/1089585569 >> 172.21.15.2:0/1089585569 conn(0x7f48700f6760 msgr2=0x7f48700f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:06.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 -- 172.21.15.2:0/1089585569 shutdown_connections 2022-03-21T14:07:06.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.988+0000 7f4875fd4700 1 -- 172.21.15.2:0/1089585569 wait complete. 2022-03-21T14:07:06.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 Processor -- start 2022-03-21T14:07:06.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 -- start start 2022-03-21T14:07:06.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:06.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48701015f0 0x7f487010f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:06.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4870103f10 0x7f48701165d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:06.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870116b10 con 0x7f48701015f0 2022-03-21T14:07:06.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870116c80 con 0x7f48700660a0 2022-03-21T14:07:06.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.989+0000 7f4875fd4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870116e40 con 0x7f4870103f10 2022-03-21T14:07:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:06.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48390/0 (socket says 172.21.15.2:48390) 2022-03-21T14:07:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 -- 172.21.15.2:0/884962905 learned_addr learned my addr 172.21.15.2:0/884962905 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486effd700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48701015f0 0x7f487010f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486ffff700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4870103f10 0x7f48701165d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:06.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 -- 172.21.15.2:0/884962905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4870103f10 msgr2=0x7f48701165d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:06.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4870103f10 0x7f48701165d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:06.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 -- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48701015f0 msgr2=0x7f487010f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:06.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48701015f0 0x7f487010f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:06.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 -- 172.21.15.2:0/884962905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600093f0 con 0x7f48700660a0 2022-03-21T14:07:06.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.990+0000 7f486f7fe700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4860006010 tx=0x7f4860005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:06.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.991+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486001d070 con 0x7f48700660a0 2022-03-21T14:07:06.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.991+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4860005040 con 0x7f48700660a0 2022-03-21T14:07:06.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.991+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48701170c0 con 0x7f48700660a0 2022-03-21T14:07:07.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.991+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860017aa0 con 0x7f48700660a0 2022-03-21T14:07:07.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.991+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48701175a0 con 0x7f48700660a0 2022-03-21T14:07:07.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.992+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4870109aa0 con 0x7f48700660a0 2022-03-21T14:07:07.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.994+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f4860029030 con 0x7f48700660a0 2022-03-21T14:07:07.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.995+0000 7f4857fff700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 0x7f4858099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:07.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.995+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f48600c0ff0 con 0x7f48700660a0 2022-03-21T14:07:07.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.995+0000 7f486effd700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 0x7f4858099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:07.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.996+0000 7f486effd700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 0x7f4858099240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48700664f0 tx=0x7f4864005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:07.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:06.997+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f486008e380 con 0x7f48700660a0 2022-03-21T14:07:07.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.240+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f48700f8110 con 0x7f48700660a0 2022-03-21T14:07:07.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.246+0000 7f4857fff700 1 -- 172.21.15.2:0/884962905 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+241809 (secure 0 0 0) 0x7f4860027020 con 0x7f48700660a0 2022-03-21T14:07:07.252 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:07.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 msgr2=0x7f4858099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:07.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 0x7f4858099240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48700664f0 tx=0x7f4864005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:07.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 msgr2=0x7f487010f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:07.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4860006010 tx=0x7f4860005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:07.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 shutdown_connections 2022-03-21T14:07:07.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48700660a0 0x7f487010f000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:07.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4858096d90 0x7f4858099240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:07.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4870103f10 0x7f48701165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:07.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 --2- 172.21.15.2:0/884962905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48701015f0 0x7f487010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:07.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 >> 172.21.15.2:0/884962905 conn(0x7f48700f6760 msgr2=0x7f4870102230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:07.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 shutdown_connections 2022-03-21T14:07:07.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:07.252+0000 7f4875fd4700 1 -- 172.21.15.2:0/884962905 wait complete. 2022-03-21T14:07:07.714 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":613532934},{"type":"v1","addr":"172.21.15.2:6801","nonce":613532934}]},"active_addr":"172.21.15.2:6801/613532934","active_change":"2022-03-21T14:00:20.257280+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24109,"name":"y","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"z","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3664388638}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":132686861}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1477304639}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":268993181}]}]}} 2022-03-21T14:07:07.721 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-03-21T14:07:07.722 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-03-21T14:07:07.722 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-03-21T14:07:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:07 smithi002 ceph-mon[17167]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2022-03-21T14:07:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:07 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/884962905' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-03-21T14:07:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:07 smithi016 ceph-mon[22010]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2022-03-21T14:07:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:07 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/884962905' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-03-21T14:07:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:07 smithi079 ceph-mon[23158]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2022-03-21T14:07:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:07 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/884962905' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-03-21T14:07:08.372 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:09.523 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 -- 172.21.15.2:0/368731310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 msgr2=0x7fa0c00fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/368731310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c00fe680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0098e0 tx=0x7fa0bc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 -- 172.21.15.2:0/368731310 shutdown_connections 2022-03-21T14:07:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/368731310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c00fe680 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/368731310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ffea0 0x7fa0c0106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/368731310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ff510 0x7fa0c00ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 -- 172.21.15.2:0/368731310 >> 172.21.15.2:0/368731310 conn(0x7fa0c00f9b20 msgr2=0x7fa0c00fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 -- 172.21.15.2:0/368731310 shutdown_connections 2022-03-21T14:07:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.064+0000 7fa0c7faa700 1 -- 172.21.15.2:0/368731310 wait complete. 2022-03-21T14:07:10.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.065+0000 7fa0c7faa700 1 Processor -- start 2022-03-21T14:07:10.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.065+0000 7fa0c7faa700 1 -- start start 2022-03-21T14:07:10.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:10.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ff510 0x7fa0c010f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:10.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ffea0 0x7fa0c0116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:10.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0116a60 con 0x7fa0c00ffea0 2022-03-21T14:07:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0116bd0 con 0x7fa0c00fe270 2022-03-21T14:07:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c7faa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0116d90 con 0x7fa0c00ff510 2022-03-21T14:07:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c5d46700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c5d46700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48396/0 (socket says 172.21.15.2:48396) 2022-03-21T14:07:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c6547700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ffea0 0x7fa0c0116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c5d46700 1 -- 172.21.15.2:0/4261226321 learned_addr learned my addr 172.21.15.2:0/4261226321 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.066+0000 7fa0c5545700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ff510 0x7fa0c010f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 -- 172.21.15.2:0/4261226321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ff510 msgr2=0x7fa0c010f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:10.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ff510 0x7fa0c010f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 -- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ffea0 msgr2=0x7fa0c0116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ffea0 0x7fa0c0116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 -- 172.21.15.2:0/4261226321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0093f0 con 0x7fa0c00fe270 2022-03-21T14:07:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c5d46700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc000e50 tx=0x7fa0bc005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:10.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc01d070 con 0x7fa0c00fe270 2022-03-21T14:07:10.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa0bc005040 con 0x7fa0c00fe270 2022-03-21T14:07:10.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c0117010 con 0x7fa0c00fe270 2022-03-21T14:07:10.075 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0179e0 con 0x7fa0c00fe270 2022-03-21T14:07:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.067+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c0117490 con 0x7fa0c00fe270 2022-03-21T14:07:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.068+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0109a80 con 0x7fa0c00fe270 2022-03-21T14:07:10.077 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.070+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa0bc028030 con 0x7fa0c00fe270 2022-03-21T14:07:10.077 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.071+0000 7fa0b67fc700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 0x7fa0ac0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.072+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa0bc0c0b80 con 0x7fa0c00fe270 2022-03-21T14:07:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.072+0000 7fa0c5545700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 0x7fa0ac0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.072+0000 7fa0c5545700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 0x7fa0ac0991c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0005fd0 tx=0x7fa0b0005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.074+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fa0bc08dfa0 con 0x7fa0c00fe270 2022-03-21T14:07:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:09 smithi002 ceph-mon[17167]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 938 B/s wr, 5 op/s 2022-03-21T14:07:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:09 smithi016 ceph-mon[22010]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 938 B/s wr, 5 op/s 2022-03-21T14:07:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:09 smithi079 ceph-mon[23158]: pgmap v276: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 938 B/s wr, 5 op/s 2022-03-21T14:07:10.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.264+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0c0105850 con 0x7fa0c00fe270 2022-03-21T14:07:10.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.266+0000 7fa0b67fc700 1 -- 172.21.15.2:0/4261226321 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21210 (secure 0 0 0) 0x7fa0bc0915c0 con 0x7fa0c00fe270 2022-03-21T14:07:10.268 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:10.269 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:55.839803+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:07:10.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.268+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 msgr2=0x7fa0ac0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:10.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.268+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 0x7fa0ac0991c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0005fd0 tx=0x7fa0b0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.268+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 msgr2=0x7fa0c010efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:10.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.268+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc000e50 tx=0x7fa0bc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:10.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 shutdown_connections 2022-03-21T14:07:10.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c00fe270 0x7fa0c010efa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa0ac096d10 0x7fa0ac0991c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0c00ffea0 0x7fa0c0116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 --2- 172.21.15.2:0/4261226321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0c00ff510 0x7fa0c010f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:10.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 >> 172.21.15.2:0/4261226321 conn(0x7fa0c00f9b20 msgr2=0x7fa0c0102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:10.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 shutdown_connections 2022-03-21T14:07:10.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:10.269+0000 7fa0c7faa700 1 -- 172.21.15.2:0/4261226321 wait complete. 2022-03-21T14:07:10.729 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-03-21T14:07:10.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-03-21T14:07:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:10 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4261226321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:10 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4261226321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:10 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4261226321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:11.389 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:11 smithi002 ceph-mon[17167]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2022-03-21T14:07:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:11 smithi016 ceph-mon[22010]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2022-03-21T14:07:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:11 smithi079 ceph-mon[23158]: pgmap v277: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2022-03-21T14:07:12.641 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:13.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 -- 172.21.15.2:0/2138064550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 msgr2=0x7f22740fc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:13.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 --2- 172.21.15.2:0/2138064550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f22740fc480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f22640098e0 tx=0x7f2264005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:13.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 -- 172.21.15.2:0/2138064550 shutdown_connections 2022-03-21T14:07:13.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 --2- 172.21.15.2:0/2138064550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 0x7f2274104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 --2- 172.21.15.2:0/2138064550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f22740fc480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 --2- 172.21.15.2:0/2138064550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f22740fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 -- 172.21.15.2:0/2138064550 >> 172.21.15.2:0/2138064550 conn(0x7f22740f7880 msgr2=0x7f22740f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:13.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 -- 172.21.15.2:0/2138064550 shutdown_connections 2022-03-21T14:07:13.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.128+0000 7f227891a700 1 -- 172.21.15.2:0/2138064550 wait complete. 2022-03-21T14:07:13.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 Processor -- start 2022-03-21T14:07:13.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 -- start start 2022-03-21T14:07:13.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:13.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f227406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:13.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 0x7f227406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:13.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227406ff70 con 0x7f22740fc070 2022-03-21T14:07:13.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22740700e0 con 0x7f22740fd310 2022-03-21T14:07:13.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.129+0000 7f227891a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2274116f30 con 0x7f22740fdca0 2022-03-21T14:07:13.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:13.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43814/0 (socket says 172.21.15.2:43814) 2022-03-21T14:07:13.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 -- 172.21.15.2:0/3479944663 learned_addr learned my addr 172.21.15.2:0/3479944663 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:13.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f2271d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f227406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:13.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f2272d9d700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 0x7f227406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:13.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 -- 172.21.15.2:0/3479944663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 msgr2=0x7f227406f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:13.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 0x7f227406f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 -- 172.21.15.2:0/3479944663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 msgr2=0x7f227406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:13.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f227406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 -- 172.21.15.2:0/3479944663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640093f0 con 0x7f22740fc070 2022-03-21T14:07:13.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f2271d9b700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f227406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:13.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.130+0000 7f227259c700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22640093c0 tx=0x7f2264005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:13.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.131+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226401d070 con 0x7f22740fc070 2022-03-21T14:07:13.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.131+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2264000c90 con 0x7f22740fc070 2022-03-21T14:07:13.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.131+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264017990 con 0x7f22740fc070 2022-03-21T14:07:13.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.131+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22741170d0 con 0x7f22740fc070 2022-03-21T14:07:13.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.131+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22741174a0 con 0x7f22740fc070 2022-03-21T14:07:13.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.132+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22741078d0 con 0x7f22740fc070 2022-03-21T14:07:13.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.134+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2264028030 con 0x7f22740fc070 2022-03-21T14:07:13.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.136+0000 7f2262ffd700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 0x7f225c099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:13.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.136+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f22640c0d30 con 0x7f22740fc070 2022-03-21T14:07:13.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.136+0000 7f2271d9b700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 0x7f225c099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:13.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.137+0000 7f2271d9b700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 0x7f225c099240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2268005fd0 tx=0x7f2268005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:13.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.139+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f226408e060 con 0x7f22740fc070 2022-03-21T14:07:13.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.327+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2274117780 con 0x7f22740fc070 2022-03-21T14:07:13.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.328+0000 7f2262ffd700 1 -- 172.21.15.2:0/3479944663 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21210 (secure 0 0 0) 0x7f2264091680 con 0x7f22740fc070 2022-03-21T14:07:13.330 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:06:55.839803+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:07:13.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 msgr2=0x7f225c099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:13.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 0x7f225c099240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2268005fd0 tx=0x7f2268005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 msgr2=0x7f2274070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:13.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22640093c0 tx=0x7f2264005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:13.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 shutdown_connections 2022-03-21T14:07:13.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f225c096d90 0x7f225c099240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22740fdca0 0x7f227406f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22740fc070 0x7f2274070cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 --2- 172.21.15.2:0/3479944663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740fd310 0x7f227406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:13.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 >> 172.21.15.2:0/3479944663 conn(0x7f22740f7880 msgr2=0x7f22740f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:13.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 shutdown_connections 2022-03-21T14:07:13.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:13.331+0000 7f227891a700 1 -- 172.21.15.2:0/3479944663 wait complete. 2022-03-21T14:07:13.846 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-03-21T14:07:13.847 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-03-21T14:07:13.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-03-21T14:07:13.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-03-21T14:07:13.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-03-21T14:07:13.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-03-21T14:07:13.850 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-03-21T14:07:13.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-03-21T14:07:13.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2022-03-21T14:07:13.852 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2022-03-21T14:07:13.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2022-03-21T14:07:13.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2022-03-21T14:07:14.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:13 smithi002 ceph-mon[17167]: pgmap v278: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2022-03-21T14:07:14.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:13 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3479944663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:13 smithi016 ceph-mon[22010]: pgmap v278: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2022-03-21T14:07:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:13 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3479944663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:13 smithi079 ceph-mon[23158]: pgmap v278: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2022-03-21T14:07:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:13 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3479944663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:15.276 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:15.550 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:15.767 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:15.913 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:15.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:15 smithi002 ceph-mon[17167]: pgmap v279: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2022-03-21T14:07:16.033 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:15 smithi016 ceph-mon[22010]: pgmap v279: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2022-03-21T14:07:16.190 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:15 smithi079 ceph-mon[23158]: pgmap v279: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2022-03-21T14:07:16.859 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:16.866 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:16.880 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:17.277 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:17.801 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:17.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 -- 172.21.15.2:0/1337849655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 msgr2=0x7fd0780fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:17.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/1337849655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 0x7fd0780fe640 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd0680098e0 tx=0x7fd068005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:17.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 -- 172.21.15.2:0/1337849655 shutdown_connections 2022-03-21T14:07:17.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/1337849655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 0x7fd078106720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:17.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/1337849655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 0x7fd0780fe640 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:17.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/1337849655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 0x7fd0780ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:17.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 -- 172.21.15.2:0/1337849655 >> 172.21.15.2:0/1337849655 conn(0x7fd0780f9b20 msgr2=0x7fd0780fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:17.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 -- 172.21.15.2:0/1337849655 shutdown_connections 2022-03-21T14:07:17.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.970+0000 7fd07dbd3700 1 -- 172.21.15.2:0/1337849655 wait complete. 2022-03-21T14:07:17.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.971+0000 7fd07dbd3700 1 Processor -- start 2022-03-21T14:07:17.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.971+0000 7fd07dbd3700 1 -- start start 2022-03-21T14:07:17.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 0x7fd07810efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:17.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 0x7fd07810f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:17.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 0x7fd078116580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:17.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078116ac0 con 0x7fd0780fe230 2022-03-21T14:07:17.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078116c30 con 0x7fd0780ffe60 2022-03-21T14:07:17.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd07dbd3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078116df0 con 0x7fd0780ff4d0 2022-03-21T14:07:17.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd076ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 0x7fd07810f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:17.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd076ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 0x7fd07810f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39674/0 (socket says 172.21.15.2:39674) 2022-03-21T14:07:17.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd076ffd700 1 -- 172.21.15.2:0/2273617880 learned_addr learned my addr 172.21.15.2:0/2273617880 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:17.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd0777fe700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 0x7fd07810efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:17.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.972+0000 7fd077fff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 0x7fd078116580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:17.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 -- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 msgr2=0x7fd07810f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:17.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0780ff4d0 0x7fd07810f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:17.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 -- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 msgr2=0x7fd07810efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:17.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0780fe230 0x7fd07810efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:17.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680093f0 con 0x7fd0780ffe60 2022-03-21T14:07:17.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.973+0000 7fd077fff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 0x7fd078116580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd07000eb50 tx=0x7fd070009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:17.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.974+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000c9d0 con 0x7fd0780ffe60 2022-03-21T14:07:17.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.974+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0781170d0 con 0x7fd0780ffe60 2022-03-21T14:07:17.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.974+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd070003840 con 0x7fd0780ffe60 2022-03-21T14:07:17.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.974+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070004450 con 0x7fd0780ffe60 2022-03-21T14:07:17.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.974+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd078117690 con 0x7fd0780ffe60 2022-03-21T14:07:17.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.975+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd07804b760 con 0x7fd0780ffe60 2022-03-21T14:07:17.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.977+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd07000c040 con 0x7fd0780ffe60 2022-03-21T14:07:17.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd05ffff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd060096d90 0x7fd060099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:17.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fd070014070 con 0x7fd0780ffe60 2022-03-21T14:07:17.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd0777fe700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd060096d90 0x7fd060099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:17.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd05ffff700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] conn(0x7fd0600a0560 0x7fd0600a2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:17.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0600a3020 con 0x7fd0600a0560 2022-03-21T14:07:17.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.979+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0700c0130 con 0x7fd0780ffe60 2022-03-21T14:07:17.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.980+0000 7fd076ffd700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] conn(0x7fd0600a0560 0x7fd0600a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:17.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.980+0000 7fd0777fe700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd060096d90 0x7fd060099240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06800f8e0 tx=0x7fd068005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:17.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.981+0000 7fd076ffd700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] conn(0x7fd0600a0560 0x7fd0600a2970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:17.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:17.982+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== osd.2 v2:172.21.15.79:6800/4178164596 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fd0600a3020 con 0x7fd0600a0560 2022-03-21T14:07:18.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.002+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 --> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd07805d6d0 con 0x7fd0600a0560 2022-03-21T14:07:18.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.002+0000 7fd05ffff700 1 -- 172.21.15.2:0/2273617880 <== osd.2 v2:172.21.15.79:6800/4178164596 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd07805d6d0 con 0x7fd0600a0560 2022-03-21T14:07:18.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.003+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] conn(0x7fd0600a0560 msgr2=0x7fd0600a2970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:18.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.003+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] conn(0x7fd0600a0560 0x7fd0600a2970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:18.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd060096d90 msgr2=0x7fd060099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:18.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd060096d90 0x7fd060099240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06800f8e0 tx=0x7fd068005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 msgr2=0x7fd078116580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 --2- 172.21.15.2:0/2273617880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0780ffe60 0x7fd078116580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd07000eb50 tx=0x7fd070009430 comp rx=0 tx=0).stop 2022-03-21T14:07:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd077fff700 1 -- 172.21.15.2:0/2273617880 reap_dead start 2022-03-21T14:07:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 shutdown_connections 2022-03-21T14:07:18.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 >> 172.21.15.2:0/2273617880 conn(0x7fd0780f9b20 msgr2=0x7fd0780fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 shutdown_connections 2022-03-21T14:07:18.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:18.004+0000 7fd07dbd3700 1 -- 172.21.15.2:0/2273617880 wait complete. 2022-03-21T14:07:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:17 smithi002 ceph-mon[17167]: pgmap v280: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:17 smithi016 ceph-mon[22010]: pgmap v280: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:17 smithi079 ceph-mon[23158]: pgmap v280: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:18.519 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:20.055 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:20.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:19 smithi002 ceph-mon[17167]: pgmap v281: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:19 smithi016 ceph-mon[22010]: pgmap v281: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:19 smithi079 ceph-mon[23158]: pgmap v281: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:21.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 -- 172.21.15.2:0/2487532232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac1013f0 msgr2=0x7fefac1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 --2- 172.21.15.2:0/2487532232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac1013f0 0x7fefac1037d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefa000b820 tx=0x7fefa0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 -- 172.21.15.2:0/2487532232 shutdown_connections 2022-03-21T14:07:21.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 --2- 172.21.15.2:0/2487532232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac103d10 0x7fefac106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 --2- 172.21.15.2:0/2487532232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fefac0655c0 0x7fefac100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 --2- 172.21.15.2:0/2487532232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac1013f0 0x7fefac1037d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.033+0000 7fefab59e700 1 -- 172.21.15.2:0/2487532232 >> 172.21.15.2:0/2487532232 conn(0x7fefac0f64a0 msgr2=0x7fefac0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.034+0000 7fefab59e700 1 -- 172.21.15.2:0/2487532232 shutdown_connections 2022-03-21T14:07:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.034+0000 7fefab59e700 1 -- 172.21.15.2:0/2487532232 wait complete. 2022-03-21T14:07:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.035+0000 7fefab59e700 1 Processor -- start 2022-03-21T14:07:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.035+0000 7fefab59e700 1 -- start start 2022-03-21T14:07:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 0x7fefac10ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac1013f0 0x7fefac10f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fefac103d10 0x7fefac116420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac116960 con 0x7fefac103d10 2022-03-21T14:07:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac116ad0 con 0x7fefac1013f0 2022-03-21T14:07:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefab59e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac116c90 con 0x7fefac0655c0 2022-03-21T14:07:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefaa59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 0x7fefac10ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefaa59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 0x7fefac10ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39678/0 (socket says 172.21.15.2:39678) 2022-03-21T14:07:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefaa59c700 1 -- 172.21.15.2:0/2833028288 learned_addr learned my addr 172.21.15.2:0/2833028288 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefaad9d700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fefac103d10 0x7fefac116420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.036+0000 7fefa9d9b700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac1013f0 0x7fefac10f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 -- 172.21.15.2:0/2833028288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac1013f0 msgr2=0x7fefac10f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefac1013f0 0x7fefac10f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 -- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fefac103d10 msgr2=0x7fefac116420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fefac103d10 0x7fefac116420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa000b360 con 0x7fefac0655c0 2022-03-21T14:07:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fefaa59c700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 0x7fefac10ee50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fefa400ebf0 tx=0x7fefa40094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.037+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa40098d0 con 0x7fefac0655c0 2022-03-21T14:07:21.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.038+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fefa4010430 con 0x7fefac0655c0 2022-03-21T14:07:21.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.038+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac116f70 con 0x7fefac0655c0 2022-03-21T14:07:21.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.038+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4018670 con 0x7fefac0655c0 2022-03-21T14:07:21.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.038+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac117400 con 0x7fefac0655c0 2022-03-21T14:07:21.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.039+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fefac04b760 con 0x7fefac0655c0 2022-03-21T14:07:21.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.041+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fefa4028070 con 0x7fefac0655c0 2022-03-21T14:07:21.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fef9affd700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fef94096d90 0x7fef94099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fefa4014070 con 0x7fefac0655c0 2022-03-21T14:07:21.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fef9affd700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] conn(0x7fef940a07d0 0x7fef940a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fefa9d9b700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fef94096d90 0x7fef94099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef940a3330 con 0x7fef940a07d0 2022-03-21T14:07:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.042+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefa40c0960 con 0x7fefac0655c0 2022-03-21T14:07:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.043+0000 7fefaad9d700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] conn(0x7fef940a07d0 0x7fef940a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.043+0000 7fefa9d9b700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fef94096d90 0x7fef94099240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fefa000fea0 tx=0x7fefa0005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.043+0000 7fefaad9d700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] conn(0x7fef940a07d0 0x7fef940a2c60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.044+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== osd.3 v2:172.21.15.2:6810/1119340852 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fef940a3330 con 0x7fef940a07d0 2022-03-21T14:07:21.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.061+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 --> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fefac05d6d0 con 0x7fef940a07d0 2022-03-21T14:07:21.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fef9affd700 1 -- 172.21.15.2:0/2833028288 <== osd.3 v2:172.21.15.2:6810/1119340852 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fefac05d6d0 con 0x7fef940a07d0 2022-03-21T14:07:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] conn(0x7fef940a07d0 msgr2=0x7fef940a2c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] conn(0x7fef940a07d0 0x7fef940a2c60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fef94096d90 msgr2=0x7fef94099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fef94096d90 0x7fef94099240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fefa000fea0 tx=0x7fefa0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 msgr2=0x7fefac10ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 --2- 172.21.15.2:0/2833028288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefac0655c0 0x7fefac10ee50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fefa400ebf0 tx=0x7fefa40094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefaad9d700 1 -- 172.21.15.2:0/2833028288 reap_dead start 2022-03-21T14:07:21.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 shutdown_connections 2022-03-21T14:07:21.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 >> 172.21.15.2:0/2833028288 conn(0x7fefac0f64a0 msgr2=0x7fefac102030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:21.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 shutdown_connections 2022-03-21T14:07:21.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.062+0000 7fefab59e700 1 -- 172.21.15.2:0/2833028288 wait complete. 2022-03-21T14:07:21.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]: dispatch 2022-03-21T14:07:21.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]: dispatch 2022-03-21T14:07:21.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2022-03-21T14:07:21.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:07:21.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]: dispatch 2022-03-21T14:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]: dispatch 2022-03-21T14:07:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2022-03-21T14:07:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:07:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:07:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]: dispatch 2022-03-21T14:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]: dispatch 2022-03-21T14:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2022-03-21T14:07:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2022-03-21T14:07:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-03-21T14:07:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-03-21T14:07:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:07:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:07:21.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 -- 172.21.15.2:0/3102945242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200ff510 msgr2=0x7f48200ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 --2- 172.21.15.2:0/3102945242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200ff510 0x7f48200ff960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f48140098e0 tx=0x7f4814005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:21.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 -- 172.21.15.2:0/3102945242 shutdown_connections 2022-03-21T14:07:21.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 --2- 172.21.15.2:0/3102945242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48200ffea0 0x7f4820106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 --2- 172.21.15.2:0/3102945242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200fe270 0x7f48200fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 --2- 172.21.15.2:0/3102945242 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200ff510 0x7f48200ff960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 -- 172.21.15.2:0/3102945242 >> 172.21.15.2:0/3102945242 conn(0x7f48200f9b20 msgr2=0x7f48200fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:21.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 -- 172.21.15.2:0/3102945242 shutdown_connections 2022-03-21T14:07:21.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.504+0000 7f482511c700 1 -- 172.21.15.2:0/3102945242 wait complete. 2022-03-21T14:07:21.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.505+0000 7f482511c700 1 Processor -- start 2022-03-21T14:07:21.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 -- start start 2022-03-21T14:07:21.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200fe270 0x7f482010efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200ff510 0x7f482010f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48200ffea0 0x7f48201165a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4820116ae0 con 0x7f48200fe270 2022-03-21T14:07:21.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4820116c50 con 0x7f48200ffea0 2022-03-21T14:07:21.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f482511c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4820116e10 con 0x7f48200ff510 2022-03-21T14:07:21.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f481ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200fe270 0x7f482010efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f481e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200ff510 0x7f482010f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f481e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200ff510 0x7f482010f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39682/0 (socket says 172.21.15.2:39682) 2022-03-21T14:07:21.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.506+0000 7f481e59c700 1 -- 172.21.15.2:0/2771340384 learned_addr learned my addr 172.21.15.2:0/2771340384 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:21.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481f59e700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48200ffea0 0x7f48201165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 -- 172.21.15.2:0/2771340384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200ff510 msgr2=0x7f482010f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48200ff510 0x7f482010f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 -- 172.21.15.2:0/2771340384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48200ffea0 msgr2=0x7f48201165a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48200ffea0 0x7f48201165a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48140093f0 con 0x7f48200fe270 2022-03-21T14:07:21.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.507+0000 7f481ed9d700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200fe270 0x7f482010efd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4810005fd0 tx=0x7f4810005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.508+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810004020 con 0x7f48200fe270 2022-03-21T14:07:21.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.508+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4810004180 con 0x7f48200fe270 2022-03-21T14:07:21.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.508+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810003b30 con 0x7f48200fe270 2022-03-21T14:07:21.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.508+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48201170f0 con 0x7f48200fe270 2022-03-21T14:07:21.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.508+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4820117610 con 0x7f48200fe270 2022-03-21T14:07:21.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.509+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4820063220 con 0x7f48200fe270 2022-03-21T14:07:21.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.511+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f4810018030 con 0x7f48200fe270 2022-03-21T14:07:21.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.512+0000 7f48077fe700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4808096d10 0x7f48080991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.512+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f48100bf880 con 0x7f48200fe270 2022-03-21T14:07:21.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f481e59c700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4808096d10 0x7f48080991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f48077fe700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] conn(0x7f48080a08c0 0x7f48080a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:21.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48080a3420 con 0x7f48080a08c0 2022-03-21T14:07:21.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48100bfc00 con 0x7f48200fe270 2022-03-21T14:07:21.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f481f59e700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] conn(0x7f48080a08c0 0x7f48080a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:21.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.513+0000 7f481e59c700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4808096d10 0x7f48080991c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4814000e50 tx=0x7f4814005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.514+0000 7f481f59e700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] conn(0x7f48080a08c0 0x7f48080a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:21.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.515+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== osd.0 v2:172.21.15.2:6802/1386749192 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f48080a3420 con 0x7f48080a08c0 2022-03-21T14:07:21.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.533+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 --> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f482005d710 con 0x7f48080a08c0 2022-03-21T14:07:21.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.534+0000 7f48077fe700 1 -- 172.21.15.2:0/2771340384 <== osd.0 v2:172.21.15.2:6802/1386749192 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f482005d710 con 0x7f48080a08c0 2022-03-21T14:07:21.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.534+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] conn(0x7f48080a08c0 msgr2=0x7f48080a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] conn(0x7f48080a08c0 0x7f48080a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4808096d10 msgr2=0x7f48080991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4808096d10 0x7f48080991c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4814000e50 tx=0x7f4814005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:21.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200fe270 msgr2=0x7f482010efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:21.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 --2- 172.21.15.2:0/2771340384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48200fe270 0x7f482010efd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4810005fd0 tx=0x7f4810005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:21.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f481f59e700 1 -- 172.21.15.2:0/2771340384 reap_dead start 2022-03-21T14:07:21.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 shutdown_connections 2022-03-21T14:07:21.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 >> 172.21.15.2:0/2771340384 conn(0x7f48200f9b20 msgr2=0x7f48200fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:21.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 shutdown_connections 2022-03-21T14:07:21.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:21.535+0000 7f482511c700 1 -- 172.21.15.2:0/2771340384 wait complete. 2022-03-21T14:07:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]': finished 2022-03-21T14:07:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2022-03-21T14:07:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2022-03-21T14:07:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-03-21T14:07:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]': finished 2022-03-21T14:07:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2022-03-21T14:07:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2022-03-21T14:07:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-03-21T14:07:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-03-21T14:07:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:21 smithi002 ceph-mon[17167]: osdmap e83: 12 total, 12 up, 12 in 2022-03-21T14:07:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]': finished 2022-03-21T14:07:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2022-03-21T14:07:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2022-03-21T14:07:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-03-21T14:07:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]': finished 2022-03-21T14:07:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2022-03-21T14:07:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2022-03-21T14:07:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-03-21T14:07:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-03-21T14:07:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:21 smithi016 ceph-mon[22010]: osdmap e83: 12 total, 12 up, 12 in 2022-03-21T14:07:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9, 5, 7]}]': finished 2022-03-21T14:07:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2022-03-21T14:07:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2022-03-21T14:07:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-03-21T14:07:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [1, 8]}]': finished 2022-03-21T14:07:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2022-03-21T14:07:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2022-03-21T14:07:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-03-21T14:07:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-03-21T14:07:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:21 smithi079 ceph-mon[23158]: osdmap e83: 12 total, 12 up, 12 in 2022-03-21T14:07:22.228 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:23.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:22 smithi002 ceph-mon[17167]: osdmap e84: 12 total, 12 up, 12 in 2022-03-21T14:07:23.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 -- 172.21.15.2:0/1456913676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 msgr2=0x7fb05c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 --2- 172.21.15.2:0/1456913676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 0x7fb05c1014f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0500098e0 tx=0x7fb050005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:23.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 -- 172.21.15.2:0/1456913676 shutdown_connections 2022-03-21T14:07:23.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 --2- 172.21.15.2:0/1456913676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c101b80 0x7fb05c104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 --2- 172.21.15.2:0/1456913676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb05c065f80 0x7fb05c0feae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 --2- 172.21.15.2:0/1456913676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 0x7fb05c1014f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 -- 172.21.15.2:0/1456913676 >> 172.21.15.2:0/1456913676 conn(0x7fb05c0f6550 msgr2=0x7fb05c0f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:23.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 -- 172.21.15.2:0/1456913676 shutdown_connections 2022-03-21T14:07:23.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.698+0000 7fb05b59e700 1 -- 172.21.15.2:0/1456913676 wait complete. 2022-03-21T14:07:23.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.699+0000 7fb05b59e700 1 Processor -- start 2022-03-21T14:07:23.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.699+0000 7fb05b59e700 1 -- start start 2022-03-21T14:07:23.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 0x7fb05c10cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:23.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 0x7fb05c10d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:23.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb05c101b80 0x7fb05c1141c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:23.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c114740 con 0x7fb05c101b80 2022-03-21T14:07:23.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c1148b0 con 0x7fb05c065f80 2022-03-21T14:07:23.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05b59e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c114a70 con 0x7fb05c0ff0b0 2022-03-21T14:07:23.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 0x7fb05c10cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:23.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb059d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 0x7fb05c10d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:23.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 0x7fb05c10cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48416/0 (socket says 172.21.15.2:48416) 2022-03-21T14:07:23.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05a59c700 1 -- 172.21.15.2:0/638907065 learned_addr learned my addr 172.21.15.2:0/638907065 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:23.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.700+0000 7fb05ad9d700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb05c101b80 0x7fb05c1141c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:23.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 -- 172.21.15.2:0/638907065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 msgr2=0x7fb05c10d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb05c0ff0b0 0x7fb05c10d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 -- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb05c101b80 msgr2=0x7fb05c1141c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb05c101b80 0x7fb05c1141c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c00b360 con 0x7fb05c065f80 2022-03-21T14:07:23.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05a59c700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 0x7fb05c10cc30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00ea30 tx=0x7fb04c005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:23.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00b6a0 con 0x7fb05c065f80 2022-03-21T14:07:23.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500093f0 con 0x7fb05c065f80 2022-03-21T14:07:23.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb04c00c9d0 con 0x7fb05c065f80 2022-03-21T14:07:23.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.701+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c003c60 con 0x7fb05c065f80 2022-03-21T14:07:23.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.702+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c1150b0 con 0x7fb05c065f80 2022-03-21T14:07:23.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.702+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb05c04b760 con 0x7fb05c065f80 2022-03-21T14:07:23.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.704+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fb04c00c040 con 0x7fb05c065f80 2022-03-21T14:07:23.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb04affd700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb044096d10 0x7fb0440991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:23.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fb04c014070 con 0x7fb05c065f80 2022-03-21T14:07:23.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb059d9b700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb044096d10 0x7fb0440991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:23.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb04affd700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] conn(0x7fb0440a08c0 0x7fb0440a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:23.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0440a3420 con 0x7fb0440a08c0 2022-03-21T14:07:23.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb04c0c0170 con 0x7fb05c065f80 2022-03-21T14:07:23.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.706+0000 7fb05ad9d700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] conn(0x7fb0440a08c0 0x7fb0440a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:23.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.707+0000 7fb059d9b700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb044096d10 0x7fb0440991c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb05000f8e0 tx=0x7fb050005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:23.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.707+0000 7fb05ad9d700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] conn(0x7fb0440a08c0 0x7fb0440a2d50 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:23.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.708+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== osd.7 v2:172.21.15.16:6816/3746016674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fb0440a3420 con 0x7fb0440a08c0 2022-03-21T14:07:23.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.724+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 --> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb05c0fe5c0 con 0x7fb0440a08c0 2022-03-21T14:07:23.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.725+0000 7fb04affd700 1 -- 172.21.15.2:0/638907065 <== osd.7 v2:172.21.15.16:6816/3746016674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb05c0fe5c0 con 0x7fb0440a08c0 2022-03-21T14:07:23.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.725+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] conn(0x7fb0440a08c0 msgr2=0x7fb0440a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.725+0000 7fb05b59e700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] conn(0x7fb0440a08c0 0x7fb0440a2d50 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb044096d10 msgr2=0x7fb0440991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fb044096d10 0x7fb0440991c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb05000f8e0 tx=0x7fb050005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:23.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 msgr2=0x7fb05c10cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:23.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 --2- 172.21.15.2:0/638907065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb05c065f80 0x7fb05c10cc30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00ea30 tx=0x7fb04c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:23.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05ad9d700 1 -- 172.21.15.2:0/638907065 reap_dead start 2022-03-21T14:07:23.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 shutdown_connections 2022-03-21T14:07:23.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 >> 172.21.15.2:0/638907065 conn(0x7fb05c0f6550 msgr2=0x7fb05c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:23.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 shutdown_connections 2022-03-21T14:07:23.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:22.726+0000 7fb05b59e700 1 -- 172.21.15.2:0/638907065 wait complete. 2022-03-21T14:07:23.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:22 smithi016 ceph-mon[22010]: osdmap e84: 12 total, 12 up, 12 in 2022-03-21T14:07:23.212 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:22 smithi079 ceph-mon[23158]: osdmap e84: 12 total, 12 up, 12 in 2022-03-21T14:07:23.583 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:24.032 INFO:teuthology.orchestra.run.smithi002.stdout:77309411388 2022-03-21T14:07:24.033 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-03-21T14:07:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:23 smithi016 ceph-mon[22010]: pgmap v285: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:24.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:23 smithi002 ceph-mon[17167]: pgmap v285: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:23 smithi079 ceph-mon[23158]: pgmap v285: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:24.333 INFO:teuthology.orchestra.run.smithi002.stdout:107374182455 2022-03-21T14:07:24.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-03-21T14:07:24.387 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:24.392 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:24.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.938+0000 7fa314a28700 1 -- 172.21.15.2:0/2738041651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 msgr2=0x7fa310104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.938+0000 7fa314a28700 1 --2- 172.21.15.2:0/2738041651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310104560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa308009ba0 tx=0x7fa308009430 comp rx=0 tx=0).stop 2022-03-21T14:07:24.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 -- 172.21.15.2:0/2738041651 shutdown_connections 2022-03-21T14:07:24.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 --2- 172.21.15.2:0/2738041651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310104560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 --2- 172.21.15.2:0/2738041651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fc070 0x7fa3100fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 --2- 172.21.15.2:0/2738041651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3100fd310 0x7fa3100fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 -- 172.21.15.2:0/2738041651 >> 172.21.15.2:0/2738041651 conn(0x7fa3100f7880 msgr2=0x7fa3100f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:24.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 -- 172.21.15.2:0/2738041651 shutdown_connections 2022-03-21T14:07:24.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.939+0000 7fa314a28700 1 -- 172.21.15.2:0/2738041651 wait complete. 2022-03-21T14:07:24.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 Processor -- start 2022-03-21T14:07:24.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 -- start start 2022-03-21T14:07:24.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3100fc070 0x7fa31010ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fd310 0x7fa31010f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3101168a0 con 0x7fa3100fc070 2022-03-21T14:07:24.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa310116a10 con 0x7fa3100fdca0 2022-03-21T14:07:24.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.940+0000 7fa314a28700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa310116bd0 con 0x7fa3100fd310 2022-03-21T14:07:24.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30dd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fd310 0x7fa31010f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30dd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fd310 0x7fa31010f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39690/0 (socket says 172.21.15.2:39690) 2022-03-21T14:07:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30dd9b700 1 -- 172.21.15.2:0/3956595054 learned_addr learned my addr 172.21.15.2:0/3956595054 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30e59c700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3100fc070 0x7fa31010ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30ed9d700 1 -- 172.21.15.2:0/3956595054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fd310 msgr2=0x7fa31010f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30ed9d700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3100fd310 0x7fa31010f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30ed9d700 1 -- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3100fc070 msgr2=0x7fa31010ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.941+0000 7fa30ed9d700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3100fc070 0x7fa31010ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.942+0000 7fa30ed9d700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040093f0 con 0x7fa3100fdca0 2022-03-21T14:07:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.942+0000 7fa30ed9d700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310116360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa308005100 tx=0x7fa308009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.942+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30801d070 con 0x7fa3100fdca0 2022-03-21T14:07:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.942+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa30800f460 con 0x7fa3100fdca0 2022-03-21T14:07:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.942+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308017670 con 0x7fa3100fdca0 2022-03-21T14:07:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.943+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3080096b0 con 0x7fa3100fdca0 2022-03-21T14:07:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.943+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa310117070 con 0x7fa3100fdca0 2022-03-21T14:07:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.944+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa310063220 con 0x7fa3100fdca0 2022-03-21T14:07:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.947+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa308027060 con 0x7fa3100fdca0 2022-03-21T14:07:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.948+0000 7fa2feffd700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa2f8096d90 0x7fa2f8099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.948+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fa3080c0fe0 con 0x7fa3100fdca0 2022-03-21T14:07:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.948+0000 7fa30e59c700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa2f8096d90 0x7fa2f8099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.948+0000 7fa2feffd700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] conn(0x7fa2f80a0940 0x7fa2f80a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.804 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.949+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2f80a34a0 con 0x7fa2f80a0940 2022-03-21T14:07:24.804 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.949+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3080c13e0 con 0x7fa3100fdca0 2022-03-21T14:07:24.804 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.949+0000 7fa30dd9b700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] conn(0x7fa2f80a0940 0x7fa2f80a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.949+0000 7fa30e59c700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa2f8096d90 0x7fa2f8099240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3100fd170 tx=0x7fa300005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.950+0000 7fa30dd9b700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] conn(0x7fa2f80a0940 0x7fa2f80a2dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.952+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== osd.9 v2:172.21.15.2:6826/2878353867 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fa2f80a34a0 con 0x7fa2f80a0940 2022-03-21T14:07:24.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.980+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 --> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa310103de0 con 0x7fa2f80a0940 2022-03-21T14:07:24.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.981+0000 7fa2feffd700 1 -- 172.21.15.2:0/3956595054 <== osd.9 v2:172.21.15.2:6826/2878353867 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa310103de0 con 0x7fa2f80a0940 2022-03-21T14:07:24.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.981+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] conn(0x7fa2f80a0940 msgr2=0x7fa2f80a2dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.981+0000 7fa314a28700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] conn(0x7fa2f80a0940 0x7fa2f80a2dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.981+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa2f8096d90 msgr2=0x7fa2f8099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa2f8096d90 0x7fa2f8099240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3100fd170 tx=0x7fa300005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 msgr2=0x7fa310116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 --2- 172.21.15.2:0/3956595054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3100fdca0 0x7fa310116360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa308005100 tx=0x7fa308009430 comp rx=0 tx=0).stop 2022-03-21T14:07:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa30ed9d700 1 -- 172.21.15.2:0/3956595054 reap_dead start 2022-03-21T14:07:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 shutdown_connections 2022-03-21T14:07:24.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 >> 172.21.15.2:0/3956595054 conn(0x7fa3100f7880 msgr2=0x7fa3100f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:24.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 shutdown_connections 2022-03-21T14:07:24.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:23.982+0000 7fa314a28700 1 -- 172.21.15.2:0/3956595054 wait complete. 2022-03-21T14:07:24.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 -- 172.21.15.2:0/4137547985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fd310 msgr2=0x7f188c0fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 --2- 172.21.15.2:0/4137547985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fd310 0x7f188c0fd760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18800098e0 tx=0x7f1880005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:24.928 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 -- 172.21.15.2:0/4137547985 shutdown_connections 2022-03-21T14:07:24.928 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 --2- 172.21.15.2:0/4137547985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fc070 0x7f188c0fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.928 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 --2- 172.21.15.2:0/4137547985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 0x7f188c104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.928 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 --2- 172.21.15.2:0/4137547985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fd310 0x7f188c0fd760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 -- 172.21.15.2:0/4137547985 >> 172.21.15.2:0/4137547985 conn(0x7f188c0f7880 msgr2=0x7f188c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:24.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 -- 172.21.15.2:0/4137547985 shutdown_connections 2022-03-21T14:07:24.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.926+0000 7f1891b1c700 1 -- 172.21.15.2:0/4137547985 wait complete. 2022-03-21T14:07:24.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.927+0000 7f1891b1c700 1 Processor -- start 2022-03-21T14:07:24.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.927+0000 7f1891b1c700 1 -- start start 2022-03-21T14:07:24.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fc070 0x7f188c10ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fd310 0x7f188c10f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 0x7f188c116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c116a70 con 0x7f188c0fc070 2022-03-21T14:07:24.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c116be0 con 0x7f188c0fd310 2022-03-21T14:07:24.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f1891b1c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c116da0 con 0x7f188c0fdca0 2022-03-21T14:07:24.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f188b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fc070 0x7f188c10ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f188affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fd310 0x7f188c10f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.933 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f188affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fd310 0x7f188c10f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48430/0 (socket says 172.21.15.2:48430) 2022-03-21T14:07:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f188affd700 1 -- 172.21.15.2:0/603258699 learned_addr learned my addr 172.21.15.2:0/603258699 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.928+0000 7f188bfff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 0x7f188c116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 -- 172.21.15.2:0/603258699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 msgr2=0x7f188c116530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 0x7f188c116530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 -- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fd310 msgr2=0x7f188c10f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f188c0fd310 0x7f188c10f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18800093f0 con 0x7f188c0fc070 2022-03-21T14:07:24.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188bfff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f188c0fdca0 0x7f188c116530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:24.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f188b7fe700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fc070 0x7f188c10ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f187c005fd0 tx=0x7f187c005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c004020 con 0x7f188c0fc070 2022-03-21T14:07:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f187c004180 con 0x7f188c0fc070 2022-03-21T14:07:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188c117080 con 0x7f188c0fc070 2022-03-21T14:07:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c003b30 con 0x7f188c0fc070 2022-03-21T14:07:24.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.929+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c117640 con 0x7f188c0fc070 2022-03-21T14:07:24.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.931+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f188c063220 con 0x7f188c0fc070 2022-03-21T14:07:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.933+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f187c018030 con 0x7f188c0fc070 2022-03-21T14:07:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f186ffff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1874096d90 0x7f1874099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f187c0bfb90 con 0x7f188c0fc070 2022-03-21T14:07:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f188affd700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1874096d90 0x7f1874099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f186ffff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] conn(0x7f18740a0940 0x7f18740a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18740a34a0 con 0x7f18740a0940 2022-03-21T14:07:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.934+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f187c0bfe80 con 0x7f188c0fc070 2022-03-21T14:07:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.935+0000 7f188bfff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] conn(0x7f18740a0940 0x7f18740a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.935+0000 7f188affd700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1874096d90 0x7f1874099240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f188000fa00 tx=0x7f1880005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.935+0000 7f188bfff700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] conn(0x7f18740a0940 0x7f18740a2dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.936+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== osd.4 v2:172.21.15.16:6808/530720703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f18740a34a0 con 0x7f18740a0940 2022-03-21T14:07:24.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.953+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 --> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f188c070c20 con 0x7f18740a0940 2022-03-21T14:07:24.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.953+0000 7f186ffff700 1 -- 172.21.15.2:0/603258699 <== osd.4 v2:172.21.15.16:6808/530720703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f188c070c20 con 0x7f18740a0940 2022-03-21T14:07:24.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] conn(0x7f18740a0940 msgr2=0x7f18740a2dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] conn(0x7f18740a0940 0x7f18740a2dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1874096d90 msgr2=0x7f1874099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1874096d90 0x7f1874099240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f188000fa00 tx=0x7f1880005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:24.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fc070 msgr2=0x7f188c10ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:24.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 --2- 172.21.15.2:0/603258699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f188c0fc070 0x7f188c10ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f187c005fd0 tx=0x7f187c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:24.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f188bfff700 1 -- 172.21.15.2:0/603258699 reap_dead start 2022-03-21T14:07:24.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 shutdown_connections 2022-03-21T14:07:24.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 >> 172.21.15.2:0/603258699 conn(0x7f188c0f7880 msgr2=0x7f188c06f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:24.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.954+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 shutdown_connections 2022-03-21T14:07:24.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.955+0000 7f1891b1c700 1 -- 172.21.15.2:0/603258699 wait complete. 2022-03-21T14:07:25.166 INFO:teuthology.orchestra.run.smithi002.stdout:214748364837 2022-03-21T14:07:25.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-03-21T14:07:25.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 -- 172.21.15.2:0/3394774934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fd070 msgr2=0x7f60540fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 --2- 172.21.15.2:0/3394774934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fd070 0x7f60540fd4e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f60480098e0 tx=0x7f6048005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:25.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 -- 172.21.15.2:0/3394774934 shutdown_connections 2022-03-21T14:07:25.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 --2- 172.21.15.2:0/3394774934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fda20 0x7f6054104350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 --2- 172.21.15.2:0/3394774934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f60540fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 --2- 172.21.15.2:0/3394774934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fd070 0x7f60540fd4e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 -- 172.21.15.2:0/3394774934 >> 172.21.15.2:0/3394774934 conn(0x7f60540f75c0 msgr2=0x7f60540f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 -- 172.21.15.2:0/3394774934 shutdown_connections 2022-03-21T14:07:25.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.655+0000 7f60596f6700 1 -- 172.21.15.2:0/3394774934 wait complete. 2022-03-21T14:07:25.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 Processor -- start 2022-03-21T14:07:25.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 -- start start 2022-03-21T14:07:25.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f6054070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fd070 0x7f605406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fda20 0x7f605406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605406ff70 con 0x7f60540fda20 2022-03-21T14:07:25.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540700e0 con 0x7f60540fd070 2022-03-21T14:07:25.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.656+0000 7f60596f6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054116cf0 con 0x7f60540fbe70 2022-03-21T14:07:25.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f6054070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f6054070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39694/0 (socket says 172.21.15.2:39694) 2022-03-21T14:07:25.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 -- 172.21.15.2:0/2590333763 learned_addr learned my addr 172.21.15.2:0/2590333763 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:25.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6058ef5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fda20 0x7f605406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f60537fe700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fd070 0x7f605406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 -- 172.21.15.2:0/2590333763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fd070 msgr2=0x7f605406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fd070 0x7f605406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 -- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fda20 msgr2=0x7f605406f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fda20 0x7f605406f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480093f0 con 0x7f60540fbe70 2022-03-21T14:07:25.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f60537fe700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60540fd070 0x7f605406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:25.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6058ef5700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60540fda20 0x7f605406f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:25.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.657+0000 7f6053fff700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f6054070cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f604400fa90 tx=0x7f6044005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.658+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400b4f0 con 0x7f60540fbe70 2022-03-21T14:07:25.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.658+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f604400dc80 con 0x7f60540fbe70 2022-03-21T14:07:25.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.658+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6054116e90 con 0x7f60540fbe70 2022-03-21T14:07:25.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.658+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044011dc0 con 0x7f60540fbe70 2022-03-21T14:07:25.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.658+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6054117310 con 0x7f60540fbe70 2022-03-21T14:07:25.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.659+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f605404b760 con 0x7f60540fbe70 2022-03-21T14:07:25.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.661+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f604400c040 con 0x7f60540fbe70 2022-03-21T14:07:25.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.662+0000 7f6050ff9700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f603c096d10 0x7f603c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.662+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f60440bfcb0 con 0x7f60540fbe70 2022-03-21T14:07:25.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f60537fe700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f603c096d10 0x7f603c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f6050ff9700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] conn(0x7f603c0a08c0 0x7f603c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f603c0a3420 con 0x7f603c0a08c0 2022-03-21T14:07:25.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60440c0000 con 0x7f60540fbe70 2022-03-21T14:07:25.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f6058ef5700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] conn(0x7f603c0a08c0 0x7f603c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.663+0000 7f60537fe700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f603c096d10 0x7f603c0991c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6048009f50 tx=0x7f604801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.664+0000 7f6058ef5700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] conn(0x7f603c0a08c0 0x7f603c0a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.665+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== osd.8 v2:172.21.15.79:6816/2185003031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f603c0a3420 con 0x7f603c0a08c0 2022-03-21T14:07:25.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.681+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 --> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f605405d6d0 con 0x7f603c0a08c0 2022-03-21T14:07:25.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.682+0000 7f6050ff9700 1 -- 172.21.15.2:0/2590333763 <== osd.8 v2:172.21.15.79:6816/2185003031 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f605405d6d0 con 0x7f603c0a08c0 2022-03-21T14:07:25.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.682+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] conn(0x7f603c0a08c0 msgr2=0x7f603c0a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.682+0000 7f60596f6700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] conn(0x7f603c0a08c0 0x7f603c0a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f603c096d10 msgr2=0x7f603c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f603c096d10 0x7f603c0991c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6048009f50 tx=0x7f604801a040 comp rx=0 tx=0).stop 2022-03-21T14:07:25.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 msgr2=0x7f6054070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 --2- 172.21.15.2:0/2590333763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f60540fbe70 0x7f6054070cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f604400fa90 tx=0x7f6044005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f6058ef5700 1 -- 172.21.15.2:0/2590333763 reap_dead start 2022-03-21T14:07:25.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 shutdown_connections 2022-03-21T14:07:25.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 >> 172.21.15.2:0/2590333763 conn(0x7f60540f75c0 msgr2=0x7f6054100330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 shutdown_connections 2022-03-21T14:07:25.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:24.683+0000 7f60596f6700 1 -- 172.21.15.2:0/2590333763 wait complete. 2022-03-21T14:07:25.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 -- 172.21.15.2:0/168047587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fd070 msgr2=0x7f0a300fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 --2- 172.21.15.2:0/168047587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fd070 0x7f0a300fd4e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a200098e0 tx=0x7f0a20005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:25.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 -- 172.21.15.2:0/168047587 shutdown_connections 2022-03-21T14:07:25.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 --2- 172.21.15.2:0/168047587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fda20 0x7f0a30104350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 --2- 172.21.15.2:0/168047587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 0x7f0a300fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 --2- 172.21.15.2:0/168047587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fd070 0x7f0a300fd4e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 -- 172.21.15.2:0/168047587 >> 172.21.15.2:0/168047587 conn(0x7f0a300f75c0 msgr2=0x7f0a300f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 -- 172.21.15.2:0/168047587 shutdown_connections 2022-03-21T14:07:25.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.379+0000 7f0a363e5700 1 -- 172.21.15.2:0/168047587 wait complete. 2022-03-21T14:07:25.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.380+0000 7f0a363e5700 1 Processor -- start 2022-03-21T14:07:25.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.380+0000 7f0a363e5700 1 -- start start 2022-03-21T14:07:25.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 0x7f0a30070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fd070 0x7f0a3006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 0x7f0a3006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3006ff70 con 0x7f0a300fd070 2022-03-21T14:07:25.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300700e0 con 0x7f0a300fda20 2022-03-21T14:07:25.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a363e5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a30116d40 con 0x7f0a300fbe70 2022-03-21T14:07:25.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a35be4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 0x7f0a3006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a34be2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fd070 0x7f0a3006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.387 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a353e3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 0x7f0a30070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.387 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a35be4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 0x7f0a3006f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48436/0 (socket says 172.21.15.2:48436) 2022-03-21T14:07:25.387 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.381+0000 7f0a35be4700 1 -- 172.21.15.2:0/2783919837 learned_addr learned my addr 172.21.15.2:0/2783919837 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:25.387 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 -- 172.21.15.2:0/2783919837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 msgr2=0x7f0a30070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 0x7f0a30070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 -- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fd070 msgr2=0x7f0a3006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fd070 0x7f0a3006f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a200093f0 con 0x7f0a300fda20 2022-03-21T14:07:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a353e3700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a300fbe70 0x7f0a30070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:25.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a34be2700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a300fd070 0x7f0a3006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:25.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.382+0000 7f0a35be4700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 0x7f0a3006f860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800ec50 tx=0x7f0a28009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.383+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800ca40 con 0x7f0a300fda20 2022-03-21T14:07:25.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.383+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a30116f40 con 0x7f0a300fda20 2022-03-21T14:07:25.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.383+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0a2800cba0 con 0x7f0a300fda20 2022-03-21T14:07:25.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.383+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28018690 con 0x7f0a300fda20 2022-03-21T14:07:25.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.383+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301173d0 con 0x7f0a300fda20 2022-03-21T14:07:25.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.384+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a3004b760 con 0x7f0a300fda20 2022-03-21T14:07:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.386+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f0a2800c040 con 0x7f0a300fda20 2022-03-21T14:07:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.387+0000 7f0a25ffb700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0a1c096d10 0x7f0a1c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.387+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f0a28014070 con 0x7f0a300fda20 2022-03-21T14:07:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.387+0000 7f0a353e3700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0a1c096d10 0x7f0a1c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.388+0000 7f0a25ffb700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] conn(0x7f0a1c0a08c0 0x7f0a1c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.388+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a1c0a3420 con 0x7f0a1c0a08c0 2022-03-21T14:07:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.388+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a280c0720 con 0x7f0a300fda20 2022-03-21T14:07:25.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.388+0000 7f0a34be2700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] conn(0x7f0a1c0a08c0 0x7f0a1c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.388+0000 7f0a353e3700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0a1c096d10 0x7f0a1c0991c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005fd0 tx=0x7f0a2c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.389+0000 7f0a34be2700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] conn(0x7f0a1c0a08c0 0x7f0a1c0a2d50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.390+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== osd.1 v2:172.21.15.16:6800/491865625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f0a1c0a3420 con 0x7f0a1c0a08c0 2022-03-21T14:07:25.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.406+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 --> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a30101a40 con 0x7f0a1c0a08c0 2022-03-21T14:07:25.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a25ffb700 1 -- 172.21.15.2:0/2783919837 <== osd.1 v2:172.21.15.16:6800/491865625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0a30101a40 con 0x7f0a1c0a08c0 2022-03-21T14:07:25.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] conn(0x7f0a1c0a08c0 msgr2=0x7f0a1c0a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a363e5700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] conn(0x7f0a1c0a08c0 0x7f0a1c0a2d50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0a1c096d10 msgr2=0x7f0a1c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a363e5700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0a1c096d10 0x7f0a1c0991c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005fd0 tx=0x7f0a2c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.407+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 msgr2=0x7f0a3006f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a363e5700 1 --2- 172.21.15.2:0/2783919837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a300fda20 0x7f0a3006f860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800ec50 tx=0x7f0a28009430 comp rx=0 tx=0).stop 2022-03-21T14:07:25.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a35be4700 1 -- 172.21.15.2:0/2783919837 reap_dead start 2022-03-21T14:07:25.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 shutdown_connections 2022-03-21T14:07:25.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 >> 172.21.15.2:0/2783919837 conn(0x7f0a300f75c0 msgr2=0x7f0a300f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 shutdown_connections 2022-03-21T14:07:25.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.408+0000 7f0a363e5700 1 -- 172.21.15.2:0/2783919837 wait complete. 2022-03-21T14:07:25.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 -- 172.21.15.2:0/2154140747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865406f900 msgr2=0x7f865406fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 --2- 172.21.15.2:0/2154140747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865406f900 0x7f865406fd70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f86480098e0 tx=0x7f8648005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:25.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 -- 172.21.15.2:0/2154140747 shutdown_connections 2022-03-21T14:07:25.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 --2- 172.21.15.2:0/2154140747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f86540ffb90 0x7f865410a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 --2- 172.21.15.2:0/2154140747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8654070ec0 0x7f865406f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 --2- 172.21.15.2:0/2154140747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865406f900 0x7f865406fd70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 -- 172.21.15.2:0/2154140747 >> 172.21.15.2:0/2154140747 conn(0x7f86540f7880 msgr2=0x7f86540f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 -- 172.21.15.2:0/2154140747 shutdown_connections 2022-03-21T14:07:25.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.869+0000 7f8659bc8700 1 -- 172.21.15.2:0/2154140747 wait complete. 2022-03-21T14:07:25.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 Processor -- start 2022-03-21T14:07:25.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 -- start start 2022-03-21T14:07:25.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 0x7f8654113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654070ec0 0x7f8654113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86540ffb90 0x7f865411a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865411af10 con 0x7f86540ffb90 2022-03-21T14:07:25.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865411b080 con 0x7f8654070ec0 2022-03-21T14:07:25.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.870+0000 7f8659bc8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865411b240 con 0x7f865406f900 2022-03-21T14:07:25.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f8653fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86540ffb90 0x7f865411a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 0x7f8654113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f8652ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654070ec0 0x7f8654113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 0x7f8654113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39706/0 (socket says 172.21.15.2:39706) 2022-03-21T14:07:25.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 -- 172.21.15.2:0/4130832092 learned_addr learned my addr 172.21.15.2:0/4130832092 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:25.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 -- 172.21.15.2:0/4130832092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654070ec0 msgr2=0x7f8654113940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654070ec0 0x7f8654113940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 -- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86540ffb90 msgr2=0x7f865411a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86540ffb90 0x7f865411a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f86537fe700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86480093f0 con 0x7f865406f900 2022-03-21T14:07:25.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.871+0000 7f8653fff700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86540ffb90 0x7f865411a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:25.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f86537fe700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 0x7f8654113400 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f864400fa90 tx=0x7f8644005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864400be00 con 0x7f865406f900 2022-03-21T14:07:25.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f864400de90 con 0x7f865406f900 2022-03-21T14:07:25.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865411b520 con 0x7f865406f900 2022-03-21T14:07:25.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644011e80 con 0x7f865406f900 2022-03-21T14:07:25.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.872+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865411ba90 con 0x7f865406f900 2022-03-21T14:07:25.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.873+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8654063220 con 0x7f865406f900 2022-03-21T14:07:25.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.875+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f864400c040 con 0x7f865406f900 2022-03-21T14:07:25.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.876+0000 7f863bfff700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f863c096d10 0x7f863c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f86440bfff0 con 0x7f865406f900 2022-03-21T14:07:25.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f8652ffd700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f863c096d10 0x7f863c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f863bfff700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] conn(0x7f863c0a08c0 0x7f863c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:25.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f863c0a3420 con 0x7f863c0a08c0 2022-03-21T14:07:25.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86440c0340 con 0x7f865406f900 2022-03-21T14:07:25.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.877+0000 7f8653fff700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] conn(0x7f863c0a08c0 0x7f863c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:25.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.878+0000 7f8652ffd700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f863c096d10 0x7f863c0991c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8648009f50 tx=0x7f8648005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.878+0000 7f8653fff700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] conn(0x7f863c0a08c0 0x7f863c0a2d50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:25.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.879+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== osd.11 v2:172.21.15.79:6824/1987240625 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f863c0a3420 con 0x7f863c0a08c0 2022-03-21T14:07:25.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.898+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 --> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f86540fcd50 con 0x7f863c0a08c0 2022-03-21T14:07:25.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f863bfff700 1 -- 172.21.15.2:0/4130832092 <== osd.11 v2:172.21.15.79:6824/1987240625 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f86540fcd50 con 0x7f863c0a08c0 2022-03-21T14:07:25.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] conn(0x7f863c0a08c0 msgr2=0x7f863c0a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] conn(0x7f863c0a08c0 0x7f863c0a2d50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f863c096d10 msgr2=0x7f863c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f863c096d10 0x7f863c0991c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8648009f50 tx=0x7f8648005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:25.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 msgr2=0x7f8654113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:25.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 --2- 172.21.15.2:0/4130832092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f865406f900 0x7f8654113400 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f864400fa90 tx=0x7f8644005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:25.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8653fff700 1 -- 172.21.15.2:0/4130832092 reap_dead start 2022-03-21T14:07:25.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 shutdown_connections 2022-03-21T14:07:25.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 >> 172.21.15.2:0/4130832092 conn(0x7f86540f7880 msgr2=0x7f86540f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:25.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 shutdown_connections 2022-03-21T14:07:25.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:25.899+0000 7f8659bc8700 1 -- 172.21.15.2:0/4130832092 wait complete. 2022-03-21T14:07:26.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:25 smithi002 ceph-mon[17167]: pgmap v286: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2022-03-21T14:07:26.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:25 smithi016 ceph-mon[22010]: pgmap v286: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2022-03-21T14:07:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:25 smithi079 ceph-mon[23158]: pgmap v286: 97 pgs: 5 peering, 92 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2022-03-21T14:07:26.402 INFO:teuthology.orchestra.run.smithi002.stdout:137438953523 2022-03-21T14:07:26.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-03-21T14:07:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:279172874268 2022-03-21T14:07:26.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.9 2022-03-21T14:07:27.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 -- 172.21.15.2:0/1959690961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070101ce0 msgr2=0x7f0070104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1959690961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070101ce0 0x7f0070104160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0068009c70 tx=0x7f00680094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 -- 172.21.15.2:0/1959690961 shutdown_connections 2022-03-21T14:07:27.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1959690961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070101ce0 0x7f0070104160 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1959690961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070065560 0x7f00700fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1959690961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00700ff250 0x7f0070101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 -- 172.21.15.2:0/1959690961 >> 172.21.15.2:0/1959690961 conn(0x7f00700f6760 msgr2=0x7f00700f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:27.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 -- 172.21.15.2:0/1959690961 shutdown_connections 2022-03-21T14:07:27.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.314+0000 7f0076f1d700 1 -- 172.21.15.2:0/1959690961 wait complete. 2022-03-21T14:07:27.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 Processor -- start 2022-03-21T14:07:27.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 -- start start 2022-03-21T14:07:27.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 0x7f007010ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00700ff250 0x7f007010f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070101ce0 0x7f0070116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701168a0 con 0x7f00700ff250 2022-03-21T14:07:27.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0070116a10 con 0x7f0070065560 2022-03-21T14:07:27.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.316+0000 7f0076f1d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0070116bd0 con 0x7f0070101ce0 2022-03-21T14:07:27.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 0x7f007010ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 0x7f007010ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48444/0 (socket says 172.21.15.2:48444) 2022-03-21T14:07:27.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f006ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00700ff250 0x7f007010f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 -- 172.21.15.2:0/1454792934 learned_addr learned my addr 172.21.15.2:0/1454792934 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:27.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f00754ba700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070101ce0 0x7f0070116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 -- 172.21.15.2:0/1454792934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070101ce0 msgr2=0x7f0070116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070101ce0 0x7f0070116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 -- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00700ff250 msgr2=0x7f007010f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00700ff250 0x7f007010f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f0074cb9700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068009780 con 0x7f0070065560 2022-03-21T14:07:27.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.317+0000 7f00754ba700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0070101ce0 0x7f0070116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:27.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f0074cb9700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 0x7f007010ed90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f006400fa90 tx=0x7f0064005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006400b4f0 con 0x7f0070065560 2022-03-21T14:07:27.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f006400dc80 con 0x7f0070065560 2022-03-21T14:07:27.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064011d50 con 0x7f0070065560 2022-03-21T14:07:27.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0070116eb0 con 0x7f0070065560 2022-03-21T14:07:27.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.318+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0070117280 con 0x7f0070065560 2022-03-21T14:07:27.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.319+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0070063220 con 0x7f0070065560 2022-03-21T14:07:27.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.322+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f006400c040 con 0x7f0070065560 2022-03-21T14:07:27.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006d7fa700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0060096d10 0x7f00600991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f00640c01b0 con 0x7f0070065560 2022-03-21T14:07:27.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006ffff700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0060096d10 0x7f00600991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006d7fa700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] conn(0x7f00600a08c0 0x7f00600a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00600a3420 con 0x7f00600a08c0 2022-03-21T14:07:27.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.324+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00640c04a0 con 0x7f0070065560 2022-03-21T14:07:27.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.325+0000 7f00754ba700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] conn(0x7f00600a08c0 0x7f00600a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.325+0000 7f006ffff700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0060096d10 0x7f00600991c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0070065330 tx=0x7f005c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.325+0000 7f00754ba700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] conn(0x7f00600a08c0 0x7f00600a2d50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.327+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== osd.6 v2:172.21.15.2:6818/2687857259 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f00600a3420 con 0x7f00600a08c0 2022-03-21T14:07:27.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.349+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 --> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0070103c30 con 0x7f00600a08c0 2022-03-21T14:07:27.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.350+0000 7f006d7fa700 1 -- 172.21.15.2:0/1454792934 <== osd.6 v2:172.21.15.2:6818/2687857259 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0070103c30 con 0x7f00600a08c0 2022-03-21T14:07:27.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] conn(0x7f00600a08c0 msgr2=0x7f00600a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] conn(0x7f00600a08c0 0x7f00600a2d50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0060096d10 msgr2=0x7f00600991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0060096d10 0x7f00600991c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0070065330 tx=0x7f005c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 msgr2=0x7f007010ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 --2- 172.21.15.2:0/1454792934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070065560 0x7f007010ed90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f006400fa90 tx=0x7f0064005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f00754ba700 1 -- 172.21.15.2:0/1454792934 reap_dead start 2022-03-21T14:07:27.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 shutdown_connections 2022-03-21T14:07:27.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 >> 172.21.15.2:0/1454792934 conn(0x7f00700f6760 msgr2=0x7f00700f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:27.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 shutdown_connections 2022-03-21T14:07:27.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:26.351+0000 7f0076f1d700 1 -- 172.21.15.2:0/1454792934 wait complete. 2022-03-21T14:07:27.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 -- 172.21.15.2:0/3554893051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c806f8a0 msgr2=0x7f57c806fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 --2- 172.21.15.2:0/3554893051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c806f8a0 0x7f57c806fd10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098e0 tx=0x7f57b8005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:27.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 -- 172.21.15.2:0/3554893051 shutdown_connections 2022-03-21T14:07:27.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 --2- 172.21.15.2:0/3554893051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c8104000 0x7f57c81063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 --2- 172.21.15.2:0/3554893051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8070e60 0x7f57c806f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 --2- 172.21.15.2:0/3554893051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c806f8a0 0x7f57c806fd10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 -- 172.21.15.2:0/3554893051 >> 172.21.15.2:0/3554893051 conn(0x7f57c80f7880 msgr2=0x7f57c80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:27.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 -- 172.21.15.2:0/3554893051 shutdown_connections 2022-03-21T14:07:27.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.227+0000 7f57cf47c700 1 -- 172.21.15.2:0/3554893051 wait complete. 2022-03-21T14:07:27.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.228+0000 7f57cf47c700 1 Processor -- start 2022-03-21T14:07:27.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.228+0000 7f57cf47c700 1 -- start start 2022-03-21T14:07:27.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.228+0000 7f57cf47c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 0x7f57c810efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cf47c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c8070e60 0x7f57c810f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cf47c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8104000 0x7f57c8116590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cf47c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8116ad0 con 0x7f57c8070e60 2022-03-21T14:07:27.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cf47c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8116c40 con 0x7f57c806f8a0 2022-03-21T14:07:27.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cf47c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c8116e00 con 0x7f57c8104000 2022-03-21T14:07:27.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cd218700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 0x7f57c810efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cd218700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 0x7f57c810efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48448/0 (socket says 172.21.15.2:48448) 2022-03-21T14:07:27.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cda19700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8104000 0x7f57c8116590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cd218700 1 -- 172.21.15.2:0/2463847499 learned_addr learned my addr 172.21.15.2:0/2463847499 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:27.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cca17700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c8070e60 0x7f57c810f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cda19700 1 -- 172.21.15.2:0/2463847499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 msgr2=0x7f57c810efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.229+0000 7f57cda19700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 0x7f57c810efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cda19700 1 -- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c8070e60 msgr2=0x7f57c810f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cda19700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c8070e60 0x7f57c810f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cda19700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80093f0 con 0x7f57c8104000 2022-03-21T14:07:27.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cd218700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c806f8a0 0x7f57c810efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:27.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cda19700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8104000 0x7f57c8116590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f57c80713e0 tx=0x7f57c00095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0003840 con 0x7f57c8104000 2022-03-21T14:07:27.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f57c0003e50 con 0x7f57c8104000 2022-03-21T14:07:27.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c81170e0 con 0x7f57c8104000 2022-03-21T14:07:27.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.230+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c000caa0 con 0x7f57c8104000 2022-03-21T14:07:27.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.231+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8117600 con 0x7f57c8104000 2022-03-21T14:07:27.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.231+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57c8063220 con 0x7f57c8104000 2022-03-21T14:07:27.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.233+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f57c0020060 con 0x7f57c8104000 2022-03-21T14:07:27.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.235+0000 7f57bdffb700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f57b4096d40 0x7f57b40991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.236+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f57c0015070 con 0x7f57c8104000 2022-03-21T14:07:27.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.236+0000 7f57cd218700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f57b4096d40 0x7f57b40991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.236+0000 7f57bdffb700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] conn(0x7f57b40a08f0 0x7f57b40a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:27.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.236+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57b40a3450 con 0x7f57b40a08f0 2022-03-21T14:07:27.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.236+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57c00bfd00 con 0x7f57c8104000 2022-03-21T14:07:27.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.237+0000 7f57cd218700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f57b4096d40 0x7f57b40991f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57c400bfd0 tx=0x7f57c4005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.237+0000 7f57cca17700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] conn(0x7f57b40a08f0 0x7f57b40a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:27.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.237+0000 7f57cca17700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] conn(0x7f57b40a08f0 0x7f57b40a2d80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:27.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.238+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== osd.5 v2:172.21.15.79:6808/1459081452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f57b40a3450 con 0x7f57b40a08f0 2022-03-21T14:07:27.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.256+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 --> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57c805d6d0 con 0x7f57b40a08f0 2022-03-21T14:07:27.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.256+0000 7f57bdffb700 1 -- 172.21.15.2:0/2463847499 <== osd.5 v2:172.21.15.79:6808/1459081452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f57c805d6d0 con 0x7f57b40a08f0 2022-03-21T14:07:27.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] conn(0x7f57b40a08f0 msgr2=0x7f57b40a2d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] conn(0x7f57b40a08f0 0x7f57b40a2d80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f57b4096d40 msgr2=0x7f57b40991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f57b4096d40 0x7f57b40991f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57c400bfd0 tx=0x7f57c4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8104000 msgr2=0x7f57c8116590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 --2- 172.21.15.2:0/2463847499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c8104000 0x7f57c8116590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f57c80713e0 tx=0x7f57c00095d0 comp rx=0 tx=0).stop 2022-03-21T14:07:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cda19700 1 -- 172.21.15.2:0/2463847499 reap_dead start 2022-03-21T14:07:27.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 shutdown_connections 2022-03-21T14:07:27.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 >> 172.21.15.2:0/2463847499 conn(0x7f57c80f7880 msgr2=0x7f57c80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:27.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.257+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 shutdown_connections 2022-03-21T14:07:27.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:27.258+0000 7f57cf47c700 1 -- 172.21.15.2:0/2463847499 wait complete. 2022-03-21T14:07:27.343 INFO:teuthology.orchestra.run.smithi002.stdout:55834574913 2022-03-21T14:07:27.344 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-03-21T14:07:27.511 INFO:teuthology.orchestra.run.smithi002.stdout:34359738437 2022-03-21T14:07:27.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-03-21T14:07:27.574 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:27.602 INFO:teuthology.orchestra.run.smithi002.stdout:330712481812 2022-03-21T14:07:27.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.11 2022-03-21T14:07:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:27 smithi002 ceph-mon[17167]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:27 smithi016 ceph-mon[22010]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:28.218 INFO:teuthology.orchestra.run.smithi002.stdout:244813135904 2022-03-21T14:07:28.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.8 2022-03-21T14:07:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:27 smithi079 ceph-mon[23158]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:28.264 INFO:teuthology.orchestra.run.smithi002.stdout:158913790000 2022-03-21T14:07:28.264 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-03-21T14:07:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:28.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.943+0000 7feea09f9700 1 -- 172.21.15.2:0/717086343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 msgr2=0x7fee9c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.943+0000 7feea09f9700 1 --2- 172.21.15.2:0/717086343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c0fd4e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee900098e0 tx=0x7fee90005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:28.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 -- 172.21.15.2:0/717086343 shutdown_connections 2022-03-21T14:07:28.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 --2- 172.21.15.2:0/717086343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c0fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 --2- 172.21.15.2:0/717086343 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee9c0fda20 0x7fee9c104350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 --2- 172.21.15.2:0/717086343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c0fd4e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 -- 172.21.15.2:0/717086343 >> 172.21.15.2:0/717086343 conn(0x7fee9c0f75c0 msgr2=0x7fee9c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:28.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 -- 172.21.15.2:0/717086343 shutdown_connections 2022-03-21T14:07:28.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.944+0000 7feea09f9700 1 -- 172.21.15.2:0/717086343 wait complete. 2022-03-21T14:07:28.948 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 Processor -- start 2022-03-21T14:07:28.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 -- start start 2022-03-21T14:07:28.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c10ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:28.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c10f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:28.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee9c0fda20 0x7fee9c1163a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:28.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c1168e0 con 0x7fee9c0fd070 2022-03-21T14:07:28.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c116a50 con 0x7fee9c0fbe70 2022-03-21T14:07:28.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.945+0000 7feea09f9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c116c10 con 0x7fee9c0fda20 2022-03-21T14:07:28.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c10ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:28.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c10ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48452/0 (socket says 172.21.15.2:48452) 2022-03-21T14:07:28.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 -- 172.21.15.2:0/3577060509 learned_addr learned my addr 172.21.15.2:0/3577060509 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:28.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9a59c700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c10f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:28.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9b59e700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee9c0fda20 0x7fee9c1163a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:28.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 -- 172.21.15.2:0/3577060509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee9c0fda20 msgr2=0x7fee9c1163a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee9c0fda20 0x7fee9c1163a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 -- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 msgr2=0x7fee9c10f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c10f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee900093f0 con 0x7fee9c0fbe70 2022-03-21T14:07:28.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9a59c700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c0fd070 0x7fee9c10f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:28.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.946+0000 7fee9ad9d700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c10ee20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00fa90 tx=0x7fee8c005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:28.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.947+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c00be00 con 0x7fee9c0fbe70 2022-03-21T14:07:28.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.947+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c116ef0 con 0x7fee9c0fbe70 2022-03-21T14:07:28.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.947+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fee8c00de90 con 0x7fee9c0fbe70 2022-03-21T14:07:28.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.947+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c011e80 con 0x7fee9c0fbe70 2022-03-21T14:07:28.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.947+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c117380 con 0x7fee9c0fbe70 2022-03-21T14:07:28.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.948+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee9c04b760 con 0x7fee9c0fbe70 2022-03-21T14:07:28.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.950+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fee8c00c040 con 0x7fee9c0fbe70 2022-03-21T14:07:28.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee837fe700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee84096d10 0x7fee840991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:28.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fee8c0bff80 con 0x7fee9c0fbe70 2022-03-21T14:07:28.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee9a59c700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee84096d10 0x7fee840991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:28.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee837fe700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] conn(0x7fee840a08c0 0x7fee840a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:28.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee840a3420 con 0x7fee840a08c0 2022-03-21T14:07:28.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.951+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee8c0c02d0 con 0x7fee9c0fbe70 2022-03-21T14:07:28.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.952+0000 7fee9b59e700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] conn(0x7fee840a08c0 0x7fee840a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:28.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.952+0000 7fee9a59c700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee84096d10 0x7fee840991c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee90009f50 tx=0x7fee9001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:28.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.952+0000 7fee9b59e700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] conn(0x7fee840a08c0 0x7fee840a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:28.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.953+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== osd.10 v2:172.21.15.16:6824/4107732214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fee840a3420 con 0x7fee840a08c0 2022-03-21T14:07:28.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.970+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 --> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fee9c05d6d0 con 0x7fee840a08c0 2022-03-21T14:07:28.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7fee837fe700 1 -- 172.21.15.2:0/3577060509 <== osd.10 v2:172.21.15.16:6824/4107732214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fee9c05d6d0 con 0x7fee840a08c0 2022-03-21T14:07:28.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] conn(0x7fee840a08c0 msgr2=0x7fee840a2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] conn(0x7fee840a08c0 0x7fee840a2d50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee84096d10 msgr2=0x7fee840991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fee84096d10 0x7fee840991c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee90009f50 tx=0x7fee9001a040 comp rx=0 tx=0).stop 2022-03-21T14:07:28.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 msgr2=0x7fee9c10ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7feea09f9700 1 --2- 172.21.15.2:0/3577060509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c0fbe70 0x7fee9c10ee20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00fa90 tx=0x7fee8c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.971+0000 7fee9b59e700 1 -- 172.21.15.2:0/3577060509 reap_dead start 2022-03-21T14:07:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.972+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 shutdown_connections 2022-03-21T14:07:28.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.972+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 >> 172.21.15.2:0/3577060509 conn(0x7fee9c0f75c0 msgr2=0x7fee9c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:28.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.972+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 shutdown_connections 2022-03-21T14:07:28.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:28.972+0000 7feea09f9700 1 -- 172.21.15.2:0/3577060509 wait complete. 2022-03-21T14:07:29.025 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.026 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.157 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.206 INFO:teuthology.orchestra.run.smithi002.stdout:193273528362 2022-03-21T14:07:29.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-03-21T14:07:29.317 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.416 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.427 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.431 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:29.455 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:30.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:29 smithi002 ceph-mon[17167]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:30.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:29 smithi016 ceph-mon[22010]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:29 smithi079 ceph-mon[23158]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:30.527 INFO:teuthology.orchestra.run.smithi002.stdout:300647710744 2022-03-21T14:07:30.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.10 2022-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:32.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:31 smithi002 ceph-mon[17167]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2022-03-21T14:07:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:31 smithi016 ceph-mon[22010]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2022-03-21T14:07:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:31 smithi079 ceph-mon[23158]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2022-03-21T14:07:32.542 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:34.028 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:33 smithi002 ceph-mon[17167]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 972 B/s rd, 1 op/s 2022-03-21T14:07:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:33 smithi016 ceph-mon[22010]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 972 B/s rd, 1 op/s 2022-03-21T14:07:34.222 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:33 smithi079 ceph-mon[23158]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 972 B/s rd, 1 op/s 2022-03-21T14:07:34.268 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:34.454 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/3352075597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f28104000 msgr2=0x7f2f281063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:34.455 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 --2- 172.21.15.2:0/3352075597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f28104000 0x7f2f281063e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009ba0 tx=0x7f2f20009430 comp rx=0 tx=0).stop 2022-03-21T14:07:34.455 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/3352075597 shutdown_connections 2022-03-21T14:07:34.455 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 --2- 172.21.15.2:0/3352075597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f28104000 0x7f2f281063e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.456 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 --2- 172.21.15.2:0/3352075597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f2806f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.456 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.239+0000 7f2f2ff9d700 1 --2- 172.21.15.2:0/3352075597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f2806f8a0 0x7f2f2806fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.456 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.240+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/3352075597 >> 172.21.15.2:0/3352075597 conn(0x7f2f280f7880 msgr2=0x7f2f280f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:34.456 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.240+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/3352075597 shutdown_connections 2022-03-21T14:07:34.457 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.240+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/3352075597 wait complete. 2022-03-21T14:07:34.457 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.241+0000 7f2f2ff9d700 1 Processor -- start 2022-03-21T14:07:34.457 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.241+0000 7f2f2ff9d700 1 -- start start 2022-03-21T14:07:34.457 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.241+0000 7f2f2ff9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:34.457 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.241+0000 7f2f2ff9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f280ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:34.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2ff9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 0x7f2f280fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:34.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2ff9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280fc6f0 con 0x7f2f28104000 2022-03-21T14:07:34.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2ff9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280fc860 con 0x7f2f2806f8a0 2022-03-21T14:07:34.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2ff9d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280fca20 con 0x7f2f28070e60 2022-03-21T14:07:34.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2dd39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:34.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2dd39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48456/0 (socket says 172.21.15.2:48456) 2022-03-21T14:07:34.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2dd39700 1 -- 172.21.15.2:0/2633840152 learned_addr learned my addr 172.21.15.2:0/2633840152 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:34.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2d538700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f280ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:34.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.242+0000 7f2f2e53a700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 0x7f2f280fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:34.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 -- 172.21.15.2:0/2633840152 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 msgr2=0x7f2f280ff9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:34.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f280ff9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 -- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 msgr2=0x7f2f280fbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:34.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 0x7f2f280fbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.461 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 -- 172.21.15.2:0/2633840152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180093f0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.461 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2d538700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f280ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:34.461 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2e53a700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 0x7f2f280fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:34.461 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.243+0000 7f2f2dd39700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400d9e0 tx=0x7f2f2400b3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:34.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.244+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400be20 con 0x7f2f2806f8a0 2022-03-21T14:07:34.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.244+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/2633840152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f200096b0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.244+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2f2400b500 con 0x7f2f2806f8a0 2022-03-21T14:07:34.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.244+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24010df0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.244+0000 7f2f2ff9d700 1 -- 172.21.15.2:0/2633840152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2811b6e0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.246+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28109ac0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.252+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2f2400c040 con 0x7f2f2806f8a0 2022-03-21T14:07:34.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.255+0000 7f2f1e7fc700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 0x7f2f14099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:34.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.255+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2f24016030 con 0x7f2f2806f8a0 2022-03-21T14:07:34.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.256+0000 7f2f2d538700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 0x7f2f14099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:34.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.256+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2f240e7fc0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.256+0000 7f2f2d538700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 0x7f2f14099240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f180093c0 tx=0x7f2f18005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:34.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.477+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2f2805dbe0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.478+0000 7f2f1e7fc700 1 -- 172.21.15.2:0/2633840152 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2f2408ccb0 con 0x7f2f2806f8a0 2022-03-21T14:07:34.479 INFO:teuthology.orchestra.run.smithi002.stdout:214748364839 2022-03-21T14:07:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.480+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 msgr2=0x7f2f14099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.480+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 0x7f2f14099240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f180093c0 tx=0x7f2f18005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.480+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 msgr2=0x7f2f280ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:34.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.480+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400d9e0 tx=0x7f2f2400b3a0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 shutdown_connections 2022-03-21T14:07:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f2806f8a0 0x7f2f280ff4a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2f14096d90 0x7f2f14099240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f28104000 0x7f2f280fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 --2- 172.21.15.2:0/2633840152 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f28070e60 0x7f2f280ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:34.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 >> 172.21.15.2:0/2633840152 conn(0x7f2f280f7880 msgr2=0x7f2f28106180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:34.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 shutdown_connections 2022-03-21T14:07:34.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:34.481+0000 7f2f0bfff700 1 -- 172.21.15.2:0/2633840152 wait complete. 2022-03-21T14:07:34.885 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:34 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2633840152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-03-21T14:07:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:34 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2633840152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-03-21T14:07:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:34 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2633840152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-03-21T14:07:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:35.884 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:36.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:35 smithi002 ceph-mon[17167]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:36.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 -- 172.21.15.2:0/3669357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fd310 msgr2=0x7faaa80fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 --2- 172.21.15.2:0/3669357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fd310 0x7faaa80fd760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faa980098e0 tx=0x7faa98005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:36.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 -- 172.21.15.2:0/3669357820 shutdown_connections 2022-03-21T14:07:36.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 --2- 172.21.15.2:0/3669357820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fdca0 0x7faaa8104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 --2- 172.21.15.2:0/3669357820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fc070 0x7faaa80fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 --2- 172.21.15.2:0/3669357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fd310 0x7faaa80fd760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 -- 172.21.15.2:0/3669357820 >> 172.21.15.2:0/3669357820 conn(0x7faaa80f7880 msgr2=0x7faaa80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 -- 172.21.15.2:0/3669357820 shutdown_connections 2022-03-21T14:07:36.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.078+0000 7faaaf763700 1 -- 172.21.15.2:0/3669357820 wait complete. 2022-03-21T14:07:36.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.079+0000 7faaaf763700 1 Processor -- start 2022-03-21T14:07:36.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.079+0000 7faaaf763700 1 -- start start 2022-03-21T14:07:36.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fc070 0x7faaa8102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fd310 0x7faaa8102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80ff530 con 0x7faaa80fc070 2022-03-21T14:07:36.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80ff6a0 con 0x7faaa80fdca0 2022-03-21T14:07:36.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaaf763700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80ff860 con 0x7faaa80fd310 2022-03-21T14:07:36.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaad4ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fc070 0x7faaa8102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaadd00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaadd00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48464/0 (socket says 172.21.15.2:48464) 2022-03-21T14:07:36.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.080+0000 7faaadd00700 1 -- 172.21.15.2:0/1113054084 learned_addr learned my addr 172.21.15.2:0/1113054084 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:36.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaaccfe700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fd310 0x7faaa8102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 -- 172.21.15.2:0/1113054084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fd310 msgr2=0x7faaa8102790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fd310 0x7faaa8102790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 -- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fc070 msgr2=0x7faaa8102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fc070 0x7faaa8102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 -- 172.21.15.2:0/1113054084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa980093f0 con 0x7faaa80fdca0 2022-03-21T14:07:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaadd00700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faaa000ec50 tx=0x7faaa0009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa000cb50 con 0x7faaa80fdca0 2022-03-21T14:07:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7faaa000ccb0 con 0x7faaa80fdca0 2022-03-21T14:07:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa0018690 con 0x7faaa80fdca0 2022-03-21T14:07:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.081+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa811b620 con 0x7faaa80fdca0 2022-03-21T14:07:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.082+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa811bb60 con 0x7faaa80fdca0 2022-03-21T14:07:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.084+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa8107910 con 0x7faaa80fdca0 2022-03-21T14:07:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.085+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7faaa000c040 con 0x7faaa80fdca0 2022-03-21T14:07:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.086+0000 7faa9dffb700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 0x7faa94099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.086+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7faaa0014070 con 0x7faaa80fdca0 2022-03-21T14:07:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.087+0000 7faaad4ff700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 0x7faa94099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.087+0000 7faaad4ff700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 0x7faa94099240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4009060 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.090+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7faaa008d0e0 con 0x7faaa80fdca0 2022-03-21T14:07:36.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:35 smithi016 ceph-mon[22010]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:35 smithi079 ceph-mon[23158]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:36.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.326+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7faaa811bd50 con 0x7faaa80fdca0 2022-03-21T14:07:36.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.327+0000 7faa9dffb700 1 -- 172.21.15.2:0/1113054084 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7faaa0090700 con 0x7faaa80fdca0 2022-03-21T14:07:36.328 INFO:teuthology.orchestra.run.smithi002.stdout:279172874270 2022-03-21T14:07:36.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.330+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 msgr2=0x7faa94099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.330+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 0x7faa94099240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4009060 comp rx=0 tx=0).stop 2022-03-21T14:07:36.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.330+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 msgr2=0x7faaa80fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.330+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faaa000ec50 tx=0x7faaa0009430 comp rx=0 tx=0).stop 2022-03-21T14:07:36.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 shutdown_connections 2022-03-21T14:07:36.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa80fdca0 0x7faaa80fed90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faa94096d90 0x7faa94099240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faaa80fc070 0x7faaa8102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 --2- 172.21.15.2:0/1113054084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faaa80fd310 0x7faaa8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 >> 172.21.15.2:0/1113054084 conn(0x7faaa80f7880 msgr2=0x7faaa80f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 shutdown_connections 2022-03-21T14:07:36.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.331+0000 7faaaf763700 1 -- 172.21.15.2:0/1113054084 wait complete. 2022-03-21T14:07:36.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364837 got 214748364839 for osd.7 2022-03-21T14:07:36.340 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:36.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 -- 172.21.15.2:0/1826759076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 msgr2=0x7f053c104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 --2- 172.21.15.2:0/1826759076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c104560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0534009c70 tx=0x7f05340094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 -- 172.21.15.2:0/1826759076 shutdown_connections 2022-03-21T14:07:36.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 --2- 172.21.15.2:0/1826759076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fc070 0x7f053c0fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 --2- 172.21.15.2:0/1826759076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c104560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 --2- 172.21.15.2:0/1826759076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f053c0fd310 0x7f053c0fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 -- 172.21.15.2:0/1826759076 >> 172.21.15.2:0/1826759076 conn(0x7f053c0f7880 msgr2=0x7f053c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 -- 172.21.15.2:0/1826759076 shutdown_connections 2022-03-21T14:07:36.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.529+0000 7f0541ee6700 1 -- 172.21.15.2:0/1826759076 wait complete. 2022-03-21T14:07:36.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.530+0000 7f0541ee6700 1 Processor -- start 2022-03-21T14:07:36.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 -- start start 2022-03-21T14:07:36.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f053c0fc070 0x7f053c10cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c1140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c114630 con 0x7f053c0fdca0 2022-03-21T14:07:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1147a0 con 0x7f053c0fd310 2022-03-21T14:07:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.531+0000 7f0541ee6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c114960 con 0x7f053c0fc070 2022-03-21T14:07:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48470/0 (socket says 172.21.15.2:48470) 2022-03-21T14:07:36.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 -- 172.21.15.2:0/1321146298 learned_addr learned my addr 172.21.15.2:0/1321146298 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:36.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053bfff700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c1140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 -- 172.21.15.2:0/1321146298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f053c0fc070 msgr2=0x7f053c10cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f053c0fc070 0x7f053c10cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 -- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 msgr2=0x7f053c1140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c1140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.532+0000 7f053affd700 1 -- 172.21.15.2:0/1321146298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534009780 con 0x7f053c0fd310 2022-03-21T14:07:36.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.537+0000 7f053affd700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f052c00bca0 tx=0x7f052c005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.537+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c00d9b0 con 0x7f053c0fd310 2022-03-21T14:07:36.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.537+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f052c013030 con 0x7f053c0fd310 2022-03-21T14:07:36.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.537+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c012ca0 con 0x7f053c0fd310 2022-03-21T14:07:36.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.537+0000 7f0541ee6700 1 -- 172.21.15.2:0/1321146298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c06f260 con 0x7f053c0fd310 2022-03-21T14:07:36.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.538+0000 7f0541ee6700 1 -- 172.21.15.2:0/1321146298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c06f720 con 0x7f053c0fd310 2022-03-21T14:07:36.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.980+0000 7f64b7429700 1 -- 172.21.15.2:0/3913819069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fdca0 msgr2=0x7f64b0104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.980+0000 7f64b7429700 1 --2- 172.21.15.2:0/3913819069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fdca0 0x7f64b0104560 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009c70 tx=0x7f64a80094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 -- 172.21.15.2:0/3913819069 shutdown_connections 2022-03-21T14:07:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 --2- 172.21.15.2:0/3913819069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fdca0 0x7f64b0104560 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 --2- 172.21.15.2:0/3913819069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b00fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 --2- 172.21.15.2:0/3913819069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fd310 0x7f64b00fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 -- 172.21.15.2:0/3913819069 >> 172.21.15.2:0/3913819069 conn(0x7f64b00f7880 msgr2=0x7f64b00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 -- 172.21.15.2:0/3913819069 shutdown_connections 2022-03-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.981+0000 7f64b7429700 1 -- 172.21.15.2:0/3913819069 wait complete. 2022-03-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.982+0000 7f64b7429700 1 Processor -- start 2022-03-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.982+0000 7f64b7429700 1 -- start start 2022-03-21T14:07:36.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b010cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fd310 0x7f64b010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 0x7f64b0114140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0114680 con 0x7f64b00fc070 2022-03-21T14:07:36.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01147f0 con 0x7f64b00fdca0 2022-03-21T14:07:36.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b7429700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01149b0 con 0x7f64b00fd310 2022-03-21T14:07:36.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b51c5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b010cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b59c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 0x7f64b0114140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b51c5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b010cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43876/0 (socket says 172.21.15.2:43876) 2022-03-21T14:07:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b49c4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fd310 0x7f64b010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.983+0000 7f64b51c5700 1 -- 172.21.15.2:0/715437270 learned_addr learned my addr 172.21.15.2:0/715437270 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 -- 172.21.15.2:0/715437270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fd310 msgr2=0x7f64b010d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fd310 0x7f64b010d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 -- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 msgr2=0x7f64b010cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b010cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 -- 172.21.15.2:0/715437270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8009780 con 0x7f64b00fdca0 2022-03-21T14:07:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.984+0000 7f64b59c6700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 0x7f64b0114140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009230 tx=0x7f64a80094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.985+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a801c070 con 0x7f64b00fdca0 2022-03-21T14:07:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.985+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f64a800f910 con 0x7f64b00fdca0 2022-03-21T14:07:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.985+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8003e40 con 0x7f64b00fdca0 2022-03-21T14:07:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.985+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b006f200 con 0x7f64b00fdca0 2022-03-21T14:07:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.985+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b006f6e0 con 0x7f64b00fdca0 2022-03-21T14:07:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.986+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0107880 con 0x7f64b00fdca0 2022-03-21T14:07:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.988+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f64a8028030 con 0x7f64b00fdca0 2022-03-21T14:07:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.990+0000 7f64a5ffb700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 0x7f649c099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.562 INFO:teuthology.orchestra.run.smithi002.stdout:137438953525 2022-03-21T14:07:36.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.990+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f64a80c0fe0 con 0x7f64b00fdca0 2022-03-21T14:07:36.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.990+0000 7f64b51c5700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 0x7f649c099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.991+0000 7f64b51c5700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 0x7f649c099240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00bdd0 tx=0x7f64ac005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:35.992+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f64a808e260 con 0x7f64b00fdca0 2022-03-21T14:07:36.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.232+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f64b0103460 con 0x7f64b00fdca0 2022-03-21T14:07:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.233+0000 7f64a5ffb700 1 -- 172.21.15.2:0/715437270 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f64a8026090 con 0x7f64b00fdca0 2022-03-21T14:07:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 msgr2=0x7f649c099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 0x7f649c099240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00bdd0 tx=0x7f64ac005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 msgr2=0x7f64b0114140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 0x7f64b0114140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009230 tx=0x7f64a80094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 shutdown_connections 2022-03-21T14:07:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64b00fdca0 0x7f64b0114140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f649c096d90 0x7f649c099240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64b00fc070 0x7f64b010cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 --2- 172.21.15.2:0/715437270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64b00fd310 0x7f64b010d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.235+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 >> 172.21.15.2:0/715437270 conn(0x7f64b00f7880 msgr2=0x7f64b0100760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.236+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 shutdown_connections 2022-03-21T14:07:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.236+0000 7f64b7429700 1 -- 172.21.15.2:0/715437270 wait complete. 2022-03-21T14:07:36.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.542+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f052c00c040 con 0x7f053c0fd310 2022-03-21T14:07:36.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.543+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c107910 con 0x7f053c0fd310 2022-03-21T14:07:36.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.545+0000 7f0523fff700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 0x7f05240992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:36.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.545+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f052c0c3830 con 0x7f053c0fd310 2022-03-21T14:07:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.545+0000 7f053b7fe700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 0x7f05240992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.546+0000 7f053b7fe700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 0x7f05240992c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f053c0fd170 tx=0x7f0530005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.549+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f052c090a50 con 0x7f053c0fd310 2022-03-21T14:07:36.741 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:36.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.777+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f053c05dc80 con 0x7f053c0fd310 2022-03-21T14:07:36.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.777+0000 7f0523fff700 1 -- 172.21.15.2:0/1321146298 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f052c094070 con 0x7f053c0fd310 2022-03-21T14:07:36.779 INFO:teuthology.orchestra.run.smithi002.stdout:107374182457 2022-03-21T14:07:36.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 msgr2=0x7f05240992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 0x7f05240992c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f053c0fd170 tx=0x7f0530005d60 comp rx=0 tx=0).stop 2022-03-21T14:07:36.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 msgr2=0x7f053c10d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:36.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f052c00bca0 tx=0x7f052c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 shutdown_connections 2022-03-21T14:07:36.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f0524096e10 0x7f05240992c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f053c0fdca0 0x7f053c1140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f053c0fc070 0x7f053c10cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 --2- 172.21.15.2:0/1321146298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c0fd310 0x7f053c10d060 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:36.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 >> 172.21.15.2:0/1321146298 conn(0x7f053c0f7880 msgr2=0x7f053c0f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:36.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 shutdown_connections 2022-03-21T14:07:36.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:36.780+0000 7f0521ffb700 1 -- 172.21.15.2:0/1321146298 wait complete. 2022-03-21T14:07:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:36 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/715437270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-03-21T14:07:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:36 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1113054084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-03-21T14:07:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:36 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1321146298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-03-21T14:07:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:36 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/715437270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-03-21T14:07:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:36 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1113054084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-03-21T14:07:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:36 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1321146298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-03-21T14:07:37.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 -- 172.21.15.2:0/3371259366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c101af0 msgr2=0x7f284c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 --2- 172.21.15.2:0/3371259366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c101af0 0x7f284c103f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2844009ba0 tx=0x7f2844009430 comp rx=0 tx=0).stop 2022-03-21T14:07:37.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 -- 172.21.15.2:0/3371259366 shutdown_connections 2022-03-21T14:07:37.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 --2- 172.21.15.2:0/3371259366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c101af0 0x7f284c103f70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 --2- 172.21.15.2:0/3371259366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c065620 0x7f284c0fea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 --2- 172.21.15.2:0/3371259366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c0ff020 0x7f284c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 -- 172.21.15.2:0/3371259366 >> 172.21.15.2:0/3371259366 conn(0x7f284c0f64a0 msgr2=0x7f284c0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:37.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 -- 172.21.15.2:0/3371259366 shutdown_connections 2022-03-21T14:07:37.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.217+0000 7f28508d5700 1 -- 172.21.15.2:0/3371259366 wait complete. 2022-03-21T14:07:37.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.218+0000 7f28508d5700 1 Processor -- start 2022-03-21T14:07:37.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.218+0000 7f28508d5700 1 -- start start 2022-03-21T14:07:37.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 0x7f284c10c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c0ff020 0x7f284c10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c1144b0 con 0x7f284c101af0 2022-03-21T14:07:37.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c114620 con 0x7f284c065620 2022-03-21T14:07:37.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f28508d5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c1147e0 con 0x7f284c0ff020 2022-03-21T14:07:37.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f284bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f284bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43890/0 (socket says 172.21.15.2:43890) 2022-03-21T14:07:37.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f284b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 0x7f284c10c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f284bfff700 1 -- 172.21.15.2:0/3064528236 learned_addr learned my addr 172.21.15.2:0/3064528236 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:37.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.219+0000 7f284affd700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c0ff020 0x7f284c10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 -- 172.21.15.2:0/3064528236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c0ff020 msgr2=0x7f284c10cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c0ff020 0x7f284c10cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 -- 172.21.15.2:0/3064528236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 msgr2=0x7f284c10c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 0x7f284c10c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 -- 172.21.15.2:0/3064528236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c00b360 con 0x7f284c101af0 2022-03-21T14:07:37.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284b7fe700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 0x7f284c10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:37.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f284bfff700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2844005100 tx=0x7f2844009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:37.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.220+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284401d070 con 0x7f284c101af0 2022-03-21T14:07:37.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.221+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f284400f460 con 0x7f284c101af0 2022-03-21T14:07:37.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.221+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844022690 con 0x7f284c101af0 2022-03-21T14:07:37.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.221+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28400093f0 con 0x7f284c101af0 2022-03-21T14:07:37.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.221+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28440096b0 con 0x7f284c101af0 2022-03-21T14:07:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:36 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/715437270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-03-21T14:07:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:36 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1113054084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-03-21T14:07:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:36 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1321146298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-03-21T14:07:37.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.222+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c107690 con 0x7f284c101af0 2022-03-21T14:07:37.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.224+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f284400c040 con 0x7f284c101af0 2022-03-21T14:07:37.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.225+0000 7f2833fff700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 0x7f2834099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.225+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f28440c1380 con 0x7f284c101af0 2022-03-21T14:07:37.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.225+0000 7f284b7fe700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 0x7f2834099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.226+0000 7f284b7fe700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 0x7f2834099310 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f284c065a10 tx=0x7f283c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:37.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.230+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f284408e600 con 0x7f284c101af0 2022-03-21T14:07:37.355 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:37.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.440+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f284c0f7e50 con 0x7f284c101af0 2022-03-21T14:07:37.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.440+0000 7f2833fff700 1 -- 172.21.15.2:0/3064528236 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2844091c20 con 0x7f284c101af0 2022-03-21T14:07:37.441 INFO:teuthology.orchestra.run.smithi002.stdout:55834574916 2022-03-21T14:07:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 msgr2=0x7f2834099310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 0x7f2834099310 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f284c065a10 tx=0x7f283c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 msgr2=0x7f284c113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2844005100 tx=0x7f2844009430 comp rx=0 tx=0).stop 2022-03-21T14:07:37.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 shutdown_connections 2022-03-21T14:07:37.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f284c065620 0x7f284c10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2834096e60 0x7f2834099310 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f284c065a10 tx=0x7f283c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c101af0 0x7f284c113f70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 --2- 172.21.15.2:0/3064528236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f284c0ff020 0x7f284c10cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.443+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 >> 172.21.15.2:0/3064528236 conn(0x7f284c0f64a0 msgr2=0x7f284c0fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:37.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.444+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 shutdown_connections 2022-03-21T14:07:37.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.444+0000 7f28508d5700 1 -- 172.21.15.2:0/3064528236 wait complete. 2022-03-21T14:07:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 -- 172.21.15.2:0/3337536561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 msgr2=0x7fd0c8101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3337536561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c8101660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0098e0 tx=0x7fd0bc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 -- 172.21.15.2:0/3337536561 shutdown_connections 2022-03-21T14:07:37.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3337536561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3337536561 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c80fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3337536561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c8101660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 -- 172.21.15.2:0/3337536561 >> 172.21.15.2:0/3337536561 conn(0x7fd0c80f6760 msgr2=0x7fd0c80f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 -- 172.21.15.2:0/3337536561 shutdown_connections 2022-03-21T14:07:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.383+0000 7fd0ce385700 1 -- 172.21.15.2:0/3337536561 wait complete. 2022-03-21T14:07:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.384+0000 7fd0ce385700 1 Processor -- start 2022-03-21T14:07:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.384+0000 7fd0ce385700 1 -- start start 2022-03-21T14:07:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c810f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c810f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8116b50 con 0x7fd0c80ff280 2022-03-21T14:07:37.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8116cc0 con 0x7fd0c8101cf0 2022-03-21T14:07:37.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0ce385700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8116e80 con 0x7fd0c80655c0 2022-03-21T14:07:37.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0c77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c810f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0cc922700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0cc922700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48478/0 (socket says 172.21.15.2:48478) 2022-03-21T14:07:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0cc922700 1 -- 172.21.15.2:0/3233341985 learned_addr learned my addr 172.21.15.2:0/3233341985 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.385+0000 7fd0c7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c810f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 -- 172.21.15.2:0/3233341985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 msgr2=0x7fd0c810f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c810f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 -- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 msgr2=0x7fd0c810f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c810f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 -- 172.21.15.2:0/3233341985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0093f0 con 0x7fd0c8101cf0 2022-03-21T14:07:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0c7fff700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0cc922700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000ec50 tx=0x7fd0c0009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000caa0 con 0x7fd0c8101cf0 2022-03-21T14:07:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd0c000cc00 con 0x7fd0c8101cf0 2022-03-21T14:07:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0018690 con 0x7fd0c8101cf0 2022-03-21T14:07:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.386+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c8117160 con 0x7fd0c8101cf0 2022-03-21T14:07:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.387+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c8117680 con 0x7fd0c8101cf0 2022-03-21T14:07:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.388+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c81078a0 con 0x7fd0c8101cf0 2022-03-21T14:07:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.390+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd0c000c040 con 0x7fd0c8101cf0 2022-03-21T14:07:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.391+0000 7fd0c4ff9700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 0x7fd0b00991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.391+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fd0c0014070 con 0x7fd0c8101cf0 2022-03-21T14:07:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.391+0000 7fd0c7fff700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 0x7fd0b00991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.392+0000 7fd0c7fff700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 0x7fd0b00991c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8005fd0 tx=0x7fd0b8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.395+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd0c008d120 con 0x7fd0c8101cf0 2022-03-21T14:07:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.648+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd0c802ab20 con 0x7fd0c8101cf0 2022-03-21T14:07:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.649+0000 7fd0c4ff9700 1 -- 172.21.15.2:0/3233341985 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0c0090740 con 0x7fd0c8101cf0 2022-03-21T14:07:37.672 INFO:teuthology.orchestra.run.smithi002.stdout:34359738440 2022-03-21T14:07:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 msgr2=0x7fd0b00991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 0x7fd0b00991c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8005fd0 tx=0x7fd0b8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 msgr2=0x7fd0c8116610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000ec50 tx=0x7fd0c0009430 comp rx=0 tx=0).stop 2022-03-21T14:07:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 shutdown_connections 2022-03-21T14:07:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c8101cf0 0x7fd0c8116610 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd0b0096d10 0x7fd0b00991c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd0c80655c0 0x7fd0c810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 --2- 172.21.15.2:0/3233341985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0c80ff280 0x7fd0c810f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 >> 172.21.15.2:0/3233341985 conn(0x7fd0c80f6760 msgr2=0x7fd0c8102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 shutdown_connections 2022-03-21T14:07:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:37.652+0000 7fd0ce385700 1 -- 172.21.15.2:0/3233341985 wait complete. 2022-03-21T14:07:38.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:37 smithi002 ceph-mon[17167]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:38.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:37 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3064528236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-03-21T14:07:38.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:37 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3233341985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-03-21T14:07:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:37 smithi016 ceph-mon[22010]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:37 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3064528236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-03-21T14:07:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:37 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3233341985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-03-21T14:07:38.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953523 got 137438953525 for osd.4 2022-03-21T14:07:38.187 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:37 smithi079 ceph-mon[23158]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:37 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3064528236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-03-21T14:07:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:37 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3233341985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-03-21T14:07:38.337 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:38.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 -- 172.21.15.2:0/2716577792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fc070 msgr2=0x7f380c0fc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2716577792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fc070 0x7f380c0fc480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0098e0 tx=0x7f37fc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:38.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 -- 172.21.15.2:0/2716577792 shutdown_connections 2022-03-21T14:07:38.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2716577792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fdca0 0x7f380c104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2716577792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fc070 0x7f380c0fc480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2716577792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fd310 0x7f380c0fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 -- 172.21.15.2:0/2716577792 >> 172.21.15.2:0/2716577792 conn(0x7f380c0f7880 msgr2=0x7f380c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:38.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.367+0000 7f3811ffc700 1 -- 172.21.15.2:0/2716577792 shutdown_connections 2022-03-21T14:07:38.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.368+0000 7f3811ffc700 1 -- 172.21.15.2:0/2716577792 wait complete. 2022-03-21T14:07:38.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.369+0000 7f3811ffc700 1 Processor -- start 2022-03-21T14:07:38.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.369+0000 7f3811ffc700 1 -- start start 2022-03-21T14:07:38.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 0x7f380c10b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 0x7f380c112190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c112710 con 0x7f380c0fdca0 2022-03-21T14:07:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c112880 con 0x7f380c0fc070 2022-03-21T14:07:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f3811ffc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c112a40 con 0x7f380c0fd310 2022-03-21T14:07:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f380b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f380b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48482/0 (socket says 172.21.15.2:48482) 2022-03-21T14:07:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f380b7fe700 1 -- 172.21.15.2:0/2750542827 learned_addr learned my addr 172.21.15.2:0/2750542827 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f380affd700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 0x7f380c10b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.370+0000 7f380bfff700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 0x7f380c112190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 -- 172.21.15.2:0/2750542827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 msgr2=0x7f380c10b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 0x7f380c10b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 -- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 msgr2=0x7f380c112190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 0x7f380c112190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 -- 172.21.15.2:0/2750542827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380000b360 con 0x7f380c0fc070 2022-03-21T14:07:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380affd700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 0x7f380c10b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380bfff700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 0x7f380c112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f380b7fe700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0093c0 tx=0x7f37fc005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.371+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc01d070 con 0x7f380c0fc070 2022-03-21T14:07:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.372+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc0093f0 con 0x7f380c0fc070 2022-03-21T14:07:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.372+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f37fc005740 con 0x7f380c0fc070 2022-03-21T14:07:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.372+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc017950 con 0x7f380c0fc070 2022-03-21T14:07:38.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.372+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c113010 con 0x7f380c0fc070 2022-03-21T14:07:38.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.373+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c105740 con 0x7f380c0fc070 2022-03-21T14:07:38.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.375+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f37fc028030 con 0x7f380c0fc070 2022-03-21T14:07:38.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.377+0000 7f37f3fff700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 0x7f37f40991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.377+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f37fc0c08e0 con 0x7f380c0fc070 2022-03-21T14:07:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.377+0000 7f380affd700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 0x7f37f40991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.378+0000 7f380affd700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 0x7f37f40991c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3800005fd0 tx=0x7f3800005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.378+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f37fc08db60 con 0x7f380c0fc070 2022-03-21T14:07:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 -- 172.21.15.2:0/1311034052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c0ff020 msgr2=0x7f2b0c101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/1311034052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c101460 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2b000098e0 tx=0x7f2b00005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 -- 172.21.15.2:0/1311034052 shutdown_connections 2022-03-21T14:07:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/1311034052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c101af0 0x7f2b0c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/1311034052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c065620 0x7f2b0c0fea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/1311034052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c101460 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 -- 172.21.15.2:0/1311034052 >> 172.21.15.2:0/1311034052 conn(0x7f2b0c0f64a0 msgr2=0x7f2b0c0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 -- 172.21.15.2:0/1311034052 shutdown_connections 2022-03-21T14:07:38.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.523+0000 7f2b0bfff700 1 -- 172.21.15.2:0/1311034052 wait complete. 2022-03-21T14:07:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 Processor -- start 2022-03-21T14:07:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 -- start start 2022-03-21T14:07:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 0x7f2b0c10ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c10f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c101af0 0x7f2b0c116430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.525+0000 7f2b0bfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c116970 con 0x7f2b0c0ff020 2022-03-21T14:07:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0bfff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c116ae0 con 0x7f2b0c101af0 2022-03-21T14:07:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0bfff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c116ca0 con 0x7f2b0c065620 2022-03-21T14:07:38.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 0x7f2b0c10ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c10f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 0x7f2b0c10ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39752/0 (socket says 172.21.15.2:39752) 2022-03-21T14:07:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0affd700 1 -- 172.21.15.2:0/960463693 learned_addr learned my addr 172.21.15.2:0/960463693 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:38.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0b7fe700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c101af0 0x7f2b0c116430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 -- 172.21.15.2:0/960463693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 msgr2=0x7f2b0c10ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 0x7f2b0c10ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 -- 172.21.15.2:0/960463693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c101af0 msgr2=0x7f2b0c116430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c101af0 0x7f2b0c116430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.526+0000 7f2b0a7fc700 1 -- 172.21.15.2:0/960463693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc00b360 con 0x7f2b0c0ff020 2022-03-21T14:07:38.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.527+0000 7f2b0a7fc700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c10f3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009780 tx=0x7f2b00005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.527+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0001d070 con 0x7f2b0c0ff020 2022-03-21T14:07:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.527+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b00000c90 con 0x7f2b0c0ff020 2022-03-21T14:07:38.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.527+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00017950 con 0x7f2b0c0ff020 2022-03-21T14:07:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.528+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b000093f0 con 0x7f2b0c0ff020 2022-03-21T14:07:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.528+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0c1171f0 con 0x7f2b0c0ff020 2022-03-21T14:07:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.530+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0c107710 con 0x7f2b0c0ff020 2022-03-21T14:07:38.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.532+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2b00029030 con 0x7f2b0c0ff020 2022-03-21T14:07:38.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.535+0000 7f2af37fe700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 0x7f2af40991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:38.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.535+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2b000c0be0 con 0x7f2b0c0ff020 2022-03-21T14:07:38.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.535+0000 7f2b0affd700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 0x7f2af40991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:38.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.536+0000 7f2b0affd700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 0x7f2af40991c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00bef0 tx=0x7f2afc005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:38.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.543+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2b0008de60 con 0x7f2b0c0ff020 2022-03-21T14:07:38.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.612+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f380c113260 con 0x7f380c0fc070 2022-03-21T14:07:38.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.612+0000 7f37f3fff700 1 -- 172.21.15.2:0/2750542827 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f37fc091180 con 0x7f380c0fc070 2022-03-21T14:07:38.614 INFO:teuthology.orchestra.run.smithi002.stdout:330712481814 2022-03-21T14:07:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 msgr2=0x7f37f40991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 0x7f37f40991c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3800005fd0 tx=0x7f3800005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 msgr2=0x7f380c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0093c0 tx=0x7f37fc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 shutdown_connections 2022-03-21T14:07:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c0fc070 0x7f380c10ac00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f37f4096d10 0x7f37f40991c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380c0fdca0 0x7f380c112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 --2- 172.21.15.2:0/2750542827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380c0fd310 0x7f380c10b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 >> 172.21.15.2:0/2750542827 conn(0x7f380c0f7880 msgr2=0x7f380c0f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:38.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 shutdown_connections 2022-03-21T14:07:38.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.616+0000 7f3811ffc700 1 -- 172.21.15.2:0/2750542827 wait complete. 2022-03-21T14:07:38.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.772+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2b0c02aa80 con 0x7f2b0c0ff020 2022-03-21T14:07:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.773+0000 7f2af37fe700 1 -- 172.21.15.2:0/960463693 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2b00091480 con 0x7f2b0c0ff020 2022-03-21T14:07:38.774 INFO:teuthology.orchestra.run.smithi002.stdout:158913790002 2022-03-21T14:07:38.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 msgr2=0x7f2af40991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 0x7f2af40991c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00bef0 tx=0x7f2afc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 msgr2=0x7f2b0c10f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:38.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c10f3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009780 tx=0x7f2b00005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 shutdown_connections 2022-03-21T14:07:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b0c101af0 0x7f2b0c116430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2af4096d10 0x7f2af40991c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b0c065620 0x7f2b0c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 --2- 172.21.15.2:0/960463693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b0c0ff020 0x7f2b0c10f3a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 >> 172.21.15.2:0/960463693 conn(0x7f2b0c0f64a0 msgr2=0x7f2b0c0f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 shutdown_connections 2022-03-21T14:07:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.776+0000 7f2b0bfff700 1 -- 172.21.15.2:0/960463693 wait complete. 2022-03-21T14:07:39.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738440 for osd.0 2022-03-21T14:07:39.044 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:39.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182455 got 107374182457 for osd.3 2022-03-21T14:07:39.490 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:39 smithi002 ceph-mon[17167]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:39.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:39 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2750542827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-03-21T14:07:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:39 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2529944040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-03-21T14:07:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:39 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/960463693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-03-21T14:07:39.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574913 got 55834574916 for osd.1 2022-03-21T14:07:39.574 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:39.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 -- 172.21.15.2:0/680486014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 msgr2=0x7f92d010a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 --2- 172.21.15.2:0/680486014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d010a5a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009c70 tx=0x7f92c80094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 -- 172.21.15.2:0/680486014 shutdown_connections 2022-03-21T14:07:39.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 --2- 172.21.15.2:0/680486014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d010a5a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 --2- 172.21.15.2:0/680486014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d0070ec0 0x7f92d006f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 --2- 172.21.15.2:0/680486014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d006f900 0x7f92d006fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 -- 172.21.15.2:0/680486014 >> 172.21.15.2:0/680486014 conn(0x7f92d00f75c0 msgr2=0x7f92d00f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:39.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 -- 172.21.15.2:0/680486014 shutdown_connections 2022-03-21T14:07:39.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.553+0000 7f92d48cd700 1 -- 172.21.15.2:0/680486014 wait complete. 2022-03-21T14:07:39.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.554+0000 7f92d48cd700 1 Processor -- start 2022-03-21T14:07:39.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.554+0000 7f92d48cd700 1 -- start start 2022-03-21T14:07:39.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 0x7f92d0113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d011ab40 con 0x7f92d0070ec0 2022-03-21T14:07:39.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d011acb0 con 0x7f92d00ff990 2022-03-21T14:07:39.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92d48cd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d011ae70 con 0x7f92d006f900 2022-03-21T14:07:39.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43906/0 (socket says 172.21.15.2:43906) 2022-03-21T14:07:39.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92ceffd700 1 -- 172.21.15.2:0/2529944040 learned_addr learned my addr 172.21.15.2:0/2529944040 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:39.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92cffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.555+0000 7f92cf7fe700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 0x7f92d0113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 -- 172.21.15.2:0/2529944040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 msgr2=0x7f92d0113030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 0x7f92d0113030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 -- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 msgr2=0x7f92d0113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 -- 172.21.15.2:0/2529944040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c400b360 con 0x7f92d00ff990 2022-03-21T14:07:39.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92ceffd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:39.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cf7fe700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 0x7f92d0113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:39.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92cffff700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d011a600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f92c8006010 tx=0x7f92c80094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:39.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c801c070 con 0x7f92d00ff990 2022-03-21T14:07:39.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f92c800f460 con 0x7f92d00ff990 2022-03-21T14:07:39.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c8009780 con 0x7f92d00ff990 2022-03-21T14:07:39.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.556+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8003cb0 con 0x7f92d00ff990 2022-03-21T14:07:39.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.557+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d011b390 con 0x7f92d00ff990 2022-03-21T14:07:39.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.558+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d010dd00 con 0x7f92d00ff990 2022-03-21T14:07:39.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.559+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f92c800c040 con 0x7f92d00ff990 2022-03-21T14:07:39.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.561+0000 7f92b7fff700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 0x7f92b80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.561+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f92c80c0f20 con 0x7f92d00ff990 2022-03-21T14:07:39.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.561+0000 7f92cf7fe700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 0x7f92b80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.562+0000 7f92cf7fe700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 0x7f92b80991c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f92c0005fd0 tx=0x7f92c0005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:39.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.565+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f92c808e140 con 0x7f92d00ff990 2022-03-21T14:07:39.630 INFO:teuthology.orchestra.run.smithi002.stdout:193273528364 2022-03-21T14:07:39.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.766+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f92d002ab20 con 0x7f92d00ff990 2022-03-21T14:07:39.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.767+0000 7f92b7fff700 1 -- 172.21.15.2:0/2529944040 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f92c8094e60 con 0x7f92d00ff990 2022-03-21T14:07:39.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.769+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 msgr2=0x7f92b80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.769+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 0x7f92b80991c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f92c0005fd0 tx=0x7f92c0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.769+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 msgr2=0x7f92d011a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.769+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d011a600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f92c8006010 tx=0x7f92c80094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 shutdown_connections 2022-03-21T14:07:39.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92d00ff990 0x7f92d011a600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f92b8096d10 0x7f92b80991c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92d006f900 0x7f92d0113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 --2- 172.21.15.2:0/2529944040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0070ec0 0x7f92d0113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 >> 172.21.15.2:0/2529944040 conn(0x7f92d00f75c0 msgr2=0x7f92d00f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:39.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 shutdown_connections 2022-03-21T14:07:39.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:38.770+0000 7f92d48cd700 1 -- 172.21.15.2:0/2529944040 wait complete. 2022-03-21T14:07:39.671 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790000 got 158913790002 for osd.5 2022-03-21T14:07:39.671 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:39.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:39 smithi016 ceph-mon[22010]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:39 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2750542827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-03-21T14:07:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:39 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2529944040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-03-21T14:07:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:39 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/960463693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-03-21T14:07:39.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 -- 172.21.15.2:0/271783128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 msgr2=0x7fd7c80fc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 --2- 172.21.15.2:0/271783128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c80fc490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0098e0 tx=0x7fd7bc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 -- 172.21.15.2:0/271783128 shutdown_connections 2022-03-21T14:07:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 --2- 172.21.15.2:0/271783128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c80fc490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 --2- 172.21.15.2:0/271783128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 0x7fd7c8104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 --2- 172.21.15.2:0/271783128 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 0x7fd7c80fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 -- 172.21.15.2:0/271783128 >> 172.21.15.2:0/271783128 conn(0x7fd7c80f7850 msgr2=0x7fd7c80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:39.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 -- 172.21.15.2:0/271783128 shutdown_connections 2022-03-21T14:07:39.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.943+0000 7fd7cef73700 1 -- 172.21.15.2:0/271783128 wait complete. 2022-03-21T14:07:39.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.944+0000 7fd7cef73700 1 Processor -- start 2022-03-21T14:07:39.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.944+0000 7fd7cef73700 1 -- start start 2022-03-21T14:07:39.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 0x7fd7c8102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 0x7fd7c80fe950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80ff060 con 0x7fd7c80fdcb0 2022-03-21T14:07:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80ff1d0 con 0x7fd7c80fc080 2022-03-21T14:07:39.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cef73700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80ff390 con 0x7fd7c80fd320 2022-03-21T14:07:39.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48494/0 (socket says 172.21.15.2:48494) 2022-03-21T14:07:39.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 -- 172.21.15.2:0/3148172426 learned_addr learned my addr 172.21.15.2:0/3148172426 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7c7fff700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 0x7fd7c8102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7cd510700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 0x7fd7c80fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 -- 172.21.15.2:0/3148172426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 msgr2=0x7fd7c8102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 0x7fd7c8102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 -- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 msgr2=0x7fd7c80fe950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 0x7fd7c80fe950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.945+0000 7fd7ccd0f700 1 -- 172.21.15.2:0/3148172426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0093f0 con 0x7fd7c80fc080 2022-03-21T14:07:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7ccd0f700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc000e50 tx=0x7fd7bc005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc01d070 con 0x7fd7c80fc080 2022-03-21T14:07:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c811b4f0 con 0x7fd7c80fc080 2022-03-21T14:07:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd7bc005040 con 0x7fd7c80fc080 2022-03-21T14:07:39.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc017a40 con 0x7fd7c80fc080 2022-03-21T14:07:39.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.946+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c811b970 con 0x7fd7c80fc080 2022-03-21T14:07:39.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.947+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c8107880 con 0x7fd7c80fc080 2022-03-21T14:07:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.949+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fd7bc028030 con 0x7fd7c80fc080 2022-03-21T14:07:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.950+0000 7fd7c57fa700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 0x7fd7b80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.951+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fd7bc0c0e00 con 0x7fd7c80fc080 2022-03-21T14:07:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.951+0000 7fd7c7fff700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 0x7fd7b80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.952+0000 7fd7c7fff700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 0x7fd7b80991c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005fd0 tx=0x7fd7b4005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:39.953+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fd7bc08e080 con 0x7fd7c80fc080 2022-03-21T14:07:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:39 smithi079 ceph-mon[23158]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:39 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2750542827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-03-21T14:07:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:39 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2529944040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-03-21T14:07:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:39 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/960463693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-03-21T14:07:40.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.159+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd7c811bb00 con 0x7fd7c80fc080 2022-03-21T14:07:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.160+0000 7fd7c57fa700 1 -- 172.21.15.2:0/3148172426 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7bc0916a0 con 0x7fd7c80fc080 2022-03-21T14:07:40.161 INFO:teuthology.orchestra.run.smithi002.stdout:77309411392 2022-03-21T14:07:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.162+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 msgr2=0x7fd7b80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.162+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 0x7fd7b80991c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005fd0 tx=0x7fd7b4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.162+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 msgr2=0x7fd7c8101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.162+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc000e50 tx=0x7fd7bc005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 shutdown_connections 2022-03-21T14:07:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c80fc080 0x7fd7c8101e10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fd7b8096d10 0x7fd7b80991c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7c80fdcb0 0x7fd7c80fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 --2- 172.21.15.2:0/3148172426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c80fd320 0x7fd7c8102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:40.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 >> 172.21.15.2:0/3148172426 conn(0x7fd7c80f7850 msgr2=0x7fd7c806f200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:40.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 shutdown_connections 2022-03-21T14:07:40.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:40.163+0000 7fd7cef73700 1 -- 172.21.15.2:0/3148172426 wait complete. 2022-03-21T14:07:40.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874268 got 279172874270 for osd.9 2022-03-21T14:07:40.277 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:40.557 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481812 got 330712481814 for osd.11 2022-03-21T14:07:40.557 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:40.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:40 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3148172426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-03-21T14:07:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:40 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3148172426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-03-21T14:07:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:40 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3148172426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-03-21T14:07:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 -- 172.21.15.2:0/4065276313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980fe0d0 msgr2=0x7efc980fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/4065276313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980fe0d0 0x7efc980fe4e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b820 tx=0x7efc94005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 -- 172.21.15.2:0/4065276313 shutdown_connections 2022-03-21T14:07:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/4065276313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980fe0d0 0x7efc980fe4e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/4065276313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ffd00 0x7efc981065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.148+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/4065276313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980ff370 0x7efc980ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.149+0000 7efc9e7f4700 1 -- 172.21.15.2:0/4065276313 >> 172.21.15.2:0/4065276313 conn(0x7efc980f9960 msgr2=0x7efc980fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.149+0000 7efc9e7f4700 1 -- 172.21.15.2:0/4065276313 shutdown_connections 2022-03-21T14:07:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.149+0000 7efc9e7f4700 1 -- 172.21.15.2:0/4065276313 wait complete. 2022-03-21T14:07:41.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.149+0000 7efc9e7f4700 1 Processor -- start 2022-03-21T14:07:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 -- start start 2022-03-21T14:07:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 0x7efc9810ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 0x7efc9810f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 0x7efc981163e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc98116920 con 0x7efc980ff370 2022-03-21T14:07:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc98116a90 con 0x7efc980ffd00 2022-03-21T14:07:41.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9e7f4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc98116c50 con 0x7efc980fe0d0 2022-03-21T14:07:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9d7f2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 0x7efc9810ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9cff1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 0x7efc9810f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9d7f2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 0x7efc9810ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39766/0 (socket says 172.21.15.2:39766) 2022-03-21T14:07:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.150+0000 7efc9d7f2700 1 -- 172.21.15.2:0/63105491 learned_addr learned my addr 172.21.15.2:0/63105491 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:41.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9dff3700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 0x7efc981163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:41.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 -- 172.21.15.2:0/63105491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 msgr2=0x7efc9810ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:41.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 0x7efc9810ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 -- 172.21.15.2:0/63105491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 msgr2=0x7efc981163e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:41.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 0x7efc981163e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 -- 172.21.15.2:0/63105491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880093f0 con 0x7efc980ff370 2022-03-21T14:07:41.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9dff3700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 0x7efc981163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:41.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.151+0000 7efc9cff1700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 0x7efc9810f350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc8800ea00 tx=0x7efc88005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:41.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.152+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc880096d0 con 0x7efc980ff370 2022-03-21T14:07:41.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.152+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9400b360 con 0x7efc980ff370 2022-03-21T14:07:41.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.152+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efc8800c820 con 0x7efc980ff370 2022-03-21T14:07:41.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.152+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88003b30 con 0x7efc980ff370 2022-03-21T14:07:41.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.152+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc981172b0 con 0x7efc980ff370 2022-03-21T14:07:41.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.153+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc981098e0 con 0x7efc980ff370 2022-03-21T14:07:41.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.155+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7efc8800c040 con 0x7efc980ff370 2022-03-21T14:07:41.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.156+0000 7efc8dffb700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 0x7efc840991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:41.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.156+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7efc88014070 con 0x7efc980ff370 2022-03-21T14:07:41.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.156+0000 7efc9d7f2700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 0x7efc840991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:41.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.157+0000 7efc9d7f2700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 0x7efc840991c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efc9400be90 tx=0x7efc94005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:41.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.160+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7efc8808cf00 con 0x7efc980ff370 2022-03-21T14:07:41.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.352+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7efc9802ac40 con 0x7efc980ff370 2022-03-21T14:07:41.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.353+0000 7efc8dffb700 1 -- 172.21.15.2:0/63105491 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efc88090520 con 0x7efc980ff370 2022-03-21T14:07:41.354 INFO:teuthology.orchestra.run.smithi002.stdout:244813135907 2022-03-21T14:07:41.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 msgr2=0x7efc840991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:41.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 0x7efc840991c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efc9400be90 tx=0x7efc94005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 msgr2=0x7efc9810f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:41.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 0x7efc9810f350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc8800ea00 tx=0x7efc88005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:41.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 shutdown_connections 2022-03-21T14:07:41.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc980ffd00 0x7efc981163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7efc84096d10 0x7efc840991c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc980fe0d0 0x7efc9810ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 --2- 172.21.15.2:0/63105491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc980ff370 0x7efc9810f350 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:41.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.355+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 >> 172.21.15.2:0/63105491 conn(0x7efc980f9960 msgr2=0x7efc981025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:41.359 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.356+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 shutdown_connections 2022-03-21T14:07:41.359 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:41.356+0000 7efc9e7f4700 1 -- 172.21.15.2:0/63105491 wait complete. 2022-03-21T14:07:41.468 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411388 got 77309411392 for osd.2 2022-03-21T14:07:41.469 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:41.470 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528362 got 193273528364 for osd.6 2022-03-21T14:07:41.470 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:41.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:41 smithi002 ceph-mon[17167]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:41.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:41 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/63105491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-03-21T14:07:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:41 smithi016 ceph-mon[22010]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:41 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/63105491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-03-21T14:07:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:41 smithi079 ceph-mon[23158]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:41 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/63105491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-03-21T14:07:42.235 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:42.245 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135904 got 244813135907 for osd.8 2022-03-21T14:07:42.245 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:42.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/2408693843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580ffea0 msgr2=0x7f5b58106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:42.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/2408693843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580ffea0 0x7f5b58106760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009c70 tx=0x7f5b500094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/2408693843 shutdown_connections 2022-03-21T14:07:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/2408693843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580ffea0 0x7f5b58106760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/2408693843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580fe270 0x7f5b580fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/2408693843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ff510 0x7f5b580ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/2408693843 >> 172.21.15.2:0/2408693843 conn(0x7f5b580f9b20 msgr2=0x7f5b580fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/2408693843 shutdown_connections 2022-03-21T14:07:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.853+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/2408693843 wait complete. 2022-03-21T14:07:42.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.854+0000 7f5b5d9bc700 1 Processor -- start 2022-03-21T14:07:42.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.854+0000 7f5b5d9bc700 1 -- start start 2022-03-21T14:07:42.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:42.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580ff510 0x7f5b5810f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:42.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:42.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581168b0 con 0x7f5b580fe270 2022-03-21T14:07:42.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58116a20 con 0x7f5b580ffea0 2022-03-21T14:07:42.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b5d9bc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58116be0 con 0x7f5b580ff510 2022-03-21T14:07:42.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b56ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:42.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:42.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48502/0 (socket says 172.21.15.2:48502) 2022-03-21T14:07:42.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b56ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43918/0 (socket says 172.21.15.2:43918) 2022-03-21T14:07:42.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b577fe700 1 -- 172.21.15.2:0/4241554974 learned_addr learned my addr 172.21.15.2:0/4241554974 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:42.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b567fc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580ff510 0x7f5b5810f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:42.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b577fe700 1 -- 172.21.15.2:0/4241554974 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580ff510 msgr2=0x7f5b5810f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:42.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.855+0000 7f5b577fe700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580ff510 0x7f5b5810f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b577fe700 1 -- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 msgr2=0x7f5b5810eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:42.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b577fe700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:42.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b577fe700 1 -- 172.21.15.2:0/4241554974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50009780 con 0x7f5b580ffea0 2022-03-21T14:07:42.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b56ffd700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:42.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b577fe700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b500094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:42.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5001c070 con 0x7f5b580ffea0 2022-03-21T14:07:42.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5b5000f460 con 0x7f5b580ffea0 2022-03-21T14:07:42.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50003bc0 con 0x7f5b580ffea0 2022-03-21T14:07:42.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.856+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b58116e60 con 0x7f5b580ffea0 2022-03-21T14:07:42.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.857+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b58117230 con 0x7f5b580ffea0 2022-03-21T14:07:42.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.858+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b58109b10 con 0x7f5b580ffea0 2022-03-21T14:07:42.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.860+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f5b5000c040 con 0x7f5b580ffea0 2022-03-21T14:07:42.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.861+0000 7f5b3f7fe700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 0x7f5b40099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:42.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.861+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f5b500c1830 con 0x7f5b580ffea0 2022-03-21T14:07:42.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.861+0000 7f5b56ffd700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 0x7f5b40099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:42.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.862+0000 7f5b56ffd700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 0x7f5b40099240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b580ff370 tx=0x7f5b48005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:42.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:42.864+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f5b5008eab0 con 0x7f5b580ffea0 2022-03-21T14:07:43.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.055+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f5b58104fa0 con 0x7f5b580ffea0 2022-03-21T14:07:43.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.056+0000 7f5b3f7fe700 1 -- 172.21.15.2:0/4241554974 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5b500920d0 con 0x7f5b580ffea0 2022-03-21T14:07:43.057 INFO:teuthology.orchestra.run.smithi002.stdout:300647710747 2022-03-21T14:07:43.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 msgr2=0x7f5b40099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:43.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 0x7f5b40099240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b580ff370 tx=0x7f5b48005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 msgr2=0x7f5b58116370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:43.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b500094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 shutdown_connections 2022-03-21T14:07:43.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b580ffea0 0x7f5b58116370 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5b40096d90 0x7f5b40099240 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b580fe270 0x7f5b5810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 --2- 172.21.15.2:0/4241554974 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b580ff510 0x7f5b5810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:43.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 >> 172.21.15.2:0/4241554974 conn(0x7f5b580f9b20 msgr2=0x7f5b580fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:43.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 shutdown_connections 2022-03-21T14:07:43.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:43.059+0000 7f5b5d9bc700 1 -- 172.21.15.2:0/4241554974 wait complete. 2022-03-21T14:07:43.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710744 got 300647710747 for osd.10 2022-03-21T14:07:43.539 DEBUG:teuthology.parallel:result is None 2022-03-21T14:07:43.540 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-03-21T14:07:43.540 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph pg dump --format=json 2022-03-21T14:07:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:43 smithi002 ceph-mon[17167]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2022-03-21T14:07:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:43 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4241554974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-03-21T14:07:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:43 smithi016 ceph-mon[22010]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2022-03-21T14:07:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:43 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4241554974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-03-21T14:07:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:43 smithi079 ceph-mon[23158]: pgmap v295: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2022-03-21T14:07:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:43 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4241554974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-03-21T14:07:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:45.444 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:45 smithi002 ceph-mon[17167]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:45.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:45 smithi016 ceph-mon[22010]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:45 smithi079 ceph-mon[23158]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:45.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.975+0000 7f4dc90cc700 1 -- 172.21.15.2:0/3181929965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc4103da0 msgr2=0x7f4dc4106180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:45.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/3181929965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc4103da0 0x7f4dc4106180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009c70 tx=0x7f4dbc0094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 -- 172.21.15.2:0/3181929965 shutdown_connections 2022-03-21T14:07:45.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/3181929965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc4103da0 0x7f4dc4106180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/3181929965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 0x7f4dc406f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/3181929965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc406f900 0x7f4dc406fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 -- 172.21.15.2:0/3181929965 >> 172.21.15.2:0/3181929965 conn(0x7f4dc40f75c0 msgr2=0x7f4dc40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:45.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 -- 172.21.15.2:0/3181929965 shutdown_connections 2022-03-21T14:07:45.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.976+0000 7f4dc90cc700 1 -- 172.21.15.2:0/3181929965 wait complete. 2022-03-21T14:07:45.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 Processor -- start 2022-03-21T14:07:45.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 -- start start 2022-03-21T14:07:45.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:45.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 0x7f4dc410f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:45.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc4103da0 0x7f4dc4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:45.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc41167e0 con 0x7f4dc406f900 2022-03-21T14:07:45.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4116950 con 0x7f4dc4103da0 2022-03-21T14:07:45.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc90cc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4116b10 con 0x7f4dc4070ec0 2022-03-21T14:07:45.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.977+0000 7f4dc3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43922/0 (socket says 172.21.15.2:43922) 2022-03-21T14:07:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 -- 172.21.15.2:0/823419348 learned_addr learned my addr 172.21.15.2:0/823419348 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc37fe700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 0x7f4dc410f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc88cb700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc4103da0 0x7f4dc4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:45.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 -- 172.21.15.2:0/823419348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 msgr2=0x7f4dc410f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:45.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 0x7f4dc410f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 -- 172.21.15.2:0/823419348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc4103da0 msgr2=0x7f4dc4116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc4103da0 0x7f4dc4116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 -- 172.21.15.2:0/823419348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009780 con 0x7f4dc406f900 2022-03-21T14:07:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.978+0000 7f4dc3fff700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4db400fa90 tx=0x7f4db4005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.979+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db400b4f0 con 0x7f4dc406f900 2022-03-21T14:07:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.979+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4db400dc00 con 0x7f4dc406f900 2022-03-21T14:07:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.979+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4116df0 con 0x7f4dc406f900 2022-03-21T14:07:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.979+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4011cf0 con 0x7f4dc406f900 2022-03-21T14:07:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.979+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc41172d0 con 0x7f4dc406f900 2022-03-21T14:07:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.980+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc41098e0 con 0x7f4dc406f900 2022-03-21T14:07:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.982+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f4db400c040 con 0x7f4dc406f900 2022-03-21T14:07:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.983+0000 7f4dc0ff9700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 0x7f4dac099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.983+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f4db40bf800 con 0x7f4dc406f900 2022-03-21T14:07:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.983+0000 7f4dc37fe700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 0x7f4dac099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.984+0000 7f4dc37fe700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 0x7f4dac099240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4110190 tx=0x7f4db8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:45.987+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f4db408ca20 con 0x7f4dc406f900 2022-03-21T14:07:46.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.168+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4dc402acb0 con 0x7f4dac096d90 2022-03-21T14:07:46.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.179+0000 7f4dc0ff9700 1 -- 172.21.15.2:0/823419348 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+338085 (secure 0 0 0) 0x7f4dc402acb0 con 0x7f4dac096d90 2022-03-21T14:07:46.187 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:46.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.185+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 msgr2=0x7f4dac099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:46.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.185+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 0x7f4dac099240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4110190 tx=0x7f4db8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.185+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 msgr2=0x7f4dc410ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:46.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.185+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4db400fa90 tx=0x7f4db4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 shutdown_connections 2022-03-21T14:07:46.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dc4103da0 0x7f4dc4116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4dac096d90 0x7f4dac099240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc406f900 0x7f4dc410ec70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 --2- 172.21.15.2:0/823419348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4dc4070ec0 0x7f4dc410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:46.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 >> 172.21.15.2:0/823419348 conn(0x7f4dc40f75c0 msgr2=0x7f4dc4105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:46.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 shutdown_connections 2022-03-21T14:07:46.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:46.186+0000 7f4dc90cc700 1 -- 172.21.15.2:0/823419348 wait complete. 2022-03-21T14:07:46.192 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-03-21T14:07:46.656 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2022-03-21T14:07:44.404086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":220,"num_read_kb":160,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78876,"kb_used_data":12348,"kb_used_omap":0,"kb_used_meta":66432,"kb_avail":1124862948,"statfs":{"total":1151940427776,"available":1151859658752,"internally_reserved":0,"allocated":12644352,"data_stored":6850557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68026368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203331+0000","last_change":"2022-03-21T14:06:54.869479+0000","last_active":"2022-03-21T14:07:21.203331+0000","last_peered":"2022-03-21T14:07:21.203331+0000","last_clean":"2022-03-21T14:07:21.203331+0000","last_became_active":"2022-03-21T14:06:54.869079+0000","last_became_peered":"2022-03-21T14:06:54.869079+0000","last_unstale":"2022-03-21T14:07:21.203331+0000","last_undegraded":"2022-03-21T14:07:21.203331+0000","last_fullsized":"2022-03-21T14:07:21.203331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:09:19.289483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.876865+0000","last_change":"2022-03-21T14:06:54.884544+0000","last_active":"2022-03-21T14:07:20.876865+0000","last_peered":"2022-03-21T14:07:20.876865+0000","last_clean":"2022-03-21T14:07:20.876865+0000","last_became_active":"2022-03-21T14:06:54.884210+0000","last_became_peered":"2022-03-21T14:06:54.884210+0000","last_unstale":"2022-03-21T14:07:20.876865+0000","last_undegraded":"2022-03-21T14:07:20.876865+0000","last_fullsized":"2022-03-21T14:07:20.876865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:48:52.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870649+0000","last_change":"2022-03-21T14:06:54.880620+0000","last_active":"2022-03-21T14:07:20.870649+0000","last_peered":"2022-03-21T14:07:20.870649+0000","last_clean":"2022-03-21T14:07:20.870649+0000","last_became_active":"2022-03-21T14:06:54.880297+0000","last_became_peered":"2022-03-21T14:06:54.880297+0000","last_unstale":"2022-03-21T14:07:20.870649+0000","last_undegraded":"2022-03-21T14:07:20.870649+0000","last_fullsized":"2022-03-21T14:07:20.870649+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:43:49.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865650+0000","last_change":"2022-03-21T14:06:54.861753+0000","last_active":"2022-03-21T14:07:20.865650+0000","last_peered":"2022-03-21T14:07:20.865650+0000","last_clean":"2022-03-21T14:07:20.865650+0000","last_became_active":"2022-03-21T14:06:54.860649+0000","last_became_peered":"2022-03-21T14:06:54.860649+0000","last_unstale":"2022-03-21T14:07:20.865650+0000","last_undegraded":"2022-03-21T14:07:20.865650+0000","last_fullsized":"2022-03-21T14:07:20.865650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:22:24.251084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866512+0000","last_change":"2022-03-21T14:06:54.875705+0000","last_active":"2022-03-21T14:07:20.866512+0000","last_peered":"2022-03-21T14:07:20.866512+0000","last_clean":"2022-03-21T14:07:20.866512+0000","last_became_active":"2022-03-21T14:06:54.875171+0000","last_became_peered":"2022-03-21T14:06:54.875171+0000","last_unstale":"2022-03-21T14:07:20.866512+0000","last_undegraded":"2022-03-21T14:07:20.866512+0000","last_fullsized":"2022-03-21T14:07:20.866512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:52:39.257984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878853+0000","last_change":"2022-03-21T14:07:21.878853+0000","last_active":"2022-03-21T14:07:21.878853+0000","last_peered":"2022-03-21T14:07:21.878853+0000","last_clean":"2022-03-21T14:07:21.878853+0000","last_became_active":"2022-03-21T14:07:21.878436+0000","last_became_peered":"2022-03-21T14:07:21.878436+0000","last_unstale":"2022-03-21T14:07:21.878853+0000","last_undegraded":"2022-03-21T14:07:21.878853+0000","last_fullsized":"2022-03-21T14:07:21.878853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:34:56.330484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869200+0000","last_change":"2022-03-21T14:06:54.856225+0000","last_active":"2022-03-21T14:07:20.869200+0000","last_peered":"2022-03-21T14:07:20.869200+0000","last_clean":"2022-03-21T14:07:20.869200+0000","last_became_active":"2022-03-21T14:06:54.855801+0000","last_became_peered":"2022-03-21T14:06:54.855801+0000","last_unstale":"2022-03-21T14:07:20.869200+0000","last_undegraded":"2022-03-21T14:07:20.869200+0000","last_fullsized":"2022-03-21T14:07:20.869200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:26:47.895423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878320+0000","last_change":"2022-03-21T14:07:21.878320+0000","last_active":"2022-03-21T14:07:21.878320+0000","last_peered":"2022-03-21T14:07:21.878320+0000","last_clean":"2022-03-21T14:07:21.878320+0000","last_became_active":"2022-03-21T14:07:21.877897+0000","last_became_peered":"2022-03-21T14:07:21.877897+0000","last_unstale":"2022-03-21T14:07:21.878320+0000","last_undegraded":"2022-03-21T14:07:21.878320+0000","last_fullsized":"2022-03-21T14:07:21.878320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:46.608506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873414+0000","last_change":"2022-03-21T14:06:54.860355+0000","last_active":"2022-03-21T14:07:20.873414+0000","last_peered":"2022-03-21T14:07:20.873414+0000","last_clean":"2022-03-21T14:07:20.873414+0000","last_became_active":"2022-03-21T14:06:54.858723+0000","last_became_peered":"2022-03-21T14:06:54.858723+0000","last_unstale":"2022-03-21T14:07:20.873414+0000","last_undegraded":"2022-03-21T14:07:20.873414+0000","last_fullsized":"2022-03-21T14:07:20.873414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:34.198090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868736+0000","last_change":"2022-03-21T14:06:54.855421+0000","last_active":"2022-03-21T14:07:20.868736+0000","last_peered":"2022-03-21T14:07:20.868736+0000","last_clean":"2022-03-21T14:07:20.868736+0000","last_became_active":"2022-03-21T14:06:54.855027+0000","last_became_peered":"2022-03-21T14:06:54.855027+0000","last_unstale":"2022-03-21T14:07:20.868736+0000","last_undegraded":"2022-03-21T14:07:20.868736+0000","last_fullsized":"2022-03-21T14:07:20.868736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:36:14.968816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870161+0000","last_change":"2022-03-21T14:06:54.869318+0000","last_active":"2022-03-21T14:07:20.870161+0000","last_peered":"2022-03-21T14:07:20.870161+0000","last_clean":"2022-03-21T14:07:20.870161+0000","last_became_active":"2022-03-21T14:06:54.868664+0000","last_became_peered":"2022-03-21T14:06:54.868664+0000","last_unstale":"2022-03-21T14:07:20.870161+0000","last_undegraded":"2022-03-21T14:07:20.870161+0000","last_fullsized":"2022-03-21T14:07:20.870161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:29.775263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865526+0000","last_change":"2022-03-21T14:06:54.862293+0000","last_active":"2022-03-21T14:07:20.865526+0000","last_peered":"2022-03-21T14:07:20.865526+0000","last_clean":"2022-03-21T14:07:20.865526+0000","last_became_active":"2022-03-21T14:06:54.861926+0000","last_became_peered":"2022-03-21T14:06:54.861926+0000","last_unstale":"2022-03-21T14:07:20.865526+0000","last_undegraded":"2022-03-21T14:07:20.865526+0000","last_fullsized":"2022-03-21T14:07:20.865526+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:16:27.088619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874217+0000","last_change":"2022-03-21T14:06:54.859631+0000","last_active":"2022-03-21T14:07:20.874217+0000","last_peered":"2022-03-21T14:07:20.874217+0000","last_clean":"2022-03-21T14:07:20.874217+0000","last_became_active":"2022-03-21T14:06:54.858657+0000","last_became_peered":"2022-03-21T14:06:54.858657+0000","last_unstale":"2022-03-21T14:07:20.874217+0000","last_undegraded":"2022-03-21T14:07:20.874217+0000","last_fullsized":"2022-03-21T14:07:20.874217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:03.821600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869888+0000","last_change":"2022-03-21T14:06:54.866492+0000","last_active":"2022-03-21T14:07:20.869888+0000","last_peered":"2022-03-21T14:07:20.869888+0000","last_clean":"2022-03-21T14:07:20.869888+0000","last_became_active":"2022-03-21T14:06:54.866091+0000","last_became_peered":"2022-03-21T14:06:54.866091+0000","last_unstale":"2022-03-21T14:07:20.869888+0000","last_undegraded":"2022-03-21T14:07:20.869888+0000","last_fullsized":"2022-03-21T14:07:20.869888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:31:33.741891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869690+0000","last_change":"2022-03-21T14:06:54.879848+0000","last_active":"2022-03-21T14:07:20.869690+0000","last_peered":"2022-03-21T14:07:20.869690+0000","last_clean":"2022-03-21T14:07:20.869690+0000","last_became_active":"2022-03-21T14:06:54.879496+0000","last_became_peered":"2022-03-21T14:06:54.879496+0000","last_unstale":"2022-03-21T14:07:20.869690+0000","last_undegraded":"2022-03-21T14:07:20.869690+0000","last_fullsized":"2022-03-21T14:07:20.869690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:18:13.350002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872165+0000","last_change":"2022-03-21T14:06:54.877838+0000","last_active":"2022-03-21T14:07:20.872165+0000","last_peered":"2022-03-21T14:07:20.872165+0000","last_clean":"2022-03-21T14:07:20.872165+0000","last_became_active":"2022-03-21T14:06:54.877444+0000","last_became_peered":"2022-03-21T14:06:54.877444+0000","last_unstale":"2022-03-21T14:07:20.872165+0000","last_undegraded":"2022-03-21T14:07:20.872165+0000","last_fullsized":"2022-03-21T14:07:20.872165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:32.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203390+0000","last_change":"2022-03-21T14:06:54.885978+0000","last_active":"2022-03-21T14:07:21.203390+0000","last_peered":"2022-03-21T14:07:21.203390+0000","last_clean":"2022-03-21T14:07:21.203390+0000","last_became_active":"2022-03-21T14:06:54.885645+0000","last_became_peered":"2022-03-21T14:06:54.885645+0000","last_unstale":"2022-03-21T14:07:21.203390+0000","last_undegraded":"2022-03-21T14:07:21.203390+0000","last_fullsized":"2022-03-21T14:07:21.203390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:51.003441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871073+0000","last_change":"2022-03-21T14:06:54.860687+0000","last_active":"2022-03-21T14:07:20.871073+0000","last_peered":"2022-03-21T14:07:20.871073+0000","last_clean":"2022-03-21T14:07:20.871073+0000","last_became_active":"2022-03-21T14:06:54.860315+0000","last_became_peered":"2022-03-21T14:06:54.860315+0000","last_unstale":"2022-03-21T14:07:20.871073+0000","last_undegraded":"2022-03-21T14:07:20.871073+0000","last_fullsized":"2022-03-21T14:07:20.871073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:27:56.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202560+0000","last_change":"2022-03-21T14:06:54.867688+0000","last_active":"2022-03-21T14:07:21.202560+0000","last_peered":"2022-03-21T14:07:21.202560+0000","last_clean":"2022-03-21T14:07:21.202560+0000","last_became_active":"2022-03-21T14:06:54.866998+0000","last_became_peered":"2022-03-21T14:06:54.866998+0000","last_unstale":"2022-03-21T14:07:21.202560+0000","last_undegraded":"2022-03-21T14:07:21.202560+0000","last_fullsized":"2022-03-21T14:07:21.202560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:17:24.979917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871202+0000","last_change":"2022-03-21T14:06:54.326883+0000","last_active":"2022-03-21T14:07:20.871202+0000","last_peered":"2022-03-21T14:07:20.871202+0000","last_clean":"2022-03-21T14:07:20.871202+0000","last_became_active":"2022-03-21T14:06:54.326424+0000","last_became_peered":"2022-03-21T14:06:54.326424+0000","last_unstale":"2022-03-21T14:07:20.871202+0000","last_undegraded":"2022-03-21T14:07:20.871202+0000","last_fullsized":"2022-03-21T14:07:20.871202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:39.818504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877195+0000","last_change":"2022-03-21T14:06:54.875068+0000","last_active":"2022-03-21T14:07:20.877195+0000","last_peered":"2022-03-21T14:07:20.877195+0000","last_clean":"2022-03-21T14:07:20.877195+0000","last_became_active":"2022-03-21T14:06:54.874715+0000","last_became_peered":"2022-03-21T14:06:54.874715+0000","last_unstale":"2022-03-21T14:07:20.877195+0000","last_undegraded":"2022-03-21T14:07:20.877195+0000","last_fullsized":"2022-03-21T14:07:20.877195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:33.864263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877308+0000","last_change":"2022-03-21T14:06:54.335459+0000","last_active":"2022-03-21T14:07:20.877308+0000","last_peered":"2022-03-21T14:07:20.877308+0000","last_clean":"2022-03-21T14:07:20.877308+0000","last_became_active":"2022-03-21T14:06:54.335109+0000","last_became_peered":"2022-03-21T14:06:54.335109+0000","last_unstale":"2022-03-21T14:07:20.877308+0000","last_undegraded":"2022-03-21T14:07:20.877308+0000","last_fullsized":"2022-03-21T14:07:20.877308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:06:56.531385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870715+0000","last_change":"2022-03-21T14:06:54.884043+0000","last_active":"2022-03-21T14:07:20.870715+0000","last_peered":"2022-03-21T14:07:20.870715+0000","last_clean":"2022-03-21T14:07:20.870715+0000","last_became_active":"2022-03-21T14:06:54.883647+0000","last_became_peered":"2022-03-21T14:06:54.883647+0000","last_unstale":"2022-03-21T14:07:20.870715+0000","last_undegraded":"2022-03-21T14:07:20.870715+0000","last_fullsized":"2022-03-21T14:07:20.870715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:32:17.243588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866752+0000","last_change":"2022-03-21T14:06:54.314357+0000","last_active":"2022-03-21T14:07:20.866752+0000","last_peered":"2022-03-21T14:07:20.866752+0000","last_clean":"2022-03-21T14:07:20.866752+0000","last_became_active":"2022-03-21T14:06:54.313855+0000","last_became_peered":"2022-03-21T14:06:54.313855+0000","last_unstale":"2022-03-21T14:07:20.866752+0000","last_undegraded":"2022-03-21T14:07:20.866752+0000","last_fullsized":"2022-03-21T14:07:20.866752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:58:09.686843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872714+0000","last_change":"2022-03-21T14:06:54.861824+0000","last_active":"2022-03-21T14:07:20.872714+0000","last_peered":"2022-03-21T14:07:20.872714+0000","last_clean":"2022-03-21T14:07:20.872714+0000","last_became_active":"2022-03-21T14:06:54.859283+0000","last_became_peered":"2022-03-21T14:06:54.859283+0000","last_unstale":"2022-03-21T14:07:20.872714+0000","last_undegraded":"2022-03-21T14:07:20.872714+0000","last_fullsized":"2022-03-21T14:07:20.872714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:00:42.993237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866520+0000","last_change":"2022-03-21T14:06:54.312853+0000","last_active":"2022-03-21T14:07:20.866520+0000","last_peered":"2022-03-21T14:07:20.866520+0000","last_clean":"2022-03-21T14:07:20.866520+0000","last_became_active":"2022-03-21T14:06:54.312474+0000","last_became_peered":"2022-03-21T14:06:54.312474+0000","last_unstale":"2022-03-21T14:07:20.866520+0000","last_undegraded":"2022-03-21T14:07:20.866520+0000","last_fullsized":"2022-03-21T14:07:20.866520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:53:19.818645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869430+0000","last_change":"2022-03-21T14:06:54.859678+0000","last_active":"2022-03-21T14:07:20.869430+0000","last_peered":"2022-03-21T14:07:20.869430+0000","last_clean":"2022-03-21T14:07:20.869430+0000","last_became_active":"2022-03-21T14:06:54.858928+0000","last_became_peered":"2022-03-21T14:06:54.858928+0000","last_unstale":"2022-03-21T14:07:20.869430+0000","last_undegraded":"2022-03-21T14:07:20.869430+0000","last_fullsized":"2022-03-21T14:07:20.869430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:10.326574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868911+0000","last_change":"2022-03-21T14:06:54.314216+0000","last_active":"2022-03-21T14:07:20.868911+0000","last_peered":"2022-03-21T14:07:20.868911+0000","last_clean":"2022-03-21T14:07:20.868911+0000","last_became_active":"2022-03-21T14:06:54.313808+0000","last_became_peered":"2022-03-21T14:06:54.313808+0000","last_unstale":"2022-03-21T14:07:20.868911+0000","last_undegraded":"2022-03-21T14:07:20.868911+0000","last_fullsized":"2022-03-21T14:07:20.868911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:52:09.937102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878771+0000","last_change":"2022-03-21T14:07:21.878771+0000","last_active":"2022-03-21T14:07:21.878771+0000","last_peered":"2022-03-21T14:07:21.878771+0000","last_clean":"2022-03-21T14:07:21.878771+0000","last_became_active":"2022-03-21T14:07:21.878460+0000","last_became_peered":"2022-03-21T14:07:21.878460+0000","last_unstale":"2022-03-21T14:07:21.878771+0000","last_undegraded":"2022-03-21T14:07:21.878771+0000","last_fullsized":"2022-03-21T14:07:21.878771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:42:20.837629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868095+0000","last_change":"2022-03-21T14:06:54.322793+0000","last_active":"2022-03-21T14:07:20.868095+0000","last_peered":"2022-03-21T14:07:20.868095+0000","last_clean":"2022-03-21T14:07:20.868095+0000","last_became_active":"2022-03-21T14:06:54.322360+0000","last_became_peered":"2022-03-21T14:06:54.322360+0000","last_unstale":"2022-03-21T14:07:20.868095+0000","last_undegraded":"2022-03-21T14:07:20.868095+0000","last_fullsized":"2022-03-21T14:07:20.868095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:15:12.220667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869003+0000","last_change":"2022-03-21T14:06:54.874475+0000","last_active":"2022-03-21T14:07:20.869003+0000","last_peered":"2022-03-21T14:07:20.869003+0000","last_clean":"2022-03-21T14:07:20.869003+0000","last_became_active":"2022-03-21T14:06:54.874113+0000","last_became_peered":"2022-03-21T14:06:54.874113+0000","last_unstale":"2022-03-21T14:07:20.869003+0000","last_undegraded":"2022-03-21T14:07:20.869003+0000","last_fullsized":"2022-03-21T14:07:20.869003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:25.860831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873527+0000","last_change":"2022-03-21T14:06:54.331708+0000","last_active":"2022-03-21T14:07:20.873527+0000","last_peered":"2022-03-21T14:07:20.873527+0000","last_clean":"2022-03-21T14:07:20.873527+0000","last_became_active":"2022-03-21T14:06:54.331347+0000","last_became_peered":"2022-03-21T14:06:54.331347+0000","last_unstale":"2022-03-21T14:07:20.873527+0000","last_undegraded":"2022-03-21T14:07:20.873527+0000","last_fullsized":"2022-03-21T14:07:20.873527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:59:42.282136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873905+0000","last_change":"2022-03-21T14:06:54.864160+0000","last_active":"2022-03-21T14:07:20.873905+0000","last_peered":"2022-03-21T14:07:20.873905+0000","last_clean":"2022-03-21T14:07:20.873905+0000","last_became_active":"2022-03-21T14:06:54.863699+0000","last_became_peered":"2022-03-21T14:06:54.863699+0000","last_unstale":"2022-03-21T14:07:20.873905+0000","last_undegraded":"2022-03-21T14:07:20.873905+0000","last_fullsized":"2022-03-21T14:07:20.873905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:40:53.670759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868147+0000","last_change":"2022-03-21T14:06:54.314228+0000","last_active":"2022-03-21T14:07:20.868147+0000","last_peered":"2022-03-21T14:07:20.868147+0000","last_clean":"2022-03-21T14:07:20.868147+0000","last_became_active":"2022-03-21T14:06:54.313667+0000","last_became_peered":"2022-03-21T14:06:54.313667+0000","last_unstale":"2022-03-21T14:07:20.868147+0000","last_undegraded":"2022-03-21T14:07:20.868147+0000","last_fullsized":"2022-03-21T14:07:20.868147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:20:10.791422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869546+0000","last_change":"2022-03-21T14:06:54.866898+0000","last_active":"2022-03-21T14:07:20.869546+0000","last_peered":"2022-03-21T14:07:20.869546+0000","last_clean":"2022-03-21T14:07:20.869546+0000","last_became_active":"2022-03-21T14:06:54.865450+0000","last_became_peered":"2022-03-21T14:06:54.865450+0000","last_unstale":"2022-03-21T14:07:20.869546+0000","last_undegraded":"2022-03-21T14:07:20.869546+0000","last_fullsized":"2022-03-21T14:07:20.869546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:39:42.699187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868385+0000","last_change":"2022-03-21T14:06:54.328313+0000","last_active":"2022-03-21T14:07:20.868385+0000","last_peered":"2022-03-21T14:07:20.868385+0000","last_clean":"2022-03-21T14:07:20.868385+0000","last_became_active":"2022-03-21T14:06:54.327989+0000","last_became_peered":"2022-03-21T14:06:54.327989+0000","last_unstale":"2022-03-21T14:07:20.868385+0000","last_undegraded":"2022-03-21T14:07:20.868385+0000","last_fullsized":"2022-03-21T14:07:20.868385+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:59:23.877877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878866+0000","last_change":"2022-03-21T14:07:21.878866+0000","last_active":"2022-03-21T14:07:21.878866+0000","last_peered":"2022-03-21T14:07:21.878866+0000","last_clean":"2022-03-21T14:07:21.878866+0000","last_became_active":"2022-03-21T14:07:21.878467+0000","last_became_peered":"2022-03-21T14:07:21.878467+0000","last_unstale":"2022-03-21T14:07:21.878866+0000","last_undegraded":"2022-03-21T14:07:21.878866+0000","last_fullsized":"2022-03-21T14:07:21.878866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:33.620541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,9],"acting":[8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869775+0000","last_change":"2022-03-21T14:06:54.317279+0000","last_active":"2022-03-21T14:07:20.869775+0000","last_peered":"2022-03-21T14:07:20.869775+0000","last_clean":"2022-03-21T14:07:20.869775+0000","last_became_active":"2022-03-21T14:06:54.316863+0000","last_became_peered":"2022-03-21T14:06:54.316863+0000","last_unstale":"2022-03-21T14:07:20.869775+0000","last_undegraded":"2022-03-21T14:07:20.869775+0000","last_fullsized":"2022-03-21T14:07:20.869775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:17.370559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:22.097382+0000","last_change":"2022-03-21T14:07:22.097382+0000","last_active":"2022-03-21T14:07:22.097382+0000","last_peered":"2022-03-21T14:07:22.097382+0000","last_clean":"2022-03-21T14:07:22.097382+0000","last_became_active":"2022-03-21T14:07:22.096919+0000","last_became_peered":"2022-03-21T14:07:22.096919+0000","last_unstale":"2022-03-21T14:07:22.097382+0000","last_undegraded":"2022-03-21T14:07:22.097382+0000","last_fullsized":"2022-03-21T14:07:22.097382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:05:40.625717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'3","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869545+0000","last_change":"2022-03-21T14:06:54.333183+0000","last_active":"2022-03-21T14:07:20.869545+0000","last_peered":"2022-03-21T14:07:20.869545+0000","last_clean":"2022-03-21T14:07:20.869545+0000","last_became_active":"2022-03-21T14:06:54.332795+0000","last_became_peered":"2022-03-21T14:06:54.332795+0000","last_unstale":"2022-03-21T14:07:20.869545+0000","last_undegraded":"2022-03-21T14:07:20.869545+0000","last_fullsized":"2022-03-21T14:07:20.869545+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:29.651861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872945+0000","last_change":"2022-03-21T14:06:54.862390+0000","last_active":"2022-03-21T14:07:20.872945+0000","last_peered":"2022-03-21T14:07:20.872945+0000","last_clean":"2022-03-21T14:07:20.872945+0000","last_became_active":"2022-03-21T14:06:54.862008+0000","last_became_peered":"2022-03-21T14:06:54.862008+0000","last_unstale":"2022-03-21T14:07:20.872945+0000","last_undegraded":"2022-03-21T14:07:20.872945+0000","last_fullsized":"2022-03-21T14:07:20.872945+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:53:50.255768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'6","reported_seq":107,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:40.421347+0000","last_change":"2022-03-21T14:06:54.315167+0000","last_active":"2022-03-21T14:07:40.421347+0000","last_peered":"2022-03-21T14:07:40.421347+0000","last_clean":"2022-03-21T14:07:40.421347+0000","last_became_active":"2022-03-21T14:06:54.314432+0000","last_became_peered":"2022-03-21T14:06:54.314432+0000","last_unstale":"2022-03-21T14:07:40.421347+0000","last_undegraded":"2022-03-21T14:07:40.421347+0000","last_fullsized":"2022-03-21T14:07:40.421347+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:48:27.504920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":88,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866571+0000","last_change":"2022-03-21T14:06:54.859057+0000","last_active":"2022-03-21T14:07:20.866571+0000","last_peered":"2022-03-21T14:07:20.866571+0000","last_clean":"2022-03-21T14:07:20.866571+0000","last_became_active":"2022-03-21T14:06:54.858674+0000","last_became_peered":"2022-03-21T14:06:54.858674+0000","last_unstale":"2022-03-21T14:07:20.866571+0000","last_undegraded":"2022-03-21T14:07:20.866571+0000","last_fullsized":"2022-03-21T14:07:20.866571+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:38:29.386832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202646+0000","last_change":"2022-03-21T14:06:54.323631+0000","last_active":"2022-03-21T14:07:21.202646+0000","last_peered":"2022-03-21T14:07:21.202646+0000","last_clean":"2022-03-21T14:07:21.202646+0000","last_became_active":"2022-03-21T14:06:54.323179+0000","last_became_peered":"2022-03-21T14:06:54.323179+0000","last_unstale":"2022-03-21T14:07:21.202646+0000","last_undegraded":"2022-03-21T14:07:21.202646+0000","last_fullsized":"2022-03-21T14:07:21.202646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:35:04.480559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873864+0000","last_change":"2022-03-21T14:06:54.878235+0000","last_active":"2022-03-21T14:07:20.873864+0000","last_peered":"2022-03-21T14:07:20.873864+0000","last_clean":"2022-03-21T14:07:20.873864+0000","last_became_active":"2022-03-21T14:06:54.877650+0000","last_became_peered":"2022-03-21T14:06:54.877650+0000","last_unstale":"2022-03-21T14:07:20.873864+0000","last_undegraded":"2022-03-21T14:07:20.873864+0000","last_fullsized":"2022-03-21T14:07:20.873864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:06:58.716437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868936+0000","last_change":"2022-03-21T14:06:54.316053+0000","last_active":"2022-03-21T14:07:20.868936+0000","last_peered":"2022-03-21T14:07:20.868936+0000","last_clean":"2022-03-21T14:07:20.868936+0000","last_became_active":"2022-03-21T14:06:54.315685+0000","last_became_peered":"2022-03-21T14:06:54.315685+0000","last_unstale":"2022-03-21T14:07:20.868936+0000","last_undegraded":"2022-03-21T14:07:20.868936+0000","last_fullsized":"2022-03-21T14:07:20.868936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:16:47.746232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869577+0000","last_change":"2022-03-21T14:06:54.878905+0000","last_active":"2022-03-21T14:07:20.869577+0000","last_peered":"2022-03-21T14:07:20.869577+0000","last_clean":"2022-03-21T14:07:20.869577+0000","last_became_active":"2022-03-21T14:06:54.878341+0000","last_became_peered":"2022-03-21T14:06:54.878341+0000","last_unstale":"2022-03-21T14:07:20.869577+0000","last_undegraded":"2022-03-21T14:07:20.869577+0000","last_fullsized":"2022-03-21T14:07:20.869577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:05:50.071721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865352+0000","last_change":"2022-03-21T14:06:54.313255+0000","last_active":"2022-03-21T14:07:20.865352+0000","last_peered":"2022-03-21T14:07:20.865352+0000","last_clean":"2022-03-21T14:07:20.865352+0000","last_became_active":"2022-03-21T14:06:54.312849+0000","last_became_peered":"2022-03-21T14:06:54.312849+0000","last_unstale":"2022-03-21T14:07:20.865352+0000","last_undegraded":"2022-03-21T14:07:20.865352+0000","last_fullsized":"2022-03-21T14:07:20.865352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:19:59.556267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869053+0000","last_change":"2022-03-21T14:06:54.857792+0000","last_active":"2022-03-21T14:07:20.869053+0000","last_peered":"2022-03-21T14:07:20.869053+0000","last_clean":"2022-03-21T14:07:20.869053+0000","last_became_active":"2022-03-21T14:06:54.857425+0000","last_became_peered":"2022-03-21T14:06:54.857425+0000","last_unstale":"2022-03-21T14:07:20.869053+0000","last_undegraded":"2022-03-21T14:07:20.869053+0000","last_fullsized":"2022-03-21T14:07:20.869053+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:51:33.943688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'8","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866165+0000","last_change":"2022-03-21T14:06:54.335082+0000","last_active":"2022-03-21T14:07:20.866165+0000","last_peered":"2022-03-21T14:07:20.866165+0000","last_clean":"2022-03-21T14:07:20.866165+0000","last_became_active":"2022-03-21T14:06:54.334703+0000","last_became_peered":"2022-03-21T14:06:54.334703+0000","last_unstale":"2022-03-21T14:07:20.866165+0000","last_undegraded":"2022-03-21T14:07:20.866165+0000","last_fullsized":"2022-03-21T14:07:20.866165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:12.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.877589+0000","last_change":"2022-03-21T14:07:21.877589+0000","last_active":"2022-03-21T14:07:21.877589+0000","last_peered":"2022-03-21T14:07:21.877589+0000","last_clean":"2022-03-21T14:07:21.877589+0000","last_became_active":"2022-03-21T14:07:21.876949+0000","last_became_peered":"2022-03-21T14:07:21.876949+0000","last_unstale":"2022-03-21T14:07:21.877589+0000","last_undegraded":"2022-03-21T14:07:21.877589+0000","last_fullsized":"2022-03-21T14:07:21.877589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:30:26.166689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866985+0000","last_change":"2022-03-21T14:06:54.315652+0000","last_active":"2022-03-21T14:07:20.866985+0000","last_peered":"2022-03-21T14:07:20.866985+0000","last_clean":"2022-03-21T14:07:20.866985+0000","last_became_active":"2022-03-21T14:06:54.315317+0000","last_became_peered":"2022-03-21T14:06:54.315317+0000","last_unstale":"2022-03-21T14:07:20.866985+0000","last_undegraded":"2022-03-21T14:07:20.866985+0000","last_fullsized":"2022-03-21T14:07:20.866985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:55:42.394760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203002+0000","last_change":"2022-03-21T14:06:54.856509+0000","last_active":"2022-03-21T14:07:21.203002+0000","last_peered":"2022-03-21T14:07:21.203002+0000","last_clean":"2022-03-21T14:07:21.203002+0000","last_became_active":"2022-03-21T14:06:54.855995+0000","last_became_peered":"2022-03-21T14:06:54.855995+0000","last_unstale":"2022-03-21T14:07:21.203002+0000","last_undegraded":"2022-03-21T14:07:21.203002+0000","last_fullsized":"2022-03-21T14:07:21.203002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:02:52.429679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871427+0000","last_change":"2022-03-21T14:06:54.336921+0000","last_active":"2022-03-21T14:07:20.871427+0000","last_peered":"2022-03-21T14:07:20.871427+0000","last_clean":"2022-03-21T14:07:20.871427+0000","last_became_active":"2022-03-21T14:06:54.336560+0000","last_became_peered":"2022-03-21T14:06:54.336560+0000","last_unstale":"2022-03-21T14:07:20.871427+0000","last_undegraded":"2022-03-21T14:07:20.871427+0000","last_fullsized":"2022-03-21T14:07:20.871427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:26:06.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868272+0000","last_change":"2022-03-21T14:06:54.865783+0000","last_active":"2022-03-21T14:07:20.868272+0000","last_peered":"2022-03-21T14:07:20.868272+0000","last_clean":"2022-03-21T14:07:20.868272+0000","last_became_active":"2022-03-21T14:06:54.865399+0000","last_became_peered":"2022-03-21T14:06:54.865399+0000","last_unstale":"2022-03-21T14:07:20.868272+0000","last_undegraded":"2022-03-21T14:07:20.868272+0000","last_fullsized":"2022-03-21T14:07:20.868272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:15:19.155641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869393+0000","last_change":"2022-03-21T14:06:54.321852+0000","last_active":"2022-03-21T14:07:20.869393+0000","last_peered":"2022-03-21T14:07:20.869393+0000","last_clean":"2022-03-21T14:07:20.869393+0000","last_became_active":"2022-03-21T14:06:54.321388+0000","last_became_peered":"2022-03-21T14:06:54.321388+0000","last_unstale":"2022-03-21T14:07:20.869393+0000","last_undegraded":"2022-03-21T14:07:20.869393+0000","last_fullsized":"2022-03-21T14:07:20.869393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:58:18.274331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868341+0000","last_change":"2022-03-21T14:06:54.871690+0000","last_active":"2022-03-21T14:07:20.868341+0000","last_peered":"2022-03-21T14:07:20.868341+0000","last_clean":"2022-03-21T14:07:20.868341+0000","last_became_active":"2022-03-21T14:06:54.871272+0000","last_became_peered":"2022-03-21T14:06:54.871272+0000","last_unstale":"2022-03-21T14:07:20.868341+0000","last_undegraded":"2022-03-21T14:07:20.868341+0000","last_fullsized":"2022-03-21T14:07:20.868341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:17.999041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868373+0000","last_change":"2022-03-21T14:06:54.860140+0000","last_active":"2022-03-21T14:07:20.868373+0000","last_peered":"2022-03-21T14:07:20.868373+0000","last_clean":"2022-03-21T14:07:20.868373+0000","last_became_active":"2022-03-21T14:06:54.859700+0000","last_became_peered":"2022-03-21T14:06:54.859700+0000","last_unstale":"2022-03-21T14:07:20.868373+0000","last_undegraded":"2022-03-21T14:07:20.868373+0000","last_fullsized":"2022-03-21T14:07:20.868373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:57:46.264617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873445+0000","last_change":"2022-03-21T14:06:54.334370+0000","last_active":"2022-03-21T14:07:20.873445+0000","last_peered":"2022-03-21T14:07:20.873445+0000","last_clean":"2022-03-21T14:07:20.873445+0000","last_became_active":"2022-03-21T14:06:54.333957+0000","last_became_peered":"2022-03-21T14:06:54.333957+0000","last_unstale":"2022-03-21T14:07:20.873445+0000","last_undegraded":"2022-03-21T14:07:20.873445+0000","last_fullsized":"2022-03-21T14:07:20.873445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:29.048808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.879017+0000","last_change":"2022-03-21T14:07:21.879017+0000","last_active":"2022-03-21T14:07:21.879017+0000","last_peered":"2022-03-21T14:07:21.879017+0000","last_clean":"2022-03-21T14:07:21.879017+0000","last_became_active":"2022-03-21T14:07:21.878560+0000","last_became_peered":"2022-03-21T14:07:21.878560+0000","last_unstale":"2022-03-21T14:07:21.879017+0000","last_undegraded":"2022-03-21T14:07:21.879017+0000","last_fullsized":"2022-03-21T14:07:21.879017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:29:57.592235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865587+0000","last_change":"2022-03-21T14:06:54.315511+0000","last_active":"2022-03-21T14:07:20.865587+0000","last_peered":"2022-03-21T14:07:20.865587+0000","last_clean":"2022-03-21T14:07:20.865587+0000","last_became_active":"2022-03-21T14:06:54.315144+0000","last_became_peered":"2022-03-21T14:06:54.315144+0000","last_unstale":"2022-03-21T14:07:20.865587+0000","last_undegraded":"2022-03-21T14:07:20.865587+0000","last_fullsized":"2022-03-21T14:07:20.865587+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:30:11.627760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869124+0000","last_change":"2022-03-21T14:05:44.276967+0000","last_active":"2022-03-21T14:07:20.869124+0000","last_peered":"2022-03-21T14:07:20.869124+0000","last_clean":"2022-03-21T14:07:20.869124+0000","last_became_active":"2022-03-21T14:05:44.276297+0000","last_became_peered":"2022-03-21T14:05:44.276297+0000","last_unstale":"2022-03-21T14:07:20.869124+0000","last_undegraded":"2022-03-21T14:07:20.869124+0000","last_fullsized":"2022-03-21T14:07:20.869124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_clean_scrub_stamp":"2022-03-21T14:02:31.386985+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:07:49.689146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873358+0000","last_change":"2022-03-21T14:06:54.860044+0000","last_active":"2022-03-21T14:07:20.873358+0000","last_peered":"2022-03-21T14:07:20.873358+0000","last_clean":"2022-03-21T14:07:20.873358+0000","last_became_active":"2022-03-21T14:06:54.859511+0000","last_became_peered":"2022-03-21T14:06:54.859511+0000","last_unstale":"2022-03-21T14:07:20.873358+0000","last_undegraded":"2022-03-21T14:07:20.873358+0000","last_fullsized":"2022-03-21T14:07:20.873358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:50:01.100612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869215+0000","last_change":"2022-03-21T14:06:54.313788+0000","last_active":"2022-03-21T14:07:20.869215+0000","last_peered":"2022-03-21T14:07:20.869215+0000","last_clean":"2022-03-21T14:07:20.869215+0000","last_became_active":"2022-03-21T14:06:54.313328+0000","last_became_peered":"2022-03-21T14:06:54.313328+0000","last_unstale":"2022-03-21T14:07:20.869215+0000","last_undegraded":"2022-03-21T14:07:20.869215+0000","last_fullsized":"2022-03-21T14:07:20.869215+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:24:17.930196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866695+0000","last_change":"2022-03-21T14:06:54.334716+0000","last_active":"2022-03-21T14:07:20.866695+0000","last_peered":"2022-03-21T14:07:20.866695+0000","last_clean":"2022-03-21T14:07:20.866695+0000","last_became_active":"2022-03-21T14:06:54.334384+0000","last_became_peered":"2022-03-21T14:06:54.334384+0000","last_unstale":"2022-03-21T14:07:20.866695+0000","last_undegraded":"2022-03-21T14:07:20.866695+0000","last_fullsized":"2022-03-21T14:07:20.866695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:29.598258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874388+0000","last_change":"2022-03-21T14:06:54.872297+0000","last_active":"2022-03-21T14:07:20.874388+0000","last_peered":"2022-03-21T14:07:20.874388+0000","last_clean":"2022-03-21T14:07:20.874388+0000","last_became_active":"2022-03-21T14:06:54.871557+0000","last_became_peered":"2022-03-21T14:06:54.871557+0000","last_unstale":"2022-03-21T14:07:20.874388+0000","last_undegraded":"2022-03-21T14:07:20.874388+0000","last_fullsized":"2022-03-21T14:07:20.874388+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:21:13.076650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870984+0000","last_change":"2022-03-21T14:06:54.327446+0000","last_active":"2022-03-21T14:07:20.870984+0000","last_peered":"2022-03-21T14:07:20.870984+0000","last_clean":"2022-03-21T14:07:20.870984+0000","last_became_active":"2022-03-21T14:06:54.327102+0000","last_became_peered":"2022-03-21T14:06:54.327102+0000","last_unstale":"2022-03-21T14:07:20.870984+0000","last_undegraded":"2022-03-21T14:07:20.870984+0000","last_fullsized":"2022-03-21T14:07:20.870984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:28:18.309364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877166+0000","last_change":"2022-03-21T14:06:54.869348+0000","last_active":"2022-03-21T14:07:20.877166+0000","last_peered":"2022-03-21T14:07:20.877166+0000","last_clean":"2022-03-21T14:07:20.877166+0000","last_became_active":"2022-03-21T14:06:54.868917+0000","last_became_peered":"2022-03-21T14:06:54.868917+0000","last_unstale":"2022-03-21T14:07:20.877166+0000","last_undegraded":"2022-03-21T14:07:20.877166+0000","last_fullsized":"2022-03-21T14:07:20.877166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:38:56.199987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866251+0000","last_change":"2022-03-21T14:06:54.321746+0000","last_active":"2022-03-21T14:07:20.866251+0000","last_peered":"2022-03-21T14:07:20.866251+0000","last_clean":"2022-03-21T14:07:20.866251+0000","last_became_active":"2022-03-21T14:06:54.321294+0000","last_became_peered":"2022-03-21T14:06:54.321294+0000","last_unstale":"2022-03-21T14:07:20.866251+0000","last_undegraded":"2022-03-21T14:07:20.866251+0000","last_fullsized":"2022-03-21T14:07:20.866251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:06:26.961696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:22.097402+0000","last_change":"2022-03-21T14:07:22.097402+0000","last_active":"2022-03-21T14:07:22.097402+0000","last_peered":"2022-03-21T14:07:22.097402+0000","last_clean":"2022-03-21T14:07:22.097402+0000","last_became_active":"2022-03-21T14:07:22.096925+0000","last_became_peered":"2022-03-21T14:07:22.096925+0000","last_unstale":"2022-03-21T14:07:22.097402+0000","last_undegraded":"2022-03-21T14:07:22.097402+0000","last_fullsized":"2022-03-21T14:07:22.097402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:49:34.539213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872194+0000","last_change":"2022-03-21T14:06:54.326806+0000","last_active":"2022-03-21T14:07:20.872194+0000","last_peered":"2022-03-21T14:07:20.872194+0000","last_clean":"2022-03-21T14:07:20.872194+0000","last_became_active":"2022-03-21T14:06:54.326405+0000","last_became_peered":"2022-03-21T14:06:54.326405+0000","last_unstale":"2022-03-21T14:07:20.872194+0000","last_undegraded":"2022-03-21T14:07:20.872194+0000","last_fullsized":"2022-03-21T14:07:20.872194+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:29.562293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866167+0000","last_change":"2022-03-21T14:06:54.860972+0000","last_active":"2022-03-21T14:07:20.866167+0000","last_peered":"2022-03-21T14:07:20.866167+0000","last_clean":"2022-03-21T14:07:20.866167+0000","last_became_active":"2022-03-21T14:06:54.860557+0000","last_became_peered":"2022-03-21T14:06:54.860557+0000","last_unstale":"2022-03-21T14:07:20.866167+0000","last_undegraded":"2022-03-21T14:07:20.866167+0000","last_fullsized":"2022-03-21T14:07:20.866167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:28:45.487047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874532+0000","last_change":"2022-03-21T14:06:54.320161+0000","last_active":"2022-03-21T14:07:20.874532+0000","last_peered":"2022-03-21T14:07:20.874532+0000","last_clean":"2022-03-21T14:07:20.874532+0000","last_became_active":"2022-03-21T14:06:54.318104+0000","last_became_peered":"2022-03-21T14:06:54.318104+0000","last_unstale":"2022-03-21T14:07:20.874532+0000","last_undegraded":"2022-03-21T14:07:20.874532+0000","last_fullsized":"2022-03-21T14:07:20.874532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:12.154964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203081+0000","last_change":"2022-03-21T14:06:54.870272+0000","last_active":"2022-03-21T14:07:21.203081+0000","last_peered":"2022-03-21T14:07:21.203081+0000","last_clean":"2022-03-21T14:07:21.203081+0000","last_became_active":"2022-03-21T14:06:54.869649+0000","last_became_peered":"2022-03-21T14:06:54.869649+0000","last_unstale":"2022-03-21T14:07:21.203081+0000","last_undegraded":"2022-03-21T14:07:21.203081+0000","last_fullsized":"2022-03-21T14:07:21.203081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:09:57.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202628+0000","last_change":"2022-03-21T14:06:54.323589+0000","last_active":"2022-03-21T14:07:21.202628+0000","last_peered":"2022-03-21T14:07:21.202628+0000","last_clean":"2022-03-21T14:07:21.202628+0000","last_became_active":"2022-03-21T14:06:54.323187+0000","last_became_peered":"2022-03-21T14:06:54.323187+0000","last_unstale":"2022-03-21T14:07:21.202628+0000","last_undegraded":"2022-03-21T14:07:21.202628+0000","last_fullsized":"2022-03-21T14:07:21.202628+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:05:33.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.876105+0000","last_change":"2022-03-21T14:06:54.880453+0000","last_active":"2022-03-21T14:07:20.876105+0000","last_peered":"2022-03-21T14:07:20.876105+0000","last_clean":"2022-03-21T14:07:20.876105+0000","last_became_active":"2022-03-21T14:06:54.880078+0000","last_became_peered":"2022-03-21T14:06:54.880078+0000","last_unstale":"2022-03-21T14:07:20.876105+0000","last_undegraded":"2022-03-21T14:07:20.876105+0000","last_fullsized":"2022-03-21T14:07:20.876105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:37:36.812969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866383+0000","last_change":"2022-03-21T14:06:54.314137+0000","last_active":"2022-03-21T14:07:20.866383+0000","last_peered":"2022-03-21T14:07:20.866383+0000","last_clean":"2022-03-21T14:07:20.866383+0000","last_became_active":"2022-03-21T14:06:54.313783+0000","last_became_peered":"2022-03-21T14:06:54.313783+0000","last_unstale":"2022-03-21T14:07:20.866383+0000","last_undegraded":"2022-03-21T14:07:20.866383+0000","last_fullsized":"2022-03-21T14:07:20.866383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:58.358466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869580+0000","last_change":"2022-03-21T14:06:54.867675+0000","last_active":"2022-03-21T14:07:20.869580+0000","last_peered":"2022-03-21T14:07:20.869580+0000","last_clean":"2022-03-21T14:07:20.869580+0000","last_became_active":"2022-03-21T14:06:54.867097+0000","last_became_peered":"2022-03-21T14:06:54.867097+0000","last_unstale":"2022-03-21T14:07:20.869580+0000","last_undegraded":"2022-03-21T14:07:20.869580+0000","last_fullsized":"2022-03-21T14:07:20.869580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:07:22.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202764+0000","last_change":"2022-03-21T14:06:54.333269+0000","last_active":"2022-03-21T14:07:21.202764+0000","last_peered":"2022-03-21T14:07:21.202764+0000","last_clean":"2022-03-21T14:07:21.202764+0000","last_became_active":"2022-03-21T14:06:54.332844+0000","last_became_peered":"2022-03-21T14:06:54.332844+0000","last_unstale":"2022-03-21T14:07:21.202764+0000","last_undegraded":"2022-03-21T14:07:21.202764+0000","last_fullsized":"2022-03-21T14:07:21.202764+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:45:33.278652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870143+0000","last_change":"2022-03-21T14:06:54.869222+0000","last_active":"2022-03-21T14:07:20.870143+0000","last_peered":"2022-03-21T14:07:20.870143+0000","last_clean":"2022-03-21T14:07:20.870143+0000","last_became_active":"2022-03-21T14:06:54.868242+0000","last_became_peered":"2022-03-21T14:06:54.868242+0000","last_unstale":"2022-03-21T14:07:20.870143+0000","last_undegraded":"2022-03-21T14:07:20.870143+0000","last_fullsized":"2022-03-21T14:07:20.870143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:28:30.012341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869309+0000","last_change":"2022-03-21T14:06:54.317425+0000","last_active":"2022-03-21T14:07:20.869309+0000","last_peered":"2022-03-21T14:07:20.869309+0000","last_clean":"2022-03-21T14:07:20.869309+0000","last_became_active":"2022-03-21T14:06:54.317035+0000","last_became_peered":"2022-03-21T14:06:54.317035+0000","last_unstale":"2022-03-21T14:07:20.869309+0000","last_undegraded":"2022-03-21T14:07:20.869309+0000","last_fullsized":"2022-03-21T14:07:20.869309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:24:33.693292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.881115+0000","last_change":"2022-03-21T14:07:21.881115+0000","last_active":"2022-03-21T14:07:21.881115+0000","last_peered":"2022-03-21T14:07:21.881115+0000","last_clean":"2022-03-21T14:07:21.881115+0000","last_became_active":"2022-03-21T14:07:21.880615+0000","last_became_peered":"2022-03-21T14:07:21.880615+0000","last_unstale":"2022-03-21T14:07:21.881115+0000","last_undegraded":"2022-03-21T14:07:21.881115+0000","last_fullsized":"2022-03-21T14:07:21.881115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:14:35.102757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874761+0000","last_change":"2022-03-21T14:06:54.326963+0000","last_active":"2022-03-21T14:07:20.874761+0000","last_peered":"2022-03-21T14:07:20.874761+0000","last_clean":"2022-03-21T14:07:20.874761+0000","last_became_active":"2022-03-21T14:06:54.326592+0000","last_became_peered":"2022-03-21T14:06:54.326592+0000","last_unstale":"2022-03-21T14:07:20.874761+0000","last_undegraded":"2022-03-21T14:07:20.874761+0000","last_fullsized":"2022-03-21T14:07:20.874761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:11:23.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870134+0000","last_change":"2022-03-21T14:06:54.865257+0000","last_active":"2022-03-21T14:07:20.870134+0000","last_peered":"2022-03-21T14:07:20.870134+0000","last_clean":"2022-03-21T14:07:20.870134+0000","last_became_active":"2022-03-21T14:06:54.864119+0000","last_became_peered":"2022-03-21T14:06:54.864119+0000","last_unstale":"2022-03-21T14:07:20.870134+0000","last_undegraded":"2022-03-21T14:07:20.870134+0000","last_fullsized":"2022-03-21T14:07:20.870134+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:45.228838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203130+0000","last_change":"2022-03-21T14:06:54.864027+0000","last_active":"2022-03-21T14:07:21.203130+0000","last_peered":"2022-03-21T14:07:21.203130+0000","last_clean":"2022-03-21T14:07:21.203130+0000","last_became_active":"2022-03-21T14:06:54.863684+0000","last_became_peered":"2022-03-21T14:06:54.863684+0000","last_unstale":"2022-03-21T14:07:21.203130+0000","last_undegraded":"2022-03-21T14:07:21.203130+0000","last_fullsized":"2022-03-21T14:07:21.203130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:11.453033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202391+0000","last_change":"2022-03-21T14:06:54.860266+0000","last_active":"2022-03-21T14:07:21.202391+0000","last_peered":"2022-03-21T14:07:21.202391+0000","last_clean":"2022-03-21T14:07:21.202391+0000","last_became_active":"2022-03-21T14:06:54.859922+0000","last_became_peered":"2022-03-21T14:06:54.859922+0000","last_unstale":"2022-03-21T14:07:21.202391+0000","last_undegraded":"2022-03-21T14:07:21.202391+0000","last_fullsized":"2022-03-21T14:07:21.202391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:23.689919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202356+0000","last_change":"2022-03-21T14:06:54.875261+0000","last_active":"2022-03-21T14:07:21.202356+0000","last_peered":"2022-03-21T14:07:21.202356+0000","last_clean":"2022-03-21T14:07:21.202356+0000","last_became_active":"2022-03-21T14:06:54.874826+0000","last_became_peered":"2022-03-21T14:06:54.874826+0000","last_unstale":"2022-03-21T14:07:21.202356+0000","last_undegraded":"2022-03-21T14:07:21.202356+0000","last_fullsized":"2022-03-21T14:07:21.202356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:02.673575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870183+0000","last_change":"2022-03-21T14:06:54.867997+0000","last_active":"2022-03-21T14:07:20.870183+0000","last_peered":"2022-03-21T14:07:20.870183+0000","last_clean":"2022-03-21T14:07:20.870183+0000","last_became_active":"2022-03-21T14:06:54.867624+0000","last_became_peered":"2022-03-21T14:06:54.867624+0000","last_unstale":"2022-03-21T14:07:20.870183+0000","last_undegraded":"2022-03-21T14:07:20.870183+0000","last_fullsized":"2022-03-21T14:07:20.870183+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:17:37.552083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874474+0000","last_change":"2022-03-21T14:06:54.876868+0000","last_active":"2022-03-21T14:07:20.874474+0000","last_peered":"2022-03-21T14:07:20.874474+0000","last_clean":"2022-03-21T14:07:20.874474+0000","last_became_active":"2022-03-21T14:06:54.876518+0000","last_became_peered":"2022-03-21T14:06:54.876518+0000","last_unstale":"2022-03-21T14:07:20.874474+0000","last_undegraded":"2022-03-21T14:07:20.874474+0000","last_fullsized":"2022-03-21T14:07:20.874474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:50:48.947076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202905+0000","last_change":"2022-03-21T14:06:54.858384+0000","last_active":"2022-03-21T14:07:21.202905+0000","last_peered":"2022-03-21T14:07:21.202905+0000","last_clean":"2022-03-21T14:07:21.202905+0000","last_became_active":"2022-03-21T14:06:54.857183+0000","last_became_peered":"2022-03-21T14:06:54.857183+0000","last_unstale":"2022-03-21T14:07:21.202905+0000","last_undegraded":"2022-03-21T14:07:21.202905+0000","last_fullsized":"2022-03-21T14:07:21.202905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:03:58.964777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202770+0000","last_change":"2022-03-21T14:06:54.859071+0000","last_active":"2022-03-21T14:07:21.202770+0000","last_peered":"2022-03-21T14:07:21.202770+0000","last_clean":"2022-03-21T14:07:21.202770+0000","last_became_active":"2022-03-21T14:06:54.858740+0000","last_became_peered":"2022-03-21T14:06:54.858740+0000","last_unstale":"2022-03-21T14:07:21.202770+0000","last_undegraded":"2022-03-21T14:07:21.202770+0000","last_fullsized":"2022-03-21T14:07:21.202770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:12.263178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877071+0000","last_change":"2022-03-21T14:06:54.877506+0000","last_active":"2022-03-21T14:07:20.877071+0000","last_peered":"2022-03-21T14:07:20.877071+0000","last_clean":"2022-03-21T14:07:20.877071+0000","last_became_active":"2022-03-21T14:06:54.877201+0000","last_became_peered":"2022-03-21T14:06:54.877201+0000","last_unstale":"2022-03-21T14:07:20.877071+0000","last_undegraded":"2022-03-21T14:07:20.877071+0000","last_fullsized":"2022-03-21T14:07:20.877071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:27:57.264889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868257+0000","last_change":"2022-03-21T14:06:54.875160+0000","last_active":"2022-03-21T14:07:20.868257+0000","last_peered":"2022-03-21T14:07:20.868257+0000","last_clean":"2022-03-21T14:07:20.868257+0000","last_became_active":"2022-03-21T14:06:54.874453+0000","last_became_peered":"2022-03-21T14:06:54.874453+0000","last_unstale":"2022-03-21T14:07:20.868257+0000","last_undegraded":"2022-03-21T14:07:20.868257+0000","last_fullsized":"2022-03-21T14:07:20.868257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:53.657732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874442+0000","last_change":"2022-03-21T14:06:54.876605+0000","last_active":"2022-03-21T14:07:20.874442+0000","last_peered":"2022-03-21T14:07:20.874442+0000","last_clean":"2022-03-21T14:07:20.874442+0000","last_became_active":"2022-03-21T14:06:54.876241+0000","last_became_peered":"2022-03-21T14:06:54.876241+0000","last_unstale":"2022-03-21T14:07:20.874442+0000","last_undegraded":"2022-03-21T14:07:20.874442+0000","last_fullsized":"2022-03-21T14:07:20.874442+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:38.713657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872417+0000","last_change":"2022-03-21T14:06:54.874849+0000","last_active":"2022-03-21T14:07:20.872417+0000","last_peered":"2022-03-21T14:07:20.872417+0000","last_clean":"2022-03-21T14:07:20.872417+0000","last_became_active":"2022-03-21T14:06:54.874203+0000","last_became_peered":"2022-03-21T14:06:54.874203+0000","last_unstale":"2022-03-21T14:07:20.872417+0000","last_undegraded":"2022-03-21T14:07:20.872417+0000","last_fullsized":"2022-03-21T14:07:20.872417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:17:32.906357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874135+0000","last_change":"2022-03-21T14:06:54.869399+0000","last_active":"2022-03-21T14:07:20.874135+0000","last_peered":"2022-03-21T14:07:20.874135+0000","last_clean":"2022-03-21T14:07:20.874135+0000","last_became_active":"2022-03-21T14:06:54.869078+0000","last_became_peered":"2022-03-21T14:06:54.869078+0000","last_unstale":"2022-03-21T14:07:20.874135+0000","last_undegraded":"2022-03-21T14:07:20.874135+0000","last_fullsized":"2022-03-21T14:07:20.874135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:39:49.492701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202767+0000","last_change":"2022-03-21T14:06:54.858492+0000","last_active":"2022-03-21T14:07:21.202767+0000","last_peered":"2022-03-21T14:07:21.202767+0000","last_clean":"2022-03-21T14:07:21.202767+0000","last_became_active":"2022-03-21T14:06:54.858113+0000","last_became_peered":"2022-03-21T14:06:54.858113+0000","last_unstale":"2022-03-21T14:07:21.202767+0000","last_undegraded":"2022-03-21T14:07:21.202767+0000","last_fullsized":"2022-03-21T14:07:21.202767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:41:03.763238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":94,"num_read_kb":51,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481816,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5956,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739196,"statfs":{"total":95995035648,"available":95988936704,"internally_reserved":0,"allocated":913408,"data_stored":438146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.502}]},{"osd":1,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.624},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72399999999999998}]},{"osd":6,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75}]},{"osd":7,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76200000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":9,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.877},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.82399999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.83799999999999997}]}]},{"osd":10,"up_from":70,"seq":300647710747,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6624,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738528,"statfs":{"total":95995035648,"available":95988252672,"internally_reserved":0,"allocated":1466368,"data_stored":968493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.82399999999999995}]},{"osd":3,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.86299999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.89600000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.88300000000000001}]},{"osd":7,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.58099999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.91900000000000004}]},{"osd":9,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.0349999999999999}]},{"osd":11,"last update":"Mon Mar 21 14:07:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.97399999999999998}]}]},{"osd":9,"up_from":65,"seq":279172874272,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6024,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739128,"statfs":{"total":95995035648,"available":95988867072,"internally_reserved":0,"allocated":917504,"data_stored":438712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86599999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72399999999999998,"15min":0.73399999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80100000000000005,"15min":0.81200000000000006},"min":{"1min":0.51400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0329999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.72599999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85599999999999998,"15min":0.86799999999999999},"min":{"1min":0.61699999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.012,"5min":1.466,"15min":1.466},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.83499999999999996,"15min":0.85899999999999999},"min":{"1min":0.496,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.048,"5min":1.605,"15min":1.605},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90900000000000003,"15min":0.92300000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.874,"15min":0.89500000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0660000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.85199999999999998,"15min":0.86799999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.96699999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.103,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.94299999999999995,"15min":0.96299999999999997},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.73099999999999998}]},{"osd":6,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0209999999999999,"15min":1.0389999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.1579999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.97799999999999998,"15min":1.0049999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.242,"5min":1.379,"15min":1.379},"last":0.746},{"interface":"front","average":{"1min":0.875,"5min":1.0389999999999999,"15min":1.0660000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1899999999999999,"5min":1.538,"15min":1.538},"last":0.65600000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.069,"15min":1.087},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2070000000000001,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.873,"5min":1.026,"15min":1.052},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.177,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.60899999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.81999999999999995}]}]},{"osd":8,"up_from":57,"seq":244813135908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6216,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738936,"statfs":{"total":95995035648,"available":95988670464,"internally_reserved":0,"allocated":917504,"data_stored":438411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.871,"5min":0.88,"15min":0.88},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81100000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.54400000000000004}]},{"osd":1,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.38700000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.876,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71699999999999997}]},{"osd":2,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59699999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.433,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64900000000000002,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.63}]},{"osd":3,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83899999999999997,"5min":0.876,"15min":0.876},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78100000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77600000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90800000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.51100000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.92300000000000004}]},{"osd":7,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.99299999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.47399999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.95799999999999996}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72999999999999998,"15min":0.72199999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.85399999999999998}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.93500000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6800,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738352,"statfs":{"total":95995035648,"available":95988072448,"internally_reserved":0,"allocated":1449984,"data_stored":967640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84199999999999997,"5min":0.88,"15min":0.88},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59399999999999997},"min":{"1min":0.441,"5min":0.38,"15min":0.38},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.441},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.45400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.53100000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.72199999999999998,"15min":0.72999999999999998},"min":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.103},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.45100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.84999999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40799999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92300000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67800000000000005,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.70599999999999996},"min":{"1min":0.56399999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94899999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.621}]},{"osd":6,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.77600000000000002},"min":{"1min":0.61699999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.91700000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.77400000000000002,"15min":0.77000000000000002},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.98999999999999999,"5min":1.024,"15min":1.024},"last":0.71799999999999997}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.73799999999999999},"min":{"1min":0.57899999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73699999999999999,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.65200000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.81799999999999995,"15min":0.81599999999999995},"min":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.87,"15min":0.873},"min":{"1min":0.69099999999999995,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.752}]},{"osd":10,"last update":"Mon Mar 21 14:06:43 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.66400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":913408,"data_stored":438146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53100000000000003,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59399999999999997,"15min":0.57599999999999996},"min":{"1min":0.39700000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.501}]},{"osd":1,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73899999999999999,"15min":0.70799999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.495},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.754}]},{"osd":2,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.85799999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78400000000000003,"15min":0.78000000000000003},"min":{"1min":0.61399999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2050000000000001,"5min":1.208,"15min":1.208},"last":1.0269999999999999}]},{"osd":3,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.72899999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.79900000000000004,"15min":0.77100000000000002},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.79400000000000004}]},{"osd":4,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.86299999999999999,"15min":0.86499999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.84299999999999997,"15min":0.82499999999999996},"min":{"1min":0.63500000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.72599999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.55500000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81999999999999995,"15min":0.81200000000000006},"min":{"1min":0.54200000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1799999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.89400000000000002,"15min":0.88700000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.81000000000000005}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.91600000000000004},"min":{"1min":0.70399999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.89600000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.004,"15min":1.0109999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.413,"5min":1.48,"15min":1.48},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0820000000000001,"15min":1.0880000000000001},"min":{"1min":0.59699999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.2519999999999998,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.94299999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:06:46 2022","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.875}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7120,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738032,"statfs":{"total":95995035648,"available":95987744768,"internally_reserved":0,"allocated":925696,"data_stored":439694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79700000000000004,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.35799999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90500000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004}]},{"osd":3,"last update":"Mon Mar 21 14:07:06 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98799999999999999,"5min":1.022,"15min":1.022},"last":0.73199999999999998}]},{"osd":4,"last update":"Mon Mar 21 14:07:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.47299999999999998,"15min":0.437},"min":{"1min":0.35499999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.748,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.49099999999999999,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Mar 21 14:06:50 2022","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.89500000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71299999999999997,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0580000000000001,"5min":1.073,"15min":1.073},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66500000000000004,"15min":0.66900000000000004},"min":{"1min":0.46400000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94599999999999995,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.64900000000000002,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.105,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.629}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.006,"15min":1.006},"last":0.747},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.68700000000000006,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.90500000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.74399999999999999,"15min":0.73899999999999999},"min":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0009999999999999,"5min":1.097,"15min":1.097},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77400000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.98299999999999998}]},{"osd":10,"last update":"Mon Mar 21 14:06:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.94499999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7112,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738040,"statfs":{"total":95995035648,"available":95987752960,"internally_reserved":0,"allocated":917504,"data_stored":438236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.58599999999999997},"min":{"1min":0.53400000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94099999999999995,"5min":1.085,"15min":1.085},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.41099999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83899999999999997,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:07:38 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.48999999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.74199999999999999,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.874}]},{"osd":3,"last update":"Mon Mar 21 14:05:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.58599999999999997,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:07:32 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.75900000000000001,"15min":0.68700000000000006},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0569999999999999,"5min":1.173,"15min":1.173},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.66700000000000004},"min":{"1min":0.61499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78400000000000003,"15min":0.77200000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.78700000000000003,"15min":0.77700000000000002},"min":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0609999999999999,"5min":1.153,"15min":1.153},"last":0.85999999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:07:15 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78800000000000003,"15min":0.754},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0700000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.85399999999999998,"15min":0.81000000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.117,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92100000000000004,"15min":0.88500000000000001},"min":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.766,"5min":5.766,"15min":5.766},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.1890000000000001,"5min":1.224,"15min":1.224},"last":0.70199999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:57 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.96599999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.27,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.997,"15min":1.0009999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.3500000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.93100000000000005}]},{"osd":9,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0640000000000001,"15min":1.0740000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.347,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.1120000000000001,"15min":1.127},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.327,"5min":1.615,"15min":1.615},"last":0.96199999999999997}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.82099999999999995}]},{"osd":11,"last update":"Mon Mar 21 14:07:03 2022","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.002}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7124,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738028,"statfs":{"total":95995035648,"available":95987740672,"internally_reserved":0,"allocated":929792,"data_stored":439728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.0629999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.879,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.40400000000000003}]},{"osd":1,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.57199999999999995,"15min":0.52900000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.93999999999999995,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.39100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.503},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Mar 21 14:06:28 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.755},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Mar 21 14:07:12 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.66400000000000003},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85099999999999998,"5min":1.165,"15min":1.165},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.96499999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.77800000000000002}]},{"osd":7,"last update":"Mon Mar 21 14:06:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.622},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:54 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.59899999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.56299999999999994}]},{"osd":9,"last update":"Mon Mar 21 14:07:25 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.71399999999999997,"15min":0.71799999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92700000000000005,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76100000000000001,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.052,"5min":1.375,"15min":1.375},"last":0.81699999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:06:44 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.65600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":921600,"data_stored":438250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.51400000000000001,"15min":0.46100000000000002},"min":{"1min":0.443,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61399999999999999,"15min":0.56399999999999995},"min":{"1min":0.499,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.70699999999999996}]},{"osd":1,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.67000000000000004,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0720000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.621},"min":{"1min":0.46999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73399999999999999,"15min":0.70399999999999996},"min":{"1min":0.503,"5min":0.441,"15min":0.441},"max":{"1min":0.98799999999999999,"5min":1.125,"15min":1.125},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.71399999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0880000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.82399999999999995}]},{"osd":4,"last update":"Mon Mar 21 14:07:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.74299999999999999,"15min":0.67700000000000005},"min":{"1min":0.53200000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73099999999999998,"15min":0.68100000000000005},"min":{"1min":0.55100000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0529999999999999,"5min":1.169,"15min":1.169},"last":0.89000000000000001}]},{"osd":5,"last update":"Mon Mar 21 14:06:49 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81100000000000005,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.77500000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.119,"15min":1.119},"last":0.96699999999999997}]},{"osd":6,"last update":"Mon Mar 21 14:07:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81100000000000005,"15min":0.80900000000000005},"min":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.89000000000000001,"15min":0.86499999999999999},"min":{"1min":0.65200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.3240000000000001,"5min":1.345,"15min":1.345},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.95099999999999996,"15min":0.90900000000000003},"min":{"1min":0.747,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.25,"5min":1.518,"15min":1.518},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.85499999999999998,"15min":0.83899999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1639999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":1.133}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.96199999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:07:19 2022","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.022,"15min":1.03},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":1},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1319999999999999,"15min":1.1439999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.93600000000000005}]},{"osd":10,"last update":"Mon Mar 21 14:06:41 2022","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.016},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":1.073}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]}]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":917504,"data_stored":438168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.40899999999999997,"15min":0.39100000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40300000000000002},"min":{"1min":0.31,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.627,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.55700000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79100000000000004,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63300000000000001,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Mar 21 14:06:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.46400000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67100000000000004,"15min":0.65800000000000003},"min":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.91000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:07:41 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59399999999999997,"15min":0.55800000000000005},"min":{"1min":0.502,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86399999999999999,"5min":1.115,"15min":1.115},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66100000000000003,"15min":0.64900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98199999999999998,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.46700000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88500000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0469999999999999,"5min":1.05,"15min":1.05},"last":0.78500000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.60099999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.012,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.88},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.61699999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.05,"5min":1.077,"15min":1.077},"last":0.628}]},{"osd":10,"last update":"Mon Mar 21 14:07:41 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.161,"15min":1.161},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81000000000000005,"15min":0.81699999999999995},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.095,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56200000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790004,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6932,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738220,"statfs":{"total":95995035648,"available":95987937280,"internally_reserved":0,"allocated":1454080,"data_stored":966933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.39500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.61399999999999999}]},{"osd":3,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Mar 21 14:07:14 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63,"15min":0.60499999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.91000000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66900000000000004}]},{"osd":7,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.44400000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94999999999999996,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.628,"15min":0.60999999999999999},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.80800000000000005}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.76100000000000001}]},{"osd":9,"last update":"Mon Mar 21 14:07:19 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.748}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:07:46.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph pg dump --format=json 2022-03-21T14:07:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:47.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:47 smithi002 ceph-mon[17167]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:47.780 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:47 smithi002 ceph-mon[17167]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:47 smithi016 ceph-mon[22010]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:47 smithi016 ceph-mon[22010]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:47 smithi079 ceph-mon[23158]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:47 smithi079 ceph-mon[23158]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:48.358 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:48.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 -- 172.21.15.2:0/2148088852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 msgr2=0x7f425c103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:48.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 --2- 172.21.15.2:0/2148088852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c103ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4254009c70 tx=0x7f42540094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 -- 172.21.15.2:0/2148088852 shutdown_connections 2022-03-21T14:07:48.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 --2- 172.21.15.2:0/2148088852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c103ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 --2- 172.21.15.2:0/2148088852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 0x7f425c0fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 --2- 172.21.15.2:0/2148088852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 0x7f425c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 -- 172.21.15.2:0/2148088852 >> 172.21.15.2:0/2148088852 conn(0x7f425c0f6520 msgr2=0x7f425c0f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:48.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 -- 172.21.15.2:0/2148088852 shutdown_connections 2022-03-21T14:07:48.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.889+0000 7f42626ed700 1 -- 172.21.15.2:0/2148088852 wait complete. 2022-03-21T14:07:48.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.890+0000 7f42626ed700 1 Processor -- start 2022-03-21T14:07:48.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.890+0000 7f42626ed700 1 -- start start 2022-03-21T14:07:48.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 0x7f425c10c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:48.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 0x7f425c10cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:48.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c1144e0 con 0x7f425c101b70 2022-03-21T14:07:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c114650 con 0x7f425c0ff0a0 2022-03-21T14:07:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42626ed700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c114810 con 0x7f425c065f80 2022-03-21T14:07:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f4261eec700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f4260eea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 0x7f425c10cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f4261eec700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43926/0 (socket says 172.21.15.2:43926) 2022-03-21T14:07:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f4261eec700 1 -- 172.21.15.2:0/868462730 learned_addr learned my addr 172.21.15.2:0/868462730 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f42616eb700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 0x7f425c10c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.891+0000 7f4261eec700 1 -- 172.21.15.2:0/868462730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 msgr2=0x7f425c10c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4261eec700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 0x7f425c10c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4261eec700 1 -- 172.21.15.2:0/868462730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 msgr2=0x7f425c10cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4261eec700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 0x7f425c10cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4261eec700 1 -- 172.21.15.2:0/868462730 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254009780 con 0x7f425c101b70 2022-03-21T14:07:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4261eec700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4254006010 tx=0x7f42540094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425401c070 con 0x7f425c101b70 2022-03-21T14:07:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f425400f460 con 0x7f425c101b70 2022-03-21T14:07:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254003ba0 con 0x7f425c101b70 2022-03-21T14:07:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.892+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c114a90 con 0x7f425c101b70 2022-03-21T14:07:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.893+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c114e60 con 0x7f425c101b70 2022-03-21T14:07:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.894+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c107650 con 0x7f425c101b70 2022-03-21T14:07:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.895+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f425400c040 con 0x7f425c101b70 2022-03-21T14:07:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.896+0000 7f4251ffb700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 0x7f42480991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:48.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.897+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f42540c0db0 con 0x7f425c101b70 2022-03-21T14:07:48.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.897+0000 7f42616eb700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 0x7f42480991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:48.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.897+0000 7f42616eb700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 0x7f42480991c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f425c066370 tx=0x7f4258005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:48.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:48.901+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f4254057f40 con 0x7f425c101b70 2022-03-21T14:07:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.083+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f425c02ab90 con 0x7f4248096d10 2022-03-21T14:07:49.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.094+0000 7f4251ffb700 1 -- 172.21.15.2:0/868462730 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+338093 (secure 0 0 0) 0x7f425c02ab90 con 0x7f4248096d10 2022-03-21T14:07:49.102 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:49.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 msgr2=0x7f42480991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:49.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 0x7f42480991c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f425c066370 tx=0x7f4258005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 msgr2=0x7f425c113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:49.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4254006010 tx=0x7f42540094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 shutdown_connections 2022-03-21T14:07:49.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4248096d10 0x7f42480991c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f425c101b70 0x7f425c113fa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f425c065f80 0x7f425c10c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 --2- 172.21.15.2:0/868462730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c0ff0a0 0x7f425c10cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:49.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 >> 172.21.15.2:0/868462730 conn(0x7f425c0f6520 msgr2=0x7f425c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:49.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 shutdown_connections 2022-03-21T14:07:49.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:49.101+0000 7f42626ed700 1 -- 172.21.15.2:0/868462730 wait complete. 2022-03-21T14:07:49.108 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-03-21T14:07:49.562 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2022-03-21T14:07:48.405233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":230,"num_read_kb":165,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78876,"kb_used_data":12348,"kb_used_omap":0,"kb_used_meta":66432,"kb_avail":1124862948,"statfs":{"total":1151940427776,"available":1151859658752,"internally_reserved":0,"allocated":12644352,"data_stored":6850557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68026368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003618"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203331+0000","last_change":"2022-03-21T14:06:54.869479+0000","last_active":"2022-03-21T14:07:21.203331+0000","last_peered":"2022-03-21T14:07:21.203331+0000","last_clean":"2022-03-21T14:07:21.203331+0000","last_became_active":"2022-03-21T14:06:54.869079+0000","last_became_peered":"2022-03-21T14:06:54.869079+0000","last_unstale":"2022-03-21T14:07:21.203331+0000","last_undegraded":"2022-03-21T14:07:21.203331+0000","last_fullsized":"2022-03-21T14:07:21.203331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:09:19.289483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.876865+0000","last_change":"2022-03-21T14:06:54.884544+0000","last_active":"2022-03-21T14:07:20.876865+0000","last_peered":"2022-03-21T14:07:20.876865+0000","last_clean":"2022-03-21T14:07:20.876865+0000","last_became_active":"2022-03-21T14:06:54.884210+0000","last_became_peered":"2022-03-21T14:06:54.884210+0000","last_unstale":"2022-03-21T14:07:20.876865+0000","last_undegraded":"2022-03-21T14:07:20.876865+0000","last_fullsized":"2022-03-21T14:07:20.876865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:48:52.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870649+0000","last_change":"2022-03-21T14:06:54.880620+0000","last_active":"2022-03-21T14:07:20.870649+0000","last_peered":"2022-03-21T14:07:20.870649+0000","last_clean":"2022-03-21T14:07:20.870649+0000","last_became_active":"2022-03-21T14:06:54.880297+0000","last_became_peered":"2022-03-21T14:06:54.880297+0000","last_unstale":"2022-03-21T14:07:20.870649+0000","last_undegraded":"2022-03-21T14:07:20.870649+0000","last_fullsized":"2022-03-21T14:07:20.870649+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:43:49.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865650+0000","last_change":"2022-03-21T14:06:54.861753+0000","last_active":"2022-03-21T14:07:20.865650+0000","last_peered":"2022-03-21T14:07:20.865650+0000","last_clean":"2022-03-21T14:07:20.865650+0000","last_became_active":"2022-03-21T14:06:54.860649+0000","last_became_peered":"2022-03-21T14:06:54.860649+0000","last_unstale":"2022-03-21T14:07:20.865650+0000","last_undegraded":"2022-03-21T14:07:20.865650+0000","last_fullsized":"2022-03-21T14:07:20.865650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:22:24.251084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866512+0000","last_change":"2022-03-21T14:06:54.875705+0000","last_active":"2022-03-21T14:07:20.866512+0000","last_peered":"2022-03-21T14:07:20.866512+0000","last_clean":"2022-03-21T14:07:20.866512+0000","last_became_active":"2022-03-21T14:06:54.875171+0000","last_became_peered":"2022-03-21T14:06:54.875171+0000","last_unstale":"2022-03-21T14:07:20.866512+0000","last_undegraded":"2022-03-21T14:07:20.866512+0000","last_fullsized":"2022-03-21T14:07:20.866512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:52:39.257984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878853+0000","last_change":"2022-03-21T14:07:21.878853+0000","last_active":"2022-03-21T14:07:21.878853+0000","last_peered":"2022-03-21T14:07:21.878853+0000","last_clean":"2022-03-21T14:07:21.878853+0000","last_became_active":"2022-03-21T14:07:21.878436+0000","last_became_peered":"2022-03-21T14:07:21.878436+0000","last_unstale":"2022-03-21T14:07:21.878853+0000","last_undegraded":"2022-03-21T14:07:21.878853+0000","last_fullsized":"2022-03-21T14:07:21.878853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:34:56.330484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869200+0000","last_change":"2022-03-21T14:06:54.856225+0000","last_active":"2022-03-21T14:07:20.869200+0000","last_peered":"2022-03-21T14:07:20.869200+0000","last_clean":"2022-03-21T14:07:20.869200+0000","last_became_active":"2022-03-21T14:06:54.855801+0000","last_became_peered":"2022-03-21T14:06:54.855801+0000","last_unstale":"2022-03-21T14:07:20.869200+0000","last_undegraded":"2022-03-21T14:07:20.869200+0000","last_fullsized":"2022-03-21T14:07:20.869200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:26:47.895423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878320+0000","last_change":"2022-03-21T14:07:21.878320+0000","last_active":"2022-03-21T14:07:21.878320+0000","last_peered":"2022-03-21T14:07:21.878320+0000","last_clean":"2022-03-21T14:07:21.878320+0000","last_became_active":"2022-03-21T14:07:21.877897+0000","last_became_peered":"2022-03-21T14:07:21.877897+0000","last_unstale":"2022-03-21T14:07:21.878320+0000","last_undegraded":"2022-03-21T14:07:21.878320+0000","last_fullsized":"2022-03-21T14:07:21.878320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:46.608506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873414+0000","last_change":"2022-03-21T14:06:54.860355+0000","last_active":"2022-03-21T14:07:20.873414+0000","last_peered":"2022-03-21T14:07:20.873414+0000","last_clean":"2022-03-21T14:07:20.873414+0000","last_became_active":"2022-03-21T14:06:54.858723+0000","last_became_peered":"2022-03-21T14:06:54.858723+0000","last_unstale":"2022-03-21T14:07:20.873414+0000","last_undegraded":"2022-03-21T14:07:20.873414+0000","last_fullsized":"2022-03-21T14:07:20.873414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:34.198090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868736+0000","last_change":"2022-03-21T14:06:54.855421+0000","last_active":"2022-03-21T14:07:20.868736+0000","last_peered":"2022-03-21T14:07:20.868736+0000","last_clean":"2022-03-21T14:07:20.868736+0000","last_became_active":"2022-03-21T14:06:54.855027+0000","last_became_peered":"2022-03-21T14:06:54.855027+0000","last_unstale":"2022-03-21T14:07:20.868736+0000","last_undegraded":"2022-03-21T14:07:20.868736+0000","last_fullsized":"2022-03-21T14:07:20.868736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:36:14.968816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870161+0000","last_change":"2022-03-21T14:06:54.869318+0000","last_active":"2022-03-21T14:07:20.870161+0000","last_peered":"2022-03-21T14:07:20.870161+0000","last_clean":"2022-03-21T14:07:20.870161+0000","last_became_active":"2022-03-21T14:06:54.868664+0000","last_became_peered":"2022-03-21T14:06:54.868664+0000","last_unstale":"2022-03-21T14:07:20.870161+0000","last_undegraded":"2022-03-21T14:07:20.870161+0000","last_fullsized":"2022-03-21T14:07:20.870161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:29.775263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865526+0000","last_change":"2022-03-21T14:06:54.862293+0000","last_active":"2022-03-21T14:07:20.865526+0000","last_peered":"2022-03-21T14:07:20.865526+0000","last_clean":"2022-03-21T14:07:20.865526+0000","last_became_active":"2022-03-21T14:06:54.861926+0000","last_became_peered":"2022-03-21T14:06:54.861926+0000","last_unstale":"2022-03-21T14:07:20.865526+0000","last_undegraded":"2022-03-21T14:07:20.865526+0000","last_fullsized":"2022-03-21T14:07:20.865526+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:16:27.088619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874217+0000","last_change":"2022-03-21T14:06:54.859631+0000","last_active":"2022-03-21T14:07:20.874217+0000","last_peered":"2022-03-21T14:07:20.874217+0000","last_clean":"2022-03-21T14:07:20.874217+0000","last_became_active":"2022-03-21T14:06:54.858657+0000","last_became_peered":"2022-03-21T14:06:54.858657+0000","last_unstale":"2022-03-21T14:07:20.874217+0000","last_undegraded":"2022-03-21T14:07:20.874217+0000","last_fullsized":"2022-03-21T14:07:20.874217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:03.821600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869888+0000","last_change":"2022-03-21T14:06:54.866492+0000","last_active":"2022-03-21T14:07:20.869888+0000","last_peered":"2022-03-21T14:07:20.869888+0000","last_clean":"2022-03-21T14:07:20.869888+0000","last_became_active":"2022-03-21T14:06:54.866091+0000","last_became_peered":"2022-03-21T14:06:54.866091+0000","last_unstale":"2022-03-21T14:07:20.869888+0000","last_undegraded":"2022-03-21T14:07:20.869888+0000","last_fullsized":"2022-03-21T14:07:20.869888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:31:33.741891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869690+0000","last_change":"2022-03-21T14:06:54.879848+0000","last_active":"2022-03-21T14:07:20.869690+0000","last_peered":"2022-03-21T14:07:20.869690+0000","last_clean":"2022-03-21T14:07:20.869690+0000","last_became_active":"2022-03-21T14:06:54.879496+0000","last_became_peered":"2022-03-21T14:06:54.879496+0000","last_unstale":"2022-03-21T14:07:20.869690+0000","last_undegraded":"2022-03-21T14:07:20.869690+0000","last_fullsized":"2022-03-21T14:07:20.869690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:18:13.350002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872165+0000","last_change":"2022-03-21T14:06:54.877838+0000","last_active":"2022-03-21T14:07:20.872165+0000","last_peered":"2022-03-21T14:07:20.872165+0000","last_clean":"2022-03-21T14:07:20.872165+0000","last_became_active":"2022-03-21T14:06:54.877444+0000","last_became_peered":"2022-03-21T14:06:54.877444+0000","last_unstale":"2022-03-21T14:07:20.872165+0000","last_undegraded":"2022-03-21T14:07:20.872165+0000","last_fullsized":"2022-03-21T14:07:20.872165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:32.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203390+0000","last_change":"2022-03-21T14:06:54.885978+0000","last_active":"2022-03-21T14:07:21.203390+0000","last_peered":"2022-03-21T14:07:21.203390+0000","last_clean":"2022-03-21T14:07:21.203390+0000","last_became_active":"2022-03-21T14:06:54.885645+0000","last_became_peered":"2022-03-21T14:06:54.885645+0000","last_unstale":"2022-03-21T14:07:21.203390+0000","last_undegraded":"2022-03-21T14:07:21.203390+0000","last_fullsized":"2022-03-21T14:07:21.203390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:51.003441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871073+0000","last_change":"2022-03-21T14:06:54.860687+0000","last_active":"2022-03-21T14:07:20.871073+0000","last_peered":"2022-03-21T14:07:20.871073+0000","last_clean":"2022-03-21T14:07:20.871073+0000","last_became_active":"2022-03-21T14:06:54.860315+0000","last_became_peered":"2022-03-21T14:06:54.860315+0000","last_unstale":"2022-03-21T14:07:20.871073+0000","last_undegraded":"2022-03-21T14:07:20.871073+0000","last_fullsized":"2022-03-21T14:07:20.871073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:27:56.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202560+0000","last_change":"2022-03-21T14:06:54.867688+0000","last_active":"2022-03-21T14:07:21.202560+0000","last_peered":"2022-03-21T14:07:21.202560+0000","last_clean":"2022-03-21T14:07:21.202560+0000","last_became_active":"2022-03-21T14:06:54.866998+0000","last_became_peered":"2022-03-21T14:06:54.866998+0000","last_unstale":"2022-03-21T14:07:21.202560+0000","last_undegraded":"2022-03-21T14:07:21.202560+0000","last_fullsized":"2022-03-21T14:07:21.202560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:17:24.979917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871202+0000","last_change":"2022-03-21T14:06:54.326883+0000","last_active":"2022-03-21T14:07:20.871202+0000","last_peered":"2022-03-21T14:07:20.871202+0000","last_clean":"2022-03-21T14:07:20.871202+0000","last_became_active":"2022-03-21T14:06:54.326424+0000","last_became_peered":"2022-03-21T14:06:54.326424+0000","last_unstale":"2022-03-21T14:07:20.871202+0000","last_undegraded":"2022-03-21T14:07:20.871202+0000","last_fullsized":"2022-03-21T14:07:20.871202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:30:39.818504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877195+0000","last_change":"2022-03-21T14:06:54.875068+0000","last_active":"2022-03-21T14:07:20.877195+0000","last_peered":"2022-03-21T14:07:20.877195+0000","last_clean":"2022-03-21T14:07:20.877195+0000","last_became_active":"2022-03-21T14:06:54.874715+0000","last_became_peered":"2022-03-21T14:06:54.874715+0000","last_unstale":"2022-03-21T14:07:20.877195+0000","last_undegraded":"2022-03-21T14:07:20.877195+0000","last_fullsized":"2022-03-21T14:07:20.877195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:33.864263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877308+0000","last_change":"2022-03-21T14:06:54.335459+0000","last_active":"2022-03-21T14:07:20.877308+0000","last_peered":"2022-03-21T14:07:20.877308+0000","last_clean":"2022-03-21T14:07:20.877308+0000","last_became_active":"2022-03-21T14:06:54.335109+0000","last_became_peered":"2022-03-21T14:06:54.335109+0000","last_unstale":"2022-03-21T14:07:20.877308+0000","last_undegraded":"2022-03-21T14:07:20.877308+0000","last_fullsized":"2022-03-21T14:07:20.877308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:06:56.531385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870715+0000","last_change":"2022-03-21T14:06:54.884043+0000","last_active":"2022-03-21T14:07:20.870715+0000","last_peered":"2022-03-21T14:07:20.870715+0000","last_clean":"2022-03-21T14:07:20.870715+0000","last_became_active":"2022-03-21T14:06:54.883647+0000","last_became_peered":"2022-03-21T14:06:54.883647+0000","last_unstale":"2022-03-21T14:07:20.870715+0000","last_undegraded":"2022-03-21T14:07:20.870715+0000","last_fullsized":"2022-03-21T14:07:20.870715+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:32:17.243588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866752+0000","last_change":"2022-03-21T14:06:54.314357+0000","last_active":"2022-03-21T14:07:20.866752+0000","last_peered":"2022-03-21T14:07:20.866752+0000","last_clean":"2022-03-21T14:07:20.866752+0000","last_became_active":"2022-03-21T14:06:54.313855+0000","last_became_peered":"2022-03-21T14:06:54.313855+0000","last_unstale":"2022-03-21T14:07:20.866752+0000","last_undegraded":"2022-03-21T14:07:20.866752+0000","last_fullsized":"2022-03-21T14:07:20.866752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:58:09.686843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872714+0000","last_change":"2022-03-21T14:06:54.861824+0000","last_active":"2022-03-21T14:07:20.872714+0000","last_peered":"2022-03-21T14:07:20.872714+0000","last_clean":"2022-03-21T14:07:20.872714+0000","last_became_active":"2022-03-21T14:06:54.859283+0000","last_became_peered":"2022-03-21T14:06:54.859283+0000","last_unstale":"2022-03-21T14:07:20.872714+0000","last_undegraded":"2022-03-21T14:07:20.872714+0000","last_fullsized":"2022-03-21T14:07:20.872714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:00:42.993237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866520+0000","last_change":"2022-03-21T14:06:54.312853+0000","last_active":"2022-03-21T14:07:20.866520+0000","last_peered":"2022-03-21T14:07:20.866520+0000","last_clean":"2022-03-21T14:07:20.866520+0000","last_became_active":"2022-03-21T14:06:54.312474+0000","last_became_peered":"2022-03-21T14:06:54.312474+0000","last_unstale":"2022-03-21T14:07:20.866520+0000","last_undegraded":"2022-03-21T14:07:20.866520+0000","last_fullsized":"2022-03-21T14:07:20.866520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:53:19.818645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869430+0000","last_change":"2022-03-21T14:06:54.859678+0000","last_active":"2022-03-21T14:07:20.869430+0000","last_peered":"2022-03-21T14:07:20.869430+0000","last_clean":"2022-03-21T14:07:20.869430+0000","last_became_active":"2022-03-21T14:06:54.858928+0000","last_became_peered":"2022-03-21T14:06:54.858928+0000","last_unstale":"2022-03-21T14:07:20.869430+0000","last_undegraded":"2022-03-21T14:07:20.869430+0000","last_fullsized":"2022-03-21T14:07:20.869430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:44:10.326574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868911+0000","last_change":"2022-03-21T14:06:54.314216+0000","last_active":"2022-03-21T14:07:20.868911+0000","last_peered":"2022-03-21T14:07:20.868911+0000","last_clean":"2022-03-21T14:07:20.868911+0000","last_became_active":"2022-03-21T14:06:54.313808+0000","last_became_peered":"2022-03-21T14:06:54.313808+0000","last_unstale":"2022-03-21T14:07:20.868911+0000","last_undegraded":"2022-03-21T14:07:20.868911+0000","last_fullsized":"2022-03-21T14:07:20.868911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:52:09.937102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878771+0000","last_change":"2022-03-21T14:07:21.878771+0000","last_active":"2022-03-21T14:07:21.878771+0000","last_peered":"2022-03-21T14:07:21.878771+0000","last_clean":"2022-03-21T14:07:21.878771+0000","last_became_active":"2022-03-21T14:07:21.878460+0000","last_became_peered":"2022-03-21T14:07:21.878460+0000","last_unstale":"2022-03-21T14:07:21.878771+0000","last_undegraded":"2022-03-21T14:07:21.878771+0000","last_fullsized":"2022-03-21T14:07:21.878771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:42:20.837629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868095+0000","last_change":"2022-03-21T14:06:54.322793+0000","last_active":"2022-03-21T14:07:20.868095+0000","last_peered":"2022-03-21T14:07:20.868095+0000","last_clean":"2022-03-21T14:07:20.868095+0000","last_became_active":"2022-03-21T14:06:54.322360+0000","last_became_peered":"2022-03-21T14:06:54.322360+0000","last_unstale":"2022-03-21T14:07:20.868095+0000","last_undegraded":"2022-03-21T14:07:20.868095+0000","last_fullsized":"2022-03-21T14:07:20.868095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:15:12.220667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869003+0000","last_change":"2022-03-21T14:06:54.874475+0000","last_active":"2022-03-21T14:07:20.869003+0000","last_peered":"2022-03-21T14:07:20.869003+0000","last_clean":"2022-03-21T14:07:20.869003+0000","last_became_active":"2022-03-21T14:06:54.874113+0000","last_became_peered":"2022-03-21T14:06:54.874113+0000","last_unstale":"2022-03-21T14:07:20.869003+0000","last_undegraded":"2022-03-21T14:07:20.869003+0000","last_fullsized":"2022-03-21T14:07:20.869003+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:25.860831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873527+0000","last_change":"2022-03-21T14:06:54.331708+0000","last_active":"2022-03-21T14:07:20.873527+0000","last_peered":"2022-03-21T14:07:20.873527+0000","last_clean":"2022-03-21T14:07:20.873527+0000","last_became_active":"2022-03-21T14:06:54.331347+0000","last_became_peered":"2022-03-21T14:06:54.331347+0000","last_unstale":"2022-03-21T14:07:20.873527+0000","last_undegraded":"2022-03-21T14:07:20.873527+0000","last_fullsized":"2022-03-21T14:07:20.873527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:59:42.282136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873905+0000","last_change":"2022-03-21T14:06:54.864160+0000","last_active":"2022-03-21T14:07:20.873905+0000","last_peered":"2022-03-21T14:07:20.873905+0000","last_clean":"2022-03-21T14:07:20.873905+0000","last_became_active":"2022-03-21T14:06:54.863699+0000","last_became_peered":"2022-03-21T14:06:54.863699+0000","last_unstale":"2022-03-21T14:07:20.873905+0000","last_undegraded":"2022-03-21T14:07:20.873905+0000","last_fullsized":"2022-03-21T14:07:20.873905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:40:53.670759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868147+0000","last_change":"2022-03-21T14:06:54.314228+0000","last_active":"2022-03-21T14:07:20.868147+0000","last_peered":"2022-03-21T14:07:20.868147+0000","last_clean":"2022-03-21T14:07:20.868147+0000","last_became_active":"2022-03-21T14:06:54.313667+0000","last_became_peered":"2022-03-21T14:06:54.313667+0000","last_unstale":"2022-03-21T14:07:20.868147+0000","last_undegraded":"2022-03-21T14:07:20.868147+0000","last_fullsized":"2022-03-21T14:07:20.868147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:20:10.791422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869546+0000","last_change":"2022-03-21T14:06:54.866898+0000","last_active":"2022-03-21T14:07:20.869546+0000","last_peered":"2022-03-21T14:07:20.869546+0000","last_clean":"2022-03-21T14:07:20.869546+0000","last_became_active":"2022-03-21T14:06:54.865450+0000","last_became_peered":"2022-03-21T14:06:54.865450+0000","last_unstale":"2022-03-21T14:07:20.869546+0000","last_undegraded":"2022-03-21T14:07:20.869546+0000","last_fullsized":"2022-03-21T14:07:20.869546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:39:42.699187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868385+0000","last_change":"2022-03-21T14:06:54.328313+0000","last_active":"2022-03-21T14:07:20.868385+0000","last_peered":"2022-03-21T14:07:20.868385+0000","last_clean":"2022-03-21T14:07:20.868385+0000","last_became_active":"2022-03-21T14:06:54.327989+0000","last_became_peered":"2022-03-21T14:06:54.327989+0000","last_unstale":"2022-03-21T14:07:20.868385+0000","last_undegraded":"2022-03-21T14:07:20.868385+0000","last_fullsized":"2022-03-21T14:07:20.868385+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:59:23.877877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.878866+0000","last_change":"2022-03-21T14:07:21.878866+0000","last_active":"2022-03-21T14:07:21.878866+0000","last_peered":"2022-03-21T14:07:21.878866+0000","last_clean":"2022-03-21T14:07:21.878866+0000","last_became_active":"2022-03-21T14:07:21.878467+0000","last_became_peered":"2022-03-21T14:07:21.878467+0000","last_unstale":"2022-03-21T14:07:21.878866+0000","last_undegraded":"2022-03-21T14:07:21.878866+0000","last_fullsized":"2022-03-21T14:07:21.878866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:33.620541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,9],"acting":[8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869775+0000","last_change":"2022-03-21T14:06:54.317279+0000","last_active":"2022-03-21T14:07:20.869775+0000","last_peered":"2022-03-21T14:07:20.869775+0000","last_clean":"2022-03-21T14:07:20.869775+0000","last_became_active":"2022-03-21T14:06:54.316863+0000","last_became_peered":"2022-03-21T14:06:54.316863+0000","last_unstale":"2022-03-21T14:07:20.869775+0000","last_undegraded":"2022-03-21T14:07:20.869775+0000","last_fullsized":"2022-03-21T14:07:20.869775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:02:17.370559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:22.097382+0000","last_change":"2022-03-21T14:07:22.097382+0000","last_active":"2022-03-21T14:07:22.097382+0000","last_peered":"2022-03-21T14:07:22.097382+0000","last_clean":"2022-03-21T14:07:22.097382+0000","last_became_active":"2022-03-21T14:07:22.096919+0000","last_became_peered":"2022-03-21T14:07:22.096919+0000","last_unstale":"2022-03-21T14:07:22.097382+0000","last_undegraded":"2022-03-21T14:07:22.097382+0000","last_fullsized":"2022-03-21T14:07:22.097382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:05:40.625717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'3","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869545+0000","last_change":"2022-03-21T14:06:54.333183+0000","last_active":"2022-03-21T14:07:20.869545+0000","last_peered":"2022-03-21T14:07:20.869545+0000","last_clean":"2022-03-21T14:07:20.869545+0000","last_became_active":"2022-03-21T14:06:54.332795+0000","last_became_peered":"2022-03-21T14:06:54.332795+0000","last_unstale":"2022-03-21T14:07:20.869545+0000","last_undegraded":"2022-03-21T14:07:20.869545+0000","last_fullsized":"2022-03-21T14:07:20.869545+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:29.651861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872945+0000","last_change":"2022-03-21T14:06:54.862390+0000","last_active":"2022-03-21T14:07:20.872945+0000","last_peered":"2022-03-21T14:07:20.872945+0000","last_clean":"2022-03-21T14:07:20.872945+0000","last_became_active":"2022-03-21T14:06:54.862008+0000","last_became_peered":"2022-03-21T14:06:54.862008+0000","last_unstale":"2022-03-21T14:07:20.872945+0000","last_undegraded":"2022-03-21T14:07:20.872945+0000","last_fullsized":"2022-03-21T14:07:20.872945+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:53:50.255768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'6","reported_seq":117,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:45.430120+0000","last_change":"2022-03-21T14:06:54.315167+0000","last_active":"2022-03-21T14:07:45.430120+0000","last_peered":"2022-03-21T14:07:45.430120+0000","last_clean":"2022-03-21T14:07:45.430120+0000","last_became_active":"2022-03-21T14:06:54.314432+0000","last_became_peered":"2022-03-21T14:06:54.314432+0000","last_unstale":"2022-03-21T14:07:45.430120+0000","last_undegraded":"2022-03-21T14:07:45.430120+0000","last_fullsized":"2022-03-21T14:07:45.430120+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:48:27.504920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":98,"num_read_kb":50,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866571+0000","last_change":"2022-03-21T14:06:54.859057+0000","last_active":"2022-03-21T14:07:20.866571+0000","last_peered":"2022-03-21T14:07:20.866571+0000","last_clean":"2022-03-21T14:07:20.866571+0000","last_became_active":"2022-03-21T14:06:54.858674+0000","last_became_peered":"2022-03-21T14:06:54.858674+0000","last_unstale":"2022-03-21T14:07:20.866571+0000","last_undegraded":"2022-03-21T14:07:20.866571+0000","last_fullsized":"2022-03-21T14:07:20.866571+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:38:29.386832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202646+0000","last_change":"2022-03-21T14:06:54.323631+0000","last_active":"2022-03-21T14:07:21.202646+0000","last_peered":"2022-03-21T14:07:21.202646+0000","last_clean":"2022-03-21T14:07:21.202646+0000","last_became_active":"2022-03-21T14:06:54.323179+0000","last_became_peered":"2022-03-21T14:06:54.323179+0000","last_unstale":"2022-03-21T14:07:21.202646+0000","last_undegraded":"2022-03-21T14:07:21.202646+0000","last_fullsized":"2022-03-21T14:07:21.202646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:35:04.480559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873864+0000","last_change":"2022-03-21T14:06:54.878235+0000","last_active":"2022-03-21T14:07:20.873864+0000","last_peered":"2022-03-21T14:07:20.873864+0000","last_clean":"2022-03-21T14:07:20.873864+0000","last_became_active":"2022-03-21T14:06:54.877650+0000","last_became_peered":"2022-03-21T14:06:54.877650+0000","last_unstale":"2022-03-21T14:07:20.873864+0000","last_undegraded":"2022-03-21T14:07:20.873864+0000","last_fullsized":"2022-03-21T14:07:20.873864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:06:58.716437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868936+0000","last_change":"2022-03-21T14:06:54.316053+0000","last_active":"2022-03-21T14:07:20.868936+0000","last_peered":"2022-03-21T14:07:20.868936+0000","last_clean":"2022-03-21T14:07:20.868936+0000","last_became_active":"2022-03-21T14:06:54.315685+0000","last_became_peered":"2022-03-21T14:06:54.315685+0000","last_unstale":"2022-03-21T14:07:20.868936+0000","last_undegraded":"2022-03-21T14:07:20.868936+0000","last_fullsized":"2022-03-21T14:07:20.868936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:16:47.746232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869577+0000","last_change":"2022-03-21T14:06:54.878905+0000","last_active":"2022-03-21T14:07:20.869577+0000","last_peered":"2022-03-21T14:07:20.869577+0000","last_clean":"2022-03-21T14:07:20.869577+0000","last_became_active":"2022-03-21T14:06:54.878341+0000","last_became_peered":"2022-03-21T14:06:54.878341+0000","last_unstale":"2022-03-21T14:07:20.869577+0000","last_undegraded":"2022-03-21T14:07:20.869577+0000","last_fullsized":"2022-03-21T14:07:20.869577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:05:50.071721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865352+0000","last_change":"2022-03-21T14:06:54.313255+0000","last_active":"2022-03-21T14:07:20.865352+0000","last_peered":"2022-03-21T14:07:20.865352+0000","last_clean":"2022-03-21T14:07:20.865352+0000","last_became_active":"2022-03-21T14:06:54.312849+0000","last_became_peered":"2022-03-21T14:06:54.312849+0000","last_unstale":"2022-03-21T14:07:20.865352+0000","last_undegraded":"2022-03-21T14:07:20.865352+0000","last_fullsized":"2022-03-21T14:07:20.865352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:19:59.556267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869053+0000","last_change":"2022-03-21T14:06:54.857792+0000","last_active":"2022-03-21T14:07:20.869053+0000","last_peered":"2022-03-21T14:07:20.869053+0000","last_clean":"2022-03-21T14:07:20.869053+0000","last_became_active":"2022-03-21T14:06:54.857425+0000","last_became_peered":"2022-03-21T14:06:54.857425+0000","last_unstale":"2022-03-21T14:07:20.869053+0000","last_undegraded":"2022-03-21T14:07:20.869053+0000","last_fullsized":"2022-03-21T14:07:20.869053+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:51:33.943688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'8","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866165+0000","last_change":"2022-03-21T14:06:54.335082+0000","last_active":"2022-03-21T14:07:20.866165+0000","last_peered":"2022-03-21T14:07:20.866165+0000","last_clean":"2022-03-21T14:07:20.866165+0000","last_became_active":"2022-03-21T14:06:54.334703+0000","last_became_peered":"2022-03-21T14:06:54.334703+0000","last_unstale":"2022-03-21T14:07:20.866165+0000","last_undegraded":"2022-03-21T14:07:20.866165+0000","last_fullsized":"2022-03-21T14:07:20.866165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:36:12.443615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.877589+0000","last_change":"2022-03-21T14:07:21.877589+0000","last_active":"2022-03-21T14:07:21.877589+0000","last_peered":"2022-03-21T14:07:21.877589+0000","last_clean":"2022-03-21T14:07:21.877589+0000","last_became_active":"2022-03-21T14:07:21.876949+0000","last_became_peered":"2022-03-21T14:07:21.876949+0000","last_unstale":"2022-03-21T14:07:21.877589+0000","last_undegraded":"2022-03-21T14:07:21.877589+0000","last_fullsized":"2022-03-21T14:07:21.877589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:30:26.166689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866985+0000","last_change":"2022-03-21T14:06:54.315652+0000","last_active":"2022-03-21T14:07:20.866985+0000","last_peered":"2022-03-21T14:07:20.866985+0000","last_clean":"2022-03-21T14:07:20.866985+0000","last_became_active":"2022-03-21T14:06:54.315317+0000","last_became_peered":"2022-03-21T14:06:54.315317+0000","last_unstale":"2022-03-21T14:07:20.866985+0000","last_undegraded":"2022-03-21T14:07:20.866985+0000","last_fullsized":"2022-03-21T14:07:20.866985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:55:42.394760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203002+0000","last_change":"2022-03-21T14:06:54.856509+0000","last_active":"2022-03-21T14:07:21.203002+0000","last_peered":"2022-03-21T14:07:21.203002+0000","last_clean":"2022-03-21T14:07:21.203002+0000","last_became_active":"2022-03-21T14:06:54.855995+0000","last_became_peered":"2022-03-21T14:06:54.855995+0000","last_unstale":"2022-03-21T14:07:21.203002+0000","last_undegraded":"2022-03-21T14:07:21.203002+0000","last_fullsized":"2022-03-21T14:07:21.203002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:02:52.429679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.871427+0000","last_change":"2022-03-21T14:06:54.336921+0000","last_active":"2022-03-21T14:07:20.871427+0000","last_peered":"2022-03-21T14:07:20.871427+0000","last_clean":"2022-03-21T14:07:20.871427+0000","last_became_active":"2022-03-21T14:06:54.336560+0000","last_became_peered":"2022-03-21T14:06:54.336560+0000","last_unstale":"2022-03-21T14:07:20.871427+0000","last_undegraded":"2022-03-21T14:07:20.871427+0000","last_fullsized":"2022-03-21T14:07:20.871427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:26:06.417226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868272+0000","last_change":"2022-03-21T14:06:54.865783+0000","last_active":"2022-03-21T14:07:20.868272+0000","last_peered":"2022-03-21T14:07:20.868272+0000","last_clean":"2022-03-21T14:07:20.868272+0000","last_became_active":"2022-03-21T14:06:54.865399+0000","last_became_peered":"2022-03-21T14:06:54.865399+0000","last_unstale":"2022-03-21T14:07:20.868272+0000","last_undegraded":"2022-03-21T14:07:20.868272+0000","last_fullsized":"2022-03-21T14:07:20.868272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:15:19.155641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869393+0000","last_change":"2022-03-21T14:06:54.321852+0000","last_active":"2022-03-21T14:07:20.869393+0000","last_peered":"2022-03-21T14:07:20.869393+0000","last_clean":"2022-03-21T14:07:20.869393+0000","last_became_active":"2022-03-21T14:06:54.321388+0000","last_became_peered":"2022-03-21T14:06:54.321388+0000","last_unstale":"2022-03-21T14:07:20.869393+0000","last_undegraded":"2022-03-21T14:07:20.869393+0000","last_fullsized":"2022-03-21T14:07:20.869393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:58:18.274331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868341+0000","last_change":"2022-03-21T14:06:54.871690+0000","last_active":"2022-03-21T14:07:20.868341+0000","last_peered":"2022-03-21T14:07:20.868341+0000","last_clean":"2022-03-21T14:07:20.868341+0000","last_became_active":"2022-03-21T14:06:54.871272+0000","last_became_peered":"2022-03-21T14:06:54.871272+0000","last_unstale":"2022-03-21T14:07:20.868341+0000","last_undegraded":"2022-03-21T14:07:20.868341+0000","last_fullsized":"2022-03-21T14:07:20.868341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:05:17.999041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868373+0000","last_change":"2022-03-21T14:06:54.860140+0000","last_active":"2022-03-21T14:07:20.868373+0000","last_peered":"2022-03-21T14:07:20.868373+0000","last_clean":"2022-03-21T14:07:20.868373+0000","last_became_active":"2022-03-21T14:06:54.859700+0000","last_became_peered":"2022-03-21T14:06:54.859700+0000","last_unstale":"2022-03-21T14:07:20.868373+0000","last_undegraded":"2022-03-21T14:07:20.868373+0000","last_fullsized":"2022-03-21T14:07:20.868373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:57:46.264617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873445+0000","last_change":"2022-03-21T14:06:54.334370+0000","last_active":"2022-03-21T14:07:20.873445+0000","last_peered":"2022-03-21T14:07:20.873445+0000","last_clean":"2022-03-21T14:07:20.873445+0000","last_became_active":"2022-03-21T14:06:54.333957+0000","last_became_peered":"2022-03-21T14:06:54.333957+0000","last_unstale":"2022-03-21T14:07:20.873445+0000","last_undegraded":"2022-03-21T14:07:20.873445+0000","last_fullsized":"2022-03-21T14:07:20.873445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:29.048808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.879017+0000","last_change":"2022-03-21T14:07:21.879017+0000","last_active":"2022-03-21T14:07:21.879017+0000","last_peered":"2022-03-21T14:07:21.879017+0000","last_clean":"2022-03-21T14:07:21.879017+0000","last_became_active":"2022-03-21T14:07:21.878560+0000","last_became_peered":"2022-03-21T14:07:21.878560+0000","last_unstale":"2022-03-21T14:07:21.879017+0000","last_undegraded":"2022-03-21T14:07:21.879017+0000","last_fullsized":"2022-03-21T14:07:21.879017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:29:57.592235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.865587+0000","last_change":"2022-03-21T14:06:54.315511+0000","last_active":"2022-03-21T14:07:20.865587+0000","last_peered":"2022-03-21T14:07:20.865587+0000","last_clean":"2022-03-21T14:07:20.865587+0000","last_became_active":"2022-03-21T14:06:54.315144+0000","last_became_peered":"2022-03-21T14:06:54.315144+0000","last_unstale":"2022-03-21T14:07:20.865587+0000","last_undegraded":"2022-03-21T14:07:20.865587+0000","last_fullsized":"2022-03-21T14:07:20.865587+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:30:11.627760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869124+0000","last_change":"2022-03-21T14:05:44.276967+0000","last_active":"2022-03-21T14:07:20.869124+0000","last_peered":"2022-03-21T14:07:20.869124+0000","last_clean":"2022-03-21T14:07:20.869124+0000","last_became_active":"2022-03-21T14:05:44.276297+0000","last_became_peered":"2022-03-21T14:05:44.276297+0000","last_unstale":"2022-03-21T14:07:20.869124+0000","last_undegraded":"2022-03-21T14:07:20.869124+0000","last_fullsized":"2022-03-21T14:07:20.869124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:02:31.386985+0000","last_clean_scrub_stamp":"2022-03-21T14:02:31.386985+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:07:49.689146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.873358+0000","last_change":"2022-03-21T14:06:54.860044+0000","last_active":"2022-03-21T14:07:20.873358+0000","last_peered":"2022-03-21T14:07:20.873358+0000","last_clean":"2022-03-21T14:07:20.873358+0000","last_became_active":"2022-03-21T14:06:54.859511+0000","last_became_peered":"2022-03-21T14:06:54.859511+0000","last_unstale":"2022-03-21T14:07:20.873358+0000","last_undegraded":"2022-03-21T14:07:20.873358+0000","last_fullsized":"2022-03-21T14:07:20.873358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:50:01.100612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869215+0000","last_change":"2022-03-21T14:06:54.313788+0000","last_active":"2022-03-21T14:07:20.869215+0000","last_peered":"2022-03-21T14:07:20.869215+0000","last_clean":"2022-03-21T14:07:20.869215+0000","last_became_active":"2022-03-21T14:06:54.313328+0000","last_became_peered":"2022-03-21T14:06:54.313328+0000","last_unstale":"2022-03-21T14:07:20.869215+0000","last_undegraded":"2022-03-21T14:07:20.869215+0000","last_fullsized":"2022-03-21T14:07:20.869215+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:24:17.930196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866695+0000","last_change":"2022-03-21T14:06:54.334716+0000","last_active":"2022-03-21T14:07:20.866695+0000","last_peered":"2022-03-21T14:07:20.866695+0000","last_clean":"2022-03-21T14:07:20.866695+0000","last_became_active":"2022-03-21T14:06:54.334384+0000","last_became_peered":"2022-03-21T14:06:54.334384+0000","last_unstale":"2022-03-21T14:07:20.866695+0000","last_undegraded":"2022-03-21T14:07:20.866695+0000","last_fullsized":"2022-03-21T14:07:20.866695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:29.598258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874388+0000","last_change":"2022-03-21T14:06:54.872297+0000","last_active":"2022-03-21T14:07:20.874388+0000","last_peered":"2022-03-21T14:07:20.874388+0000","last_clean":"2022-03-21T14:07:20.874388+0000","last_became_active":"2022-03-21T14:06:54.871557+0000","last_became_peered":"2022-03-21T14:06:54.871557+0000","last_unstale":"2022-03-21T14:07:20.874388+0000","last_undegraded":"2022-03-21T14:07:20.874388+0000","last_fullsized":"2022-03-21T14:07:20.874388+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:21:13.076650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870984+0000","last_change":"2022-03-21T14:06:54.327446+0000","last_active":"2022-03-21T14:07:20.870984+0000","last_peered":"2022-03-21T14:07:20.870984+0000","last_clean":"2022-03-21T14:07:20.870984+0000","last_became_active":"2022-03-21T14:06:54.327102+0000","last_became_peered":"2022-03-21T14:06:54.327102+0000","last_unstale":"2022-03-21T14:07:20.870984+0000","last_undegraded":"2022-03-21T14:07:20.870984+0000","last_fullsized":"2022-03-21T14:07:20.870984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:28:18.309364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877166+0000","last_change":"2022-03-21T14:06:54.869348+0000","last_active":"2022-03-21T14:07:20.877166+0000","last_peered":"2022-03-21T14:07:20.877166+0000","last_clean":"2022-03-21T14:07:20.877166+0000","last_became_active":"2022-03-21T14:06:54.868917+0000","last_became_peered":"2022-03-21T14:06:54.868917+0000","last_unstale":"2022-03-21T14:07:20.877166+0000","last_undegraded":"2022-03-21T14:07:20.877166+0000","last_fullsized":"2022-03-21T14:07:20.877166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:38:56.199987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866251+0000","last_change":"2022-03-21T14:06:54.321746+0000","last_active":"2022-03-21T14:07:20.866251+0000","last_peered":"2022-03-21T14:07:20.866251+0000","last_clean":"2022-03-21T14:07:20.866251+0000","last_became_active":"2022-03-21T14:06:54.321294+0000","last_became_peered":"2022-03-21T14:06:54.321294+0000","last_unstale":"2022-03-21T14:07:20.866251+0000","last_undegraded":"2022-03-21T14:07:20.866251+0000","last_fullsized":"2022-03-21T14:07:20.866251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:06:26.961696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:22.097402+0000","last_change":"2022-03-21T14:07:22.097402+0000","last_active":"2022-03-21T14:07:22.097402+0000","last_peered":"2022-03-21T14:07:22.097402+0000","last_clean":"2022-03-21T14:07:22.097402+0000","last_became_active":"2022-03-21T14:07:22.096925+0000","last_became_peered":"2022-03-21T14:07:22.096925+0000","last_unstale":"2022-03-21T14:07:22.097402+0000","last_undegraded":"2022-03-21T14:07:22.097402+0000","last_fullsized":"2022-03-21T14:07:22.097402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:49:34.539213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872194+0000","last_change":"2022-03-21T14:06:54.326806+0000","last_active":"2022-03-21T14:07:20.872194+0000","last_peered":"2022-03-21T14:07:20.872194+0000","last_clean":"2022-03-21T14:07:20.872194+0000","last_became_active":"2022-03-21T14:06:54.326405+0000","last_became_peered":"2022-03-21T14:06:54.326405+0000","last_unstale":"2022-03-21T14:07:20.872194+0000","last_undegraded":"2022-03-21T14:07:20.872194+0000","last_fullsized":"2022-03-21T14:07:20.872194+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:36:29.562293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866167+0000","last_change":"2022-03-21T14:06:54.860972+0000","last_active":"2022-03-21T14:07:20.866167+0000","last_peered":"2022-03-21T14:07:20.866167+0000","last_clean":"2022-03-21T14:07:20.866167+0000","last_became_active":"2022-03-21T14:06:54.860557+0000","last_became_peered":"2022-03-21T14:06:54.860557+0000","last_unstale":"2022-03-21T14:07:20.866167+0000","last_undegraded":"2022-03-21T14:07:20.866167+0000","last_fullsized":"2022-03-21T14:07:20.866167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:28:45.487047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874532+0000","last_change":"2022-03-21T14:06:54.320161+0000","last_active":"2022-03-21T14:07:20.874532+0000","last_peered":"2022-03-21T14:07:20.874532+0000","last_clean":"2022-03-21T14:07:20.874532+0000","last_became_active":"2022-03-21T14:06:54.318104+0000","last_became_peered":"2022-03-21T14:06:54.318104+0000","last_unstale":"2022-03-21T14:07:20.874532+0000","last_undegraded":"2022-03-21T14:07:20.874532+0000","last_fullsized":"2022-03-21T14:07:20.874532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:12.154964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203081+0000","last_change":"2022-03-21T14:06:54.870272+0000","last_active":"2022-03-21T14:07:21.203081+0000","last_peered":"2022-03-21T14:07:21.203081+0000","last_clean":"2022-03-21T14:07:21.203081+0000","last_became_active":"2022-03-21T14:06:54.869649+0000","last_became_peered":"2022-03-21T14:06:54.869649+0000","last_unstale":"2022-03-21T14:07:21.203081+0000","last_undegraded":"2022-03-21T14:07:21.203081+0000","last_fullsized":"2022-03-21T14:07:21.203081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:09:57.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202628+0000","last_change":"2022-03-21T14:06:54.323589+0000","last_active":"2022-03-21T14:07:21.202628+0000","last_peered":"2022-03-21T14:07:21.202628+0000","last_clean":"2022-03-21T14:07:21.202628+0000","last_became_active":"2022-03-21T14:06:54.323187+0000","last_became_peered":"2022-03-21T14:06:54.323187+0000","last_unstale":"2022-03-21T14:07:21.202628+0000","last_undegraded":"2022-03-21T14:07:21.202628+0000","last_fullsized":"2022-03-21T14:07:21.202628+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:05:33.201278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.876105+0000","last_change":"2022-03-21T14:06:54.880453+0000","last_active":"2022-03-21T14:07:20.876105+0000","last_peered":"2022-03-21T14:07:20.876105+0000","last_clean":"2022-03-21T14:07:20.876105+0000","last_became_active":"2022-03-21T14:06:54.880078+0000","last_became_peered":"2022-03-21T14:06:54.880078+0000","last_unstale":"2022-03-21T14:07:20.876105+0000","last_undegraded":"2022-03-21T14:07:20.876105+0000","last_fullsized":"2022-03-21T14:07:20.876105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:37:36.812969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.866383+0000","last_change":"2022-03-21T14:06:54.314137+0000","last_active":"2022-03-21T14:07:20.866383+0000","last_peered":"2022-03-21T14:07:20.866383+0000","last_clean":"2022-03-21T14:07:20.866383+0000","last_became_active":"2022-03-21T14:06:54.313783+0000","last_became_peered":"2022-03-21T14:06:54.313783+0000","last_unstale":"2022-03-21T14:07:20.866383+0000","last_undegraded":"2022-03-21T14:07:20.866383+0000","last_fullsized":"2022-03-21T14:07:20.866383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T14:57:58.358466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869580+0000","last_change":"2022-03-21T14:06:54.867675+0000","last_active":"2022-03-21T14:07:20.869580+0000","last_peered":"2022-03-21T14:07:20.869580+0000","last_clean":"2022-03-21T14:07:20.869580+0000","last_became_active":"2022-03-21T14:06:54.867097+0000","last_became_peered":"2022-03-21T14:06:54.867097+0000","last_unstale":"2022-03-21T14:07:20.869580+0000","last_undegraded":"2022-03-21T14:07:20.869580+0000","last_fullsized":"2022-03-21T14:07:20.869580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:07:22.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202764+0000","last_change":"2022-03-21T14:06:54.333269+0000","last_active":"2022-03-21T14:07:21.202764+0000","last_peered":"2022-03-21T14:07:21.202764+0000","last_clean":"2022-03-21T14:07:21.202764+0000","last_became_active":"2022-03-21T14:06:54.332844+0000","last_became_peered":"2022-03-21T14:06:54.332844+0000","last_unstale":"2022-03-21T14:07:21.202764+0000","last_undegraded":"2022-03-21T14:07:21.202764+0000","last_fullsized":"2022-03-21T14:07:21.202764+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:45:33.278652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870143+0000","last_change":"2022-03-21T14:06:54.869222+0000","last_active":"2022-03-21T14:07:20.870143+0000","last_peered":"2022-03-21T14:07:20.870143+0000","last_clean":"2022-03-21T14:07:20.870143+0000","last_became_active":"2022-03-21T14:06:54.868242+0000","last_became_peered":"2022-03-21T14:06:54.868242+0000","last_unstale":"2022-03-21T14:07:20.870143+0000","last_undegraded":"2022-03-21T14:07:20.870143+0000","last_fullsized":"2022-03-21T14:07:20.870143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:28:30.012341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.869309+0000","last_change":"2022-03-21T14:06:54.317425+0000","last_active":"2022-03-21T14:07:20.869309+0000","last_peered":"2022-03-21T14:07:20.869309+0000","last_clean":"2022-03-21T14:07:20.869309+0000","last_became_active":"2022-03-21T14:06:54.317035+0000","last_became_peered":"2022-03-21T14:06:54.317035+0000","last_unstale":"2022-03-21T14:07:20.869309+0000","last_undegraded":"2022-03-21T14:07:20.869309+0000","last_fullsized":"2022-03-21T14:07:20.869309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:24:33.693292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.881115+0000","last_change":"2022-03-21T14:07:21.881115+0000","last_active":"2022-03-21T14:07:21.881115+0000","last_peered":"2022-03-21T14:07:21.881115+0000","last_clean":"2022-03-21T14:07:21.881115+0000","last_became_active":"2022-03-21T14:07:21.880615+0000","last_became_peered":"2022-03-21T14:07:21.880615+0000","last_unstale":"2022-03-21T14:07:21.881115+0000","last_undegraded":"2022-03-21T14:07:21.881115+0000","last_fullsized":"2022-03-21T14:07:21.881115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T18:14:35.102757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874761+0000","last_change":"2022-03-21T14:06:54.326963+0000","last_active":"2022-03-21T14:07:20.874761+0000","last_peered":"2022-03-21T14:07:20.874761+0000","last_clean":"2022-03-21T14:07:20.874761+0000","last_became_active":"2022-03-21T14:06:54.326592+0000","last_became_peered":"2022-03-21T14:06:54.326592+0000","last_unstale":"2022-03-21T14:07:20.874761+0000","last_undegraded":"2022-03-21T14:07:20.874761+0000","last_fullsized":"2022-03-21T14:07:20.874761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:53.063705+0000","last_clean_scrub_stamp":"2022-03-21T14:06:53.063705+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:11:23.098573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870134+0000","last_change":"2022-03-21T14:06:54.865257+0000","last_active":"2022-03-21T14:07:20.870134+0000","last_peered":"2022-03-21T14:07:20.870134+0000","last_clean":"2022-03-21T14:07:20.870134+0000","last_became_active":"2022-03-21T14:06:54.864119+0000","last_became_peered":"2022-03-21T14:06:54.864119+0000","last_unstale":"2022-03-21T14:07:20.870134+0000","last_undegraded":"2022-03-21T14:07:20.870134+0000","last_fullsized":"2022-03-21T14:07:20.870134+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T23:19:45.228838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.203130+0000","last_change":"2022-03-21T14:06:54.864027+0000","last_active":"2022-03-21T14:07:21.203130+0000","last_peered":"2022-03-21T14:07:21.203130+0000","last_clean":"2022-03-21T14:07:21.203130+0000","last_became_active":"2022-03-21T14:06:54.863684+0000","last_became_peered":"2022-03-21T14:06:54.863684+0000","last_unstale":"2022-03-21T14:07:21.203130+0000","last_undegraded":"2022-03-21T14:07:21.203130+0000","last_fullsized":"2022-03-21T14:07:21.203130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T22:47:11.453033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202391+0000","last_change":"2022-03-21T14:06:54.860266+0000","last_active":"2022-03-21T14:07:21.202391+0000","last_peered":"2022-03-21T14:07:21.202391+0000","last_clean":"2022-03-21T14:07:21.202391+0000","last_became_active":"2022-03-21T14:06:54.859922+0000","last_became_peered":"2022-03-21T14:06:54.859922+0000","last_unstale":"2022-03-21T14:07:21.202391+0000","last_undegraded":"2022-03-21T14:07:21.202391+0000","last_fullsized":"2022-03-21T14:07:21.202391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T17:47:23.689919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202356+0000","last_change":"2022-03-21T14:06:54.875261+0000","last_active":"2022-03-21T14:07:21.202356+0000","last_peered":"2022-03-21T14:07:21.202356+0000","last_clean":"2022-03-21T14:07:21.202356+0000","last_became_active":"2022-03-21T14:06:54.874826+0000","last_became_peered":"2022-03-21T14:06:54.874826+0000","last_unstale":"2022-03-21T14:07:21.202356+0000","last_undegraded":"2022-03-21T14:07:21.202356+0000","last_fullsized":"2022-03-21T14:07:21.202356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:44:02.673575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.870183+0000","last_change":"2022-03-21T14:06:54.867997+0000","last_active":"2022-03-21T14:07:20.870183+0000","last_peered":"2022-03-21T14:07:20.870183+0000","last_clean":"2022-03-21T14:07:20.870183+0000","last_became_active":"2022-03-21T14:06:54.867624+0000","last_became_peered":"2022-03-21T14:06:54.867624+0000","last_unstale":"2022-03-21T14:07:20.870183+0000","last_undegraded":"2022-03-21T14:07:20.870183+0000","last_fullsized":"2022-03-21T14:07:20.870183+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:17:37.552083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874474+0000","last_change":"2022-03-21T14:06:54.876868+0000","last_active":"2022-03-21T14:07:20.874474+0000","last_peered":"2022-03-21T14:07:20.874474+0000","last_clean":"2022-03-21T14:07:20.874474+0000","last_became_active":"2022-03-21T14:06:54.876518+0000","last_became_peered":"2022-03-21T14:06:54.876518+0000","last_unstale":"2022-03-21T14:07:20.874474+0000","last_undegraded":"2022-03-21T14:07:20.874474+0000","last_fullsized":"2022-03-21T14:07:20.874474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T16:50:48.947076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202905+0000","last_change":"2022-03-21T14:06:54.858384+0000","last_active":"2022-03-21T14:07:21.202905+0000","last_peered":"2022-03-21T14:07:21.202905+0000","last_clean":"2022-03-21T14:07:21.202905+0000","last_became_active":"2022-03-21T14:06:54.857183+0000","last_became_peered":"2022-03-21T14:06:54.857183+0000","last_unstale":"2022-03-21T14:07:21.202905+0000","last_undegraded":"2022-03-21T14:07:21.202905+0000","last_fullsized":"2022-03-21T14:07:21.202905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T00:03:58.964777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202770+0000","last_change":"2022-03-21T14:06:54.859071+0000","last_active":"2022-03-21T14:07:21.202770+0000","last_peered":"2022-03-21T14:07:21.202770+0000","last_clean":"2022-03-21T14:07:21.202770+0000","last_became_active":"2022-03-21T14:06:54.858740+0000","last_became_peered":"2022-03-21T14:06:54.858740+0000","last_unstale":"2022-03-21T14:07:21.202770+0000","last_undegraded":"2022-03-21T14:07:21.202770+0000","last_fullsized":"2022-03-21T14:07:21.202770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T15:16:12.263178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.877071+0000","last_change":"2022-03-21T14:06:54.877506+0000","last_active":"2022-03-21T14:07:20.877071+0000","last_peered":"2022-03-21T14:07:20.877071+0000","last_clean":"2022-03-21T14:07:20.877071+0000","last_became_active":"2022-03-21T14:06:54.877201+0000","last_became_peered":"2022-03-21T14:06:54.877201+0000","last_unstale":"2022-03-21T14:07:20.877071+0000","last_undegraded":"2022-03-21T14:07:20.877071+0000","last_fullsized":"2022-03-21T14:07:20.877071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:27:57.264889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.868257+0000","last_change":"2022-03-21T14:06:54.875160+0000","last_active":"2022-03-21T14:07:20.868257+0000","last_peered":"2022-03-21T14:07:20.868257+0000","last_clean":"2022-03-21T14:07:20.868257+0000","last_became_active":"2022-03-21T14:06:54.874453+0000","last_became_peered":"2022-03-21T14:06:54.874453+0000","last_unstale":"2022-03-21T14:07:20.868257+0000","last_undegraded":"2022-03-21T14:07:20.868257+0000","last_fullsized":"2022-03-21T14:07:20.868257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:53.657732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874442+0000","last_change":"2022-03-21T14:06:54.876605+0000","last_active":"2022-03-21T14:07:20.874442+0000","last_peered":"2022-03-21T14:07:20.874442+0000","last_clean":"2022-03-21T14:07:20.874442+0000","last_became_active":"2022-03-21T14:06:54.876241+0000","last_became_peered":"2022-03-21T14:06:54.876241+0000","last_unstale":"2022-03-21T14:07:20.874442+0000","last_undegraded":"2022-03-21T14:07:20.874442+0000","last_fullsized":"2022-03-21T14:07:20.874442+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:14:38.713657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.872417+0000","last_change":"2022-03-21T14:06:54.874849+0000","last_active":"2022-03-21T14:07:20.872417+0000","last_peered":"2022-03-21T14:07:20.872417+0000","last_clean":"2022-03-21T14:07:20.872417+0000","last_became_active":"2022-03-21T14:06:54.874203+0000","last_became_peered":"2022-03-21T14:06:54.874203+0000","last_unstale":"2022-03-21T14:07:20.872417+0000","last_undegraded":"2022-03-21T14:07:20.872417+0000","last_fullsized":"2022-03-21T14:07:20.872417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-23T01:17:32.906357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:20.874135+0000","last_change":"2022-03-21T14:06:54.869399+0000","last_active":"2022-03-21T14:07:20.874135+0000","last_peered":"2022-03-21T14:07:20.874135+0000","last_clean":"2022-03-21T14:07:20.874135+0000","last_became_active":"2022-03-21T14:06:54.869078+0000","last_became_peered":"2022-03-21T14:06:54.869078+0000","last_unstale":"2022-03-21T14:07:20.874135+0000","last_undegraded":"2022-03-21T14:07:20.874135+0000","last_fullsized":"2022-03-21T14:07:20.874135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T19:39:49.492701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-03-21T14:07:21.202767+0000","last_change":"2022-03-21T14:06:54.858492+0000","last_active":"2022-03-21T14:07:21.202767+0000","last_peered":"2022-03-21T14:07:21.202767+0000","last_clean":"2022-03-21T14:07:21.202767+0000","last_became_active":"2022-03-21T14:06:54.858113+0000","last_became_peered":"2022-03-21T14:06:54.858113+0000","last_unstale":"2022-03-21T14:07:21.202767+0000","last_undegraded":"2022-03-21T14:07:21.202767+0000","last_fullsized":"2022-03-21T14:07:21.202767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:06:54.292267+0000","last_clean_scrub_stamp":"2022-03-21T14:06:54.292267+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T20:41:03.763238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":104,"num_read_kb":56,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481816,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5956,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739196,"statfs":{"total":95995035648,"available":95988936704,"internally_reserved":0,"allocated":913408,"data_stored":438146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.502}]},{"osd":1,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.624},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72399999999999998}]},{"osd":6,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.75}]},{"osd":7,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76200000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":9,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.877},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.82399999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:07:04 2022","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.83799999999999997}]}]},{"osd":10,"up_from":70,"seq":300647710748,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6624,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738528,"statfs":{"total":95995035648,"available":95988252672,"internally_reserved":0,"allocated":1466368,"data_stored":968493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.442},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.502}]},{"osd":2,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.76000000000000001}]},{"osd":3,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68300000000000005,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94299999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68100000000000005,"15min":0.68500000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.022,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81399999999999995,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.40400000000000003}]},{"osd":5,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.433,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95399999999999996,"5min":1.087,"15min":1.087},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82199999999999995,"15min":0.82799999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81200000000000006,"15min":0.81999999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93899999999999995,"5min":1.389,"15min":1.389},"last":0.62}]},{"osd":7,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.58099999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.76700000000000002,"15min":0.76500000000000001},"min":{"1min":0.49399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.96799999999999997,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.59399999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.94199999999999995,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.66600000000000004}]},{"osd":9,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.72199999999999998}]},{"osd":11,"last update":"Mon Mar 21 14:07:06 2022","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.79900000000000004}]}]},{"osd":9,"up_from":65,"seq":279172874273,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6024,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739128,"statfs":{"total":95995035648,"available":95988867072,"internally_reserved":0,"allocated":917504,"data_stored":438712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86599999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.88500000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72399999999999998,"15min":0.73399999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80100000000000005,"15min":0.81200000000000006},"min":{"1min":0.51400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0329999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.77700000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85599999999999998,"15min":0.86799999999999999},"min":{"1min":0.61699999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.012,"5min":1.466,"15min":1.466},"last":0.83699999999999997}]},{"osd":3,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.83499999999999996,"15min":0.85899999999999999},"min":{"1min":0.496,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.048,"5min":1.605,"15min":1.605},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90900000000000003,"15min":0.92300000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.874,"15min":0.89500000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0660000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.85199999999999998,"15min":0.86799999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.87}]},{"osd":5,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.96699999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.103,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.94299999999999995,"15min":0.96299999999999997},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0209999999999999,"15min":1.0389999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.1579999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.97799999999999998,"15min":1.0049999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.242,"5min":1.379,"15min":1.379},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.875,"5min":1.0389999999999999,"15min":1.0660000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1899999999999999,"5min":1.538,"15min":1.538},"last":0.91600000000000004}]},{"osd":8,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.069,"15min":1.087},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2070000000000001,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.081},{"interface":"front","average":{"1min":0.873,"5min":1.026,"15min":1.052},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.177,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.98899999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:07:44 2022","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.254,"5min":1.512,"15min":1.512},"last":1.121},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.753,"5min":0.752,"15min":0.752},"max":{"1min":1.244,"5min":1.411,"15min":1.411},"last":1.01}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6216,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738936,"statfs":{"total":95995035648,"available":95988670464,"internally_reserved":0,"allocated":917504,"data_stored":438411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.871,"5min":0.88,"15min":0.88},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.81100000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.38700000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.876,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59699999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64900000000000002,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.439}]},{"osd":3,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83899999999999997,"5min":0.876,"15min":0.876},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78100000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77600000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90800000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.746},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.51100000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.77700000000000002}]},{"osd":7,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.99299999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.879},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.47399999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.84199999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72999999999999998,"15min":0.72199999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79400000000000004}]},{"osd":10,"last update":"Mon Mar 21 14:07:45 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.75900000000000001,"15min":0.74299999999999999},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.73699999999999999,"15min":0.72199999999999998},"min":{"1min":0.56299999999999994,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6800,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738352,"statfs":{"total":95995035648,"available":95988072448,"internally_reserved":0,"allocated":1449984,"data_stored":967640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84199999999999997,"5min":0.88,"15min":0.88},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59399999999999997},"min":{"1min":0.441,"5min":0.38,"15min":0.38},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.45400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.53100000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.72199999999999998,"15min":0.72999999999999998},"min":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.103},"last":0.82499999999999996}]},{"osd":4,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.45100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.84999999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40799999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92300000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.433}]},{"osd":5,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67800000000000005,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84599999999999997,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.70599999999999996},"min":{"1min":0.56399999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94899999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.753}]},{"osd":6,"last update":"Mon Mar 21 14:07:39 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.77600000000000002},"min":{"1min":0.61699999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.91700000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.77400000000000002,"15min":0.77000000000000002},"min":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.98999999999999999,"5min":1.024,"15min":1.024},"last":0.73999999999999999}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.73799999999999999},"min":{"1min":0.57899999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73699999999999999,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.61699999999999999}]},{"osd":9,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.81799999999999995,"15min":0.81599999999999995},"min":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.87,"15min":0.873},"min":{"1min":0.69099999999999995,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.86399999999999999}]},{"osd":10,"last update":"Mon Mar 21 14:07:44 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.81499999999999995,"15min":0.82299999999999995},"min":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0620000000000001,"5min":1.077,"15min":1.077},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.79300000000000004,"15min":0.79800000000000004},"min":{"1min":0.58299999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.79400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6212,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":913408,"data_stored":438146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53100000000000003,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59399999999999997,"15min":0.57599999999999996},"min":{"1min":0.39700000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.501}]},{"osd":1,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73899999999999999,"15min":0.70799999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.495},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.754}]},{"osd":2,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.85799999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78400000000000003,"15min":0.78000000000000003},"min":{"1min":0.61399999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2050000000000001,"5min":1.208,"15min":1.208},"last":1.0269999999999999}]},{"osd":3,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.72899999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.79900000000000004,"15min":0.77100000000000002},"min":{"1min":0.47699999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.79400000000000004}]},{"osd":4,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.86299999999999999,"15min":0.86499999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.84299999999999997,"15min":0.82499999999999996},"min":{"1min":0.63500000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Mar 21 14:07:13 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.72599999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.55500000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81999999999999995,"15min":0.81200000000000006},"min":{"1min":0.54200000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1799999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.89400000000000002,"15min":0.88700000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.81000000000000005}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.91600000000000004},"min":{"1min":0.70399999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.93999999999999995,"15min":0.93400000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.89600000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.004,"15min":1.0109999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.413,"5min":1.48,"15min":1.48},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0820000000000001,"15min":1.0880000000000001},"min":{"1min":0.59699999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.2519999999999998,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.94299999999999995}]},{"osd":10,"last update":"Mon Mar 21 14:06:46 2022","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.875}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7120,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738032,"statfs":{"total":95995035648,"available":95987744768,"internally_reserved":0,"allocated":925696,"data_stored":439694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79700000000000004,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.35799999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90500000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.73399999999999999}]},{"osd":2,"last update":"Mon Mar 21 14:07:43 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.58299999999999996,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.58799999999999997,"15min":0.53800000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Mar 21 14:07:06 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89900000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98799999999999999,"5min":1.022,"15min":1.022},"last":0.79200000000000004}]},{"osd":4,"last update":"Mon Mar 21 14:07:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.47299999999999998,"15min":0.437},"min":{"1min":0.35499999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.748,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.754},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.49099999999999999,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.42099999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:50 2022","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.64900000000000002,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83699999999999997}]},{"osd":6,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71299999999999997,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0580000000000001,"5min":1.073,"15min":1.073},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66500000000000004,"15min":0.66900000000000004},"min":{"1min":0.46400000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94599999999999995,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.64900000000000002,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.105,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.623}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":1.006,"15min":1.006},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.68700000000000006,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.81799999999999995}]},{"osd":9,"last update":"Mon Mar 21 14:07:21 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.74399999999999999,"15min":0.73899999999999999},"min":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0009999999999999,"5min":1.097,"15min":1.097},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77400000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.98299999999999998}]},{"osd":10,"last update":"Mon Mar 21 14:06:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.93300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7112,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738040,"statfs":{"total":95995035648,"available":95987752960,"internally_reserved":0,"allocated":917504,"data_stored":438236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.58599999999999997},"min":{"1min":0.53400000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94099999999999995,"5min":1.085,"15min":1.085},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.41099999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83899999999999997,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Mar 21 14:07:38 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.48999999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.74199999999999999,"15min":0.69899999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.746}]},{"osd":3,"last update":"Mon Mar 21 14:05:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.58599999999999997,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.503},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.83699999999999997}]},{"osd":4,"last update":"Mon Mar 21 14:07:32 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.75900000000000001,"15min":0.68700000000000006},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0569999999999999,"5min":1.173,"15min":1.173},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.73699999999999999,"15min":0.66700000000000004},"min":{"1min":0.61499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78400000000000003,"15min":0.77200000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.78700000000000003,"15min":0.77700000000000002},"min":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0609999999999999,"5min":1.153,"15min":1.153},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:07:15 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78800000000000003,"15min":0.754},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0700000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.85399999999999998,"15min":0.81000000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.117,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.86499999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:06:32 2022","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92100000000000004,"15min":0.88500000000000001},"min":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.766,"5min":5.766,"15min":5.766},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.1890000000000001,"5min":1.224,"15min":1.224},"last":0.70199999999999996}]},{"osd":8,"last update":"Mon Mar 21 14:06:57 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.96599999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.27,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.997,"15min":1.0009999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.3500000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.81899999999999995}]},{"osd":9,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0640000000000001,"15min":1.0740000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.347,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.02,"5min":1.1120000000000001,"15min":1.127},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.327,"5min":1.615,"15min":1.615},"last":0.88500000000000001}]},{"osd":10,"last update":"Mon Mar 21 14:06:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.65900000000000003}]},{"osd":11,"last update":"Mon Mar 21 14:07:03 2022","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.996},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.93500000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7124,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738028,"statfs":{"total":95995035648,"available":95987740672,"internally_reserved":0,"allocated":929792,"data_stored":439728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.0629999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.879,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.82799999999999996}]},{"osd":1,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.57199999999999995,"15min":0.52900000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.93999999999999995,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54900000000000004,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.39100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.63900000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Mon Mar 21 14:06:28 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Mar 21 14:06:47 2022","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.432},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:07:12 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.66400000000000003},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85099999999999998,"5min":1.165,"15min":1.165},"last":0.755},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68999999999999995,"15min":0.68400000000000005},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.96499999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.77300000000000002}]},{"osd":7,"last update":"Mon Mar 21 14:06:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.622},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Mon Mar 21 14:06:54 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.59899999999999998},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.53900000000000003}]},{"osd":9,"last update":"Mon Mar 21 14:07:25 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.71399999999999997,"15min":0.71799999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92700000000000005,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76100000000000001,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.052,"5min":1.375,"15min":1.375},"last":0.80700000000000005}]},{"osd":10,"last update":"Mon Mar 21 14:07:44 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.96399999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91200000000000003,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.65400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":921600,"data_stored":438250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.51400000000000001,"15min":0.46100000000000002},"min":{"1min":0.443,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61399999999999999,"15min":0.56399999999999995},"min":{"1min":0.499,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.432}]},{"osd":1,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.67000000000000004,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0720000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.621},"min":{"1min":0.46999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Mar 21 14:07:09 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73399999999999999,"15min":0.70399999999999996},"min":{"1min":0.503,"5min":0.441,"15min":0.441},"max":{"1min":0.98799999999999999,"5min":1.125,"15min":1.125},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.71399999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0880000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.871}]},{"osd":4,"last update":"Mon Mar 21 14:07:30 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.74299999999999999,"15min":0.67700000000000005},"min":{"1min":0.53200000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.749},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73099999999999998,"15min":0.68100000000000005},"min":{"1min":0.55100000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0529999999999999,"5min":1.169,"15min":1.169},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:49 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81100000000000005,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.77500000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.119,"15min":1.119},"last":0.77200000000000002}]},{"osd":6,"last update":"Mon Mar 21 14:07:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81100000000000005,"15min":0.80900000000000005},"min":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.89000000000000001,"15min":0.86499999999999999},"min":{"1min":0.65200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.3240000000000001,"5min":1.345,"15min":1.345},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Mar 21 14:07:36 2022","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.95099999999999996,"15min":0.90900000000000003},"min":{"1min":0.747,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.25,"5min":1.518,"15min":1.518},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.85499999999999998,"15min":0.83899999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1639999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.80600000000000005}]},{"osd":8,"last update":"Mon Mar 21 14:05:52 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.96199999999999997}]},{"osd":9,"last update":"Mon Mar 21 14:07:19 2022","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.022,"15min":1.03},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.623},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1319999999999999,"15min":1.1439999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.91300000000000003}]},{"osd":10,"last update":"Mon Mar 21 14:07:42 2022","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.1699999999999999,"15min":1.1899999999999999},"min":{"1min":0.91000000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.288,"5min":1.575,"15min":1.575},"last":1.016},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93100000000000005},"min":{"1min":0.76300000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2010000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.84499999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]}]},{"osd":4,"up_from":32,"seq":137438953528,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":917504,"data_stored":438168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:25 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.40899999999999997,"15min":0.39100000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40300000000000002},"min":{"1min":0.31,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.627,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.55700000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79100000000000004,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Mar 21 14:07:26 2022","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63300000000000001,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Mar 21 14:06:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.46400000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Mar 21 14:07:16 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67100000000000004,"15min":0.65800000000000003},"min":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.91000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73699999999999999}]},{"osd":7,"last update":"Mon Mar 21 14:07:41 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59399999999999997,"15min":0.55800000000000005},"min":{"1min":0.502,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86399999999999999,"5min":1.115,"15min":1.115},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66100000000000003,"15min":0.64900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98199999999999998,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.752}]},{"osd":8,"last update":"Mon Mar 21 14:06:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88500000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0469999999999999,"5min":1.05,"15min":1.05},"last":0.77500000000000002}]},{"osd":9,"last update":"Mon Mar 21 14:07:20 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.60099999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.012,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.61699999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.05,"5min":1.077,"15min":1.077},"last":0.89200000000000002}]},{"osd":10,"last update":"Mon Mar 21 14:07:41 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.161,"15min":1.161},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81000000000000005,"15min":0.81699999999999995},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.095,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.93000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":5,"up_from":37,"seq":158913790005,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6932,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738220,"statfs":{"total":95995035648,"available":95987937280,"internally_reserved":0,"allocated":1454080,"data_stored":966933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.39500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39500000000000002,"15min":0.38500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.44},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76500000000000001,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Mar 21 14:06:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Mar 21 14:07:14 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63,"15min":0.60499999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.91000000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Mar 21 14:07:40 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.44400000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94999999999999996,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.628,"15min":0.60999999999999999},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.75800000000000001}]},{"osd":8,"last update":"Mon Mar 21 14:06:56 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.67600000000000005}]},{"osd":9,"last update":"Mon Mar 21 14:07:19 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.83199999999999996}]},{"osd":10,"last update":"Mon Mar 21 14:07:45 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.67000000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.54800000000000004,"5min":0.503,"15min":0.503},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:07:49.570 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-03-21T14:07:49.571 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-21T14:07:49.571 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-03-21T14:07:49.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph health --format=json 2022-03-21T14:07:49.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:49 smithi002 ceph-mon[17167]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:49.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:49 smithi002 ceph-mon[17167]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:49 smithi016 ceph-mon[22010]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:49 smithi016 ceph-mon[22010]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:49 smithi079 ceph-mon[23158]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:49 smithi079 ceph-mon[23158]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-03-21T14:07:50.221 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:07:51.365 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:51.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 -- 172.21.15.2:0/2174533571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800660c0 msgr2=0x7f34800fec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:51.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 --2- 172.21.15.2:0/2174533571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800660c0 0x7f34800fec90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f34700098e0 tx=0x7f3470005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 -- 172.21.15.2:0/2174533571 shutdown_connections 2022-03-21T14:07:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 --2- 172.21.15.2:0/2174533571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 0x7f3480104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 --2- 172.21.15.2:0/2174533571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800660c0 0x7f34800fec90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:51.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 --2- 172.21.15.2:0/2174533571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800ff260 0x7f3480101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 -- 172.21.15.2:0/2174533571 >> 172.21.15.2:0/2174533571 conn(0x7f34800f6760 msgr2=0x7f34800f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 -- 172.21.15.2:0/2174533571 shutdown_connections 2022-03-21T14:07:51.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.859+0000 7f3485956700 1 -- 172.21.15.2:0/2174533571 wait complete. 2022-03-21T14:07:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.860+0000 7f3485956700 1 Processor -- start 2022-03-21T14:07:51.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.860+0000 7f3485956700 1 -- start start 2022-03-21T14:07:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.860+0000 7f3485956700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f3485956700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800ff260 0x7f348010f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f3485956700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 0x7f34801165f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:51.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f3485956700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480116b30 con 0x7f34800660c0 2022-03-21T14:07:51.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f3485956700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480116ca0 con 0x7f3480101cf0 2022-03-21T14:07:51.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f3485956700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480116e60 con 0x7f34800ff260 2022-03-21T14:07:51.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:51.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800ff260 0x7f348010f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43932/0 (socket says 172.21.15.2:43932) 2022-03-21T14:07:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 -- 172.21.15.2:0/12324909 learned_addr learned my addr 172.21.15.2:0/12324909 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347f7fe700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 0x7f34801165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:51.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 -- 172.21.15.2:0/12324909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800ff260 msgr2=0x7f348010f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:51.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800ff260 0x7f348010f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:51.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 -- 172.21.15.2:0/12324909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 msgr2=0x7f34801165f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:51.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 0x7f34801165f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:51.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.861+0000 7f347effd700 1 -- 172.21.15.2:0/12324909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34700093f0 con 0x7f34800660c0 2022-03-21T14:07:51.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f347effd700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3470006010 tx=0x7f3470005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:51.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347001d070 con 0x7f34800660c0 2022-03-21T14:07:51.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3470005040 con 0x7f34800660c0 2022-03-21T14:07:51.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470017a00 con 0x7f34800660c0 2022-03-21T14:07:51.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34801170e0 con 0x7f34800660c0 2022-03-21T14:07:51.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.862+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3480117510 con 0x7f34800660c0 2022-03-21T14:07:51.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.863+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34801078a0 con 0x7f34800660c0 2022-03-21T14:07:51.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:51 smithi002 ceph-mon[17167]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:51.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.865+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f3470029030 con 0x7f34800660c0 2022-03-21T14:07:51.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.866+0000 7f34677fe700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 0x7f34680991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:51.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.867+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f34700c0da0 con 0x7f34800660c0 2022-03-21T14:07:51.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.867+0000 7f347e7fc700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 0x7f34680991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:51.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.867+0000 7f347e7fc700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 0x7f34680991c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f34800664f0 tx=0x7f3474005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:51.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:51.871+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f347008dfc0 con 0x7f34800660c0 2022-03-21T14:07:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:51 smithi016 ceph-mon[22010]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:51 smithi079 ceph-mon[23158]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:52.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.098+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f34800fc010 con 0x7f34800660c0 2022-03-21T14:07:52.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.099+0000 7f34677fe700 1 -- 172.21.15.2:0/12324909 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f34700915e0 con 0x7f34800660c0 2022-03-21T14:07:52.100 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:52.100 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-21T14:07:52.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 msgr2=0x7f34680991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 0x7f34680991c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f34800664f0 tx=0x7f3474005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 msgr2=0x7f348010f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3470006010 tx=0x7f3470005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:52.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 shutdown_connections 2022-03-21T14:07:52.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3480101cf0 0x7f34801165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3468096d10 0x7f34680991c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34800660c0 0x7f348010f020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 --2- 172.21.15.2:0/12324909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34800ff260 0x7f348010f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.101+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 >> 172.21.15.2:0/12324909 conn(0x7f34800f6760 msgr2=0x7f34800f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:52.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.102+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 shutdown_connections 2022-03-21T14:07:52.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.102+0000 7f3485956700 1 -- 172.21.15.2:0/12324909 wait complete. 2022-03-21T14:07:52.570 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-03-21T14:07:52.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:07:52.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.780+0000 7f4454a11700 1 -- 172.21.15.2:0/735672416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 msgr2=0x7f44500fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 --2- 172.21.15.2:0/735672416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f44500fe6a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4440009c70 tx=0x7f44400094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 -- 172.21.15.2:0/735672416 shutdown_connections 2022-03-21T14:07:52.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 --2- 172.21.15.2:0/735672416 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 0x7f4450106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 --2- 172.21.15.2:0/735672416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f44500fe6a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 --2- 172.21.15.2:0/735672416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 0x7f44500ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 -- 172.21.15.2:0/735672416 >> 172.21.15.2:0/735672416 conn(0x7f44500f9990 msgr2=0x7f44500fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:52.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 -- 172.21.15.2:0/735672416 shutdown_connections 2022-03-21T14:07:52.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.781+0000 7f4454a11700 1 -- 172.21.15.2:0/735672416 wait complete. 2022-03-21T14:07:52.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 Processor -- start 2022-03-21T14:07:52.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 -- start start 2022-03-21T14:07:52.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:52.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 0x7f445010f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:52.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 0x7f4450116580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:52.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450116ac0 con 0x7f44500fe290 2022-03-21T14:07:52.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450116c30 con 0x7f44500ff530 2022-03-21T14:07:52.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f4454a11700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450116df0 con 0x7f44500ffec0 2022-03-21T14:07:52.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f444f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:52.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f444f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43936/0 (socket says 172.21.15.2:43936) 2022-03-21T14:07:52.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f444f7fe700 1 -- 172.21.15.2:0/737024994 learned_addr learned my addr 172.21.15.2:0/737024994 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:52.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f444ffff700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 0x7f4450116580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:52.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.782+0000 7f444effd700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 0x7f445010f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:52.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 -- 172.21.15.2:0/737024994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 msgr2=0x7f4450116580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 0x7f4450116580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 -- 172.21.15.2:0/737024994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 msgr2=0x7f445010f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 0x7f445010f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 -- 172.21.15.2:0/737024994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4440009780 con 0x7f44500fe290 2022-03-21T14:07:52.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f444f7fe700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f44400095d0 tx=0x7f44400094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:52.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444001c070 con 0x7f44500fe290 2022-03-21T14:07:52.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4440005ac0 con 0x7f44500fe290 2022-03-21T14:07:52.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450117070 con 0x7f44500fe290 2022-03-21T14:07:52.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.783+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440003ab0 con 0x7f44500fe290 2022-03-21T14:07:52.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.784+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44501174f0 con 0x7f44500fe290 2022-03-21T14:07:52.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.785+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4450109af0 con 0x7f44500fe290 2022-03-21T14:07:52.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.786+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f444000c040 con 0x7f44500fe290 2022-03-21T14:07:52.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.787+0000 7f4437fff700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 0x7f44380991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:52.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.788+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f44400c1400 con 0x7f44500fe290 2022-03-21T14:07:52.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.788+0000 7f444effd700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 0x7f44380991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:52.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.788+0000 7f444effd700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 0x7f44380991c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4444005fd0 tx=0x7f4444005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:52.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.791+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f444008e620 con 0x7f44500fe290 2022-03-21T14:07:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:52 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/12324909' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-03-21T14:07:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:52 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/12324909' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-03-21T14:07:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:52 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/12324909' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-03-21T14:07:52.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.979+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4450117800 con 0x7f44500fe290 2022-03-21T14:07:52.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.980+0000 7f4437fff700 1 -- 172.21.15.2:0/737024994 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+21650 (secure 0 0 0) 0x7f4440091c40 con 0x7f44500fe290 2022-03-21T14:07:52.982 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:52.982 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":84,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:07:21.862918+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.16","mappings":[{"from":1,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:07:52.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 msgr2=0x7f44380991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 0x7f44380991c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4444005fd0 tx=0x7f4444005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 msgr2=0x7f445010efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f44400095d0 tx=0x7f44400094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 shutdown_connections 2022-03-21T14:07:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f4438096d10 0x7f44380991c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44500ffec0 0x7f4450116580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44500fe290 0x7f445010efb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 --2- 172.21.15.2:0/737024994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44500ff530 0x7f445010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.982+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 >> 172.21.15.2:0/737024994 conn(0x7f44500f9990 msgr2=0x7f44500fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.983+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 shutdown_connections 2022-03-21T14:07:52.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:52.983+0000 7f4454a11700 1 -- 172.21.15.2:0/737024994 wait complete. 2022-03-21T14:07:53.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-03-21T14:06:52.815266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-03-21T14:06:54.292010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-03-21T14:07:53.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-03-21T14:07:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 -- 172.21.15.2:0/1167800706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 msgr2=0x7f6f180fe5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 --2- 172.21.15.2:0/1167800706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f180fe5d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f080098e0 tx=0x7f6f08005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 -- 172.21.15.2:0/1167800706 shutdown_connections 2022-03-21T14:07:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 --2- 172.21.15.2:0/1167800706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f180fe5d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 --2- 172.21.15.2:0/1167800706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ffdf0 0x7f6f181066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 --2- 172.21.15.2:0/1167800706 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ff460 0x7f6f180ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 -- 172.21.15.2:0/1167800706 >> 172.21.15.2:0/1167800706 conn(0x7f6f180f9990 msgr2=0x7f6f180fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 -- 172.21.15.2:0/1167800706 shutdown_connections 2022-03-21T14:07:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.218+0000 7f6f17fff700 1 -- 172.21.15.2:0/1167800706 wait complete. 2022-03-21T14:07:53.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.219+0000 7f6f17fff700 1 Processor -- start 2022-03-21T14:07:53.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.219+0000 7f6f17fff700 1 -- start start 2022-03-21T14:07:53.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.219+0000 7f6f17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f17fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 0x7f6f1810f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f17fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ffdf0 0x7f6f18116550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f17fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18116a90 con 0x7f6f180ff460 2022-03-21T14:07:53.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f17fff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18116c00 con 0x7f6f180fe1c0 2022-03-21T14:07:53.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f17fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18116dc0 con 0x7f6f180ffdf0 2022-03-21T14:07:53.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f16ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 0x7f6f1810f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f16ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48524/0 (socket says 172.21.15.2:48524) 2022-03-21T14:07:53.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f16ffd700 1 -- 172.21.15.2:0/345023436 learned_addr learned my addr 172.21.15.2:0/345023436 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:53.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f177fe700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ffdf0 0x7f6f18116550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 -- 172.21.15.2:0/345023436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ffdf0 msgr2=0x7f6f18116550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ffdf0 0x7f6f18116550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 -- 172.21.15.2:0/345023436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 msgr2=0x7f6f1810ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f167fc700 1 -- 172.21.15.2:0/345023436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c00b360 con 0x7f6f180ff460 2022-03-21T14:07:53.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.220+0000 7f6f16ffd700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:53.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6f167fc700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 0x7f6f1810f4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e9a0 tx=0x7f6f0c005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:53.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c00b640 con 0x7f6f180ff460 2022-03-21T14:07:53.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f100096b0 con 0x7f6f180ff460 2022-03-21T14:07:53.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6f0c00c9d0 con 0x7f6f180ff460 2022-03-21T14:07:53.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c003ac0 con 0x7f6f180ff460 2022-03-21T14:07:53.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.221+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f080093f0 con 0x7f6f180ff460 2022-03-21T14:07:53.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.222+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f18109a20 con 0x7f6f180ff460 2022-03-21T14:07:53.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.224+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f6f0c00c040 con 0x7f6f180ff460 2022-03-21T14:07:53.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.225+0000 7f6eff7fe700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 0x7f6f000991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.225+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f6f0c014070 con 0x7f6f180ff460 2022-03-21T14:07:53.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.226+0000 7f6f16ffd700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 0x7f6f000991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.226+0000 7f6f16ffd700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 0x7f6f000991c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800f8e0 tx=0x7f6f0801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:53.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.229+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f6f0c08cda0 con 0x7f6f180ff460 2022-03-21T14:07:53.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.416+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f18109bb0 con 0x7f6f180ff460 2022-03-21T14:07:53.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.416+0000 7f6eff7fe700 1 -- 172.21.15.2:0/345023436 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f18109bb0 con 0x7f6f180ff460 2022-03-21T14:07:53.418 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:07:53.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 msgr2=0x7f6f000991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 0x7f6f000991c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800f8e0 tx=0x7f6f0801a040 comp rx=0 tx=0).stop 2022-03-21T14:07:53.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 msgr2=0x7f6f1810f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 0x7f6f1810f4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e9a0 tx=0x7f6f0c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 shutdown_connections 2022-03-21T14:07:53.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f180fe1c0 0x7f6f1810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6f00096d10 0x7f6f000991c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6f180ffdf0 0x7f6f18116550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 --2- 172.21.15.2:0/345023436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180ff460 0x7f6f1810f4c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 >> 172.21.15.2:0/345023436 conn(0x7f6f180f9990 msgr2=0x7f6f18102690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:53.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 shutdown_connections 2022-03-21T14:07:53.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.419+0000 7f6f17fff700 1 -- 172.21.15.2:0/345023436 wait complete. 2022-03-21T14:07:53.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-03-21T14:07:53.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.654+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/1268006887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fbfd0 msgr2=0x7f2d680fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.654+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/1268006887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fbfd0 0x7f2d680fc3e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2d580098e0 tx=0x7f2d58005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:53.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/1268006887 shutdown_connections 2022-03-21T14:07:53.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/1268006887 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fdc00 0x7f2d681044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/1268006887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fbfd0 0x7f2d680fc3e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/1268006887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 0x7f2d680fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/1268006887 >> 172.21.15.2:0/1268006887 conn(0x7f2d680f7780 msgr2=0x7f2d680f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/1268006887 shutdown_connections 2022-03-21T14:07:53.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.655+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/1268006887 wait complete. 2022-03-21T14:07:53.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 Processor -- start 2022-03-21T14:07:53.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 -- start start 2022-03-21T14:07:53.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 0x7f2d6810d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 0x7f2d681144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d681149e0 con 0x7f2d680fdc00 2022-03-21T14:07:53.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68114b50 con 0x7f2d680fd270 2022-03-21T14:07:53.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6d3a0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68114d10 con 0x7f2d680fbfd0 2022-03-21T14:07:53.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d67fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d67fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39794/0 (socket says 172.21.15.2:39794) 2022-03-21T14:07:53.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d67fff700 1 -- 172.21.15.2:0/2379237593 learned_addr learned my addr 172.21.15.2:0/2379237593 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d6cb9f700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 0x7f2d681144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.656+0000 7f2d677fe700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 0x7f2d6810d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 -- 172.21.15.2:0/2379237593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 msgr2=0x7f2d6810d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 0x7f2d6810d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 -- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 msgr2=0x7f2d681144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 0x7f2d681144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 -- 172.21.15.2:0/2379237593 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580093f0 con 0x7f2d680fbfd0 2022-03-21T14:07:53.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d6cb9f700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 0x7f2d681144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d67fff700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800f790 tx=0x7f2d58005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5801d070 con 0x7f2d680fbfd0 2022-03-21T14:07:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2d580054f0 con 0x7f2d680fbfd0 2022-03-21T14:07:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d68114f90 con 0x7f2d680fbfd0 2022-03-21T14:07:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58017c00 con 0x7f2d680fbfd0 2022-03-21T14:07:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.657+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d681152e0 con 0x7f2d680fbfd0 2022-03-21T14:07:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.658+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d681077c0 con 0x7f2d680fbfd0 2022-03-21T14:07:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.660+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2d58028030 con 0x7f2d680fbfd0 2022-03-21T14:07:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.661+0000 7f2d64ff9700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 0x7f2d500991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.661+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2d580c0c30 con 0x7f2d680fbfd0 2022-03-21T14:07:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.661+0000 7f2d677fe700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 0x7f2d500991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.662+0000 7f2d677fe700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 0x7f2d500991c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005c00 tx=0x7f2d5c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:53.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.664+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2d5808de50 con 0x7f2d680fbfd0 2022-03-21T14:07:53.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.851+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d68107950 con 0x7f2d680fbfd0 2022-03-21T14:07:53.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.852+0000 7f2d64ff9700 1 -- 172.21.15.2:0/2379237593 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2d58026020 con 0x7f2d680fbfd0 2022-03-21T14:07:53.854 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-03-21T14:07:53.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.854+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 msgr2=0x7f2d500991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.854+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 0x7f2d500991c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005c00 tx=0x7f2d5c005b50 comp rx=0 tx=0).stop 2022-03-21T14:07:53.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.854+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 msgr2=0x7f2d6810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:53.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.854+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800f790 tx=0x7f2d58005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:53.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 shutdown_connections 2022-03-21T14:07:53.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2d50096d10 0x7f2d500991c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d680fdc00 0x7f2d681144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d680fbfd0 0x7f2d6810ced0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 --2- 172.21.15.2:0/2379237593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680fd270 0x7f2d6810d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:53.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 >> 172.21.15.2:0/2379237593 conn(0x7f2d680f7780 msgr2=0x7f2d681004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:53.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 shutdown_connections 2022-03-21T14:07:53.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:53.855+0000 7f2d6d3a0700 1 -- 172.21.15.2:0/2379237593 wait complete. 2022-03-21T14:07:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:53 smithi002 ceph-mon[17167]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:53 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/737024994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:53 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/345023436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:07:53.882 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-03-21T14:07:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:53 smithi016 ceph-mon[22010]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:53 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/737024994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:53 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/345023436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:07:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:53 smithi079 ceph-mon[23158]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:53 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/737024994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:07:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:53 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/345023436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 -- 172.21.15.2:0/1656631891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00fe1f0 msgr2=0x7f55a00fe600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 --2- 172.21.15.2:0/1656631891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00fe1f0 0x7f55a00fe600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f55900098e0 tx=0x7f5590005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:54.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 -- 172.21.15.2:0/1656631891 shutdown_connections 2022-03-21T14:07:54.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 --2- 172.21.15.2:0/1656631891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a01066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 --2- 172.21.15.2:0/1656631891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00fe1f0 0x7f55a00fe600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 --2- 172.21.15.2:0/1656631891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00ff490 0x7f55a00ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 -- 172.21.15.2:0/1656631891 >> 172.21.15.2:0/1656631891 conn(0x7f55a00f99c0 msgr2=0x7f55a00fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:54.093 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 -- 172.21.15.2:0/1656631891 shutdown_connections 2022-03-21T14:07:54.093 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.089+0000 7f559ffff700 1 -- 172.21.15.2:0/1656631891 wait complete. 2022-03-21T14:07:54.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.090+0000 7f559ffff700 1 Processor -- start 2022-03-21T14:07:54.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.090+0000 7f559ffff700 1 -- start start 2022-03-21T14:07:54.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.090+0000 7f559ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.090+0000 7f559ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 0x7f55a010f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a0116540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0116a80 con 0x7f55a00ffe20 2022-03-21T14:07:54.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559ffff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0116bf0 con 0x7f55a00fe1f0 2022-03-21T14:07:54.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559ffff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0116db0 con 0x7f55a00ff490 2022-03-21T14:07:54.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 0x7f55a010f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a0116540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 0x7f55a010f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39798/0 (socket says 172.21.15.2:39798) 2022-03-21T14:07:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48534/0 (socket says 172.21.15.2:48534) 2022-03-21T14:07:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559e7fc700 1 -- 172.21.15.2:0/2055516090 learned_addr learned my addr 172.21.15.2:0/2055516090 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 -- 172.21.15.2:0/2055516090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 msgr2=0x7f55a010f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 0x7f55a010f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 -- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 msgr2=0x7f55a0116540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a0116540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559effd700 1 -- 172.21.15.2:0/2055516090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900093f0 con 0x7f55a00fe1f0 2022-03-21T14:07:54.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.091+0000 7f559f7fe700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a0116540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:54.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f559effd700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5590009780 tx=0x7f5590005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:54.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559001d070 con 0x7f55a00fe1f0 2022-03-21T14:07:54.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a0117030 con 0x7f55a00fe1f0 2022-03-21T14:07:54.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5590000c90 con 0x7f55a00fe1f0 2022-03-21T14:07:54.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590017a20 con 0x7f55a00fe1f0 2022-03-21T14:07:54.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.092+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a01174c0 con 0x7f55a00fe1f0 2022-03-21T14:07:54.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.093+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a0109a50 con 0x7f55a00fe1f0 2022-03-21T14:07:54.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.095+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f5590029030 con 0x7f55a00fe1f0 2022-03-21T14:07:54.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.097+0000 7f55877fe700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 0x7f55880991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.097+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f55900c1070 con 0x7f55a00fe1f0 2022-03-21T14:07:54.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.097+0000 7f559e7fc700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 0x7f55880991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.097+0000 7f559e7fc700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 0x7f55880991c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f559400be10 tx=0x7f5594005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:54.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.098+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f559008e290 con 0x7f55a00fe1f0 2022-03-21T14:07:54.291 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.289+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55a0109be0 con 0x7f55a00fe1f0 2022-03-21T14:07:54.292 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.290+0000 7f55877fe700 1 -- 172.21.15.2:0/2055516090 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5590027020 con 0x7f55a00fe1f0 2022-03-21T14:07:54.292 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-03-21T14:07:54.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 msgr2=0x7f55880991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 0x7f55880991c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f559400be10 tx=0x7f5594005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 msgr2=0x7f55a010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5590009780 tx=0x7f5590005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:54.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 shutdown_connections 2022-03-21T14:07:54.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a00fe1f0 0x7f55a010ef70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5588096d10 0x7f55880991c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55a00ffe20 0x7f55a0116540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 --2- 172.21.15.2:0/2055516090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55a00ff490 0x7f55a010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 >> 172.21.15.2:0/2055516090 conn(0x7f55a00f99c0 msgr2=0x7f55a00fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:54.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 shutdown_connections 2022-03-21T14:07:54.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.293+0000 7f559ffff700 1 -- 172.21.15.2:0/2055516090 wait complete. 2022-03-21T14:07:54.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:07:54.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.522+0000 7f822e0b8700 1 -- 172.21.15.2:0/4241980100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280fe270 msgr2=0x7f82280fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.522+0000 7f822e0b8700 1 --2- 172.21.15.2:0/4241980100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280fe270 0x7f82280fe680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f82240098e0 tx=0x7f8224005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:54.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 -- 172.21.15.2:0/4241980100 shutdown_connections 2022-03-21T14:07:54.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 --2- 172.21.15.2:0/4241980100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280ffea0 0x7f8228106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 --2- 172.21.15.2:0/4241980100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280fe270 0x7f82280fe680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 --2- 172.21.15.2:0/4241980100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ff510 0x7f82280ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 -- 172.21.15.2:0/4241980100 >> 172.21.15.2:0/4241980100 conn(0x7f82280f9990 msgr2=0x7f82280fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:54.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 -- 172.21.15.2:0/4241980100 shutdown_connections 2022-03-21T14:07:54.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 -- 172.21.15.2:0/4241980100 wait complete. 2022-03-21T14:07:54.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.523+0000 7f822e0b8700 1 Processor -- start 2022-03-21T14:07:54.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 -- start start 2022-03-21T14:07:54.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280fe270 0x7f822810ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ffea0 0x7f8228116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228116a60 con 0x7f82280ffea0 2022-03-21T14:07:54.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228116bd0 con 0x7f82280fe270 2022-03-21T14:07:54.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822e0b8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228116d90 con 0x7f82280ff510 2022-03-21T14:07:54.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822c8b5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822c8b5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39802/0 (socket says 172.21.15.2:39802) 2022-03-21T14:07:54.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822c8b5700 1 -- 172.21.15.2:0/248537126 learned_addr learned my addr 172.21.15.2:0/248537126 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:54.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822d8b7700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ffea0 0x7f8228116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.524+0000 7f822d0b6700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280fe270 0x7f822810ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 -- 172.21.15.2:0/248537126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280fe270 msgr2=0x7f822810ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280fe270 0x7f822810ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 -- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ffea0 msgr2=0x7f8228116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ffea0 0x7f8228116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 -- 172.21.15.2:0/248537126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82240093f0 con 0x7f82280ff510 2022-03-21T14:07:54.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822c8b5700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f821800fa90 tx=0x7f8218005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:54.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800b4f0 con 0x7f82280ff510 2022-03-21T14:07:54.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f821800dc80 con 0x7f82280ff510 2022-03-21T14:07:54.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8228117070 con 0x7f82280ff510 2022-03-21T14:07:54.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8218011dc0 con 0x7f82280ff510 2022-03-21T14:07:54.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.525+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8228117630 con 0x7f82280ff510 2022-03-21T14:07:54.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.526+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8228109ab0 con 0x7f82280ff510 2022-03-21T14:07:54.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.529+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f821800c040 con 0x7f82280ff510 2022-03-21T14:07:54.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.530+0000 7f821dffb700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 0x7f8214099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:54.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.530+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f82180bfdd0 con 0x7f82280ff510 2022-03-21T14:07:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.530+0000 7f822d0b6700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 0x7f8214099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.531+0000 7f822d0b6700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 0x7f8214099240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f822400f8e0 tx=0x7f8224005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:54.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.532+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f821808d050 con 0x7f82280ff510 2022-03-21T14:07:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:54 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2379237593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:54 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2055516090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.761+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8228109c40 con 0x7f82280ff510 2022-03-21T14:07:54.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.762+0000 7f821dffb700 1 -- 172.21.15.2:0/248537126 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10650 (secure 0 0 0) 0x7f8228109c40 con 0x7f82280ff510 2022-03-21T14:07:54.764 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:07:54.764 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-03-21T14:06:54.837756+0000","modified":"2022-03-21T14:06:59.487834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/1907413140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1907413140},{"type":"v1","addr":"172.21.15.79:6839","nonce":1907413140}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-03-21T14:07:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 msgr2=0x7f8214099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 0x7f8214099240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f822400f8e0 tx=0x7f8224005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 msgr2=0x7f822810f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f821800fa90 tx=0x7f8218005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 shutdown_connections 2022-03-21T14:07:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82280fe270 0x7f822810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8214096d90 0x7f8214099240 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82280ffea0 0x7f8228116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 --2- 172.21.15.2:0/248537126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82280ff510 0x7f822810f490 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 >> 172.21.15.2:0/248537126 conn(0x7f82280f9990 msgr2=0x7f8228102740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 shutdown_connections 2022-03-21T14:07:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.764+0000 7f822e0b8700 1 -- 172.21.15.2:0/248537126 wait complete. 2022-03-21T14:07:54.769 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 20 2022-03-21T14:07:54.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-03-21T14:06:54.837756+0000', 'modified': '2022-03-21T14:06:59.487834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6837/2188072757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6836', 'nonce': 2188072757}, {'type': 'v1', 'addr': '172.21.15.79:6837', 'nonce': 2188072757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.79:6839/1907413140', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6838', 'nonce': 1907413140}, {'type': 'v1', 'addr': '172.21.15.79:6839', 'nonce': 1907413140}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-03-21T14:07:54.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-03-21T14:07:54.792 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-03-21T14:07:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:54 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2379237593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:54 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2055516090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:54 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2379237593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:54 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2055516090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:07:54.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.996+0000 7f2b299aa700 1 -- 172.21.15.2:0/3794255325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fdc00 msgr2=0x7f2b241044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:54.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.996+0000 7f2b299aa700 1 --2- 172.21.15.2:0/3794255325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fdc00 0x7f2b241044c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009c70 tx=0x7f2b1c0094e0 comp rx=0 tx=0).stop 2022-03-21T14:07:54.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 -- 172.21.15.2:0/3794255325 shutdown_connections 2022-03-21T14:07:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 --2- 172.21.15.2:0/3794255325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b240fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 --2- 172.21.15.2:0/3794255325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fdc00 0x7f2b241044c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 --2- 172.21.15.2:0/3794255325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fd270 0x7f2b240fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 -- 172.21.15.2:0/3794255325 >> 172.21.15.2:0/3794255325 conn(0x7f2b240f7780 msgr2=0x7f2b240f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 -- 172.21.15.2:0/3794255325 shutdown_connections 2022-03-21T14:07:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.997+0000 7f2b299aa700 1 -- 172.21.15.2:0/3794255325 wait complete. 2022-03-21T14:07:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 Processor -- start 2022-03-21T14:07:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 -- start start 2022-03-21T14:07:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 0x7f2b2410d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 0x7f2b241140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24114620 con 0x7f2b240fd270 2022-03-21T14:07:55.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24114790 con 0x7f2b240fbfd0 2022-03-21T14:07:55.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b299aa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24114950 con 0x7f2b240fdc00 2022-03-21T14:07:55.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b23fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 0x7f2b2410d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b291a9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 0x7f2b241140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48544/0 (socket says 172.21.15.2:48544) 2022-03-21T14:07:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b289a8700 1 -- 172.21.15.2:0/578922178 learned_addr learned my addr 172.21.15.2:0/578922178 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.998+0000 7f2b291a9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 0x7f2b241140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39806/0 (socket says 172.21.15.2:39806) 2022-03-21T14:07:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 -- 172.21.15.2:0/578922178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 msgr2=0x7f2b241140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 0x7f2b241140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 -- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 msgr2=0x7f2b2410d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 0x7f2b2410d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 -- 172.21.15.2:0/578922178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c009780 con 0x7f2b240fbfd0 2022-03-21T14:07:55.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b23fff700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 0x7f2b2410d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:55.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b289a8700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800f860 tx=0x7f2b18005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:54.999+0000 7f2b217fa700 1 -- 172.21.15.2:0/578922178 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18004020 con 0x7f2b240fbfd0 2022-03-21T14:07:55.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.000+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b24114c30 con 0x7f2b240fbfd0 2022-03-21T14:07:55.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.000+0000 7f2b217fa700 1 -- 172.21.15.2:0/578922178 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b18004180 con 0x7f2b240fbfd0 2022-03-21T14:07:55.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.000+0000 7f2b217fa700 1 -- 172.21.15.2:0/578922178 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180039d0 con 0x7f2b240fbfd0 2022-03-21T14:07:55.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.000+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b24115170 con 0x7f2b240fbfd0 2022-03-21T14:07:55.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.003+0000 7f2b217fa700 1 -- 172.21.15.2:0/578922178 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2b1800c040 con 0x7f2b240fbfd0 2022-03-21T14:07:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.004+0000 7f2b3457db80 1 Processor -- start 2022-03-21T14:07:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 -- start start 2022-03-21T14:07:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b217fa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 0x7f2b140991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 0x55f2c2660760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c2660d80 0x55f2c26611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c25de240 con 0x55f2c2660d80 2022-03-21T14:07:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c252e090 con 0x55f2c265fe20 2022-03-21T14:07:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b3457db80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c25b3bf0 con 0x55f2c265f4d0 2022-03-21T14:07:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.005+0000 7f2b217fa700 1 -- 172.21.15.2:0/578922178 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2b180bfac0 con 0x7f2b240fbfd0 2022-03-21T14:07:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39808/0 (socket says 172.21.15.2:39808) 2022-03-21T14:07:55.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 -- 172.21.15.2:0/1800248839 learned_addr learned my addr 172.21.15.2:0/1800248839 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:55.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b23fff700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 0x55f2c2660760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b291a9700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 0x7f2b140991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b291a9700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c2660d80 0x55f2c26611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 -- 172.21.15.2:0/1800248839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 msgr2=0x55f2c2660760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 0x55f2c2660760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 -- 172.21.15.2:0/1800248839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c2660d80 msgr2=0x55f2c26611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c2660d80 0x55f2c26611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 -- 172.21.15.2:0/1800248839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2c2444360 con 0x55f2c265f4d0 2022-03-21T14:07:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b23fff700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 0x55f2c2660760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:07:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.006+0000 7f2b289a8700 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b180906a0 tx=0x7f2b18011f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10129d4b2bdf2ad1 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b291a9700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 0x7f2b140991c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0095d0 tx=0x7f2b1c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b067fc700 1 -- 172.21.15.2:0/1800248839 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1808cce0 con 0x55f2c265f4d0 2022-03-21T14:07:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b067fc700 1 -- 172.21.15.2:0/1800248839 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b180bd6f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b067fc700 1 -- 172.21.15.2:0/1800248839 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1808ed40 con 0x55f2c265f4d0 2022-03-21T14:07:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 -- 172.21.15.2:0/1800248839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 msgr2=0x55f2c265f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b180906a0 tx=0x7f2b18011f70 comp rx=0 tx=0).stop 2022-03-21T14:07:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 -- 172.21.15.2:0/1800248839 shutdown_connections 2022-03-21T14:07:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c2660d80 0x55f2c26611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c265f8e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 --2- 172.21.15.2:0/1800248839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265fe20 0x55f2c2660760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.007+0000 7f2b3457db80 1 -- 172.21.15.2:0/1800248839 >> 172.21.15.2:0/1800248839 conn(0x55f2c265a340 msgr2=0x55f2c265a730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.008+0000 7f2b3457db80 1 -- 172.21.15.2:0/1800248839 shutdown_connections 2022-03-21T14:07:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.008+0000 7f2b3457db80 1 -- 172.21.15.2:0/1800248839 wait complete. 2022-03-21T14:07:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.008+0000 7f2b3457db80 1 -- start start 2022-03-21T14:07:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.008+0000 7f2b3457db80 1 Processor -- start 2022-03-21T14:07:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c26692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c2670310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c25adcd0 con 0x55f2c265fe20 2022-03-21T14:07:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c2446650 con 0x55f2c2660d80 2022-03-21T14:07:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c2670940 con 0x55f2c265f4d0 2022-03-21T14:07:55.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b3457db80 10 client.0 _ll_register_callbacks cb 0x55f2c23bf7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-03-21T14:07:55.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b23fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c26692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39810/0 (socket says 172.21.15.2:39810) 2022-03-21T14:07:55.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b289a8700 1 -- 172.21.15.2:0/3313482515 learned_addr learned my addr 172.21.15.2:0/3313482515 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:55.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b23fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c26692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43960/0 (socket says 172.21.15.2:43960) 2022-03-21T14:07:55.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.009+0000 7f2b291a9700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c2670310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 -- 172.21.15.2:0/3313482515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 msgr2=0x55f2c2670310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c2670310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 -- 172.21.15.2:0/3313482515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 msgr2=0x55f2c26692b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c26692b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 -- 172.21.15.2:0/3313482515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1000bd20 con 0x55f2c265f4d0 2022-03-21T14:07:55.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b291a9700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c2670310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:55.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b289a8700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800fe50 tx=0x7f2b18011f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddc71cb5ff42e296 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2aff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2022-03-21T14:07:55.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180042f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mon.2 v2:172.21.15.79:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2b180666f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2c2652ac0 con 0x55f2c265f4d0 2022-03-21T14:07:55.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b180bd1a0 con 0x55f2c265f4d0 2022-03-21T14:07:55.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.010+0000 7f2b3457db80 10 client.24908 fetch_fsmap learned FSMap version 20 2022-03-21T14:07:55.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7f2b24115170 con 0x55f2c265f4d0 2022-03-21T14:07:55.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mon.2 v2:172.21.15.79:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55f2c2652ac0 con 0x55f2c265f4d0 2022-03-21T14:07:55.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mon.2 v2:172.21.15.79:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f2b18096c10 con 0x55f2c265f4d0 2022-03-21T14:07:55.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 10 client.24908 fetch_fsmap finished waiting for FSMap version 20 2022-03-21T14:07:55.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 10 client.24908 resolve_mds: resolved name 'l' to daemon mds.l 2022-03-21T14:07:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 20 client.24908 populate_metadata read hostname 'smithi002' 2022-03-21T14:07:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c2673b80 0x55f2c2678e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 4 client.24908 mds_command: new command op to 24587 tid=0[{"prefix": "get_command_descriptions"}] 2022-03-21T14:07:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 --> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2c262f640 con 0x55f2c2673b80 2022-03-21T14:07:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.011+0000 7f2b23fff700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c2673b80 0x55f2c2678e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.012+0000 7f2b23fff700 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c2673b80 0x55f2c2678e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f6e546d2b2f739 server_cookie=d11a6ef2f86dd238 in_seq=0 out_seq=0 2022-03-21T14:07:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.012+0000 7f2aff7fe700 10 client.24908 ms_handle_connect on v2:172.21.15.79:6838/1907413140 2022-03-21T14:07:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2aff7fe700 1 -- 172.21.15.2:0/3313482515 <== mds.0 v2:172.21.15.79:6838/1907413140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17246 (crc 0 0 0) 0x55f2c262f640 con 0x55f2c2673b80 2022-03-21T14:07:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2aff7fe700 10 client.24908 handle_command_reply: tid=0 2022-03-21T14:07:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 client.24908 shutdown 2022-03-21T14:07:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 msgr2=0x55f2c2668d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800fe50 tx=0x7f2b18011f70 comp rx=0 tx=0).stop 2022-03-21T14:07:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 shutdown_connections 2022-03-21T14:07:55.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c2673b80 0x55f2c2678e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c2670310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2668d70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 --2- 172.21.15.2:0/3313482515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c26692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2aff7fe700 0 client.24908 ms_handle_reset on v2:172.21.15.79:6838/1907413140 2022-03-21T14:07:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 >> 172.21.15.2:0/3313482515 conn(0x55f2c265a340 msgr2=0x55f2c2659600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 shutdown_connections 2022-03-21T14:07:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 1 -- 172.21.15.2:0/3313482515 wait complete. 2022-03-21T14:07:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.013+0000 7f2b3457db80 20 client.24908 trim_cache size 0 max 16384 2022-03-21T14:07:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 Processor -- start 2022-03-21T14:07:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 -- start start 2022-03-21T14:07:55.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2661150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c267ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c2565780 con 0x55f2c265fe20 2022-03-21T14:07:55.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c255c140 con 0x55f2c2660d80 2022-03-21T14:07:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.031+0000 7f2b3457db80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c2667d10 con 0x55f2c265f4d0 2022-03-21T14:07:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2661150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43964/0 (socket says 172.21.15.2:43964) 2022-03-21T14:07:55.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 -- 172.21.15.2:0/2195856740 learned_addr learned my addr 172.21.15.2:0/2195856740 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:55.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b291a9700 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c267ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 -- 172.21.15.2:0/2195856740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 msgr2=0x55f2c2661150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2661150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 -- 172.21.15.2:0/2195856740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 msgr2=0x55f2c267ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c267ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 -- 172.21.15.2:0/2195856740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2c23eefe0 con 0x55f2c265fe20 2022-03-21T14:07:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.032+0000 7f2b23fff700 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b10005fd0 tx=0x7f2b10005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e03be8d579946e8 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b06ffd700 1 -- 172.21.15.2:0/2195856740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10017070 con 0x55f2c265fe20 2022-03-21T14:07:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b06ffd700 1 -- 172.21.15.2:0/2195856740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b10018040 con 0x55f2c265fe20 2022-03-21T14:07:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b06ffd700 1 -- 172.21.15.2:0/2195856740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10004b30 con 0x55f2c265fe20 2022-03-21T14:07:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 -- 172.21.15.2:0/2195856740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 msgr2=0x55f2c265a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b10005fd0 tx=0x7f2b10005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 -- 172.21.15.2:0/2195856740 shutdown_connections 2022-03-21T14:07:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c2660d80 0x55f2c267ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c265f4d0 0x55f2c2661150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 --2- 172.21.15.2:0/2195856740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c265a710 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 -- 172.21.15.2:0/2195856740 >> 172.21.15.2:0/2195856740 conn(0x55f2c265a340 msgr2=0x55f2c2659600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 -- 172.21.15.2:0/2195856740 shutdown_connections 2022-03-21T14:07:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.033+0000 7f2b3457db80 1 -- 172.21.15.2:0/2195856740 wait complete. 2022-03-21T14:07:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 -- start start 2022-03-21T14:07:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 Processor -- start 2022-03-21T14:07:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c266d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c2660d80 0x55f2c266dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c26ca120 con 0x55f2c265fe20 2022-03-21T14:07:55.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.034+0000 7f2b3457db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c26ca290 con 0x55f2c265f4d0 2022-03-21T14:07:55.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b3457db80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2c26ca450 con 0x55f2c2660d80 2022-03-21T14:07:55.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b3457db80 10 client.0 _ll_register_callbacks cb 0x55f2c23bf7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-03-21T14:07:55.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48552/0 (socket says 172.21.15.2:48552) 2022-03-21T14:07:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 -- 172.21.15.2:0/4136243599 learned_addr learned my addr 172.21.15.2:0/4136243599 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b23fff700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c266d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b291a9700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c2660d80 0x55f2c266dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 -- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c2660d80 msgr2=0x55f2c266dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c2660d80 0x55f2c266dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 -- 172.21.15.2:0/4136243599 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 msgr2=0x55f2c266d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:07:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c266d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.035+0000 7f2b289a8700 1 -- 172.21.15.2:0/4136243599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b10011770 con 0x55f2c265f4d0 2022-03-21T14:07:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2b289a8700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2b18090470 tx=0x7f2b18011f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f2202d8e74c4c80 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2022-03-21T14:07:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180aa060 con 0x55f2c265f4d0 2022-03-21T14:07:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b180bd6f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mon.1 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2b180042f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b18090130 con 0x55f2c265f4d0 2022-03-21T14:07:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2b3457db80 10 client.24904 fetch_fsmap learned FSMap version 20 2022-03-21T14:07:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f2c2416f80 con 0x55f2c265f4d0 2022-03-21T14:07:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mon.1 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b180bd6f0 con 0x55f2c265f4d0 2022-03-21T14:07:55.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mon.1 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f2b180950d0 con 0x55f2c265f4d0 2022-03-21T14:07:55.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.036+0000 7f2b3457db80 10 client.24904 fetch_fsmap finished waiting for FSMap version 20 2022-03-21T14:07:55.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b3457db80 10 client.24904 resolve_mds: resolved name 'l' to daemon mds.l 2022-03-21T14:07:55.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b3457db80 20 client.24904 populate_metadata read hostname 'smithi002' 2022-03-21T14:07:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c26ceb90 0x55f2c26cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b3457db80 4 client.24904 mds_command: new command op to 24587 tid=0[{"prefix": "status"}] 2022-03-21T14:07:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 --> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2c256b5f0 con 0x55f2c26ceb90 2022-03-21T14:07:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b23fff700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c26ceb90 0x55f2c26cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2b23fff700 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c26ceb90 0x55f2c26cb500 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62b11b83288577a server_cookie=619ee4c807022631 in_seq=0 out_seq=0 2022-03-21T14:07:55.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.037+0000 7f2aff7fe700 10 client.24904 ms_handle_connect on v2:172.21.15.79:6838/1907413140 2022-03-21T14:07:55.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2aff7fe700 1 -- 172.21.15.2:0/4136243599 <== mds.0 v2:172.21.15.79:6838/1907413140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f2c256b5f0 con 0x55f2c26ceb90 2022-03-21T14:07:55.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2aff7fe700 10 client.24904 handle_command_reply: tid=0 2022-03-21T14:07:55.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 client.24904 shutdown 2022-03-21T14:07:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 msgr2=0x55f2c266d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2b18090470 tx=0x7f2b18011f70 comp rx=0 tx=0).stop 2022-03-21T14:07:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 shutdown_connections 2022-03-21T14:07:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x55f2c26ceb90 0x55f2c26cb500 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2c265f4d0 0x55f2c266d1e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2aff7fe700 0 client.24904 ms_handle_reset on v2:172.21.15.79:6838/1907413140 2022-03-21T14:07:55.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2c2660d80 0x55f2c266dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 --2- 172.21.15.2:0/4136243599 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f2c265fe20 0x55f2c266d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 >> 172.21.15.2:0/4136243599 conn(0x55f2c265a340 msgr2=0x55f2c265a710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 shutdown_connections 2022-03-21T14:07:55.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 1 -- 172.21.15.2:0/4136243599 wait complete. 2022-03-21T14:07:55.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.038+0000 7f2b3457db80 20 client.24904 trim_cache size 0 max 16384 2022-03-21T14:07:55.065 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T14:07:55.065 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "ee5e78f2-a91e-11ec-8c35-001a4aab830c", 2022-03-21T14:07:55.065 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2022-03-21T14:07:55.065 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24587, 2022-03-21T14:07:55.065 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-03-21T14:07:55.066 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-03-21T14:07:55.066 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-03-21T14:07:55.066 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 60.197968314000001, 2022-03-21T14:07:55.066 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 20, 2022-03-21T14:07:55.067 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 84, 2022-03-21T14:07:55.067 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 82, 2022-03-21T14:07:55.067 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 67.886411777000006 2022-03-21T14:07:55.067 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T14:07:55.069 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 msgr2=0x7f2b140991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.069 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 0x7f2b140991c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0095d0 tx=0x7f2b1c019040 comp rx=0 tx=0).stop 2022-03-21T14:07:55.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 msgr2=0x7f2b2410cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:55.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800f860 tx=0x7f2b18005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 shutdown_connections 2022-03-21T14:07:55.070 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b240fbfd0 0x7f2b2410cb10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2b14096d10 0x7f2b140991c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b240fdc00 0x7f2b241140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 --2- 172.21.15.2:0/578922178 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b240fd270 0x7f2b2410d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 >> 172.21.15.2:0/578922178 conn(0x7f2b240f7780 msgr2=0x7f2b240f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 shutdown_connections 2022-03-21T14:07:55.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:55.039+0000 7f2b299aa700 1 -- 172.21.15.2:0/578922178 wait complete. 2022-03-21T14:07:55.072 INFO:tasks.cephadm:Setup complete, yielding 2022-03-21T14:07:55.072 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-03-21T14:07:55.082 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2022-03-21T14:07:55.083 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph orch status' 2022-03-21T14:07:55.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:55 smithi002 ceph-mon[17167]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:55.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:55 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/248537126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:07:55.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:07:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:55 smithi016 ceph-mon[22010]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:55 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/248537126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:07:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:07:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:55 smithi079 ceph-mon[23158]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:55 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/248537126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:07:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:07:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:07:56.717 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:07:57.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 -- 172.21.15.2:0/315732183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fbde0 msgr2=0x7f6e180fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:57.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/315732183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fbde0 0x7f6e180fc1f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0098e0 tx=0x7f6e0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:07:57.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 -- 172.21.15.2:0/315732183 shutdown_connections 2022-03-21T14:07:57.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/315732183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fbde0 0x7f6e180fc1f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/315732183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fd990 0x7f6e181042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/315732183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fcfe0 0x7f6e180fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.485+0000 7f6e1dc27700 1 -- 172.21.15.2:0/315732183 >> 172.21.15.2:0/315732183 conn(0x7f6e180f75f0 msgr2=0x7f6e180f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:57.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.486+0000 7f6e1dc27700 1 -- 172.21.15.2:0/315732183 shutdown_connections 2022-03-21T14:07:57.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.486+0000 7f6e1dc27700 1 -- 172.21.15.2:0/315732183 wait complete. 2022-03-21T14:07:57.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 Processor -- start 2022-03-21T14:07:57.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 -- start start 2022-03-21T14:07:57.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 0x7f6e18070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:57.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:57.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fd990 0x7f6e1806f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:57.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1806ff70 con 0x7f6e180fbde0 2022-03-21T14:07:57.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180700e0 con 0x7f6e180fcfe0 2022-03-21T14:07:57.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e1dc27700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18116c90 con 0x7f6e180fd990 2022-03-21T14:07:57.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.487+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:57.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e1d426700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fd990 0x7f6e1806f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:57.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48556/0 (socket says 172.21.15.2:48556) 2022-03-21T14:07:57.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 -- 172.21.15.2:0/786529247 learned_addr learned my addr 172.21.15.2:0/786529247 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:07:57.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e1cc25700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 0x7f6e18070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:57.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 -- 172.21.15.2:0/786529247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fd990 msgr2=0x7f6e1806f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:57.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fd990 0x7f6e1806f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 -- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 msgr2=0x7f6e18070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:57.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 0x7f6e18070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 -- 172.21.15.2:0/786529247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c0093f0 con 0x7f6e180fcfe0 2022-03-21T14:07:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e1cc25700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 0x7f6e18070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:07:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.488+0000 7f6e17fff700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400fa90 tx=0x7f6e04005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.489+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400be00 con 0x7f6e180fcfe0 2022-03-21T14:07:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.489+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6e0400de30 con 0x7f6e180fcfe0 2022-03-21T14:07:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.489+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e18116e90 con 0x7f6e180fcfe0 2022-03-21T14:07:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.489+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04011e80 con 0x7f6e180fcfe0 2022-03-21T14:07:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.489+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e18117340 con 0x7f6e180fcfe0 2022-03-21T14:07:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.490+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e18107610 con 0x7f6e180fcfe0 2022-03-21T14:07:57.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.492+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f6e0400c040 con 0x7f6e180fcfe0 2022-03-21T14:07:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.493+0000 7f6e157fa700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 0x7f6e08099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:07:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.494+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f6e040bf8a0 con 0x7f6e180fcfe0 2022-03-21T14:07:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.494+0000 7f6e1cc25700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 0x7f6e08099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:07:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.494+0000 7f6e1cc25700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 0x7f6e08099240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009f50 tx=0x7f6e0c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:07:57.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.497+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f6e0408cb20 con 0x7f6e180fcfe0 2022-03-21T14:07:57.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.737+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e1802ac70 con 0x7f6e08096d90 2022-03-21T14:07:57.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.738+0000 7f6e157fa700 1 -- 172.21.15.2:0/786529247 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6e1802ac70 con 0x7f6e08096d90 2022-03-21T14:07:57.740 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2022-03-21T14:07:57.740 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2022-03-21T14:07:57.740 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2022-03-21T14:07:57.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.740+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 msgr2=0x7f6e08099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:57.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.740+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 0x7f6e08099240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009f50 tx=0x7f6e0c01a040 comp rx=0 tx=0).stop 2022-03-21T14:07:57.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.740+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 msgr2=0x7f6e1806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:07:57.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.740+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400fa90 tx=0x7f6e04005eb0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 shutdown_connections 2022-03-21T14:07:57.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f6e08096d90 0x7f6e08099240 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e180fd990 0x7f6e1806f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e180fbde0 0x7f6e18070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 --2- 172.21.15.2:0/786529247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e180fcfe0 0x7f6e1806f320 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:07:57.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 >> 172.21.15.2:0/786529247 conn(0x7f6e180f75f0 msgr2=0x7f6e181002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:07:57.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 shutdown_connections 2022-03-21T14:07:57.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:07:57.741+0000 7f6e1dc27700 1 -- 172.21.15.2:0/786529247 wait complete. 2022-03-21T14:07:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:57 smithi002 ceph-mon[17167]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:57 smithi016 ceph-mon[22010]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:57 smithi079 ceph-mon[23158]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:07:58.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph orch ps' 2022-03-21T14:07:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:58 smithi016 ceph-mon[22010]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:07:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:58 smithi079 ceph-mon[23158]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:07:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:07:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:58 smithi002 ceph-mon[17167]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:07:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:07:59 smithi016 ceph-mon[22010]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:59.945 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:07:59 smithi002 ceph-mon[17167]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:07:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:07:59 smithi079 ceph-mon[23158]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:00.069 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:00.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.909+0000 7fe215225700 1 -- 172.21.15.2:0/949767567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 msgr2=0x7fe2100fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:00.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.909+0000 7fe215225700 1 --2- 172.21.15.2:0/949767567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe2100fc1f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe2000098e0 tx=0x7fe200005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:00.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 -- 172.21.15.2:0/949767567 shutdown_connections 2022-03-21T14:08:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 --2- 172.21.15.2:0/949767567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 0x7fe2101042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 --2- 172.21.15.2:0/949767567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe2100fc1f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 --2- 172.21.15.2:0/949767567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe2100fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 -- 172.21.15.2:0/949767567 >> 172.21.15.2:0/949767567 conn(0x7fe2100f75f0 msgr2=0x7fe2100f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:00.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 -- 172.21.15.2:0/949767567 shutdown_connections 2022-03-21T14:08:00.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.910+0000 7fe215225700 1 -- 172.21.15.2:0/949767567 wait complete. 2022-03-21T14:08:00.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 Processor -- start 2022-03-21T14:08:00.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 -- start start 2022-03-21T14:08:00.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:00.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe21010f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:00.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 0x7fe210116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:00.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210116860 con 0x7fe2100fbde0 2022-03-21T14:08:00.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101169d0 con 0x7fe2100fcfe0 2022-03-21T14:08:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.911+0000 7fe215225700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210116b90 con 0x7fe2100fd990 2022-03-21T14:08:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe21010f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43974/0 (socket says 172.21.15.2:43974) 2022-03-21T14:08:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20ffff700 1 -- 172.21.15.2:0/1397064003 learned_addr learned my addr 172.21.15.2:0/1397064003 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:00.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe214a24700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 0x7fe210116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:00.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 -- 172.21.15.2:0/1397064003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 msgr2=0x7fe210116320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:00.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 0x7fe210116320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:00.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 -- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 msgr2=0x7fe21010ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:00.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:00.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20f7fe700 1 -- 172.21.15.2:0/1397064003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2000093f0 con 0x7fe2100fcfe0 2022-03-21T14:08:00.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.912+0000 7fe20ffff700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:00.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe20f7fe700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe21010f290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe20400f830 tx=0x7fe204005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:00.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204005020 con 0x7fe2100fcfe0 2022-03-21T14:08:00.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe204011720 con 0x7fe2100fcfe0 2022-03-21T14:08:00.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204004a60 con 0x7fe2100fcfe0 2022-03-21T14:08:00.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe210116e70 con 0x7fe2100fcfe0 2022-03-21T14:08:00.922 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.913+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2101172d0 con 0x7fe2100fcfe0 2022-03-21T14:08:00.922 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.914+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2101075c0 con 0x7fe2100fcfe0 2022-03-21T14:08:00.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.918+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fe204018030 con 0x7fe2100fcfe0 2022-03-21T14:08:00.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.919+0000 7fe20cff9700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 0x7fe1f8099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:00.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.919+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fe2040c0010 con 0x7fe2100fcfe0 2022-03-21T14:08:00.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.919+0000 7fe20ffff700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 0x7fe1f8099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:00.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.920+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fe20406ef40 con 0x7fe2100fcfe0 2022-03-21T14:08:00.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:00.920+0000 7fe20ffff700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 0x7fe1f8099240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe20001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:01.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.119+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe210103b50 con 0x7fe1f8096d90 2022-03-21T14:08:01.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.132+0000 7fe20cff9700 1 -- 172.21.15.2:0/1397064003 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4061 (secure 0 0 0) 0x7fe210103b50 con 0x7fe1f8096d90 2022-03-21T14:08:01.134 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-03-21T14:08:01.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.a smithi002 running (104s) 1s ago 103s - - 17.0.0-11208-gd3799b8c 024bbad3f2a7 d91edd3edd02 2022-03-21T14:08:01.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.b smithi016 running (90s) 3s ago 89s 14.0M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 1162df6bc57e 2022-03-21T14:08:01.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.c smithi079 running (81s) 3s ago 80s 13.6M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 2cc4d73ec679 2022-03-21T14:08:01.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.d smithi002 running (100s) 1s ago 100s - - 17.0.0-11208-gd3799b8c 024bbad3f2a7 6bc40f5ef0d7 2022-03-21T14:08:01.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.e smithi016 running (87s) 3s ago 87s 15.4M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 71cfba316875 2022-03-21T14:08:01.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.f smithi079 running (78s) 3s ago 78s 16.1M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 d3558fe48edf 2022-03-21T14:08:01.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.g smithi002 running (96s) 1s ago 96s - - 17.0.0-11208-gd3799b8c 024bbad3f2a7 23ee8a62fac3 2022-03-21T14:08:01.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.h smithi016 running (85s) 3s ago 85s 14.0M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 5949428548d3 2022-03-21T14:08:01.137 INFO:teuthology.orchestra.run.smithi002.stdout:mds.i smithi079 running (76s) 3s ago 76s 18.5M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 9cdefcb8a7ea 2022-03-21T14:08:01.137 INFO:teuthology.orchestra.run.smithi002.stdout:mds.j smithi002 running (93s) 1s ago 92s - - 17.0.0-11208-gd3799b8c 024bbad3f2a7 d4ba25ec22af 2022-03-21T14:08:01.137 INFO:teuthology.orchestra.run.smithi002.stdout:mds.k smithi016 running (83s) 3s ago 82s 13.1M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 19b39b3e2972 2022-03-21T14:08:01.137 INFO:teuthology.orchestra.run.smithi002.stdout:mds.l smithi079 running (74s) 3s ago 73s 20.6M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 f62ac7f0d07e 2022-03-21T14:08:01.137 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi002 *:9283 running (8m) 1s ago 8m - - 17.0.0-11208-gd3799b8c 024bbad3f2a7 73577d22a6e5 2022-03-21T14:08:01.138 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi016 *:8443 running (6m) 3s ago 6m 422M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 e1000d7d6b12 2022-03-21T14:08:01.138 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi079 *:8443 running (6m) 3s ago 6m 419M - 17.0.0-11208-gd3799b8c 024bbad3f2a7 7ea299894367 2022-03-21T14:08:01.138 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (8m) 1s ago 8m - 2048M 17.0.0-11208-gd3799b8c 024bbad3f2a7 a9136a71a95b 2022-03-21T14:08:01.138 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi016 running (7m) 3s ago 7m 63.8M 2048M 17.0.0-11208-gd3799b8c 024bbad3f2a7 263ca86d67cf 2022-03-21T14:08:01.139 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi079 running (6m) 3s ago 6m 63.9M 2048M 17.0.0-11208-gd3799b8c 024bbad3f2a7 e4669b1f1575 2022-03-21T14:08:01.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 1s ago 6m - 4319M 17.0.0-11208-gd3799b8c 024bbad3f2a7 95a95147c402 2022-03-21T14:08:01.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi016 running (5m) 3s ago 5m 66.0M 4294M 17.0.0-11208-gd3799b8c 024bbad3f2a7 b20dbc2a7945 2022-03-21T14:08:01.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 smithi016 running (2m) 3s ago 2m 59.7M 4294M 17.0.0-11208-gd3799b8c 024bbad3f2a7 48998e08f06c 2022-03-21T14:08:01.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 smithi079 running (2m) 3s ago 2m 53.5M 4293M 17.0.0-11208-gd3799b8c 024bbad3f2a7 fcbe0b58120e 2022-03-21T14:08:01.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi079 running (5m) 3s ago 5m 63.7M 4293M 17.0.0-11208-gd3799b8c 024bbad3f2a7 43b5d645e8ad 2022-03-21T14:08:01.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (5m) 1s ago 5m - 4319M 17.0.0-11208-gd3799b8c 024bbad3f2a7 15fd6efa8f80 2022-03-21T14:08:01.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi016 running (4m) 3s ago 4m 53.2M 4294M 17.0.0-11208-gd3799b8c 024bbad3f2a7 3d7f32d74f97 2022-03-21T14:08:01.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi079 running (4m) 3s ago 4m 58.8M 4293M 17.0.0-11208-gd3799b8c 024bbad3f2a7 1f9534bf2830 2022-03-21T14:08:01.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi002 running (3m) 1s ago 3m - 4319M 17.0.0-11208-gd3799b8c 024bbad3f2a7 f1f6daa7755d 2022-03-21T14:08:01.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi016 running (3m) 3s ago 3m 57.9M 4294M 17.0.0-11208-gd3799b8c 024bbad3f2a7 06d975959529 2022-03-21T14:08:01.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 smithi079 running (3m) 3s ago 3m 57.1M 4293M 17.0.0-11208-gd3799b8c 024bbad3f2a7 afae1c95c46a 2022-03-21T14:08:01.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 smithi002 running (2m) 1s ago 2m - 4319M 17.0.0-11208-gd3799b8c 024bbad3f2a7 2fac45d96a6f 2022-03-21T14:08:01.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 msgr2=0x7fe1f8099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 0x7fe1f8099240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe20001a040 comp rx=0 tx=0).stop 2022-03-21T14:08:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 msgr2=0x7fe21010f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe21010f290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe20400f830 tx=0x7fe204005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 shutdown_connections 2022-03-21T14:08:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fe1f8096d90 0x7fe1f8099240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:01.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2100fd990 0x7fe210116320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:01.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2100fbde0 0x7fe21010ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:01.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 --2- 172.21.15.2:0/1397064003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2100fcfe0 0x7fe21010f290 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:01.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 >> 172.21.15.2:0/1397064003 conn(0x7fe2100f75f0 msgr2=0x7fe2100f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:01.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 shutdown_connections 2022-03-21T14:08:01.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:01.135+0000 7fe215225700 1 -- 172.21.15.2:0/1397064003 wait complete. 2022-03-21T14:08:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:01.344 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:01.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:01.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph orch ls' 2022-03-21T14:08:02.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:01 smithi002 ceph-mon[17167]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:02.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:01 smithi016 ceph-mon[22010]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:01 smithi079 ceph-mon[23158]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:08:03.082 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:03 smithi002 ceph-mon[17167]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:03 smithi016 ceph-mon[22010]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:03 smithi079 ceph-mon[23158]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:03.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4001203326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 msgr2=0x7fbb040fe560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:03.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4001203326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb040fe560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbb000098e0 tx=0x7fbb00005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:03.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4001203326 shutdown_connections 2022-03-21T14:08:03.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4001203326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb04106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4001203326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb040fe560 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4001203326 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 0x7fbb040ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4001203326 >> 172.21.15.2:0/4001203326 conn(0x7fbb040f9960 msgr2=0x7fbb040fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:03.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4001203326 shutdown_connections 2022-03-21T14:08:03.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.623+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4001203326 wait complete. 2022-03-21T14:08:03.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 Processor -- start 2022-03-21T14:08:03.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 -- start start 2022-03-21T14:08:03.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:03.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 0x7fbb0410f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:03.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb041164c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:03.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04116a00 con 0x7fbb040fe150 2022-03-21T14:08:03.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04116b70 con 0x7fbb040ffd80 2022-03-21T14:08:03.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.624+0000 7fbb0c31a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb04116d30 con 0x7fbb040ff3f0 2022-03-21T14:08:03.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a0b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:03.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb041164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:03.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a0b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43978/0 (socket says 172.21.15.2:43978) 2022-03-21T14:08:03.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a0b6700 1 -- 172.21.15.2:0/4243403502 learned_addr learned my addr 172.21.15.2:0/4243403502 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:03.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb098b5700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 0x7fbb0410f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:03.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 -- 172.21.15.2:0/4243403502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 msgr2=0x7fbb0410f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:03.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 0x7fbb0410f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 -- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 msgr2=0x7fbb0410eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:03.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a8b7700 1 -- 172.21.15.2:0/4243403502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000093f0 con 0x7fbb040ffd80 2022-03-21T14:08:03.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.625+0000 7fbb0a0b6700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:03.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbb0a8b7700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb041164c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00eb50 tx=0x7fbafc009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:03.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc00c9d0 con 0x7fbb040ffd80 2022-03-21T14:08:03.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbafc003840 con 0x7fbb040ffd80 2022-03-21T14:08:03.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb04116fb0 con 0x7fbb040ffd80 2022-03-21T14:08:03.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0044d0 con 0x7fbb040ffd80 2022-03-21T14:08:03.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.626+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb041174a0 con 0x7fbb040ffd80 2022-03-21T14:08:03.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.627+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb04109990 con 0x7fbb040ffd80 2022-03-21T14:08:03.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.629+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fbafc00c040 con 0x7fbb040ffd80 2022-03-21T14:08:03.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.631+0000 7fbafaffd700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 0x7fbaf00991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:03.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.631+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fbafc014070 con 0x7fbb040ffd80 2022-03-21T14:08:03.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.631+0000 7fbb0a0b6700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 0x7fbaf00991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.631+0000 7fbb0a0b6700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 0x7fbaf00991c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000f8e0 tx=0x7fbb0001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.632+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fbafc08d030 con 0x7fbb040ffd80 2022-03-21T14:08:03.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.830+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb0405d410 con 0x7fbaf0096d10 2022-03-21T14:08:03.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.836+0000 7fbafaffd700 1 -- 172.21.15.2:0/4243403502 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7fbb0405d410 con 0x7fbaf0096d10 2022-03-21T14:08:03.838 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-21T14:08:03.838 INFO:teuthology.orchestra.run.smithi002.stdout:mds.all 12/12 6s ago 110s smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi079=c;smithi079=f;smithi079=i;smithi079=l;count:12 2022-03-21T14:08:03.838 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 6s ago 6m smithi002=x;smithi016=y;smithi079=z;count:3 2022-03-21T14:08:03.839 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 6s ago 7m smithi002:172.21.15.2=a;smithi016:172.21.15.16=b;smithi079:172.21.15.79=c;count:3 2022-03-21T14:08:03.839 INFO:teuthology.orchestra.run.smithi002.stdout:osd 12 6s ago - 2022-03-21T14:08:03.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 msgr2=0x7fbaf00991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:03.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 0x7fbaf00991c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000f8e0 tx=0x7fbb0001a040 comp rx=0 tx=0).stop 2022-03-21T14:08:03.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 msgr2=0x7fbb041164c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:03.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb041164c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00eb50 tx=0x7fbafc009430 comp rx=0 tx=0).stop 2022-03-21T14:08:03.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 shutdown_connections 2022-03-21T14:08:03.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb040ffd80 0x7fbb041164c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fbaf0096d10 0x7fbaf00991c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb040fe150 0x7fbb0410eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 --2- 172.21.15.2:0/4243403502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb040ff3f0 0x7fbb0410f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:03.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 >> 172.21.15.2:0/4243403502 conn(0x7fbb040f9960 msgr2=0x7fbb040fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:03.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 shutdown_connections 2022-03-21T14:08:03.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:03.838+0000 7fbb0c31a700 1 -- 172.21.15.2:0/4243403502 wait complete. 2022-03-21T14:08:04.282 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:04 smithi002 ceph-mon[17167]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:04.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph orch host ls' 2022-03-21T14:08:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:04 smithi016 ceph-mon[22010]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:04 smithi079 ceph-mon[23158]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:05.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:05 smithi002 ceph-mon[17167]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:05 smithi016 ceph-mon[22010]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:05 smithi079 ceph-mon[23158]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:05.799 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:06.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 -- 172.21.15.2:0/2797938565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fbf50 msgr2=0x7faf1c0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:06.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 --2- 172.21.15.2:0/2797938565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fbf50 0x7faf1c0fc360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0098e0 tx=0x7faf0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:06.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 -- 172.21.15.2:0/2797938565 shutdown_connections 2022-03-21T14:08:06.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 --2- 172.21.15.2:0/2797938565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fbf50 0x7faf1c0fc360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 --2- 172.21.15.2:0/2797938565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fdb80 0x7faf1c104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 --2- 172.21.15.2:0/2797938565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fd1f0 0x7faf1c0fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 -- 172.21.15.2:0/2797938565 >> 172.21.15.2:0/2797938565 conn(0x7faf1c0f7720 msgr2=0x7faf1c0f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:06.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 -- 172.21.15.2:0/2797938565 shutdown_connections 2022-03-21T14:08:06.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.334+0000 7faf211b4700 1 -- 172.21.15.2:0/2797938565 wait complete. 2022-03-21T14:08:06.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.335+0000 7faf211b4700 1 Processor -- start 2022-03-21T14:08:06.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 -- start start 2022-03-21T14:08:06.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:06.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fd1f0 0x7faf1c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:06.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fdb80 0x7faf1c11aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:06.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c11b010 con 0x7faf1c0fdb80 2022-03-21T14:08:06.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c11b180 con 0x7faf1c0fd1f0 2022-03-21T14:08:06.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf211b4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c11b340 con 0x7faf1c0fbf50 2022-03-21T14:08:06.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf1ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:06.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf1ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39832/0 (socket says 172.21.15.2:39832) 2022-03-21T14:08:06.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf1ad9d700 1 -- 172.21.15.2:0/3863427814 learned_addr learned my addr 172.21.15.2:0/3863427814 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:06.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf1a59c700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fd1f0 0x7faf1c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:06.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.336+0000 7faf1b59e700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fdb80 0x7faf1c11aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:06.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 -- 172.21.15.2:0/3863427814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fd1f0 msgr2=0x7faf1c113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:06.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fd1f0 0x7faf1c113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 -- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fdb80 msgr2=0x7faf1c11aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:06.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fdb80 0x7faf1c11aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 -- 172.21.15.2:0/3863427814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c0093f0 con 0x7faf1c0fbf50 2022-03-21T14:08:06.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.337+0000 7faf1ad9d700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0093c0 tx=0x7faf0c005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:06.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.338+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c01d070 con 0x7faf1c0fbf50 2022-03-21T14:08:06.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.338+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7faf0c005740 con 0x7faf1c0fbf50 2022-03-21T14:08:06.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.338+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c017950 con 0x7faf1c0fbf50 2022-03-21T14:08:06.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.338+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c11b5c0 con 0x7faf1c0fbf50 2022-03-21T14:08:06.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.338+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c11ba50 con 0x7faf1c0fbf50 2022-03-21T14:08:06.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.339+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c107740 con 0x7faf1c0fbf50 2022-03-21T14:08:06.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.341+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7faf0c028030 con 0x7faf1c0fbf50 2022-03-21T14:08:06.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.342+0000 7faf037fe700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 0x7faf04099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:06.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.342+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7faf0c0c0890 con 0x7faf1c0fbf50 2022-03-21T14:08:06.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.342+0000 7faf1a59c700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 0x7faf04099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:06.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.343+0000 7faf1a59c700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 0x7faf04099240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf10005fd0 tx=0x7faf10005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:06.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.344+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7faf0c08dab0 con 0x7faf1c0fbf50 2022-03-21T14:08:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:06 smithi002 ceph-mon[17167]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:06 smithi016 ceph-mon[22010]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:06 smithi079 ceph-mon[23158]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:06.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.538+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faf1c02aaf0 con 0x7faf04096d90 2022-03-21T14:08:06.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.539+0000 7faf037fe700 1 -- 172.21.15.2:0/3863427814 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7faf1c02aaf0 con 0x7faf04096d90 2022-03-21T14:08:06.540 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2022-03-21T14:08:06.540 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2022-03-21T14:08:06.540 INFO:teuthology.orchestra.run.smithi002.stdout:smithi016 172.21.15.16 2022-03-21T14:08:06.541 INFO:teuthology.orchestra.run.smithi002.stdout:smithi079 172.21.15.79 2022-03-21T14:08:06.541 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2022-03-21T14:08:06.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 msgr2=0x7faf04099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:06.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 0x7faf04099240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf10005fd0 tx=0x7faf10005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 msgr2=0x7faf1c113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:06.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0093c0 tx=0x7faf0c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:06.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 shutdown_connections 2022-03-21T14:08:06.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.541+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7faf04096d90 0x7faf04099240 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf1c0fdb80 0x7faf1c11aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf1c0fbf50 0x7faf1c113500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 --2- 172.21.15.2:0/3863427814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf1c0fd1f0 0x7faf1c113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:06.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 >> 172.21.15.2:0/3863427814 conn(0x7faf1c0f7720 msgr2=0x7faf1c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:06.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 shutdown_connections 2022-03-21T14:08:06.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:06.542+0000 7faf211b4700 1 -- 172.21.15.2:0/3863427814 wait complete. 2022-03-21T14:08:07.002 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph orch device ls' 2022-03-21T14:08:08.333 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:08 smithi002 ceph-mon[17167]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:08.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:08 smithi002 ceph-mon[17167]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:08 smithi016 ceph-mon[22010]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:08 smithi016 ceph-mon[22010]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:08 smithi079 ceph-mon[23158]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:08 smithi079 ceph-mon[23158]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:08.515 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:09.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.042+0000 7fa7f5403700 1 -- 172.21.15.2:0/3411757021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f0070ec0 msgr2=0x7fa7f006f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:09.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.042+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3411757021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f0070ec0 0x7fa7f006f3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00098e0 tx=0x7fa7e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:09.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 -- 172.21.15.2:0/3411757021 shutdown_connections 2022-03-21T14:08:09.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3411757021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f0070ec0 0x7fa7f006f3c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3411757021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0103e80 0x7fa7f0106260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3411757021 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f006f900 0x7fa7f006fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 -- 172.21.15.2:0/3411757021 >> 172.21.15.2:0/3411757021 conn(0x7fa7f00f7720 msgr2=0x7fa7f00f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:09.093 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 -- 172.21.15.2:0/3411757021 shutdown_connections 2022-03-21T14:08:09.093 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.043+0000 7fa7f5403700 1 -- 172.21.15.2:0/3411757021 wait complete. 2022-03-21T14:08:09.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 Processor -- start 2022-03-21T14:08:09.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 -- start start 2022-03-21T14:08:09.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:09.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 0x7fa7f010f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:09.099 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f0103e80 0x7fa7f0116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0116a40 con 0x7fa7f0070ec0 2022-03-21T14:08:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0116bb0 con 0x7fa7f006f900 2022-03-21T14:08:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7f5403700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0116d70 con 0x7fa7f0103e80 2022-03-21T14:08:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.044+0000 7fa7eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7ee7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 0x7fa7f010f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48574/0 (socket says 172.21.15.2:48574) 2022-03-21T14:08:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 -- 172.21.15.2:0/3896897893 learned_addr learned my addr 172.21.15.2:0/3896897893 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7ef7fe700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f0103e80 0x7fa7f0116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 -- 172.21.15.2:0/3896897893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f0103e80 msgr2=0x7fa7f0116500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f0103e80 0x7fa7f0116500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 -- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 msgr2=0x7fa7f010f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 0x7fa7f010f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 -- 172.21.15.2:0/3896897893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00093f0 con 0x7fa7f006f900 2022-03-21T14:08:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7ee7fc700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 0x7fa7f010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:08:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7eeffd700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000f590 tx=0x7fa7e0005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.045+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e001d070 con 0x7fa7f006f900 2022-03-21T14:08:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.046+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa7e0005760 con 0x7fa7f006f900 2022-03-21T14:08:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.046+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f0116ff0 con 0x7fa7f006f900 2022-03-21T14:08:09.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.046+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0017c00 con 0x7fa7f006f900 2022-03-21T14:08:09.106 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.046+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f0117390 con 0x7fa7f006f900 2022-03-21T14:08:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.047+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f0109930 con 0x7fa7f006f900 2022-03-21T14:08:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.049+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa7e000c040 con 0x7fa7f006f900 2022-03-21T14:08:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.050+0000 7fa7d77fe700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 0x7fa7d8099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.050+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fa7e00c1130 con 0x7fa7f006f900 2022-03-21T14:08:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.050+0000 7fa7ee7fc700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 0x7fa7d8099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.051+0000 7fa7ee7fc700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 0x7fa7d8099240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005c00 tx=0x7fa7e4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.054+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fa7e008e3b0 con 0x7fa7f006f900 2022-03-21T14:08:09.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.247+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 --> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7f002acb0 con 0x7fa7d8096d90 2022-03-21T14:08:09.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.249+0000 7fa7d77fe700 1 -- 172.21.15.2:0/3896897893 <== mgr.14150 v2:172.21.15.2:6800/613532934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7fa7f002acb0 con 0x7fa7d8096d90 2022-03-21T14:08:09.250 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-21T14:08:09.250 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 480G 2m ago LVM detected, locked 2022-03-21T14:08:09.250 INFO:teuthology.orchestra.run.smithi002.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 480G 2m ago LVM detected, locked 2022-03-21T14:08:09.251 INFO:teuthology.orchestra.run.smithi002.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 400G 2m ago LVM detected, locked 2022-03-21T14:08:09.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 msgr2=0x7fa7d8099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:09.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 0x7fa7d8099240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005c00 tx=0x7fa7e4005f90 comp rx=0 tx=0).stop 2022-03-21T14:08:09.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 msgr2=0x7fa7f010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:09.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000f590 tx=0x7fa7e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:09.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 shutdown_connections 2022-03-21T14:08:09.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f006f900 0x7fa7f010ef30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa7d8096d90 0x7fa7d8099240 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7f0103e80 0x7fa7f0116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 --2- 172.21.15.2:0/3896897893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7f0070ec0 0x7fa7f010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:09.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 >> 172.21.15.2:0/3896897893 conn(0x7fa7f00f7720 msgr2=0x7fa7f00f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:09.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 shutdown_connections 2022-03-21T14:08:09.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:09.251+0000 7fa7f5403700 1 -- 172.21.15.2:0/3896897893 wait complete. 2022-03-21T14:08:09.721 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-03-21T14:08:09.731 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2022-03-21T14:08:09.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph fs dump' 2022-03-21T14:08:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:10 smithi002 ceph-mon[17167]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:10 smithi016 ceph-mon[22010]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:10 smithi079 ceph-mon[23158]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:11 smithi002 ceph-mon[17167]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:11 smithi016 ceph-mon[22010]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:11 smithi079 ceph-mon[23158]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-03-21T14:08:11.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 -- 172.21.15.2:0/3441867723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 msgr2=0x7f32d80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:11.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3441867723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d80fe3b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f32c80098e0 tx=0x7f32c8005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:11.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 -- 172.21.15.2:0/3441867723 shutdown_connections 2022-03-21T14:08:11.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3441867723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80ffa10 0x7f32d8106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:11.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3441867723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d80fe3b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:11.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3441867723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80fe8f0 0x7f32d80ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:11.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 -- 172.21.15.2:0/3441867723 >> 172.21.15.2:0/3441867723 conn(0x7f32d80f9960 msgr2=0x7f32d80fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:11.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 -- 172.21.15.2:0/3441867723 shutdown_connections 2022-03-21T14:08:11.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.868+0000 7f32d7fff700 1 -- 172.21.15.2:0/3441867723 wait complete. 2022-03-21T14:08:11.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.869+0000 7f32d7fff700 1 Processor -- start 2022-03-21T14:08:11.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.869+0000 7f32d7fff700 1 -- start start 2022-03-21T14:08:11.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:11.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 0x7f32d810f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:11.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 0x7f32d81163c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:11.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8116900 con 0x7f32d80fdfa0 2022-03-21T14:08:11.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8116a70 con 0x7f32d80fe8f0 2022-03-21T14:08:11.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d7fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8116c30 con 0x7f32d80ffa10 2022-03-21T14:08:11.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:11.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43992/0 (socket says 172.21.15.2:43992) 2022-03-21T14:08:11.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 0x7f32d81163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:11.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d6ffd700 1 -- 172.21.15.2:0/3747912597 learned_addr learned my addr 172.21.15.2:0/3747912597 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:11.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.870+0000 7f32d67fc700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 0x7f32d810f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:11.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 -- 172.21.15.2:0/3747912597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 msgr2=0x7f32d81163c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:11.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 0x7f32d81163c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:11.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 -- 172.21.15.2:0/3747912597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 msgr2=0x7f32d810f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:11.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 0x7f32d810f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:11.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 -- 172.21.15.2:0/3747912597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c80093f0 con 0x7f32d80fdfa0 2022-03-21T14:08:11.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d77fe700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 0x7f32d81163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:11.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d67fc700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 0x7f32d810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:11.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d6ffd700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006010 tx=0x7f32c8005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:11.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c801d070 con 0x7f32d80fdfa0 2022-03-21T14:08:11.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f32c8005040 con 0x7f32d80fdfa0 2022-03-21T14:08:11.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8017a00 con 0x7f32d80fdfa0 2022-03-21T14:08:11.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.871+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8116eb0 con 0x7f32d80fdfa0 2022-03-21T14:08:11.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.872+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d8117200 con 0x7f32d80fdfa0 2022-03-21T14:08:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.872+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8109760 con 0x7f32d80fdfa0 2022-03-21T14:08:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.875+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f32c8029030 con 0x7f32d80fdfa0 2022-03-21T14:08:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.877+0000 7f32bf7fe700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 0x7f32c0099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.877+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f32c80c0cc0 con 0x7f32d80fdfa0 2022-03-21T14:08:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.877+0000 7f32d67fc700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 0x7f32c0099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:11.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.878+0000 7f32d67fc700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 0x7f32c0099240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f32d80fece0 tx=0x7f32cc005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:11.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:11.879+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f32c808dee0 con 0x7f32d80fdfa0 2022-03-21T14:08:12.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.109+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f32d802ad30 con 0x7f32d80fdfa0 2022-03-21T14:08:12.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.110+0000 7f32bf7fe700 1 -- 172.21.15.2:0/3747912597 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+2811 (secure 0 0 0) 0x7f32c8027b10 con 0x7f32d80fdfa0 2022-03-21T14:08:12.111 INFO:teuthology.orchestra.run.smithi002.stdout:e20 2022-03-21T14:08:12.111 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-03-21T14:08:12.112 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-03-21T14:08:12.112 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2022-03-21T14:08:12.112 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:12.112 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2022-03-21T14:08:12.113 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2022-03-21T14:08:12.113 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 20 2022-03-21T14:08:12.113 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-03-21T14:08:12.113 INFO:teuthology.orchestra.run.smithi002.stdout:created 2022-03-21T14:06:54.837756+0000 2022-03-21T14:08:12.114 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2022-03-21T14:06:59.487834+0000 2022-03-21T14:08:12.114 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2022-03-21T14:08:12.114 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2022-03-21T14:08:12.114 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 300 2022-03-21T14:08:12.114 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2022-03-21T14:08:12.115 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2022-03-21T14:08:12.115 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2022-03-21T14:08:12.115 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2022-03-21T14:08:12.115 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2022-03-21T14:08:12.116 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-03-21T14:08:12.116 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2022-03-21T14:08:12.116 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2022-03-21T14:08:12.116 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24587} 2022-03-21T14:08:12.116 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2022-03-21T14:08:12.117 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2022-03-21T14:08:12.117 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2022-03-21T14:08:12.117 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2022-03-21T14:08:12.117 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2022-03-21T14:08:12.117 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2022-03-21T14:08:12.118 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2022-03-21T14:08:12.118 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2022-03-21T14:08:12.118 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.l{0:24587} state up:active seq 3 addr [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.118 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.i{0:24575} state up:standby-replay seq 1 addr [v2:172.21.15.79:6836/2188072757,v1:172.21.15.79:6837/2188072757] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.119 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:12.119 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:12.119 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2022-03-21T14:08:12.119 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:12.119 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.g{-1:14424} state up:standby seq 1 addr [v2:172.21.15.2:6838/2330341423,v1:172.21.15.2:6839/2330341423] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.120 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.j{-1:14442} state up:standby seq 1 addr [v2:172.21.15.2:6840/1904085966,v1:172.21.15.2:6841/1904085966] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.120 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a{-1:24461} state up:standby seq 1 addr [v2:172.21.15.2:6834/2676603000,v1:172.21.15.2:6835/2676603000] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.120 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d{-1:24479} state up:standby seq 1 addr [v2:172.21.15.2:6836/595788412,v1:172.21.15.2:6837/595788412] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.120 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b{-1:24505} state up:standby seq 1 addr [v2:172.21.15.16:6832/3142985186,v1:172.21.15.16:6833/3142985186] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.e{-1:24517} state up:standby seq 1 addr [v2:172.21.15.16:6834/2493160775,v1:172.21.15.16:6835/2493160775] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.h{-1:24529} state up:standby seq 1 addr [v2:172.21.15.16:6836/2678336191,v1:172.21.15.16:6837/2678336191] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.k{-1:24538} state up:standby seq 1 addr [v2:172.21.15.16:6838/4173795056,v1:172.21.15.16:6839/4173795056] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c{-1:24557} state up:standby seq 1 addr [v2:172.21.15.79:6832/2500439348,v1:172.21.15.79:6833/2500439348] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.121 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.f{-1:24569} state up:standby seq 1 addr [v2:172.21.15.79:6834/805336294,v1:172.21.15.79:6835/805336294] compat {c=[1],r=[1],i=[7ff]}] 2022-03-21T14:08:12.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 msgr2=0x7f32c0099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:12.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 0x7f32c0099240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f32d80fece0 tx=0x7f32cc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:12.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 msgr2=0x7f32d810edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:12.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006010 tx=0x7f32c8005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:12.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 shutdown_connections 2022-03-21T14:08:12.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f32c0096d90 0x7f32c0099240 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:12.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32d80ffa10 0x7f32d81163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:12.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fdfa0 0x7f32d810edf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:12.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 --2- 172.21.15.2:0/3747912597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d80fe8f0 0x7f32d810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:12.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.112+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 >> 172.21.15.2:0/3747912597 conn(0x7f32d80f9960 msgr2=0x7f32d8102510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:12.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.113+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 shutdown_connections 2022-03-21T14:08:12.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:12.113+0000 7f32d7fff700 1 -- 172.21.15.2:0/3747912597 wait complete. 2022-03-21T14:08:12.125 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 20 2022-03-21T14:08:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:12 smithi002 ceph-mon[17167]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:12 smithi016 ceph-mon[22010]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:12 smithi079 ceph-mon[23158]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:12.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- bash -c 'ceph osd dump' 2022-03-21T14:08:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:13 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3747912597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-03-21T14:08:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:13 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3747912597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-03-21T14:08:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:13 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3747912597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-03-21T14:08:14.255 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:14 smithi002 ceph-mon[17167]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:14 smithi016 ceph-mon[22010]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:14 smithi079 ceph-mon[23158]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:14.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 -- 172.21.15.2:0/1148235054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 msgr2=0x7f36dc0ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:14.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 --2- 172.21.15.2:0/1148235054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc0ff840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36c80098e0 tx=0x7f36c8005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:14.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 -- 172.21.15.2:0/1148235054 shutdown_connections 2022-03-21T14:08:14.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 --2- 172.21.15.2:0/1148235054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc0fe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:14.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 --2- 172.21.15.2:0/1148235054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 0x7f36dc106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:14.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 --2- 172.21.15.2:0/1148235054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc0ff840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:14.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 -- 172.21.15.2:0/1148235054 >> 172.21.15.2:0/1148235054 conn(0x7f36dc0f9960 msgr2=0x7f36dc0fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:14.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 -- 172.21.15.2:0/1148235054 shutdown_connections 2022-03-21T14:08:14.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.830+0000 7f36e31bf700 1 -- 172.21.15.2:0/1148235054 wait complete. 2022-03-21T14:08:14.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.831+0000 7f36e31bf700 1 Processor -- start 2022-03-21T14:08:14.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.831+0000 7f36e31bf700 1 -- start start 2022-03-21T14:08:14.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:14.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc10f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:14.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 0x7f36dc1164b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:14.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc1169f0 con 0x7f36dc0ff3f0 2022-03-21T14:08:14.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc116b60 con 0x7f36dc0fe150 2022-03-21T14:08:14.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e31bf700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc116d20 con 0x7f36dc0ffd80 2022-03-21T14:08:14.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e0f5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:14.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36dbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc10f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:14.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e0f5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48582/0 (socket says 172.21.15.2:48582) 2022-03-21T14:08:14.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e0f5b700 1 -- 172.21.15.2:0/3578750180 learned_addr learned my addr 172.21.15.2:0/3578750180 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:14.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e175c700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 0x7f36dc1164b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:14.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e0f5b700 1 -- 172.21.15.2:0/3578750180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 msgr2=0x7f36dc1164b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:14.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.832+0000 7f36e0f5b700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 0x7f36dc1164b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:14.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36e0f5b700 1 -- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 msgr2=0x7f36dc10f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:14.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36e0f5b700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc10f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:14.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36e0f5b700 1 -- 172.21.15.2:0/3578750180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d40096b0 con 0x7f36dc0fe150 2022-03-21T14:08:14.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36dbfff700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc10f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:14.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36e0f5b700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36d000f860 tx=0x7f36d0005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:14.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0004020 con 0x7f36dc0fe150 2022-03-21T14:08:14.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f36d0004180 con 0x7f36dc0fe150 2022-03-21T14:08:14.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00039e0 con 0x7f36dc0fe150 2022-03-21T14:08:14.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.833+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c80093f0 con 0x7f36dc0fe150 2022-03-21T14:08:14.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.834+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36dc1172a0 con 0x7f36dc0fe150 2022-03-21T14:08:14.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.834+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36dc1099f0 con 0x7f36dc0fe150 2022-03-21T14:08:14.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.836+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f36d0022060 con 0x7f36dc0fe150 2022-03-21T14:08:14.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.838+0000 7f36d97fa700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 0x7f36cc0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:14.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.838+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f36d00bf560 con 0x7f36dc0fe150 2022-03-21T14:08:14.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.838+0000 7f36dbfff700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 0x7f36cc0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:14.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.839+0000 7f36dbfff700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 0x7f36cc0991c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009f50 tx=0x7f36c801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:14.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:14.840+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f36d008c7e0 con 0x7f36dc0fe150 2022-03-21T14:08:15.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.033+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f36dc02ac60 con 0x7f36dc0fe150 2022-03-21T14:08:15.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.034+0000 7f36d97fa700 1 -- 172.21.15.2:0/3578750180 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v84) v1 ==== 56+0+5186 (secure 0 0 0) 0x7f36d008fe00 con 0x7f36dc0fe150 2022-03-21T14:08:15.035 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 84 2022-03-21T14:08:15.036 INFO:teuthology.orchestra.run.smithi002.stdout:fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c 2022-03-21T14:08:15.036 INFO:teuthology.orchestra.run.smithi002.stdout:created 2022-03-21T13:59:17.765843+0000 2022-03-21T14:08:15.036 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2022-03-21T14:07:21.862918+0000 2022-03-21T14:08:15.036 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2022-03-21T14:08:15.037 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 26 2022-03-21T14:08:15.037 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2022-03-21T14:08:15.037 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2022-03-21T14:08:15.038 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2022-03-21T14:08:15.038 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2022-03-21T14:08:15.039 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client luminous 2022-03-21T14:08:15.039 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release quincy 2022-03-21T14:08:15.039 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2022-03-21T14:08:15.039 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2022-03-21T14:08:15.040 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs 2022-03-21T14:08:15.040 INFO:teuthology.orchestra.run.smithi002.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs 2022-03-21T14:08:15.040 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 12 2022-03-21T14:08:15.040 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/1386749192,v1:172.21.15.2:6803/1386749192] [v2:172.21.15.2:6804/1386749192,v1:172.21.15.2:6805/1386749192] exists,up 13405266-d8e8-4542-bf91-9450e11d2ecd 2022-03-21T14:08:15.040 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 13 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/491865625,v1:172.21.15.16:6801/491865625] [v2:172.21.15.16:6802/491865625,v1:172.21.15.16:6803/491865625] exists,up e1d0a7a0-443f-4117-aed8-0078291c7c93 2022-03-21T14:08:15.041 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 18 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6800/4178164596,v1:172.21.15.79:6801/4178164596] [v2:172.21.15.79:6802/4178164596,v1:172.21.15.79:6803/4178164596] exists,up c1f5778a-ca84-43c0-8233-005fbbf6da43 2022-03-21T14:08:15.041 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 25 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/1119340852,v1:172.21.15.2:6811/1119340852] [v2:172.21.15.2:6812/1119340852,v1:172.21.15.2:6813/1119340852] exists,up c84afd54-8463-4dc6-ac84-f8c56d385bc4 2022-03-21T14:08:15.041 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 32 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/530720703,v1:172.21.15.16:6809/530720703] [v2:172.21.15.16:6810/530720703,v1:172.21.15.16:6811/530720703] exists,up e7283335-3a3f-4840-b028-f85250ba113e 2022-03-21T14:08:15.041 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 37 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6808/1459081452,v1:172.21.15.79:6809/1459081452] [v2:172.21.15.79:6810/1459081452,v1:172.21.15.79:6811/1459081452] exists,up 574f01bb-e554-4aa2-bb51-627091c73352 2022-03-21T14:08:15.042 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 45 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/2687857259,v1:172.21.15.2:6819/2687857259] [v2:172.21.15.2:6820/2687857259,v1:172.21.15.2:6821/2687857259] exists,up b9e16547-b033-47a4-a8a6-8b17c953ce5b 2022-03-21T14:08:15.042 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 50 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/3746016674,v1:172.21.15.16:6817/3746016674] [v2:172.21.15.16:6818/3746016674,v1:172.21.15.16:6819/3746016674] exists,up 65e72a50-adfb-492a-b31e-68e1fa96fb5e 2022-03-21T14:08:15.042 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 57 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6816/2185003031,v1:172.21.15.79:6817/2185003031] [v2:172.21.15.79:6818/2185003031,v1:172.21.15.79:6819/2185003031] exists,up 680e2f3a-ec15-482d-9734-1691c5c013d6 2022-03-21T14:08:15.042 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 65 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6826/2878353867,v1:172.21.15.2:6827/2878353867] [v2:172.21.15.2:6828/2878353867,v1:172.21.15.2:6829/2878353867] exists,up c640ec62-e489-4b6d-bd4e-3e4b609505ea 2022-03-21T14:08:15.043 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 70 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/4107732214,v1:172.21.15.16:6825/4107732214] [v2:172.21.15.16:6826/4107732214,v1:172.21.15.16:6827/4107732214] exists,up ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0 2022-03-21T14:08:15.043 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 77 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6824/1987240625,v1:172.21.15.79:6825/1987240625] [v2:172.21.15.79:6826/1987240625,v1:172.21.15.79:6827/1987240625] exists,up 31a5878a-28c3-42a5-80c4-a032b07fc71b 2022-03-21T14:08:15.043 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.2 [3,9,5,7] 2022-03-21T14:08:15.043 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.6 [4,6] 2022-03-21T14:08:15.043 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.c [3,9] 2022-03-21T14:08:15.044 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.11 [4,8] 2022-03-21T14:08:15.044 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.16 [1,8] 2022-03-21T14:08:15.044 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.17 [4,6] 2022-03-21T14:08:15.044 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.1a [5,7] 2022-03-21T14:08:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.38 [3,9] 2022-03-21T14:08:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.3a [4,8] 2022-03-21T14:08:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3018413483 expires 2022-03-22T14:00:20.256801+0000 2022-03-21T14:08:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/225365632 expires 2022-03-22T14:00:20.256801+0000 2022-03-21T14:08:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/1086540904 expires 2022-03-22T14:00:20.256801+0000 2022-03-21T14:08:15.046 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3665116503 expires 2022-03-22T14:00:20.256801+0000 2022-03-21T14:08:15.046 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/2638362969 expires 2022-03-22T13:59:54.290862+0000 2022-03-21T14:08:15.046 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2302345157 expires 2022-03-22T13:59:54.290862+0000 2022-03-21T14:08:15.046 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/1086540904 expires 2022-03-22T14:00:20.256801+0000 2022-03-21T14:08:15.046 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/2638362969 expires 2022-03-22T13:59:54.290862+0000 2022-03-21T14:08:15.047 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2219526494 expires 2022-03-22T13:59:54.290862+0000 2022-03-21T14:08:15.047 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/1953025189 expires 2022-03-22T13:59:54.290862+0000 2022-03-21T14:08:15.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 msgr2=0x7f36cc0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:15.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 0x7f36cc0991c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009f50 tx=0x7f36c801a040 comp rx=0 tx=0).stop 2022-03-21T14:08:15.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 msgr2=0x7f36dc10eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36d000f860 tx=0x7f36d0005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 shutdown_connections 2022-03-21T14:08:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc0fe150 0x7f36dc10eee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f36cc096d10 0x7f36cc0991c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36dc0ffd80 0x7f36dc1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 --2- 172.21.15.2:0/3578750180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36dc0ff3f0 0x7f36dc10f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 >> 172.21.15.2:0/3578750180 conn(0x7f36dc0f9960 msgr2=0x7f36dc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:15.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 shutdown_connections 2022-03-21T14:08:15.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.037+0000 7f36e31bf700 1 -- 172.21.15.2:0/3578750180 wait complete. 2022-03-21T14:08:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:15 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3578750180' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-03-21T14:08:15.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:15 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3578750180' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-03-21T14:08:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:15 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3578750180' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-03-21T14:08:15.662 INFO:teuthology.run_tasks:Running task fs.ready... 2022-03-21T14:08:15.673 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:08:15.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 -- 172.21.15.2:0/4070687207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c070ec0 msgr2=0x7f192c06f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:15.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 --2- 172.21.15.2:0/4070687207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c070ec0 0x7f192c06f3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f191c0098e0 tx=0x7f191c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:15.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 -- 172.21.15.2:0/4070687207 shutdown_connections 2022-03-21T14:08:15.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 --2- 172.21.15.2:0/4070687207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c103ee0 0x7f192c1062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 --2- 172.21.15.2:0/4070687207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c070ec0 0x7f192c06f3c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 --2- 172.21.15.2:0/4070687207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c06f900 0x7f192c06fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 -- 172.21.15.2:0/4070687207 >> 172.21.15.2:0/4070687207 conn(0x7f192c0f7760 msgr2=0x7f192c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:15.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 -- 172.21.15.2:0/4070687207 shutdown_connections 2022-03-21T14:08:15.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.882+0000 7f1931020700 1 -- 172.21.15.2:0/4070687207 wait complete. 2022-03-21T14:08:15.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 Processor -- start 2022-03-21T14:08:15.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 -- start start 2022-03-21T14:08:15.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:15.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c070ec0 0x7f192c10f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:15.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c103ee0 0x7f192c116550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:15.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c116a90 con 0x7f192c06f900 2022-03-21T14:08:15.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c116c00 con 0x7f192c103ee0 2022-03-21T14:08:15.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.884+0000 7f1931020700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c116dc0 con 0x7f192c070ec0 2022-03-21T14:08:15.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:15.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44000/0 (socket says 172.21.15.2:44000) 2022-03-21T14:08:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 -- 172.21.15.2:0/1144976126 learned_addr learned my addr 172.21.15.2:0/1144976126 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f193081f700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c103ee0 0x7f192c116550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192b7fe700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c070ec0 0x7f192c10f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:15.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 -- 172.21.15.2:0/1144976126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c070ec0 msgr2=0x7f192c10f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c070ec0 0x7f192c10f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 -- 172.21.15.2:0/1144976126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c103ee0 msgr2=0x7f192c116550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c103ee0 0x7f192c116550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 -- 172.21.15.2:0/1144976126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0093f0 con 0x7f192c06f900 2022-03-21T14:08:15.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.885+0000 7f192bfff700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f191c0093c0 tx=0x7f191c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:15.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.886+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c01d070 con 0x7f192c06f900 2022-03-21T14:08:15.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.886+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f191c005040 con 0x7f192c06f900 2022-03-21T14:08:15.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.886+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c117040 con 0x7f192c06f900 2022-03-21T14:08:15.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.886+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c017a20 con 0x7f192c06f900 2022-03-21T14:08:15.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.886+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c117520 con 0x7f192c06f900 2022-03-21T14:08:15.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.887+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c1099e0 con 0x7f192c06f900 2022-03-21T14:08:15.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.888+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f191c029030 con 0x7f192c06f900 2022-03-21T14:08:15.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.890+0000 7f1928ff9700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 0x7f1914099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:15.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.890+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f191c0c0c70 con 0x7f192c06f900 2022-03-21T14:08:15.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.890+0000 7f192b7fe700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 0x7f1914099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:15.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.891+0000 7f192b7fe700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 0x7f1914099240 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1920005fd0 tx=0x7f1920005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:15.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:15.894+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f191c058f40 con 0x7f192c06f900 2022-03-21T14:08:16.082 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.081+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f192c1024b0 con 0x7f192c06f900 2022-03-21T14:08:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.082+0000 7f1928ff9700 1 -- 172.21.15.2:0/1144976126 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+21650 (secure 0 0 0) 0x7f191c027090 con 0x7f192c06f900 2022-03-21T14:08:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:16.084 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":84,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:07:21.862918+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.16","mappings":[{"from":1,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:08:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 msgr2=0x7f1914099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 0x7f1914099240 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1920005fd0 tx=0x7f1920005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 msgr2=0x7f192c10ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f191c0093c0 tx=0x7f191c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 shutdown_connections 2022-03-21T14:08:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f192c103ee0 0x7f192c116550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f1914096d90 0x7f1914099240 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c06f900 0x7f192c10ef80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 --2- 172.21.15.2:0/1144976126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f192c070ec0 0x7f192c10f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 >> 172.21.15.2:0/1144976126 conn(0x7f192c0f7760 msgr2=0x7f192c0f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 shutdown_connections 2022-03-21T14:08:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.085+0000 7f1931020700 1 -- 172.21.15.2:0/1144976126 wait complete. 2022-03-21T14:08:16.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-03-21T14:06:52.815266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-03-21T14:06:54.292010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-03-21T14:08:16.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-03-21T14:08:16.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 -- 172.21.15.2:0/1898704900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428066050 msgr2=0x7f34280fec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 --2- 172.21.15.2:0/1898704900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428066050 0x7f34280fec40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34180098e0 tx=0x7f3418005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:16.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 -- 172.21.15.2:0/1898704900 shutdown_connections 2022-03-21T14:08:16.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 --2- 172.21.15.2:0/1898704900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3428101ca0 0x7f3428104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 --2- 172.21.15.2:0/1898704900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428066050 0x7f34280fec40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 --2- 172.21.15.2:0/1898704900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34280ff210 0x7f3428101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 -- 172.21.15.2:0/1898704900 >> 172.21.15.2:0/1898704900 conn(0x7f34280f66f0 msgr2=0x7f34280f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:16.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 -- 172.21.15.2:0/1898704900 shutdown_connections 2022-03-21T14:08:16.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.322+0000 7f342759e700 1 -- 172.21.15.2:0/1898704900 wait complete. 2022-03-21T14:08:16.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 Processor -- start 2022-03-21T14:08:16.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 -- start start 2022-03-21T14:08:16.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34280ff210 0x7f342810d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 0x7f3428114530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428114a70 con 0x7f3428101ca0 2022-03-21T14:08:16.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428114be0 con 0x7f3428066050 2022-03-21T14:08:16.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.323+0000 7f342759e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428114da0 con 0x7f34280ff210 2022-03-21T14:08:16.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48590/0 (socket says 172.21.15.2:48590) 2022-03-21T14:08:16.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 -- 172.21.15.2:0/443622237 learned_addr learned my addr 172.21.15.2:0/443622237 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:16.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f3425d9b700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34280ff210 0x7f342810d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f3426d9d700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 0x7f3428114530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 -- 172.21.15.2:0/443622237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34280ff210 msgr2=0x7f342810d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34280ff210 0x7f342810d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 -- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 msgr2=0x7f3428114530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 0x7f3428114530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 -- 172.21.15.2:0/443622237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180093f0 con 0x7f3428066050 2022-03-21T14:08:16.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f3426d9d700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 0x7f3428114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:08:16.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.324+0000 7f342659c700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f341800f590 tx=0x7f3418005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:16.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.325+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341801d070 con 0x7f3428066050 2022-03-21T14:08:16.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.325+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3418005760 con 0x7f3428066050 2022-03-21T14:08:16.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.325+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3428115020 con 0x7f3428066050 2022-03-21T14:08:16.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.325+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418017c00 con 0x7f3428066050 2022-03-21T14:08:16.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.325+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3428115420 con 0x7f3428066050 2022-03-21T14:08:16.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.326+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34281077e0 con 0x7f3428066050 2022-03-21T14:08:16.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.328+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f341800c040 con 0x7f3428066050 2022-03-21T14:08:16.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.329+0000 7f3416ffd700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 0x7f34100991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.329+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f34180c1180 con 0x7f3428066050 2022-03-21T14:08:16.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.329+0000 7f3425d9b700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 0x7f34100991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.330+0000 7f3425d9b700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 0x7f34100991c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3428066480 tx=0x7f341c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:16.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.331+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f341808e400 con 0x7f3428066050 2022-03-21T14:08:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:16 smithi002 ceph-mon[17167]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:16 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1144976126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:16 smithi016 ceph-mon[22010]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:16 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1144976126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:16 smithi079 ceph-mon[23158]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:16 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1144976126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:16.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.522+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34280f80a0 con 0x7f3428066050 2022-03-21T14:08:16.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.523+0000 7f3416ffd700 1 -- 172.21.15.2:0/443622237 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3418091a20 con 0x7f3428066050 2022-03-21T14:08:16.524 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:08:16.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 msgr2=0x7f34100991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 0x7f34100991c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3428066480 tx=0x7f341c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 msgr2=0x7f342810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f341800f590 tx=0x7f3418005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:16.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 shutdown_connections 2022-03-21T14:08:16.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3428066050 0x7f342810cf60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f3410096d10 0x7f34100991c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3428101ca0 0x7f3428114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 --2- 172.21.15.2:0/443622237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34280ff210 0x7f342810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 >> 172.21.15.2:0/443622237 conn(0x7f34280f66f0 msgr2=0x7f34280fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:16.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 shutdown_connections 2022-03-21T14:08:16.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.525+0000 7f342759e700 1 -- 172.21.15.2:0/443622237 wait complete. 2022-03-21T14:08:16.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-03-21T14:08:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/695601815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fdc00 msgr2=0x7fcfe01044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/695601815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fdc00 0x7fcfe01044c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009c70 tx=0x7fcfd80094e0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/695601815 shutdown_connections 2022-03-21T14:08:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/695601815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fdc00 0x7fcfe01044c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/695601815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe00fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/695601815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fd270 0x7fcfe00fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/695601815 >> 172.21.15.2:0/695601815 conn(0x7fcfe00f7780 msgr2=0x7fcfe00f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/695601815 shutdown_connections 2022-03-21T14:08:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.759+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/695601815 wait complete. 2022-03-21T14:08:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 Processor -- start 2022-03-21T14:08:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 -- start start 2022-03-21T14:08:16.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fd270 0x7fcfe006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fdc00 0x7fcfe006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe006ff70 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00700e0 con 0x7fcfe00fdc00 2022-03-21T14:08:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe6e8f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0116cb0 con 0x7fcfe00fd270 2022-03-21T14:08:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe5e8d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe5e8d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44008/0 (socket says 172.21.15.2:44008) 2022-03-21T14:08:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.760+0000 7fcfe5e8d700 1 -- 172.21.15.2:0/19097496 learned_addr learned my addr 172.21.15.2:0/19097496 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe568c700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fd270 0x7fcfe006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe668e700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fdc00 0x7fcfe006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 -- 172.21.15.2:0/19097496 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fd270 msgr2=0x7fcfe006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fd270 0x7fcfe006f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 -- 172.21.15.2:0/19097496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fdc00 msgr2=0x7fcfe006f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fdc00 0x7fcfe006f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 -- 172.21.15.2:0/19097496 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8009780 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfe5e8d700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00f860 tx=0x7fcfdc005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.761+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004020 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.762+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fcfdc004180 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.762+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0039f0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.762+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe0116eb0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.762+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0117200 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.763+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0107870 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.765+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fcfdc00c040 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.772 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.766+0000 7fcfd67fc700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 0x7fcfcc0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:16.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.766+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fcfdc0bfba0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.766+0000 7fcfe568c700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 0x7fcfcc0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:16.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.767+0000 7fcfe568c700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 0x7fcfcc0991c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000b840 tx=0x7fcfd000b320 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:16.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.770+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fcfdc08cdc0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.956+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfe002acc0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.956+0000 7fcfd67fc700 1 -- 172.21.15.2:0/19097496 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcfdc0903e0 con 0x7fcfe00fbfd0 2022-03-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-03-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 msgr2=0x7fcfcc0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 0x7fcfcc0991c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000b840 tx=0x7fcfd000b320 comp rx=0 tx=0).stop 2022-03-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 msgr2=0x7fcfe0070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00f860 tx=0x7fcfdc005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 shutdown_connections 2022-03-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfe00fdc00 0x7fcfe006f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fcfcc096d10 0x7fcfcc0991c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfe00fbfd0 0x7fcfe0070cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 --2- 172.21.15.2:0/19097496 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfe00fd270 0x7fcfe006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 >> 172.21.15.2:0/19097496 conn(0x7fcfe00f7780 msgr2=0x7fcfe0100620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 shutdown_connections 2022-03-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:16.959+0000 7fcfe6e8f700 1 -- 172.21.15.2:0/19097496 wait complete. 2022-03-21T14:08:16.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-03-21T14:08:17.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 -- 172.21.15.2:0/2343682194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ff490 msgr2=0x7f98e40ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 --2- 172.21.15.2:0/2343682194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ff490 0x7f98e40ff8e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b820 tx=0x7f98d4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 -- 172.21.15.2:0/2343682194 shutdown_connections 2022-03-21T14:08:17.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 --2- 172.21.15.2:0/2343682194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ffe20 0x7f98e41066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 --2- 172.21.15.2:0/2343682194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e40fe600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 --2- 172.21.15.2:0/2343682194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ff490 0x7f98e40ff8e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 -- 172.21.15.2:0/2343682194 >> 172.21.15.2:0/2343682194 conn(0x7f98e40f99c0 msgr2=0x7f98e40fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:17.197 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 -- 172.21.15.2:0/2343682194 shutdown_connections 2022-03-21T14:08:17.197 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.194+0000 7f98ea653700 1 -- 172.21.15.2:0/2343682194 wait complete. 2022-03-21T14:08:17.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 Processor -- start 2022-03-21T14:08:17.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 -- start start 2022-03-21T14:08:17.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ff490 0x7f98e410f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ffe20 0x7f98e41164c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4116a00 con 0x7f98e40fe1f0 2022-03-21T14:08:17.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4116b70 con 0x7f98e40ffe20 2022-03-21T14:08:17.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.195+0000 7f98ea653700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4116d30 con 0x7f98e40ff490 2022-03-21T14:08:17.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.202 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44012/0 (socket says 172.21.15.2:44012) 2022-03-21T14:08:17.202 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 -- 172.21.15.2:0/2738663988 learned_addr learned my addr 172.21.15.2:0/2738663988 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:17.202 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9e52700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ffe20 0x7f98e41164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e8e50700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ff490 0x7f98e410f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 -- 172.21.15.2:0/2738663988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ff490 msgr2=0x7f98e410f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ff490 0x7f98e410f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 -- 172.21.15.2:0/2738663988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ffe20 msgr2=0x7f98e41164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ffe20 0x7f98e41164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 -- 172.21.15.2:0/2738663988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00093f0 con 0x7f98e40fe1f0 2022-03-21T14:08:17.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.196+0000 7f98e9651700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98e000ea80 tx=0x7f98e0005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:17.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.197+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e000ee60 con 0x7f98e40fe1f0 2022-03-21T14:08:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.197+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d400b360 con 0x7f98e40fe1f0 2022-03-21T14:08:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.197+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f98e000ce80 con 0x7f98e40fe1f0 2022-03-21T14:08:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.197+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0018420 con 0x7f98e40fe1f0 2022-03-21T14:08:17.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.197+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e4117300 con 0x7f98e40fe1f0 2022-03-21T14:08:17.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.198+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4109a90 con 0x7f98e40fe1f0 2022-03-21T14:08:17.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.200+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f98e000c040 con 0x7f98e40fe1f0 2022-03-21T14:08:17.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.201+0000 7f98d9ffb700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 0x7f98d00991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.201+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f98e0014070 con 0x7f98e40fe1f0 2022-03-21T14:08:17.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.201+0000 7f98e8e50700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 0x7f98d00991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.202+0000 7f98e8e50700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 0x7f98d00991c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b6f0 tx=0x7f98d4005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:17.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.203+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f98e008d230 con 0x7f98e40fe1f0 2022-03-21T14:08:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:17 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/443622237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:17 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/19097496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.389+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98e4105150 con 0x7f98e40fe1f0 2022-03-21T14:08:17.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.389+0000 7f98d9ffb700 1 -- 172.21.15.2:0/2738663988 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98e0090850 con 0x7f98e40fe1f0 2022-03-21T14:08:17.390 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-03-21T14:08:17.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 msgr2=0x7f98d00991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 0x7f98d00991c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b6f0 tx=0x7f98d4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 msgr2=0x7f98e410eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98e000ea80 tx=0x7f98e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:17.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 shutdown_connections 2022-03-21T14:08:17.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e40ffe20 0x7f98e41164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f98d0096d10 0x7f98d00991c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98e40fe1f0 0x7f98e410eef0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 --2- 172.21.15.2:0/2738663988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f98e40ff490 0x7f98e410f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 >> 172.21.15.2:0/2738663988 conn(0x7f98e40f99c0 msgr2=0x7f98e40fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:17.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 shutdown_connections 2022-03-21T14:08:17.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.392+0000 7f98ea653700 1 -- 172.21.15.2:0/2738663988 wait complete. 2022-03-21T14:08:17.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:17 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/443622237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:17 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/19097496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:08:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:17 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/443622237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:17 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/19097496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:17.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 -- 172.21.15.2:0/283401642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec103f40 msgr2=0x7ff6ec106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/283401642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec103f40 0x7ff6ec106320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009ba0 tx=0x7ff6e4009430 comp rx=0 tx=0).stop 2022-03-21T14:08:17.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 -- 172.21.15.2:0/283401642 shutdown_connections 2022-03-21T14:08:17.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/283401642 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec103f40 0x7ff6ec106320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/283401642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec070ea0 0x7ff6ec06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/283401642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec06f8e0 0x7ff6ec06fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 -- 172.21.15.2:0/283401642 >> 172.21.15.2:0/283401642 conn(0x7ff6ec0f77e0 msgr2=0x7ff6ec0f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:17.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 -- 172.21.15.2:0/283401642 shutdown_connections 2022-03-21T14:08:17.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.622+0000 7ff6f24d2700 1 -- 172.21.15.2:0/283401642 wait complete. 2022-03-21T14:08:17.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.623+0000 7ff6f24d2700 1 Processor -- start 2022-03-21T14:08:17.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.623+0000 7ff6f24d2700 1 -- start start 2022-03-21T14:08:17.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec06f8e0 0x7ff6ec10ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec070ea0 0x7ff6ec10f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec116800 con 0x7ff6ec103f40 2022-03-21T14:08:17.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec116970 con 0x7ff6ec070ea0 2022-03-21T14:08:17.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f24d2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec116b30 con 0x7ff6ec06f8e0 2022-03-21T14:08:17.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f1cd1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f14d0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec06f8e0 0x7ff6ec10ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f1cd1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44016/0 (socket says 172.21.15.2:44016) 2022-03-21T14:08:17.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f1cd1700 1 -- 172.21.15.2:0/2606107912 learned_addr learned my addr 172.21.15.2:0/2606107912 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:17.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.624+0000 7ff6f0ccf700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec070ea0 0x7ff6ec10f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 -- 172.21.15.2:0/2606107912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec06f8e0 msgr2=0x7ff6ec10ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec06f8e0 0x7ff6ec10ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 -- 172.21.15.2:0/2606107912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec070ea0 msgr2=0x7ff6ec10f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec070ea0 0x7ff6ec10f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 -- 172.21.15.2:0/2606107912 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0093f0 con 0x7ff6ec103f40 2022-03-21T14:08:17.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f1cd1700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009a40 tx=0x7ff6e4009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:17.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e401d070 con 0x7ff6ec103f40 2022-03-21T14:08:17.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7ff6e4022470 con 0x7ff6ec103f40 2022-03-21T14:08:17.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40096b0 con 0x7ff6ec103f40 2022-03-21T14:08:17.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.625+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400f6d0 con 0x7ff6ec103f40 2022-03-21T14:08:17.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.626+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec116fd0 con 0x7ff6ec103f40 2022-03-21T14:08:17.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.627+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec109a30 con 0x7ff6ec103f40 2022-03-21T14:08:17.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.628+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7ff6e4029030 con 0x7ff6ec103f40 2022-03-21T14:08:17.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.630+0000 7ff6e1ffb700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 0x7ff6d80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:17.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.630+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7ff6e40c10b0 con 0x7ff6ec103f40 2022-03-21T14:08:17.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.630+0000 7ff6f14d0700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 0x7ff6d80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:17.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.630+0000 7ff6f14d0700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 0x7ff6d80991c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec10fb80 tx=0x7ff6e8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:17.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.632+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7ff6e408e330 con 0x7ff6ec103f40 2022-03-21T14:08:17.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.866+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6ec102510 con 0x7ff6ec103f40 2022-03-21T14:08:17.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.867+0000 7ff6e1ffb700 1 -- 172.21.15.2:0/2606107912 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10650 (secure 0 0 0) 0x7ff6e4027090 con 0x7ff6ec103f40 2022-03-21T14:08:17.868 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:17.868 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-03-21T14:06:54.837756+0000","modified":"2022-03-21T14:06:59.487834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/1907413140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1907413140},{"type":"v1","addr":"172.21.15.79:6839","nonce":1907413140}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-03-21T14:08:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 msgr2=0x7ff6d80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 0x7ff6d80991c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec10fb80 tx=0x7ff6e8005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 msgr2=0x7ff6ec116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:17.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009a40 tx=0x7ff6e4009430 comp rx=0 tx=0).stop 2022-03-21T14:08:17.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 shutdown_connections 2022-03-21T14:08:17.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ff6d8096d10 0x7ff6d80991c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6ec103f40 0x7ff6ec116250 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6ec06f8e0 0x7ff6ec10ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 --2- 172.21.15.2:0/2606107912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6ec070ea0 0x7ff6ec10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:17.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.869+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 >> 172.21.15.2:0/2606107912 conn(0x7ff6ec0f77e0 msgr2=0x7ff6ec106030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:17.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.870+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 shutdown_connections 2022-03-21T14:08:17.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:17.870+0000 7ff6f24d2700 1 -- 172.21.15.2:0/2606107912 wait complete. 2022-03-21T14:08:17.873 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 20 2022-03-21T14:08:17.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:08:18.102 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 -- 172.21.15.2:0/1444124928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 msgr2=0x7f8e38101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 --2- 172.21.15.2:0/1444124928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e38101580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0098e0 tx=0x7f8e2c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:18.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 -- 172.21.15.2:0/1444124928 shutdown_connections 2022-03-21T14:08:18.103 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 --2- 172.21.15.2:0/1444124928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e38101c10 0x7f8e38104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 --2- 172.21.15.2:0/1444124928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e380651a0 0x7f8e380febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 --2- 172.21.15.2:0/1444124928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e38101580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.104 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.101+0000 7f8e37fff700 1 -- 172.21.15.2:0/1444124928 >> 172.21.15.2:0/1444124928 conn(0x7f8e380f6640 msgr2=0x7f8e380f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:18.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.102+0000 7f8e37fff700 1 -- 172.21.15.2:0/1444124928 shutdown_connections 2022-03-21T14:08:18.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.102+0000 7f8e37fff700 1 -- 172.21.15.2:0/1444124928 wait complete. 2022-03-21T14:08:18.105 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.102+0000 7f8e37fff700 1 Processor -- start 2022-03-21T14:08:18.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 -- start start 2022-03-21T14:08:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e380faad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e38101c10 0x7f8e380fb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380fb720 con 0x7f8e38101c10 2022-03-21T14:08:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380fb890 con 0x7f8e380ff1a0 2022-03-21T14:08:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e37fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38116fd0 con 0x7f8e380651a0 2022-03-21T14:08:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e36ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e36ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39868/0 (socket says 172.21.15.2:39868) 2022-03-21T14:08:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e377fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e38101c10 0x7f8e380fb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e36ffd700 1 -- 172.21.15.2:0/2003268646 learned_addr learned my addr 172.21.15.2:0/2003268646 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.103+0000 7f8e367fc700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e380faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 -- 172.21.15.2:0/2003268646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 msgr2=0x7f8e380faad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e380faad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 -- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e38101c10 msgr2=0x7f8e380fb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e38101c10 0x7f8e380fb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 -- 172.21.15.2:0/2003268646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c0093f0 con 0x7f8e380651a0 2022-03-21T14:08:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e367fc700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e380faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:08:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e36ffd700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800fa90 tx=0x7f8e28005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800be00 con 0x7f8e380651a0 2022-03-21T14:08:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e38117170 con 0x7f8e380651a0 2022-03-21T14:08:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8e2800de90 con 0x7f8e380651a0 2022-03-21T14:08:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e28011e80 con 0x7f8e380651a0 2022-03-21T14:08:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.104+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e38117660 con 0x7f8e380651a0 2022-03-21T14:08:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.105+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e38107790 con 0x7f8e380651a0 2022-03-21T14:08:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.107+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f8e2800c040 con 0x7f8e380651a0 2022-03-21T14:08:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.108+0000 7f8e1f7fe700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 0x7f8e200991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.108+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f8e280bfa80 con 0x7f8e380651a0 2022-03-21T14:08:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.108+0000 7f8e367fc700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 0x7f8e200991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.109+0000 7f8e367fc700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 0x7f8e200991c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0098b0 tx=0x7f8e2c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.111+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f8e2808cca0 con 0x7f8e380651a0 2022-03-21T14:08:18.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.299+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e380f7ff0 con 0x7f8e380651a0 2022-03-21T14:08:18.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.300+0000 7f8e1f7fe700 1 -- 172.21.15.2:0/2003268646 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+21650 (secure 0 0 0) 0x7f8e280902c0 con 0x7f8e380651a0 2022-03-21T14:08:18.301 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":84,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:07:21.862918+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.16","mappings":[{"from":1,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:08:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 msgr2=0x7f8e200991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 0x7f8e200991c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0098b0 tx=0x7f8e2c01a040 comp rx=0 tx=0).stop 2022-03-21T14:08:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 msgr2=0x7f8e380fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800fa90 tx=0x7f8e28005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 shutdown_connections 2022-03-21T14:08:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f8e20096d10 0x7f8e200991c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e38101c10 0x7f8e380fb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e380651a0 0x7f8e380fc470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 --2- 172.21.15.2:0/2003268646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e380ff1a0 0x7f8e380faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 >> 172.21.15.2:0/2003268646 conn(0x7f8e380f6640 msgr2=0x7f8e380fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:18.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 shutdown_connections 2022-03-21T14:08:18.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.303+0000 7f8e37fff700 1 -- 172.21.15.2:0/2003268646 wait complete. 2022-03-21T14:08:18.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:18 smithi002 ceph-mon[17167]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:18.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:18 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2738663988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:18.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:18 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2606107912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:18.329 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:02:31.166181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-03-21T14:06:52.815266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-03-21T14:06:54.292010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-03-21T14:08:18.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-03-21T14:08:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:18 smithi016 ceph-mon[22010]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:18 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2738663988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:18 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2606107912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:18 smithi079 ceph-mon[23158]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:18 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2738663988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:18 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2606107912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:18.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 -- 172.21.15.2:0/2041319907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fdc00 msgr2=0x7f05801044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 --2- 172.21.15.2:0/2041319907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fdc00 0x7f05801044c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0578009c70 tx=0x7f05780094e0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 -- 172.21.15.2:0/2041319907 shutdown_connections 2022-03-21T14:08:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 --2- 172.21.15.2:0/2041319907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f05800fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 --2- 172.21.15.2:0/2041319907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fdc00 0x7f05801044c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 --2- 172.21.15.2:0/2041319907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fd270 0x7f05800fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 -- 172.21.15.2:0/2041319907 >> 172.21.15.2:0/2041319907 conn(0x7f05800f7780 msgr2=0x7f05800f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 -- 172.21.15.2:0/2041319907 shutdown_connections 2022-03-21T14:08:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.539+0000 7f0587590700 1 -- 172.21.15.2:0/2041319907 wait complete. 2022-03-21T14:08:18.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.540+0000 7f0587590700 1 Processor -- start 2022-03-21T14:08:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.540+0000 7f0587590700 1 -- start start 2022-03-21T14:08:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f058010ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 0x7f058010f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 0x7f05801162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580116840 con 0x7f05800fd270 2022-03-21T14:08:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801169b0 con 0x7f05800fbfd0 2022-03-21T14:08:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0587590700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580116b70 con 0x7f05800fdc00 2022-03-21T14:08:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f058658e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f058010ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0585d8d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 0x7f058010f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0585d8d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 0x7f058010f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44026/0 (socket says 172.21.15.2:44026) 2022-03-21T14:08:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0586d8f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 0x7f05801162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f058658e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f058010ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48610/0 (socket says 172.21.15.2:48610) 2022-03-21T14:08:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0585d8d700 1 -- 172.21.15.2:0/3578602366 learned_addr learned my addr 172.21.15.2:0/3578602366 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0586d8f700 1 -- 172.21.15.2:0/3578602366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 msgr2=0x7f058010ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0586d8f700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f058010ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0586d8f700 1 -- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 msgr2=0x7f058010f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.541+0000 7f0586d8f700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 0x7f058010f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0586d8f700 1 -- 172.21.15.2:0/3578602366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c00b360 con 0x7f05800fdc00 2022-03-21T14:08:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0586d8f700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 0x7f05801162c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0578009230 tx=0x7f05780094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057801c070 con 0x7f05800fdc00 2022-03-21T14:08:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0578009780 con 0x7f05800fdc00 2022-03-21T14:08:18.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580117060 con 0x7f05800fdc00 2022-03-21T14:08:18.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f057800bca0 con 0x7f05800fdc00 2022-03-21T14:08:18.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.542+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578003d70 con 0x7f05800fdc00 2022-03-21T14:08:18.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.544+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580107830 con 0x7f05800fdc00 2022-03-21T14:08:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.545+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f057800c040 con 0x7f05800fdc00 2022-03-21T14:08:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.546+0000 7f0576ffd700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 0x7f056c0991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.546+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f05780c10b0 con 0x7f05800fdc00 2022-03-21T14:08:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.547+0000 7f058658e700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 0x7f056c0991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.547+0000 7f058658e700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 0x7f056c0991c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0580110220 tx=0x7f057c005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.550+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f057808e2d0 con 0x7f05800fdc00 2022-03-21T14:08:18.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.737+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05801079c0 con 0x7f05800fdc00 2022-03-21T14:08:18.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.738+0000 7f0576ffd700 1 -- 172.21.15.2:0/3578602366 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05801079c0 con 0x7f05800fdc00 2022-03-21T14:08:18.739 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-03-21T14:08:18.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 msgr2=0x7f056c0991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 0x7f056c0991c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0580110220 tx=0x7f057c005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 msgr2=0x7f05801162c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 0x7f05801162c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0578009230 tx=0x7f05780094e0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 shutdown_connections 2022-03-21T14:08:18.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05800fbfd0 0x7f058010ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f056c096d10 0x7f056c0991c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05800fdc00 0x7f05801162c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 --2- 172.21.15.2:0/3578602366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05800fd270 0x7f058010f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 >> 172.21.15.2:0/3578602366 conn(0x7f05800f7780 msgr2=0x7f05800f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:18.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 shutdown_connections 2022-03-21T14:08:18.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.740+0000 7f0587590700 1 -- 172.21.15.2:0/3578602366 wait complete. 2022-03-21T14:08:18.766 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-03-21T14:08:18.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 -- 172.21.15.2:0/621728144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fdc00 msgr2=0x7f44f01044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 --2- 172.21.15.2:0/621728144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fdc00 0x7f44f01044c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009ba0 tx=0x7f44e8009430 comp rx=0 tx=0).stop 2022-03-21T14:08:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 -- 172.21.15.2:0/621728144 shutdown_connections 2022-03-21T14:08:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 --2- 172.21.15.2:0/621728144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fdc00 0x7f44f01044c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 --2- 172.21.15.2:0/621728144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fbfd0 0x7f44f00fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 --2- 172.21.15.2:0/621728144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fd270 0x7f44f00fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 -- 172.21.15.2:0/621728144 >> 172.21.15.2:0/621728144 conn(0x7f44f00f7780 msgr2=0x7f44f00f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 -- 172.21.15.2:0/621728144 shutdown_connections 2022-03-21T14:08:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.974+0000 7f44ef59e700 1 -- 172.21.15.2:0/621728144 wait complete. 2022-03-21T14:08:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 Processor -- start 2022-03-21T14:08:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 -- start start 2022-03-21T14:08:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fbfd0 0x7f44f0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fdc00 0x7f44f006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006ff70 con 0x7f44f00fd270 2022-03-21T14:08:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00700e0 con 0x7f44f00fbfd0 2022-03-21T14:08:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.975+0000 7f44ef59e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f0116bd0 con 0x7f44f00fdc00 2022-03-21T14:08:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44030/0 (socket says 172.21.15.2:44030) 2022-03-21T14:08:18.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 -- 172.21.15.2:0/115168200 learned_addr learned my addr 172.21.15.2:0/115168200 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:18.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44ee59c700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fbfd0 0x7f44f0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44eed9d700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fdc00 0x7f44f006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 -- 172.21.15.2:0/115168200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fdc00 msgr2=0x7f44f006f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fdc00 0x7f44f006f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 -- 172.21.15.2:0/115168200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fbfd0 msgr2=0x7f44f0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:18.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fbfd0 0x7f44f0070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:18.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 -- 172.21.15.2:0/115168200 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80096b0 con 0x7f44f00fd270 2022-03-21T14:08:18.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.976+0000 7f44edd9b700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f44e400fa90 tx=0x7f44e4005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.977+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e400b4f0 con 0x7f44f00fd270 2022-03-21T14:08:18.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.977+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f44e400dba0 con 0x7f44f00fd270 2022-03-21T14:08:18.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.977+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f0116dd0 con 0x7f44f00fd270 2022-03-21T14:08:18.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.977+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4011c90 con 0x7f44f00fd270 2022-03-21T14:08:18.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.977+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f01172e0 con 0x7f44f00fd270 2022-03-21T14:08:18.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.978+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f0107870 con 0x7f44f00fd270 2022-03-21T14:08:18.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.980+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f44e400c040 con 0x7f44f00fd270 2022-03-21T14:08:18.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.981+0000 7f44deffd700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 0x7f44d80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:18.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.981+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f44e40bf830 con 0x7f44f00fd270 2022-03-21T14:08:18.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.981+0000 7f44ee59c700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 0x7f44d80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:18.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.982+0000 7f44ee59c700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 0x7f44d80991c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44f00fd0d0 tx=0x7f44e0005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:18.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:18.985+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f44e408ca50 con 0x7f44f00fd270 2022-03-21T14:08:19.172 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.171+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44f002ac40 con 0x7f44f00fd270 2022-03-21T14:08:19.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.172+0000 7f44deffd700 1 -- 172.21.15.2:0/115168200 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44e4090070 con 0x7f44f00fd270 2022-03-21T14:08:19.174 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-03-21T14:08:19.176 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 msgr2=0x7f44d80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.176 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 0x7f44d80991c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44f00fd0d0 tx=0x7f44e0005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:19.177 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 msgr2=0x7f44f006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.178 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f44e400fa90 tx=0x7f44e4005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 shutdown_connections 2022-03-21T14:08:19.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f00fbfd0 0x7f44f0070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f44d8096d10 0x7f44d80991c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44f00fdc00 0x7f44f006f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 --2- 172.21.15.2:0/115168200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44f00fd270 0x7f44f006f320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.174+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 >> 172.21.15.2:0/115168200 conn(0x7f44f00f7780 msgr2=0x7f44f00f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:19.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.175+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 shutdown_connections 2022-03-21T14:08:19.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.175+0000 7f44ef59e700 1 -- 172.21.15.2:0/115168200 wait complete. 2022-03-21T14:08:19.200 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-03-21T14:08:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:19 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/2003268646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:19 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3578602366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:19.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.411+0000 7ffa3f168700 1 -- 172.21.15.2:0/1901405017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa380ff160 msgr2=0x7ffa381015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.411+0000 7ffa3f168700 1 --2- 172.21.15.2:0/1901405017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa380ff160 0x7ffa381015a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa280098e0 tx=0x7ffa28005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:19.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 -- 172.21.15.2:0/1901405017 shutdown_connections 2022-03-21T14:08:19.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 --2- 172.21.15.2:0/1901405017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa38065200 0x7ffa380feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 --2- 172.21.15.2:0/1901405017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38101c30 0x7ffa381040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 --2- 172.21.15.2:0/1901405017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa380ff160 0x7ffa381015a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 -- 172.21.15.2:0/1901405017 >> 172.21.15.2:0/1901405017 conn(0x7ffa380f66a0 msgr2=0x7ffa380f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:19.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 -- 172.21.15.2:0/1901405017 shutdown_connections 2022-03-21T14:08:19.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.412+0000 7ffa3f168700 1 -- 172.21.15.2:0/1901405017 wait complete. 2022-03-21T14:08:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:19 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/2003268646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:19 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3578602366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:19.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 Processor -- start 2022-03-21T14:08:19.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 -- start start 2022-03-21T14:08:19.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 0x7ffa3810efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa38101c30 0x7ffa38116580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38116ac0 con 0x7ffa38101c30 2022-03-21T14:08:19.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38116c30 con 0x7ffa380ff160 2022-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3f168700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38116df0 con 0x7ffa38065200 2022-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3d965700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3e967700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa38101c30 0x7ffa38116580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3d965700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.2:48618/0 (socket says 172.21.15.2:48618) 2022-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3d965700 1 -- 172.21.15.2:0/4208865439 learned_addr learned my addr 172.21.15.2:0/4208865439 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:19.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.413+0000 7ffa3e166700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 0x7ffa3810efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 -- 172.21.15.2:0/4208865439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 msgr2=0x7ffa3810efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 0x7ffa3810efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.421 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 -- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa38101c30 msgr2=0x7ffa38116580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa38101c30 0x7ffa38116580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 -- 172.21.15.2:0/4208865439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280093f0 con 0x7ffa380ff160 2022-03-21T14:08:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3e166700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 0x7ffa3810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa3d965700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffa280093c0 tx=0x7ffa28005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.414+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2801d070 con 0x7ffa380ff160 2022-03-21T14:08:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.415+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa38117070 con 0x7ffa380ff160 2022-03-21T14:08:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.415+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7ffa28000c90 con 0x7ffa380ff160 2022-03-21T14:08:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.415+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28017990 con 0x7ffa380ff160 2022-03-21T14:08:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.415+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa38117500 con 0x7ffa380ff160 2022-03-21T14:08:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.416+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa381077b0 con 0x7ffa380ff160 2022-03-21T14:08:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.417+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7ffa28028030 con 0x7ffa380ff160 2022-03-21T14:08:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.418+0000 7ffa2effd700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 0x7ffa240991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.419+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7ffa280c09c0 con 0x7ffa380ff160 2022-03-21T14:08:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.419+0000 7ffa3e166700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 0x7ffa240991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.420+0000 7ffa3e166700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 0x7ffa240991c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffa380655f0 tx=0x7ffa34005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.423+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7ffa2808dc40 con 0x7ffa380ff160 2022-03-21T14:08:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:19 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/2003268646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:19 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3578602366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-03-21T14:08:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.609+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa38107940 con 0x7ffa380ff160 2022-03-21T14:08:19.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.610+0000 7ffa2effd700 1 -- 172.21.15.2:0/4208865439 <== mon.1 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffa38107940 con 0x7ffa380ff160 2022-03-21T14:08:19.611 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-03-21T14:08:19.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 msgr2=0x7ffa240991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 0x7ffa240991c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffa380655f0 tx=0x7ffa34005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 msgr2=0x7ffa3810f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffa280093c0 tx=0x7ffa28005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:19.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 shutdown_connections 2022-03-21T14:08:19.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7ffa24096d10 0x7ffa240991c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa38101c30 0x7ffa38116580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38065200 0x7ffa3810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 --2- 172.21.15.2:0/4208865439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa380ff160 0x7ffa3810f4f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 >> 172.21.15.2:0/4208865439 conn(0x7ffa380f66a0 msgr2=0x7ffa380f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:19.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 shutdown_connections 2022-03-21T14:08:19.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.612+0000 7ffa3f168700 1 -- 172.21.15.2:0/4208865439 wait complete. 2022-03-21T14:08:19.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-03-21T14:08:19.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 -- 172.21.15.2:0/4076460882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f04066080 msgr2=0x7f2f040fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 --2- 172.21.15.2:0/4076460882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f04066080 0x7f2f040fec30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000b820 tx=0x7f2f00005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 -- 172.21.15.2:0/4076460882 shutdown_connections 2022-03-21T14:08:19.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 --2- 172.21.15.2:0/4076460882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f04066080 0x7f2f040fec30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 --2- 172.21.15.2:0/4076460882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04101c20 0x7f2f041040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 --2- 172.21.15.2:0/4076460882 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f040ff170 0x7f2f04101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 -- 172.21.15.2:0/4076460882 >> 172.21.15.2:0/4076460882 conn(0x7f2f040f66a0 msgr2=0x7f2f040f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:19.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 -- 172.21.15.2:0/4076460882 shutdown_connections 2022-03-21T14:08:19.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.843+0000 7f2f0a281700 1 -- 172.21.15.2:0/4076460882 wait complete. 2022-03-21T14:08:19.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.844+0000 7f2f0a281700 1 Processor -- start 2022-03-21T14:08:19.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.844+0000 7f2f0a281700 1 -- start start 2022-03-21T14:08:19.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f040ff170 0x7f2f0410f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 0x7f2f04116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0927f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04116a60 con 0x7f2f04101c20 2022-03-21T14:08:19.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04116bd0 con 0x7f2f040ff170 2022-03-21T14:08:19.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0a281700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04116d90 con 0x7f2f04066080 2022-03-21T14:08:19.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0927f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39884/0 (socket says 172.21.15.2:39884) 2022-03-21T14:08:19.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f0927f700 1 -- 172.21.15.2:0/1819779330 learned_addr learned my addr 172.21.15.2:0/1819779330 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:19.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f09a80700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 0x7f2f04116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.845+0000 7f2f08a7e700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f040ff170 0x7f2f0410f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 -- 172.21.15.2:0/1819779330 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 msgr2=0x7f2f0410ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 -- 172.21.15.2:0/1819779330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f040ff170 msgr2=0x7f2f0410f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:19.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f040ff170 0x7f2f0410f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:19.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 -- 172.21.15.2:0/1819779330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40093f0 con 0x7f2f04101c20 2022-03-21T14:08:19.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f0927f700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:19.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f09a80700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 0x7f2f04116520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00ebf0 tx=0x7f2efc009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:19.853 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc009990 con 0x7f2f04101c20 2022-03-21T14:08:19.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.846+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc0096b0 con 0x7f2f04101c20 2022-03-21T14:08:19.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.847+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2efc004a80 con 0x7f2f04101c20 2022-03-21T14:08:19.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.847+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc005860 con 0x7f2f04101c20 2022-03-21T14:08:19.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.847+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0000b360 con 0x7f2f04101c20 2022-03-21T14:08:19.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.848+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f041077b0 con 0x7f2f04101c20 2022-03-21T14:08:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.849+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f2efc00c040 con 0x7f2f04101c20 2022-03-21T14:08:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.851+0000 7f2ef9ffb700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 0x7f2ef00991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.851+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f2efc014070 con 0x7f2f04101c20 2022-03-21T14:08:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.851+0000 7f2f0927f700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 0x7f2ef00991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.852+0000 7f2f0927f700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 0x7f2ef00991c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000be90 tx=0x7f2f00005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:19.855+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f2efc08d580 con 0x7f2f04101c20 2022-03-21T14:08:20.084 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.083+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f04107940 con 0x7f2f04101c20 2022-03-21T14:08:20.085 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.084+0000 7f2ef9ffb700 1 -- 172.21.15.2:0/1819779330 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10650 (secure 0 0 0) 0x7f2f04107940 con 0x7f2f04101c20 2022-03-21T14:08:20.085 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:20.085 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2330341423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2330341423},{"type":"v1","addr":"172.21.15.2:6839","nonce":2330341423}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14442,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/1904085966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1904085966},{"type":"v1","addr":"172.21.15.2:6841","nonce":1904085966}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2676603000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2676603000},{"type":"v1","addr":"172.21.15.2:6835","nonce":2676603000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/595788412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":595788412},{"type":"v1","addr":"172.21.15.2:6837","nonce":595788412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/3142985186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3142985186},{"type":"v1","addr":"172.21.15.16:6833","nonce":3142985186}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2493160775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2493160775},{"type":"v1","addr":"172.21.15.16:6835","nonce":2493160775}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24529,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2678336191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2678336191},{"type":"v1","addr":"172.21.15.16:6837","nonce":2678336191}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4173795056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4173795056},{"type":"v1","addr":"172.21.15.16:6839","nonce":4173795056}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2500439348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2500439348},{"type":"v1","addr":"172.21.15.79:6833","nonce":2500439348}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/805336294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":805336294},{"type":"v1","addr":"172.21.15.79:6835","nonce":805336294}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-03-21T14:06:54.837756+0000","modified":"2022-03-21T14:06:59.487834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.79:6837/2188072757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2188072757},{"type":"v1","addr":"172.21.15.79:6837","nonce":2188072757}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.79:6839/1907413140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1907413140},{"type":"v1","addr":"172.21.15.79:6839","nonce":1907413140}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-03-21T14:08:20.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 msgr2=0x7f2ef00991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 0x7f2ef00991c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000be90 tx=0x7f2f00005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.088 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 msgr2=0x7f2f04116520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 0x7f2f04116520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00ebf0 tx=0x7f2efc009430 comp rx=0 tx=0).stop 2022-03-21T14:08:20.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 shutdown_connections 2022-03-21T14:08:20.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f2ef0096d10 0x7f2ef00991c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.089 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f04101c20 0x7f2f04116520 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04066080 0x7f2f0410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 --2- 172.21.15.2:0/1819779330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f040ff170 0x7f2f0410f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 >> 172.21.15.2:0/1819779330 conn(0x7f2f040f66a0 msgr2=0x7f2f040f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 shutdown_connections 2022-03-21T14:08:20.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.087+0000 7f2f0a281700 1 -- 172.21.15.2:0/1819779330 wait complete. 2022-03-21T14:08:20.091 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 20 2022-03-21T14:08:20.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-03-21T14:08:20.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.318+0000 7f524c93f700 1 -- 172.21.15.2:0/1137682748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fd250 msgr2=0x7f52480fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.318+0000 7f524c93f700 1 --2- 172.21.15.2:0/1137682748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fd250 0x7f52480fd6a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f523c0098e0 tx=0x7f523c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.318+0000 7f524c93f700 1 -- 172.21.15.2:0/1137682748 shutdown_connections 2022-03-21T14:08:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.318+0000 7f524c93f700 1 --2- 172.21.15.2:0/1137682748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fbfb0 0x7f52480fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.318+0000 7f524c93f700 1 --2- 172.21.15.2:0/1137682748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fdbe0 0x7f52481044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.319+0000 7f524c93f700 1 --2- 172.21.15.2:0/1137682748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fd250 0x7f52480fd6a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.319+0000 7f524c93f700 1 -- 172.21.15.2:0/1137682748 >> 172.21.15.2:0/1137682748 conn(0x7f52480f7760 msgr2=0x7f52480f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.319+0000 7f524c93f700 1 -- 172.21.15.2:0/1137682748 shutdown_connections 2022-03-21T14:08:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.319+0000 7f524c93f700 1 -- 172.21.15.2:0/1137682748 wait complete. 2022-03-21T14:08:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.319+0000 7f524c93f700 1 Processor -- start 2022-03-21T14:08:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 -- start start 2022-03-21T14:08:20.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 0x7f5248102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fdbe0 0x7f52480fe880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480fef90 con 0x7f52480fbfb0 2022-03-21T14:08:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480ff100 con 0x7f52480fd250 2022-03-21T14:08:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524c93f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480ff2c0 con 0x7f52480fdbe0 2022-03-21T14:08:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f5246d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f5246d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44044/0 (socket says 172.21.15.2:44044) 2022-03-21T14:08:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f5246d9d700 1 -- 172.21.15.2:0/3441697902 learned_addr learned my addr 172.21.15.2:0/3441697902 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524759e700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fdbe0 0x7f52480fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.320+0000 7f524659c700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 0x7f5248102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 -- 172.21.15.2:0/3441697902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fdbe0 msgr2=0x7f52480fe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fdbe0 0x7f52480fe880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 -- 172.21.15.2:0/3441697902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 msgr2=0x7f5248102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 0x7f5248102280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 -- 172.21.15.2:0/3441697902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0093f0 con 0x7f52480fbfb0 2022-03-21T14:08:20.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f524659c700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 0x7f5248102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:08:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f5246d9d700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f523800f860 tx=0x7f5238005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238004020 con 0x7f52480fbfb0 2022-03-21T14:08:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524811b410 con 0x7f52480fbfb0 2022-03-21T14:08:20.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5238004180 con 0x7f52480fbfb0 2022-03-21T14:08:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380039f0 con 0x7f52480fbfb0 2022-03-21T14:08:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.321+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524811b9d0 con 0x7f52480fbfb0 2022-03-21T14:08:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.322+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52481077a0 con 0x7f52480fbfb0 2022-03-21T14:08:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.324+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7f5238022060 con 0x7f52480fbfb0 2022-03-21T14:08:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.325+0000 7f522f7fe700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 0x7f52300991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.326+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7f52380bf550 con 0x7f52480fbfb0 2022-03-21T14:08:20.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.326+0000 7f524659c700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 0x7f52300991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.326+0000 7f524659c700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 0x7f52300991c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f523c009f50 tx=0x7f523c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.329+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7f523808c770 con 0x7f52480fbfb0 2022-03-21T14:08:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:20 smithi002 ceph-mon[17167]: pgmap v313: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:20 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/115168200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:20 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4208865439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:20 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/1819779330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:20 smithi016 ceph-mon[22010]: pgmap v313: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:20 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/115168200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:20 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4208865439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:20 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/1819779330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:20 smithi079 ceph-mon[23158]: pgmap v313: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:20 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/115168200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:20 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4208865439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-03-21T14:08:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:20 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/1819779330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-03-21T14:08:20.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.521+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5248107930 con 0x7f52480fbfb0 2022-03-21T14:08:20.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.522+0000 7f522f7fe700 1 -- 172.21.15.2:0/3441697902 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+21650 (secure 0 0 0) 0x7f5248107930 con 0x7f52480fbfb0 2022-03-21T14:08:20.523 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-03-21T14:08:20.524 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":84,"fsid":"ee5e78f2-a91e-11ec-8c35-001a4aab830c","created":"2022-03-21T13:59:17.765843+0000","modified":"2022-03-21T14:07:21.862918+0000","last_up_change":"2022-03-21T14:06:02.453955+0000","last_in_change":"2022-03-21T14:05:44.266362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:02:31.166181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-03-21T14:06:52.815266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-03-21T14:06:54.292010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"13405266-d8e8-4542-bf91-9450e11d2ecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6803","nonce":1386749192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6805","nonce":1386749192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6809","nonce":1386749192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1386749192},{"type":"v1","addr":"172.21.15.2:6807","nonce":1386749192}]},"public_addr":"172.21.15.2:6803/1386749192","cluster_addr":"172.21.15.2:6805/1386749192","heartbeat_back_addr":"172.21.15.2:6809/1386749192","heartbeat_front_addr":"172.21.15.2:6807/1386749192","state":["exists","up"]},{"osd":1,"uuid":"e1d0a7a0-443f-4117-aed8-0078291c7c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6801","nonce":491865625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6803","nonce":491865625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6807","nonce":491865625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":491865625},{"type":"v1","addr":"172.21.15.16:6805","nonce":491865625}]},"public_addr":"172.21.15.16:6801/491865625","cluster_addr":"172.21.15.16:6803/491865625","heartbeat_back_addr":"172.21.15.16:6807/491865625","heartbeat_front_addr":"172.21.15.16:6805/491865625","state":["exists","up"]},{"osd":2,"uuid":"c1f5778a-ca84-43c0-8233-005fbbf6da43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6801","nonce":4178164596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6803","nonce":4178164596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6807","nonce":4178164596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":4178164596},{"type":"v1","addr":"172.21.15.79:6805","nonce":4178164596}]},"public_addr":"172.21.15.79:6801/4178164596","cluster_addr":"172.21.15.79:6803/4178164596","heartbeat_back_addr":"172.21.15.79:6807/4178164596","heartbeat_front_addr":"172.21.15.79:6805/4178164596","state":["exists","up"]},{"osd":3,"uuid":"c84afd54-8463-4dc6-ac84-f8c56d385bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6811","nonce":1119340852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6813","nonce":1119340852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6817","nonce":1119340852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1119340852},{"type":"v1","addr":"172.21.15.2:6815","nonce":1119340852}]},"public_addr":"172.21.15.2:6811/1119340852","cluster_addr":"172.21.15.2:6813/1119340852","heartbeat_back_addr":"172.21.15.2:6817/1119340852","heartbeat_front_addr":"172.21.15.2:6815/1119340852","state":["exists","up"]},{"osd":4,"uuid":"e7283335-3a3f-4840-b028-f85250ba113e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6809","nonce":530720703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6811","nonce":530720703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6815","nonce":530720703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":530720703},{"type":"v1","addr":"172.21.15.16:6813","nonce":530720703}]},"public_addr":"172.21.15.16:6809/530720703","cluster_addr":"172.21.15.16:6811/530720703","heartbeat_back_addr":"172.21.15.16:6815/530720703","heartbeat_front_addr":"172.21.15.16:6813/530720703","state":["exists","up"]},{"osd":5,"uuid":"574f01bb-e554-4aa2-bb51-627091c73352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6809","nonce":1459081452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6811","nonce":1459081452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6815","nonce":1459081452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1459081452},{"type":"v1","addr":"172.21.15.79:6813","nonce":1459081452}]},"public_addr":"172.21.15.79:6809/1459081452","cluster_addr":"172.21.15.79:6811/1459081452","heartbeat_back_addr":"172.21.15.79:6815/1459081452","heartbeat_front_addr":"172.21.15.79:6813/1459081452","state":["exists","up"]},{"osd":6,"uuid":"b9e16547-b033-47a4-a8a6-8b17c953ce5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6819","nonce":2687857259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6821","nonce":2687857259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6825","nonce":2687857259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2687857259},{"type":"v1","addr":"172.21.15.2:6823","nonce":2687857259}]},"public_addr":"172.21.15.2:6819/2687857259","cluster_addr":"172.21.15.2:6821/2687857259","heartbeat_back_addr":"172.21.15.2:6825/2687857259","heartbeat_front_addr":"172.21.15.2:6823/2687857259","state":["exists","up"]},{"osd":7,"uuid":"65e72a50-adfb-492a-b31e-68e1fa96fb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6817","nonce":3746016674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6819","nonce":3746016674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6823","nonce":3746016674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3746016674},{"type":"v1","addr":"172.21.15.16:6821","nonce":3746016674}]},"public_addr":"172.21.15.16:6817/3746016674","cluster_addr":"172.21.15.16:6819/3746016674","heartbeat_back_addr":"172.21.15.16:6823/3746016674","heartbeat_front_addr":"172.21.15.16:6821/3746016674","state":["exists","up"]},{"osd":8,"uuid":"680e2f3a-ec15-482d-9734-1691c5c013d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6817","nonce":2185003031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6819","nonce":2185003031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6823","nonce":2185003031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2185003031},{"type":"v1","addr":"172.21.15.79:6821","nonce":2185003031}]},"public_addr":"172.21.15.79:6817/2185003031","cluster_addr":"172.21.15.79:6819/2185003031","heartbeat_back_addr":"172.21.15.79:6823/2185003031","heartbeat_front_addr":"172.21.15.79:6821/2185003031","state":["exists","up"]},{"osd":9,"uuid":"c640ec62-e489-4b6d-bd4e-3e4b609505ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6827","nonce":2878353867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6829","nonce":2878353867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6833","nonce":2878353867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":2878353867},{"type":"v1","addr":"172.21.15.2:6831","nonce":2878353867}]},"public_addr":"172.21.15.2:6827/2878353867","cluster_addr":"172.21.15.2:6829/2878353867","heartbeat_back_addr":"172.21.15.2:6833/2878353867","heartbeat_front_addr":"172.21.15.2:6831/2878353867","state":["exists","up"]},{"osd":10,"uuid":"ebc0ecb2-703e-4108-8e30-c6f9f4b10cd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6825","nonce":4107732214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6827","nonce":4107732214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6831","nonce":4107732214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":4107732214},{"type":"v1","addr":"172.21.15.16:6829","nonce":4107732214}]},"public_addr":"172.21.15.16:6825/4107732214","cluster_addr":"172.21.15.16:6827/4107732214","heartbeat_back_addr":"172.21.15.16:6831/4107732214","heartbeat_front_addr":"172.21.15.16:6829/4107732214","state":["exists","up"]},{"osd":11,"uuid":"31a5878a-28c3-42a5-80c4-a032b07fc71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6825","nonce":1987240625}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6827","nonce":1987240625}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6831","nonce":1987240625}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":1987240625},{"type":"v1","addr":"172.21.15.79:6829","nonce":1987240625}]},"public_addr":"172.21.15.79:6825/1987240625","cluster_addr":"172.21.15.79:6827/1987240625","heartbeat_back_addr":"172.21.15.79:6831/1987240625","heartbeat_front_addr":"172.21.15.79:6829/1987240625","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:01:47.238720+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:08.716387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:29.288488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:02:58.407056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:19.933426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:03:40.819285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:07.291109+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:28.778065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:04:49.849890+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:16.910083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:05:39.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:06:00.664045+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.16","mappings":[{"from":1,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3018413483":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/225365632":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:0/3665116503":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6800/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2302345157":"2022-03-22T13:59:54.290862+0000","172.21.15.2:6801/1086540904":"2022-03-22T14:00:20.256801+0000","172.21.15.2:6801/2638362969":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/2219526494":"2022-03-22T13:59:54.290862+0000","172.21.15.2:0/1953025189":"2022-03-22T13:59:54.290862+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:08:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 msgr2=0x7f52300991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 0x7f52300991c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f523c009f50 tx=0x7f523c01a040 comp rx=0 tx=0).stop 2022-03-21T14:08:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 msgr2=0x7f5248101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f523800f860 tx=0x7f5238005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 shutdown_connections 2022-03-21T14:08:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7f5230096d10 0x7f52300991c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52480fdbe0 0x7f52480fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52480fbfb0 0x7f5248101d40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 --2- 172.21.15.2:0/3441697902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52480fd250 0x7f5248102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 >> 172.21.15.2:0/3441697902 conn(0x7f52480f7760 msgr2=0x7f52480f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 shutdown_connections 2022-03-21T14:08:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.525+0000 7f524c93f700 1 -- 172.21.15.2:0/3441697902 wait complete. 2022-03-21T14:08:20.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-03-21T14:06:54.837756+0000', 'modified': '2022-03-21T14:06:59.487834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.79:6837/2188072757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6836', 'nonce': 2188072757}, {'type': 'v1', 'addr': '172.21.15.79:6837', 'nonce': 2188072757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.79:6839/1907413140', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6838', 'nonce': 1907413140}, {'type': 'v1', 'addr': '172.21.15.79:6839', 'nonce': 1907413140}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-03-21T14:08:20.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-03-21T14:08:20.552 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-03-21T14:08:20.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 -- 172.21.15.2:0/4286457453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 msgr2=0x7fa9300fe5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 --2- 172.21.15.2:0/4286457453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa9300fe5d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0098e0 tx=0x7fa92c005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:20.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 -- 172.21.15.2:0/4286457453 shutdown_connections 2022-03-21T14:08:20.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 --2- 172.21.15.2:0/4286457453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa9300fe5d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 --2- 172.21.15.2:0/4286457453 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ffdf0 0x7fa9301066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 --2- 172.21.15.2:0/4286457453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ff460 0x7fa9300ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 -- 172.21.15.2:0/4286457453 >> 172.21.15.2:0/4286457453 conn(0x7fa9300f9990 msgr2=0x7fa9300fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 -- 172.21.15.2:0/4286457453 shutdown_connections 2022-03-21T14:08:20.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.757+0000 7fa937519700 1 -- 172.21.15.2:0/4286457453 wait complete. 2022-03-21T14:08:20.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.758+0000 7fa937519700 1 Processor -- start 2022-03-21T14:08:20.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.758+0000 7fa937519700 1 -- start start 2022-03-21T14:08:20.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa93010ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ff460 0x7fa93010f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930116a10 con 0x7fa9300ffdf0 2022-03-21T14:08:20.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930116b80 con 0x7fa9300fe1c0 2022-03-21T14:08:20.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa937519700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930116d40 con 0x7fa9300ff460 2022-03-21T14:08:20.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa936d18700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa936d18700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44048/0 (socket says 172.21.15.2:44048) 2022-03-21T14:08:20.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa936517700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa93010ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa935d16700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ff460 0x7fa93010f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.759+0000 7fa936d18700 1 -- 172.21.15.2:0/1134096054 learned_addr learned my addr 172.21.15.2:0/1134096054 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 -- 172.21.15.2:0/1134096054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ff460 msgr2=0x7fa93010f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ff460 0x7fa93010f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 -- 172.21.15.2:0/1134096054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 msgr2=0x7fa93010ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa93010ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 -- 172.21.15.2:0/1134096054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0093f0 con 0x7fa9300ffdf0 2022-03-21T14:08:20.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936517700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa93010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:20.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa936d18700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa92800eaf0 tx=0x7fa9280096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa926ffd700 1 -- 172.21.15.2:0/1134096054 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92800c970 con 0x7fa9300ffdf0 2022-03-21T14:08:20.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa926ffd700 1 -- 172.21.15.2:0/1134096054 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa928003840 con 0x7fa9300ffdf0 2022-03-21T14:08:20.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa926ffd700 1 -- 172.21.15.2:0/1134096054 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9280043c0 con 0x7fa9300ffdf0 2022-03-21T14:08:20.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa930116fc0 con 0x7fa9300ffdf0 2022-03-21T14:08:20.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.760+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9301174b0 con 0x7fa9300ffdf0 2022-03-21T14:08:20.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.763+0000 7fa926ffd700 1 -- 172.21.15.2:0/1134096054 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fa928020060 con 0x7fa9300ffdf0 2022-03-21T14:08:20.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.764+0000 7fa9420ecb80 1 Processor -- start 2022-03-21T14:08:20.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 -- start start 2022-03-21T14:08:20.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa926ffd700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 0x7fa91c099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa926ffd700 1 -- 172.21.15.2:0/1134096054 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fa928015070 con 0x7fa9300ffdf0 2022-03-21T14:08:20.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c81760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06bff240 con 0x559e06c80e20 2022-03-21T14:08:20.778 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b4f090 con 0x559e06c804d0 2022-03-21T14:08:20.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06bd4bf0 con 0x559e06c81d80 2022-03-21T14:08:20.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa935d16700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 0x7fa91c099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c81760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa936d18700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa936d18700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39898/0 (socket says 172.21.15.2:39898) 2022-03-21T14:08:20.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c81760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44050/0 (socket says 172.21.15.2:44050) 2022-03-21T14:08:20.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa936d18700 1 -- 172.21.15.2:0/1631168592 learned_addr learned my addr 172.21.15.2:0/1631168592 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.765+0000 7fa936517700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 -- 172.21.15.2:0/1631168592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 msgr2=0x559e06c808e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c808e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 -- 172.21.15.2:0/1631168592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 msgr2=0x559e06c81760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c81760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 -- 172.21.15.2:0/1631168592 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e06a65360 con 0x559e06c81d80 2022-03-21T14:08:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa936d18700 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa928090710 tx=0x7fa92808cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b11e4accb30b077b server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa935d16700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 0x7fa91c099240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa92000b5d0 tx=0x7fa920005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa913fff700 1 -- 172.21.15.2:0/1631168592 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928090780 con 0x559e06c81d80 2022-03-21T14:08:20.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa913fff700 1 -- 172.21.15.2:0/1631168592 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa9280a3070 con 0x559e06c81d80 2022-03-21T14:08:20.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.766+0000 7fa913fff700 1 -- 172.21.15.2:0/1631168592 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9280951c0 con 0x559e06c81d80 2022-03-21T14:08:20.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1631168592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 msgr2=0x559e06c821d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.783 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa928090710 tx=0x7fa92808cb70 comp rx=0 tx=0).stop 2022-03-21T14:08:20.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1631168592 shutdown_connections 2022-03-21T14:08:20.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c808e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c821d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1631168592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c81760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1631168592 >> 172.21.15.2:0/1631168592 conn(0x559e06c7b340 msgr2=0x559e06c7b730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1631168592 shutdown_connections 2022-03-21T14:08:20.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.767+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1631168592 wait complete. 2022-03-21T14:08:20.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.768+0000 7fa9420ecb80 1 -- start start 2022-03-21T14:08:20.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.768+0000 7fa9420ecb80 1 Processor -- start 2022-03-21T14:08:20.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c804d0 0x559e06c89c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06bcecd0 con 0x559e06c80e20 2022-03-21T14:08:20.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06a67650 con 0x559e06c81d80 2022-03-21T14:08:20.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c917e0 con 0x559e06c804d0 2022-03-21T14:08:20.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa9420ecb80 10 client.0 _ll_register_callbacks cb 0x559e069e07b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-03-21T14:08:20.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44054/0 (socket says 172.21.15.2:44054) 2022-03-21T14:08:20.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa935d16700 1 -- 172.21.15.2:0/4201606482 learned_addr learned my addr 172.21.15.2:0/4201606482 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936517700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c804d0 0x559e06c89c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 -- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c804d0 msgr2=0x559e06c89c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c804d0 0x559e06c89c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 -- 172.21.15.2:0/4201606482 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 msgr2=0x559e06c8a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa936d18700 1 -- 172.21.15.2:0/4201606482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9301174b0 con 0x559e06c81d80 2022-03-21T14:08:20.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.769+0000 7fa935d16700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:20.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa936d18700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c911b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa92808cc80 tx=0x7fa92808cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10b1c62bcc054f2 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9117fa700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2022-03-21T14:08:20.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mon.1 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928090780 con 0x559e06c81d80 2022-03-21T14:08:20.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mon.1 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fa92808eb20 con 0x559e06c81d80 2022-03-21T14:08:20.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e06c73ac0 con 0x559e06c81d80 2022-03-21T14:08:20.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mon.1 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa928017540 con 0x559e06c81d80 2022-03-21T14:08:20.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9420ecb80 10 client.25024 fetch_fsmap learned FSMap version 20 2022-03-21T14:08:20.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559e06a0ffe0 con 0x559e06c81d80 2022-03-21T14:08:20.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.770+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mon.1 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x559e06c73ac0 con 0x559e06c81d80 2022-03-21T14:08:20.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mon.1 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fa92808fd70 con 0x559e06c81d80 2022-03-21T14:08:20.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 10 client.25024 fetch_fsmap finished waiting for FSMap version 20 2022-03-21T14:08:20.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 10 client.25024 resolve_mds: resolved name 'l' to daemon mds.l 2022-03-21T14:08:20.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 20 client.25024 populate_metadata read hostname 'smithi002' 2022-03-21T14:08:20.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06c94900 0x559e06c99be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 4 client.25024 mds_command: new command op to 24587 tid=0[{"prefix": "get_command_descriptions"}] 2022-03-21T14:08:20.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 --> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e06c50640 con 0x559e06c94900 2022-03-21T14:08:20.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.771+0000 7fa936517700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06c94900 0x559e06c99be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.772+0000 7fa9117fa700 10 client.25024 ms_handle_connect on v2:172.21.15.79:6838/1907413140 2022-03-21T14:08:20.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.772+0000 7fa936517700 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06c94900 0x559e06c99be0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fe9202e44dce464 server_cookie=87ecbe386fc83ccd in_seq=0 out_seq=0 2022-03-21T14:08:20.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.772+0000 7fa9117fa700 1 -- 172.21.15.2:0/4201606482 <== mds.0 v2:172.21.15.79:6838/1907413140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17246 (crc 0 0 0) 0x559e06c50640 con 0x559e06c94900 2022-03-21T14:08:20.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.772+0000 7fa9117fa700 10 client.25024 handle_command_reply: tid=0 2022-03-21T14:08:20.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.772+0000 7fa9420ecb80 1 client.25024 shutdown 2022-03-21T14:08:20.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 msgr2=0x559e06c911b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c911b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa92808cc80 tx=0x7fa92808cb70 comp rx=0 tx=0).stop 2022-03-21T14:08:20.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 shutdown_connections 2022-03-21T14:08:20.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06c94900 0x559e06c99be0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c911b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9117fa700 0 client.25024 ms_handle_reset on v2:172.21.15.79:6838/1907413140 2022-03-21T14:08:20.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c804d0 0x559e06c89c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/4201606482 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 >> 172.21.15.2:0/4201606482 conn(0x559e06c7b340 msgr2=0x559e06c7a600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 shutdown_connections 2022-03-21T14:08:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 1 -- 172.21.15.2:0/4201606482 wait complete. 2022-03-21T14:08:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.773+0000 7fa9420ecb80 20 client.25024 trim_cache size 0 max 16384 2022-03-21T14:08:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 Processor -- start 2022-03-21T14:08:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 -- start start 2022-03-21T14:08:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c89bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c7a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c8ab30 con 0x559e06c80e20 2022-03-21T14:08:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c50640 con 0x559e06c804d0 2022-03-21T14:08:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c80190 con 0x559e06c81d80 2022-03-21T14:08:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa935d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44056/0 (socket says 172.21.15.2:44056) 2022-03-21T14:08:20.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa935d16700 1 -- 172.21.15.2:0/1027925103 learned_addr learned my addr 172.21.15.2:0/1027925103 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa936517700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c89bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.791+0000 7fa936d18700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c7a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 -- 172.21.15.2:0/1027925103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 msgr2=0x559e06c89bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c89bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 -- 172.21.15.2:0/1027925103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 msgr2=0x559e06c8a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 -- 172.21.15.2:0/1027925103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e06a37f80 con 0x559e06c81d80 2022-03-21T14:08:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936517700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c89bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa935d16700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-03-21T14:08:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa936d18700 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c7a600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa928090710 tx=0x7fa92808cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c1dd76e661bbb9 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa924b97700 1 -- 172.21.15.2:0/1027925103 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928017540 con 0x559e06c81d80 2022-03-21T14:08:20.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa924b97700 1 -- 172.21.15.2:0/1027925103 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa928003d00 con 0x559e06c81d80 2022-03-21T14:08:20.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.792+0000 7fa924b97700 1 -- 172.21.15.2:0/1027925103 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928096be0 con 0x559e06c81d80 2022-03-21T14:08:20.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1027925103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 msgr2=0x559e06c7a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c7a600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa928090710 tx=0x7fa92808cb70 comp rx=0 tx=0).stop 2022-03-21T14:08:20.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1027925103 shutdown_connections 2022-03-21T14:08:20.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c804d0 0x559e06c89bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c81d80 0x559e06c7a600 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/1027925103 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c80e20 0x559e06c8a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.813 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1027925103 >> 172.21.15.2:0/1027925103 conn(0x559e06c7b340 msgr2=0x559e06c84990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.813 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1027925103 shutdown_connections 2022-03-21T14:08:20.813 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- 172.21.15.2:0/1027925103 wait complete. 2022-03-21T14:08:20.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 -- start start 2022-03-21T14:08:20.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.793+0000 7fa9420ecb80 1 Processor -- start 2022-03-21T14:08:20.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c80e20 0x559e06c8d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c8dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c8e460 con 0x559e06c804d0 2022-03-21T14:08:20.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06ce6a40 con 0x559e06c81d80 2022-03-21T14:08:20.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06ce6b80 con 0x559e06c80e20 2022-03-21T14:08:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa9420ecb80 10 client.0 _ll_register_callbacks cb 0x559e069e07b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-03-21T14:08:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa936517700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa936517700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44058/0 (socket says 172.21.15.2:44058) 2022-03-21T14:08:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa936517700 1 -- 172.21.15.2:0/142170311 learned_addr learned my addr 172.21.15.2:0/142170311 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa935d16700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c80e20 0x559e06c8d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.794+0000 7fa936d18700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c8dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 -- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c80e20 msgr2=0x559e06c8d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c80e20 0x559e06c8d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 -- 172.21.15.2:0/142170311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 msgr2=0x559e06c8dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-03-21T14:08:20.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c8dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 -- 172.21.15.2:0/142170311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa930106fa0 con 0x559e06c804d0 2022-03-21T14:08:20.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936d18700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c8dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-03-21T14:08:20.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa936517700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0098e0 tx=0x7fa92c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd81d5bf78059ec server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:20.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9117fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-03-21T14:08:20.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c01d040 con 0x559e06c804d0 2022-03-21T14:08:20.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9280a3070 con 0x559e06c804d0 2022-03-21T14:08:20.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8244+0+0 (secure 0 0 0) 0x7fa92c00f610 con 0x559e06c804d0 2022-03-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa92c00f7d0 con 0x559e06c804d0 2022-03-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9420ecb80 10 client.14955 fetch_fsmap learned FSMap version 20 2022-03-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.795+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7fa92001a040 con 0x559e06c804d0 2022-03-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa92c037020 con 0x559e06c804d0 2022-03-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fa92c009910 con 0x559e06c804d0 2022-03-21T14:08:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 10 client.14955 fetch_fsmap finished waiting for FSMap version 20 2022-03-21T14:08:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 10 client.14955 resolve_mds: resolved name 'l' to daemon mds.l 2022-03-21T14:08:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 20 client.14955 populate_metadata read hostname 'smithi002' 2022-03-21T14:08:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06ceb770 0x559e06ce80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 4 client.14955 mds_command: new command op to 24587 tid=0[{"prefix": "status"}] 2022-03-21T14:08:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 --> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559e06b8c5f0 con 0x559e06ceb770 2022-03-21T14:08:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.796+0000 7fa935d16700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06ceb770 0x559e06ce80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:20.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa935d16700 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06ceb770 0x559e06ce80c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b22200cce8779 server_cookie=eb162b874cb26d4c in_seq=0 out_seq=0 2022-03-21T14:08:20.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9117fa700 10 client.14955 ms_handle_connect on v2:172.21.15.79:6838/1907413140 2022-03-21T14:08:20.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9117fa700 1 -- 172.21.15.2:0/142170311 <== mds.0 v2:172.21.15.79:6838/1907413140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559e06b8c5f0 con 0x559e06ceb770 2022-03-21T14:08:20.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9117fa700 10 client.14955 handle_command_reply: tid=0 2022-03-21T14:08:20.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 client.14955 shutdown 2022-03-21T14:08:20.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 msgr2=0x559e06c8eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0098e0 tx=0x7fa92c005040 comp rx=0 tx=0).stop 2022-03-21T14:08:20.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 shutdown_connections 2022-03-21T14:08:20.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] conn(0x559e06ceb770 0x559e06ce80c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e06c81d80 0x559e06c8dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9117fa700 0 client.14955 ms_handle_reset on v2:172.21.15.79:6838/1907413140 2022-03-21T14:08:20.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e06c804d0 0x559e06c8eeb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 --2- 172.21.15.2:0/142170311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559e06c80e20 0x559e06c8d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 >> 172.21.15.2:0/142170311 conn(0x559e06c7b340 msgr2=0x559e06c84990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.797+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 shutdown_connections 2022-03-21T14:08:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa9420ecb80 1 -- 172.21.15.2:0/142170311 wait complete. 2022-03-21T14:08:20.826 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa9420ecb80 20 client.14955 trim_cache size 0 max 16384 2022-03-21T14:08:20.826 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T14:08:20.827 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "ee5e78f2-a91e-11ec-8c35-001a4aab830c", 2022-03-21T14:08:20.827 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2022-03-21T14:08:20.827 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24587, 2022-03-21T14:08:20.827 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-03-21T14:08:20.828 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-03-21T14:08:20.828 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-03-21T14:08:20.828 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 85.956300585999998, 2022-03-21T14:08:20.828 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 20, 2022-03-21T14:08:20.829 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 84, 2022-03-21T14:08:20.829 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 82, 2022-03-21T14:08:20.829 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 93.644744071999995 2022-03-21T14:08:20.829 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T14:08:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 msgr2=0x7fa91c099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 0x7fa91c099240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa92000b5d0 tx=0x7fa920005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 msgr2=0x7fa9301164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa92800eaf0 tx=0x7fa9280096d0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 shutdown_connections 2022-03-21T14:08:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9300fe1c0 0x7fa93010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fa91c096d90 0x7fa91c099240 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9300ffdf0 0x7fa9301164d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 --2- 172.21.15.2:0/1134096054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9300ff460 0x7fa93010f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:20.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.798+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 >> 172.21.15.2:0/1134096054 conn(0x7fa9300f9990 msgr2=0x7fa930102690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:20.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.799+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 shutdown_connections 2022-03-21T14:08:20.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:20.799+0000 7fa937519700 1 -- 172.21.15.2:0/1134096054 wait complete. 2022-03-21T14:08:20.835 INFO:teuthology.run_tasks:Running task kclient... 2022-03-21T14:08:20.863 INFO:tasks.kclient:Mounting kernel clients... 2022-03-21T14:08:20.863 INFO:tasks.kclient:config is {'client.0': {}} 2022-03-21T14:08:20.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:20.864 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2022-03-21T14:08:20.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:20.911 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2022-03-21T14:08:21.025 INFO:teuthology.orchestra.run.smithi002.stderr:Cannot find device "ceph-brx" 2022-03-21T14:08:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T14:08:21.029 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1'} 2022-03-21T14:08:21.030 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph fs ls 2022-03-21T14:08:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:08:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/3441697902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2022-03-21T14:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-03-21T14:08:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2022-03-21T14:08:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-03-21T14:08:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/3441697902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2022-03-21T14:08:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-03-21T14:08:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2022-03-21T14:08:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-03-21T14:08:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:08:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:08:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/3441697902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-03-21T14:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2022-03-21T14:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-03-21T14:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2022-03-21T14:08:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2022-03-21T14:08:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2022-03-21T14:08:21.686 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T14:08:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: pgmap v314: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2022-03-21T14:08:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2022-03-21T14:08:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-03-21T14:08:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2022-03-21T14:08:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-03-21T14:08:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-03-21T14:08:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:22 smithi002 ceph-mon[17167]: osdmap e85: 12 total, 12 up, 12 in 2022-03-21T14:08:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: pgmap v314: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2022-03-21T14:08:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2022-03-21T14:08:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-03-21T14:08:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2022-03-21T14:08:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-03-21T14:08:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-03-21T14:08:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:22 smithi016 ceph-mon[22010]: osdmap e85: 12 total, 12 up, 12 in 2022-03-21T14:08:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: pgmap v314: 97 pgs: 97 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2022-03-21T14:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2022-03-21T14:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-03-21T14:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2022-03-21T14:08:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2022-03-21T14:08:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2022-03-21T14:08:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:22 smithi079 ceph-mon[23158]: osdmap e85: 12 total, 12 up, 12 in 2022-03-21T14:08:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T14:08:23.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.229+0000 7fc95fc48700 1 -- 172.21.15.2:0/3231160693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fd310 msgr2=0x7fc9580fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:23.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.229+0000 7fc95fc48700 1 --2- 172.21.15.2:0/3231160693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fd310 0x7fc9580fd760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc9480098e0 tx=0x7fc948005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:23.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 -- 172.21.15.2:0/3231160693 shutdown_connections 2022-03-21T14:08:23.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 --2- 172.21.15.2:0/3231160693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fdca0 0x7fc958104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 --2- 172.21.15.2:0/3231160693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fc070 0x7fc9580fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 --2- 172.21.15.2:0/3231160693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fd310 0x7fc9580fd760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 -- 172.21.15.2:0/3231160693 >> 172.21.15.2:0/3231160693 conn(0x7fc9580f7880 msgr2=0x7fc9580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:23.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 -- 172.21.15.2:0/3231160693 shutdown_connections 2022-03-21T14:08:23.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.230+0000 7fc95fc48700 1 -- 172.21.15.2:0/3231160693 wait complete. 2022-03-21T14:08:23.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 Processor -- start 2022-03-21T14:08:23.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 -- start start 2022-03-21T14:08:23.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:23.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fd310 0x7fc95810f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:23.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fdca0 0x7fc958116540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:23.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958116a80 con 0x7fc9580fdca0 2022-03-21T14:08:23.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958116bf0 con 0x7fc9580fd310 2022-03-21T14:08:23.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95fc48700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc958116db0 con 0x7fc9580fc070 2022-03-21T14:08:23.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95d9e4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:23.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95d9e4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.2:39908/0 (socket says 172.21.15.2:39908) 2022-03-21T14:08:23.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95d9e4700 1 -- 172.21.15.2:0/4108847634 learned_addr learned my addr 172.21.15.2:0/4108847634 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-03-21T14:08:23.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95e1e5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fdca0 0x7fc958116540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:23.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.231+0000 7fc95d1e3700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fd310 0x7fc95810f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:23.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 -- 172.21.15.2:0/4108847634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fd310 msgr2=0x7fc95810f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:23.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fd310 0x7fc95810f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 -- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fdca0 msgr2=0x7fc958116540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:23.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fdca0 0x7fc958116540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 -- 172.21.15.2:0/4108847634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95400b360 con 0x7fc9580fc070 2022-03-21T14:08:23.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95d9e4700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc95400ea50 tx=0x7fc954005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:23.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95400b6e0 con 0x7fc9580fc070 2022-03-21T14:08:23.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9480093f0 con 0x7fc9580fc070 2022-03-21T14:08:23.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc95400ce80 con 0x7fc9580fc070 2022-03-21T14:08:23.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.232+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954018420 con 0x7fc9580fc070 2022-03-21T14:08:23.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.233+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc958117470 con 0x7fc9580fc070 2022-03-21T14:08:23.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.234+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc958107810 con 0x7fc9580fc070 2022-03-21T14:08:23.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.235+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 125139+0+0 (secure 0 0 0) 0x7fc95400c040 con 0x7fc9580fc070 2022-03-21T14:08:23.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.236+0000 7fc94e7fc700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 0x7fc9440991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-03-21T14:08:23.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.236+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8418+0+0 (secure 0 0 0) 0x7fc954014070 con 0x7fc9580fc070 2022-03-21T14:08:23.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.237+0000 7fc95d1e3700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 0x7fc9440991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-03-21T14:08:23.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.237+0000 7fc95d1e3700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 0x7fc9440991c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc94800fa00 tx=0x7fc948005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-03-21T14:08:23.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.240+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163627 (secure 0 0 0) 0x7fc95408ce50 con 0x7fc9580fc070 2022-03-21T14:08:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:23 smithi002 ceph-mon[17167]: osdmap e86: 12 total, 12 up, 12 in 2022-03-21T14:08:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:23 smithi002 ceph-mon[17167]: pgmap v317: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:23 smithi016 ceph-mon[22010]: osdmap e86: 12 total, 12 up, 12 in 2022-03-21T14:08:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:23 smithi016 ceph-mon[22010]: pgmap v317: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:23.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.470+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc9581179e0 con 0x7fc9580fc070 2022-03-21T14:08:23.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.471+0000 7fc94e7fc700 1 -- 172.21.15.2:0/4108847634 <== mon.2 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc954090470 con 0x7fc9580fc070 2022-03-21T14:08:23.472 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2022-03-21T14:08:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:23 smithi079 ceph-mon[23158]: osdmap e86: 12 total, 12 up, 12 in 2022-03-21T14:08:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:23 smithi079 ceph-mon[23158]: pgmap v317: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:23.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 msgr2=0x7fc9440991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:23.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 0x7fc9440991c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc94800fa00 tx=0x7fc948005f70 comp rx=0 tx=0).stop 2022-03-21T14:08:23.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 msgr2=0x7fc95810ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-03-21T14:08:23.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc95400ea50 tx=0x7fc954005eb0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 shutdown_connections 2022-03-21T14:08:23.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:6800/613532934,v1:172.21.15.2:6801/613532934] conn(0x7fc944096d10 0x7fc9440991c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9580fdca0 0x7fc958116540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9580fc070 0x7fc95810ef70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 --2- 172.21.15.2:0/4108847634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580fd310 0x7fc95810f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-03-21T14:08:23.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 >> 172.21.15.2:0/4108847634 conn(0x7fc9580f7880 msgr2=0x7fc9580f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-03-21T14:08:23.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 shutdown_connections 2022-03-21T14:08:23.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-03-21T14:08:23.473+0000 7fc95fc48700 1 -- 172.21.15.2:0/4108847634 wait complete. 2022-03-21T14:08:24.031 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-03-21T14:08:24.031 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-03-21T14:08:24.032 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2022-03-21T14:08:24.032 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-03-21T14:08:24.032 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:24.032 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-03-21T14:08:24.033 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-03-21T14:08:24.033 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-03-21T14:08:24.033 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-03-21T14:08:24.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:24.034 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2022-03-21T14:08:24.080 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-21T14:08:24.080 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-21T14:08:24.081 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2022-03-21T14:08:24.081 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:08:24.081 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2022-03-21T14:08:24.081 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:08:24.082 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-03-21T14:08:24.082 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:08:24.082 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-03-21T14:08:24.082 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:08:24.083 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-21T14:08:24.083 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:08:24.083 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2022-03-21T14:08:24.083 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42015sec preferred_lft 42015sec 2022-03-21T14:08:24.083 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2022-03-21T14:08:24.084 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:08:24.084 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-03-21T14:08:24.084 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:08:24.085 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-03-21T14:08:24.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:24.086 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-03-21T14:08:24.087 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-03-21T14:08:24.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2022-03-21T14:08:24.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2022-03-21T14:08:24.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2022-03-21T14:08:24.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-03-21T14:08:24.088 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-03-21T14:08:24.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:24 smithi002 ceph-mon[17167]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-03-21T14:08:24.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:24 smithi002 ceph-mon[17167]: from='client.? 172.21.15.2:0/4108847634' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-03-21T14:08:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:24 smithi016 ceph-mon[22010]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-03-21T14:08:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:24 smithi016 ceph-mon[22010]: from='client.? 172.21.15.2:0/4108847634' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-03-21T14:08:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:24 smithi079 ceph-mon[23158]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-03-21T14:08:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:24 smithi079 ceph-mon[23158]: from='client.? 172.21.15.2:0/4108847634' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-03-21T14:08:24.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:24.521 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-03-21T14:08:24.593 INFO:teuthology.orchestra.run.smithi002.stdout:1 2022-03-21T14:08:24.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:24.598 DEBUG:teuthology.orchestra.run.smithi002:> route 2022-03-21T14:08:24.658 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2022-03-21T14:08:24.659 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-03-21T14:08:24.659 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2022-03-21T14:08:24.659 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2022-03-21T14:08:24.660 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-03-21T14:08:24.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:24.662 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-03-21T14:08:24.662 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-03-21T14:08:24.662 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2022-03-21T14:08:24.663 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2022-03-21T14:08:24.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2022-03-21T14:08:24.664 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-03-21T14:08:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:25 smithi002 ceph-mon[17167]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:25 smithi016 ceph-mon[22010]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:25.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:25.415 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2022-03-21T14:08:25.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:25.463 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2022-03-21T14:08:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:25 smithi079 ceph-mon[23158]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:25.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:25.508 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-03-21T14:08:25.509 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-03-21T14:08:25.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-03-21T14:08:25.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-03-21T14:08:25.509 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-03-21T14:08:25.698 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-03-21T14:08:25.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:25.699 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-03-21T14:08:25.700 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-03-21T14:08:25.700 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-03-21T14:08:25.700 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-03-21T14:08:25.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-03-21T14:08:25.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-03-21T14:08:25.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-03-21T14:08:25.702 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-03-21T14:08:26.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:26.015 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-03-21T14:08:26.016 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-03-21T14:08:26.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2022-03-21T14:08:26.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2022-03-21T14:08:26.017 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-03-21T14:08:26.201 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-03-21T14:08:26.201 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:26.247 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-03-21T14:08:26.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-03-21T14:08:26.248 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:26.333 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2022-03-21T14:08:26.334 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-03-21T14:08:26.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2022-03-21T14:08:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2022-03-21T14:08:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2022-03-21T14:08:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2022-03-21T14:08:26.933 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2022-03-21T14:08:26.934 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2022-03-21T14:08:26.934 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2022-03-21T14:08:26.934 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.2:6789,172.21.15.16:6789,172.21.15.79:6789:/ 2022-03-21T14:08:26.934 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=ee5e78f2-a91e-11ec-8c35-001a4aab830c" will pass to kernel 2022-03-21T14:08:26.942 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-03-21T14:08:26.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-03-21T14:08:26.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:27.016 INFO:teuthology.run_tasks:Running task check-counter... 2022-03-21T14:08:27.028 INFO:teuthology.run_tasks:Running task workunit... 2022-03-21T14:08:27.065 INFO:tasks.workunit:Pulling workunits from ref d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T14:08:27.066 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-03-21T14:08:27.067 INFO:tasks.workunit:timeout=3h 2022-03-21T14:08:27.067 INFO:tasks.workunit:cleanup=True 2022-03-21T14:08:27.067 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:27.116 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-03-21T14:08:27.116 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 78h/120d Inode: 1 Links: 2 2022-03-21T14:08:27.116 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:08:27.116 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:cephfs_t:s0 2022-03-21T14:08:27.117 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-03-21T14:08:27.117 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-03-21 14:06:54.842740419 +0000 2022-03-21T14:08:27.117 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-03-21 14:08:27.010227104 +0000 2022-03-21T14:08:27.117 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-03-21T14:08:27.118 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-03-21T14:08:27.118 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-03-21T14:08:27.233 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d3799b8cfa5046465710eadeea2bed7bb4681409 2022-03-21T14:08:27.281 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-03-21T14:08:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:27 smithi079 ceph-mon[23158]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:27 smithi002 ceph-mon[17167]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:27 smithi016 ceph-mon[22010]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:28 smithi002 ceph-mon[17167]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-03-21T14:08:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:28 smithi002 ceph-mon[17167]: Cluster is now healthy 2022-03-21T14:08:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:28 smithi016 ceph-mon[22010]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-03-21T14:08:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:28 smithi016 ceph-mon[22010]: Cluster is now healthy 2022-03-21T14:08:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:28 smithi079 ceph-mon[23158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-03-21T14:08:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:28 smithi079 ceph-mon[23158]: Cluster is now healthy 2022-03-21T14:08:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:29 smithi002 ceph-mon[17167]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:29.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:29 smithi016 ceph-mon[22010]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:29.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:29 smithi079 ceph-mon[23158]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 73 B/s, 0 objects/s recovering 2022-03-21T14:08:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:31 smithi002 ceph-mon[17167]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 440 B/s rd, 0 op/s; 63 B/s, 0 objects/s recovering 2022-03-21T14:08:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:31 smithi016 ceph-mon[22010]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 440 B/s rd, 0 op/s; 63 B/s, 0 objects/s recovering 2022-03-21T14:08:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:31 smithi079 ceph-mon[23158]: pgmap v321: 97 pgs: 97 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 440 B/s rd, 0 op/s; 63 B/s, 0 objects/s recovering 2022-03-21T14:08:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:33 smithi002 ceph-mon[17167]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 596 B/s wr, 2 op/s; 57 B/s, 0 objects/s recovering 2022-03-21T14:08:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:33 smithi016 ceph-mon[22010]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 596 B/s wr, 2 op/s; 57 B/s, 0 objects/s recovering 2022-03-21T14:08:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:33 smithi079 ceph-mon[23158]: pgmap v322: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 596 B/s wr, 2 op/s; 57 B/s, 0 objects/s recovering 2022-03-21T14:08:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:35 smithi002 ceph-mon[17167]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 597 B/s wr, 2 op/s; 48 B/s, 0 objects/s recovering 2022-03-21T14:08:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:35 smithi016 ceph-mon[22010]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 597 B/s wr, 2 op/s; 48 B/s, 0 objects/s recovering 2022-03-21T14:08:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:35 smithi079 ceph-mon[23158]: pgmap v323: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 597 B/s wr, 2 op/s; 48 B/s, 0 objects/s recovering 2022-03-21T14:08:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:37 smithi002 ceph-mon[17167]: pgmap v324: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:37 smithi016 ceph-mon[22010]: pgmap v324: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:37 smithi079 ceph-mon[23158]: pgmap v324: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:39.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:39 smithi002 ceph-mon[17167]: pgmap v325: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:39 smithi016 ceph-mon[22010]: pgmap v325: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:39 smithi079 ceph-mon[23158]: pgmap v325: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2022-03-21T14:08:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:41 smithi002 ceph-mon[17167]: pgmap v326: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2022-03-21T14:08:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:41 smithi016 ceph-mon[22010]: pgmap v326: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2022-03-21T14:08:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:41 smithi079 ceph-mon[23158]: pgmap v326: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2022-03-21T14:08:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:43 smithi002 ceph-mon[17167]: pgmap v327: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2022-03-21T14:08:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:43 smithi016 ceph-mon[22010]: pgmap v327: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2022-03-21T14:08:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:43 smithi079 ceph-mon[23158]: pgmap v327: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2022-03-21T14:08:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:45 smithi002 ceph-mon[17167]: pgmap v328: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:08:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:45 smithi016 ceph-mon[22010]: pgmap v328: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:08:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:45 smithi079 ceph-mon[23158]: pgmap v328: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:08:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:47 smithi002 ceph-mon[17167]: pgmap v329: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:47 smithi016 ceph-mon[22010]: pgmap v329: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:47 smithi079 ceph-mon[23158]: pgmap v329: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:49.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:49 smithi002 ceph-mon[17167]: pgmap v330: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:49 smithi016 ceph-mon[22010]: pgmap v330: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:49 smithi079 ceph-mon[23158]: pgmap v330: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:51 smithi002 ceph-mon[17167]: pgmap v331: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:51 smithi016 ceph-mon[22010]: pgmap v331: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:51 smithi079 ceph-mon[23158]: pgmap v331: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:53 smithi002 ceph-mon[17167]: pgmap v332: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:53 smithi016 ceph-mon[22010]: pgmap v332: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:53 smithi079 ceph-mon[23158]: pgmap v332: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:55 smithi002 ceph-mon[17167]: pgmap v333: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:55 smithi016 ceph-mon[22010]: pgmap v333: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:55 smithi079 ceph-mon[23158]: pgmap v333: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:57 smithi002 ceph-mon[17167]: pgmap v334: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:57 smithi016 ceph-mon[22010]: pgmap v334: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:57 smithi079 ceph-mon[23158]: pgmap v334: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:08:59.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:08:59 smithi002 ceph-mon[17167]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:59.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:08:59 smithi016 ceph-mon[22010]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:08:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:08:59 smithi079 ceph-mon[23158]: pgmap v335: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:01 smithi002 ceph-mon[17167]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:09:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:09:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:09:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:01 smithi016 ceph-mon[22010]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:09:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:09:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:09:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:01 smithi079 ceph-mon[23158]: pgmap v336: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:09:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:09:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:09:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:03 smithi002 ceph-mon[17167]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:09:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:03 smithi016 ceph-mon[22010]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:09:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:03 smithi079 ceph-mon[23158]: pgmap v337: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:09:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:05 smithi002 ceph-mon[17167]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:05 smithi016 ceph-mon[22010]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:05 smithi079 ceph-mon[23158]: pgmap v338: 97 pgs: 97 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:09:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:07 smithi079 ceph-mon[23158]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:07 smithi002 ceph-mon[17167]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:07 smithi016 ceph-mon[22010]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:09 smithi079 ceph-mon[23158]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:09 smithi002 ceph-mon[17167]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:09 smithi016 ceph-mon[22010]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:11 smithi079 ceph-mon[23158]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:11 smithi002 ceph-mon[17167]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:11 smithi016 ceph-mon[22010]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:13 smithi079 ceph-mon[23158]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 426 B/s wr, 3 op/s 2022-03-21T14:09:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:13 smithi002 ceph-mon[17167]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 426 B/s wr, 3 op/s 2022-03-21T14:09:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:13 smithi016 ceph-mon[22010]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 426 B/s wr, 3 op/s 2022-03-21T14:09:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:15 smithi079 ceph-mon[23158]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:15 smithi002 ceph-mon[17167]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:16.158 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 61% (6041/9786) Updating files: 62% (6068/9786) Updating files: 63% (6166/9786) Updating files: 64% (6264/9786) Updating files: 65% (6361/9786) Updating files: 66% (6459/9786) Updating files: 67% (6557/9786) Updating files: 68% (6655/9786) Updating files: 69% (6753/9786) Updating files: 70% (6851/9786) Updating files: 71% (6949/9786) Updating files: 72% (7046/9786) Updating files: 73% (7144/9786) Updating files: 74% (7242/9786) Updating files: 75% (7340/9786) Updating files: 76% (7438/9786) Updating files: 77% (7536/9786) Updating files: 78% (7634/9786) Updating files: 79% (7731/9786) Updating files: 80% (7829/9786) Updating files: 81% (7927/9786) Updating files: 82% (8025/9786) Updating files: 83% (8123/9786) Updating files: 84% (8221/9786) Updating files: 85% (8319/9786) Updating files: 86% (8416/9786) Updating files: 87% (8514/9786) Updating files: 88% (8612/9786) Updating files: 89% (8710/9786) Updating files: 90% (8808/9786) Updating files: 91% (8906/9786) Updating files: 92% (9004/9786) Updating files: 92% (9056/9786) Updating files: 93% (9101/9786) Updating files: 94% (9199/9786) Updating files: 95% (9297/9786) Updating files: 96% (9395/9786) Updating files: 97% (9493/9786) Updating files: 98% (9591/9786) Updating files: 99% (9689/9786) Updating files: 100% (9786/9786) Updating files: 100% (9786/9786), done. 2022-03-21T14:09:16.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:15 smithi016 ceph-mon[22010]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2022-03-21T14:09:17.885 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to 'd3799b8cfa5046465710eadeea2bed7bb4681409'. 2022-03-21T14:09:17.886 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.886 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-21T14:09:17.886 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-21T14:09:17.887 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2022-03-21T14:09:17.887 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.887 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-21T14:09:17.887 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-21T14:09:17.887 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.888 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2022-03-21T14:09:17.888 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.888 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2022-03-21T14:09:17.888 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.889 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2022-03-21T14:09:17.889 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.889 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-21T14:09:17.889 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-03-21T14:09:17.889 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at d3799b8cfa5 Revert "mds: kill session when mds do ms_handle_remote_reset" 2022-03-21T14:09:17.933 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-03-21T14:09:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:17 smithi079 ceph-mon[23158]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:17.986 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-03-21T14:09:17.992 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-03-21T14:09:17.992 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-03-21T14:09:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:17 smithi002 ceph-mon[17167]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:17 smithi016 ceph-mon[22010]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:09:18.615 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-03-21T14:09:18.712 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-03-21T14:09:18.807 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-03-21T14:09:18.812 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-03-21T14:09:18.812 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-03-21T14:09:18.893 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-03-21T14:09:18.903 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-03-21T14:09:18.904 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-03-21T14:09:18.949 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2022-03-21T14:09:18.951 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2022-03-21T14:09:18.951 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2022-03-21T14:09:19.050 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2022-03-21T14:09:19.050 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p testdir 2022-03-21T14:09:19.054 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd testdir 2022-03-21T14:09:19.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2022-03-21T14:09:19.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfacl -d -m u:nobody:rw . 2022-03-21T14:09:19.066 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:09:19.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2022-03-21T14:09:19.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=0 2022-03-21T14:09:19.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 0 -lt 100 ']' 2022-03-21T14:09:19.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 0 + 1 2022-03-21T14:09:19.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=1 2022-03-21T14:09:19.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:19.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:19.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:19.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:19.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:19.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:20.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:19 smithi002 ceph-mon[17167]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:20.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:19 smithi016 ceph-mon[22010]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:19 smithi079 ceph-mon[23158]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-03-21T14:09:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:09:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:09:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:09:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:09:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:09:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:09:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:21 smithi002 ceph-mon[17167]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:22.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:21 smithi016 ceph-mon[22010]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:21 smithi079 ceph-mon[23158]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:09:22.328 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:22.329 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:22.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:22.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:22.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:22.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:22.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 1 -lt 100 ']' 2022-03-21T14:09:22.520 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 1 + 1 2022-03-21T14:09:22.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=2 2022-03-21T14:09:22.552 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:22.581 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:22.582 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:22.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:22.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:22.586 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:23.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:23.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:23.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:23.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:23.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:23.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:23.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -lt 100 ']' 2022-03-21T14:09:23.194 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 2 + 1 2022-03-21T14:09:23.227 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=3 2022-03-21T14:09:23.227 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:23.265 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:23.265 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:23.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:23.269 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:23.270 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:23.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:23.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:23 smithi002 ceph-mon[17167]: pgmap v347: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 938 B/s wr, 2 op/s 2022-03-21T14:09:23.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:23.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:23.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:23.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:23.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 3 -lt 100 ']' 2022-03-21T14:09:23.886 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 3 + 1 2022-03-21T14:09:23.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=4 2022-03-21T14:09:23.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:23.939 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:23.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:23.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:23.944 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:23.944 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:23 smithi016 ceph-mon[22010]: pgmap v347: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 938 B/s wr, 2 op/s 2022-03-21T14:09:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:23 smithi079 ceph-mon[23158]: pgmap v347: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 938 B/s wr, 2 op/s 2022-03-21T14:09:24.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:24.370 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:24.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:24.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:24.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:24.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:24.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 4 -lt 100 ']' 2022-03-21T14:09:24.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 4 + 1 2022-03-21T14:09:24.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=5 2022-03-21T14:09:24.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:24.534 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:24.535 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:24.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:24.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:24.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:25.028 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:25.028 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:25.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:25.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:25.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:25.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:25.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 5 -lt 100 ']' 2022-03-21T14:09:25.135 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 5 + 1 2022-03-21T14:09:25.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=6 2022-03-21T14:09:25.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:25.191 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:25.191 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:25.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:25.195 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:25.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:25.770 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:25.771 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:25 smithi002 ceph-mon[17167]: pgmap v348: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:25.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:25.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:25.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:25.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:25.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 6 -lt 100 ']' 2022-03-21T14:09:25.877 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 6 + 1 2022-03-21T14:09:25.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=7 2022-03-21T14:09:25.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:25.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:25.928 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:25.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:25.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:25.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:25 smithi016 ceph-mon[22010]: pgmap v348: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:25 smithi079 ceph-mon[23158]: pgmap v348: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:26.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:26.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:26.450 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:26.451 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:26.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:26.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:26.460 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 7 -lt 100 ']' 2022-03-21T14:09:26.460 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 7 + 1 2022-03-21T14:09:26.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=8 2022-03-21T14:09:26.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:26.509 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:26.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:26.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:26.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:26.514 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:26.978 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:26.978 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:27.075 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:27.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:27.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:27.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:27.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 8 -lt 100 ']' 2022-03-21T14:09:27.085 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 8 + 1 2022-03-21T14:09:27.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=9 2022-03-21T14:09:27.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:27.135 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:27.136 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:27.139 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:27.139 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:27.140 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:27.611 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:27.611 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:27.775 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:27.775 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:27.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:27.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:27.789 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 9 -lt 100 ']' 2022-03-21T14:09:27.789 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 9 + 1 2022-03-21T14:09:27.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=10 2022-03-21T14:09:27.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:27.840 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:27.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:27.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:27.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:27.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:28.040 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:27 smithi002 ceph-mon[17167]: pgmap v349: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:27 smithi016 ceph-mon[22010]: pgmap v349: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:27 smithi079 ceph-mon[23158]: pgmap v349: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:28.244 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:28.244 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:28.367 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:28.367 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:28.379 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:28.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:28.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 10 -lt 100 ']' 2022-03-21T14:09:28.380 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 10 + 1 2022-03-21T14:09:28.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=11 2022-03-21T14:09:28.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:28.433 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:28.433 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:28.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:28.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:28.438 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:28.861 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:28.861 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:28.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:28.967 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:28.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:28.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:28.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 11 -lt 100 ']' 2022-03-21T14:09:28.977 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 11 + 1 2022-03-21T14:09:29.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=12 2022-03-21T14:09:29.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:29.033 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:29.033 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:29.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:29.037 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:29.037 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:29.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:29.478 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:29.567 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:29.567 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:29.575 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:29.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:29.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 12 -lt 100 ']' 2022-03-21T14:09:29.576 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 12 + 1 2022-03-21T14:09:29.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=13 2022-03-21T14:09:29.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:29.633 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:29.634 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:29.636 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:29.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:29.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:30.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:29 smithi002 ceph-mon[17167]: pgmap v350: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:30.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:29 smithi016 ceph-mon[22010]: pgmap v350: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:29 smithi079 ceph-mon[23158]: pgmap v350: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:30.277 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:30.278 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:30.375 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:30.375 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:30.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:30.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:30.385 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 13 -lt 100 ']' 2022-03-21T14:09:30.385 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 13 + 1 2022-03-21T14:09:30.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=14 2022-03-21T14:09:30.418 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:30.435 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:30.436 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:30.439 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:30.439 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:30.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:31.002 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:31.002 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:31.083 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:31.083 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:31.091 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:31.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:31.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 14 -lt 100 ']' 2022-03-21T14:09:31.093 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 14 + 1 2022-03-21T14:09:31.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=15 2022-03-21T14:09:31.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:31.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:31.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:31.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:31.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:31.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:31.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:31.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:31.658 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:31.658 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:31.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:31.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:31.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 15 -lt 100 ']' 2022-03-21T14:09:31.668 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 15 + 1 2022-03-21T14:09:31.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=16 2022-03-21T14:09:31.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:31.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:31.725 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:31.728 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:31.728 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:31.729 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:32.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:31 smithi002 ceph-mon[17167]: pgmap v351: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:32.127 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:32.127 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:32.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:31 smithi016 ceph-mon[22010]: pgmap v351: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:31 smithi079 ceph-mon[23158]: pgmap v351: 97 pgs: 97 active+clean; 593 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2022-03-21T14:09:32.224 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:32.225 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:32.233 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:32.233 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:32.233 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 16 -lt 100 ']' 2022-03-21T14:09:32.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 16 + 1 2022-03-21T14:09:32.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=17 2022-03-21T14:09:32.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:32.290 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:32.290 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:32.294 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:32.294 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:32.295 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:32.744 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:32.744 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:32.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:32.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:32.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:32.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:32.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 17 -lt 100 ']' 2022-03-21T14:09:32.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 17 + 1 2022-03-21T14:09:32.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=18 2022-03-21T14:09:32.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:32.893 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:32.894 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:32.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:32.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:32.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:33.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:33.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:33.583 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:33.583 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:33.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:33.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:33.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 18 -lt 100 ']' 2022-03-21T14:09:33.592 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 18 + 1 2022-03-21T14:09:33.625 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=19 2022-03-21T14:09:33.625 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:33.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:33.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:33.654 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:33.655 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:33.655 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:34.080 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:33 smithi002 ceph-mon[17167]: pgmap v352: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2022-03-21T14:09:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:33 smithi016 ceph-mon[22010]: pgmap v352: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2022-03-21T14:09:34.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:34.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:33 smithi079 ceph-mon[23158]: pgmap v352: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2022-03-21T14:09:34.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:34.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:34.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:34.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:34.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 19 -lt 100 ']' 2022-03-21T14:09:34.284 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 19 + 1 2022-03-21T14:09:34.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=20 2022-03-21T14:09:34.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:34.378 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:34.378 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:34.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:34.382 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:34.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:34.885 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:34.885 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:34.982 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:34.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:34.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:34.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:34.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 20 -lt 100 ']' 2022-03-21T14:09:34.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 20 + 1 2022-03-21T14:09:35.025 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=21 2022-03-21T14:09:35.025 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:35.049 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:35.049 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:35.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:35.053 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:35.054 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:35.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:35.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:36.099 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:36.100 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:36.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:36.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:36.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 21 -lt 100 ']' 2022-03-21T14:09:36.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 21 + 1 2022-03-21T14:09:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:35 smithi002 ceph-mon[17167]: pgmap v353: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s 2022-03-21T14:09:36.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=22 2022-03-21T14:09:36.150 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:35 smithi016 ceph-mon[22010]: pgmap v353: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s 2022-03-21T14:09:36.173 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:36.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:36.177 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:36.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:36.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:35 smithi079 ceph-mon[23158]: pgmap v353: 97 pgs: 97 active+clean; 657 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s 2022-03-21T14:09:36.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:36.785 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:36.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:36.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:36.882 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:36.883 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:36.883 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 22 -lt 100 ']' 2022-03-21T14:09:36.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 22 + 1 2022-03-21T14:09:36.916 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=23 2022-03-21T14:09:36.916 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:36.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:36.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:36.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:36.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:36.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:37.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:37.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:37.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:37.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:37.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:37.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:37.475 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 23 -lt 100 ']' 2022-03-21T14:09:37.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 23 + 1 2022-03-21T14:09:37.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=24 2022-03-21T14:09:37.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:37.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:37.532 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:37.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:37.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:37.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:37.976 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:37.976 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:38.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:38.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:38.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:38.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:38.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 24 -lt 100 ']' 2022-03-21T14:09:38.083 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 24 + 1 2022-03-21T14:09:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:37 smithi002 ceph-mon[17167]: pgmap v354: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:38.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=25 2022-03-21T14:09:38.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:38.140 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:38.140 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:38.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:38.144 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:38.144 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:37 smithi016 ceph-mon[22010]: pgmap v354: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:37 smithi079 ceph-mon[23158]: pgmap v354: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:38.626 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:38.626 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:38.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:38.741 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:38.749 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:38.749 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:38.750 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 25 -lt 100 ']' 2022-03-21T14:09:38.750 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 25 + 1 2022-03-21T14:09:38.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=26 2022-03-21T14:09:38.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:38.806 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:38.807 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:38.809 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:38.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:38.811 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:39.259 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:39.259 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:39.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:39.349 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:39.356 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:39.357 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:39.357 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 26 -lt 100 ']' 2022-03-21T14:09:39.357 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 26 + 1 2022-03-21T14:09:39.390 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=27 2022-03-21T14:09:39.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:39.412 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:39.413 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:39.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:39.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:39.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:39.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:39.843 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:39.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:39.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:39.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:39.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:39.941 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 27 -lt 100 ']' 2022-03-21T14:09:39.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 27 + 1 2022-03-21T14:09:39.974 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=28 2022-03-21T14:09:39.975 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:39.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:39.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:39.995 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:39.996 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:39.997 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:39 smithi002 ceph-mon[17167]: pgmap v355: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:39 smithi016 ceph-mon[22010]: pgmap v355: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:39 smithi079 ceph-mon[23158]: pgmap v355: 97 pgs: 97 active+clean; 714 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:40.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:40.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:40.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:40.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:40.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:40.441 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:40.441 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 28 -lt 100 ']' 2022-03-21T14:09:40.441 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 28 + 1 2022-03-21T14:09:40.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=29 2022-03-21T14:09:40.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:40.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:40.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:40.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:40.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:40.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:41.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:41.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:41.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:41.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:41.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:41.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:41.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 29 -lt 100 ']' 2022-03-21T14:09:41.258 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 29 + 1 2022-03-21T14:09:41.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=30 2022-03-21T14:09:41.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:41.314 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:41.314 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:41.317 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:41.318 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:41.319 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:41.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:41.876 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:41.965 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:41.965 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:41.973 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:41.973 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:41.974 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 30 -lt 100 ']' 2022-03-21T14:09:41.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 30 + 1 2022-03-21T14:09:42.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=31 2022-03-21T14:09:42.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:42.024 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:42.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:42.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:42.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:42.029 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:41 smithi002 ceph-mon[17167]: pgmap v356: 97 pgs: 97 active+clean; 715 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:41 smithi016 ceph-mon[22010]: pgmap v356: 97 pgs: 97 active+clean; 715 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:41 smithi079 ceph-mon[23158]: pgmap v356: 97 pgs: 97 active+clean; 715 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:09:42.467 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:42.467 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:42.590 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:42.590 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:42.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:42.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:42.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 31 -lt 100 ']' 2022-03-21T14:09:42.599 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 31 + 1 2022-03-21T14:09:42.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=32 2022-03-21T14:09:42.632 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:42.655 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:42.656 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:42.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:42.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:43.208 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:43.209 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:43.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:43.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:43.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:43.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:43.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 32 -lt 100 ']' 2022-03-21T14:09:43.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 32 + 1 2022-03-21T14:09:43.357 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=33 2022-03-21T14:09:43.357 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:43.380 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:43.381 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:43.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:43.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:43.385 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:43.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:43.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:43.881 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:43.881 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:43.890 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:43.890 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:43.890 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 33 -lt 100 ']' 2022-03-21T14:09:43.891 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 33 + 1 2022-03-21T14:09:43.923 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=34 2022-03-21T14:09:43.923 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:43.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:43.942 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:43.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:43.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:43.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:43 smithi002 ceph-mon[17167]: pgmap v357: 97 pgs: 97 active+clean; 775 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:43 smithi016 ceph-mon[22010]: pgmap v357: 97 pgs: 97 active+clean; 775 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:43 smithi079 ceph-mon[23158]: pgmap v357: 97 pgs: 97 active+clean; 775 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:44.542 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:44.542 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:44.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:44.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:44.639 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:44.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:44.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 34 -lt 100 ']' 2022-03-21T14:09:44.640 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 34 + 1 2022-03-21T14:09:44.673 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=35 2022-03-21T14:09:44.674 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:44.691 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:44.691 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:44.694 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:44.695 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:44.695 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:45.233 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:45.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:45.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:45.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:45.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:45.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:45.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 35 -lt 100 ']' 2022-03-21T14:09:45.332 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 35 + 1 2022-03-21T14:09:45.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=36 2022-03-21T14:09:45.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:45.395 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:45.395 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:45.398 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:45.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:45.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:45.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:45.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:45.964 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:45.964 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:45.972 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:45.973 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:45.973 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 36 -lt 100 ']' 2022-03-21T14:09:45.973 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 36 + 1 2022-03-21T14:09:46.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=37 2022-03-21T14:09:46.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:46.030 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:46.030 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:46.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:46.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:46.035 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:45 smithi002 ceph-mon[17167]: pgmap v358: 97 pgs: 97 active+clean; 776 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 4 op/s 2022-03-21T14:09:46.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:45 smithi016 ceph-mon[22010]: pgmap v358: 97 pgs: 97 active+clean; 776 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 4 op/s 2022-03-21T14:09:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:45 smithi079 ceph-mon[23158]: pgmap v358: 97 pgs: 97 active+clean; 776 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 4 op/s 2022-03-21T14:09:46.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:46.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:46.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:46.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:46.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:46.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:46.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 37 -lt 100 ']' 2022-03-21T14:09:46.566 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 37 + 1 2022-03-21T14:09:46.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=38 2022-03-21T14:09:46.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:46.615 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:46.616 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:46.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:46.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:46.620 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:47.200 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:47.200 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:47.339 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:47.339 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:47.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:47.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:47.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 38 -lt 100 ']' 2022-03-21T14:09:47.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 38 + 1 2022-03-21T14:09:47.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=39 2022-03-21T14:09:47.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:47.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:47.405 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:47.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:47.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:47.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:47.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:47.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:47.922 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:47.922 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:47.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:47.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:47.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 39 -lt 100 ']' 2022-03-21T14:09:47.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 39 + 1 2022-03-21T14:09:47.964 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=40 2022-03-21T14:09:47.965 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:47.983 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:47.983 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:47.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:47.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:47.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:47 smithi002 ceph-mon[17167]: pgmap v359: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:47 smithi016 ceph-mon[22010]: pgmap v359: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:47 smithi079 ceph-mon[23158]: pgmap v359: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:48.424 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:48.425 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:48.514 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:48.514 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:48.523 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:48.524 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:48.524 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 40 -lt 100 ']' 2022-03-21T14:09:48.524 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 40 + 1 2022-03-21T14:09:48.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=41 2022-03-21T14:09:48.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:48.581 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:48.581 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:48.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:48.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:48.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:49.016 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:49.016 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:49.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:49.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:49.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:49.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:49.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 41 -lt 100 ']' 2022-03-21T14:09:49.124 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 41 + 1 2022-03-21T14:09:49.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=42 2022-03-21T14:09:49.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:49.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:49.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:49.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:49.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:49.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:49.716 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:49.716 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:49.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:49.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:49.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:49.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:49.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 42 -lt 100 ']' 2022-03-21T14:09:49.856 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 42 + 1 2022-03-21T14:09:49.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=43 2022-03-21T14:09:49.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:49.997 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:49.997 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:50.000 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:50.001 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:50.002 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:49 smithi002 ceph-mon[17167]: pgmap v360: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:49 smithi016 ceph-mon[22010]: pgmap v360: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:49 smithi079 ceph-mon[23158]: pgmap v360: 97 pgs: 97 active+clean; 839 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:51.516 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:51.516 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:51.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:51.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:51.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:51.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:51.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 43 -lt 100 ']' 2022-03-21T14:09:51.999 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 43 + 1 2022-03-21T14:09:52.047 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=44 2022-03-21T14:09:52.047 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:52.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:52.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:52.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:52.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:52.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:51 smithi002 ceph-mon[17167]: pgmap v361: 97 pgs: 97 active+clean; 840 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:52.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:51 smithi016 ceph-mon[22010]: pgmap v361: 97 pgs: 97 active+clean; 840 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:51 smithi079 ceph-mon[23158]: pgmap v361: 97 pgs: 97 active+clean; 840 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:09:53.365 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:53.366 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:53.488 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:53.488 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:53.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:53.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:53.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 44 -lt 100 ']' 2022-03-21T14:09:53.497 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 44 + 1 2022-03-21T14:09:53.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=45 2022-03-21T14:09:53.531 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:53.554 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:53.554 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:53.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:53.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:53.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:53.965 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:53.966 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:54.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:54.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:54.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:54.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:54.080 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 45 -lt 100 ']' 2022-03-21T14:09:54.081 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 45 + 1 2022-03-21T14:09:54.113 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=46 2022-03-21T14:09:54.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:53 smithi002 ceph-mon[17167]: pgmap v362: 97 pgs: 97 active+clean; 904 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:09:54.137 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:54.138 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:54.141 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:54.141 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:54.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:53 smithi016 ceph-mon[22010]: pgmap v362: 97 pgs: 97 active+clean; 904 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:09:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:53 smithi079 ceph-mon[23158]: pgmap v362: 97 pgs: 97 active+clean; 904 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:09:54.573 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:54.574 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:54.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:54.688 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:54.696 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:54.696 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:54.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 46 -lt 100 ']' 2022-03-21T14:09:54.697 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 46 + 1 2022-03-21T14:09:54.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=47 2022-03-21T14:09:54.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:54.753 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:54.754 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:54.757 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:54.757 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:54.758 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:55.248 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:55.249 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:55.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:55.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:55.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:55.371 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:55.371 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 47 -lt 100 ']' 2022-03-21T14:09:55.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 47 + 1 2022-03-21T14:09:55.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=48 2022-03-21T14:09:55.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:55.428 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:55.429 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:55.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:55.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:55.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:55.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:55.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:56.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:56.013 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:56.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:56.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:56.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 48 -lt 100 ']' 2022-03-21T14:09:56.031 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 48 + 1 2022-03-21T14:09:56.082 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=49 2022-03-21T14:09:56.083 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:56.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:56.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:56.107 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:56.107 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:56.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:55 smithi002 ceph-mon[17167]: pgmap v363: 97 pgs: 97 active+clean; 905 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 4 op/s 2022-03-21T14:09:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:55 smithi016 ceph-mon[22010]: pgmap v363: 97 pgs: 97 active+clean; 905 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 4 op/s 2022-03-21T14:09:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:55 smithi079 ceph-mon[23158]: pgmap v363: 97 pgs: 97 active+clean; 905 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 4 op/s 2022-03-21T14:09:56.448 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:56.449 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:56.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:56.744 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:56.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:56.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:56.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 49 -lt 100 ']' 2022-03-21T14:09:56.746 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 49 + 1 2022-03-21T14:09:56.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=50 2022-03-21T14:09:56.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:56.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:56.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:56.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:56.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:56.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:57.090 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:57.090 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:57.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:57.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:57.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:57.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:57.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 50 -lt 100 ']' 2022-03-21T14:09:57.197 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 50 + 1 2022-03-21T14:09:57.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=51 2022-03-21T14:09:57.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:57.254 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:57.254 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:57.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:57.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:57.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:57.681 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:57.682 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:57.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:57.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:57.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:57.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:57.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 51 -lt 100 ']' 2022-03-21T14:09:57.788 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 51 + 1 2022-03-21T14:09:57.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=52 2022-03-21T14:09:57.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:57.844 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:57.845 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:57.848 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:57.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:57.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:58.077 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:57 smithi002 ceph-mon[17167]: pgmap v364: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:57 smithi016 ceph-mon[22010]: pgmap v364: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:57 smithi079 ceph-mon[23158]: pgmap v364: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 4 op/s 2022-03-21T14:09:58.223 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:58.223 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:58.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:58.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:58.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:58.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:58.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 52 -lt 100 ']' 2022-03-21T14:09:58.323 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 52 + 1 2022-03-21T14:09:58.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=53 2022-03-21T14:09:58.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:58.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:58.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:58.382 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:58.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:58.384 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:58.798 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:58.798 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:58.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:58.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:58.904 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:58.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:58.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 53 -lt 100 ']' 2022-03-21T14:09:58.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 53 + 1 2022-03-21T14:09:58.929 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=54 2022-03-21T14:09:58.929 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:58.955 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:58.955 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:58.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:58.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:58.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:59.323 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:59.323 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:59.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:59.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:59.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:59.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:59.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 54 -lt 100 ']' 2022-03-21T14:09:59.414 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 54 + 1 2022-03-21T14:09:59.446 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=55 2022-03-21T14:09:59.446 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:09:59.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:59.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:59.480 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:09:59.480 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:09:59.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:09:59.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:09:59.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:09:59.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:09:59.971 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:09:59.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:59.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:09:59.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 55 -lt 100 ']' 2022-03-21T14:09:59.989 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 55 + 1 2022-03-21T14:10:00.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=56 2022-03-21T14:10:00.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:00.039 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:00.039 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:00.042 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:00.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:00.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:09:59 smithi002 ceph-mon[17167]: pgmap v365: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:00.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:09:59 smithi016 ceph-mon[22010]: pgmap v365: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:00.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:09:59 smithi079 ceph-mon[23158]: pgmap v365: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:00.556 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:00.556 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:00.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:00.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:00.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:00.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:00.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 56 -lt 100 ']' 2022-03-21T14:10:00.698 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 56 + 1 2022-03-21T14:10:00.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=57 2022-03-21T14:10:00.755 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:00.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:00.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:00.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:00.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:00.767 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:01.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:00 smithi002 ceph-mon[17167]: overall HEALTH_OK 2022-03-21T14:10:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:00 smithi016 ceph-mon[22010]: overall HEALTH_OK 2022-03-21T14:10:01.206 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:01.206 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:00 smithi079 ceph-mon[23158]: overall HEALTH_OK 2022-03-21T14:10:01.295 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:01.295 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:01.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:01.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:01.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 57 -lt 100 ']' 2022-03-21T14:10:01.305 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 57 + 1 2022-03-21T14:10:01.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=58 2022-03-21T14:10:01.339 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:01.356 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:01.357 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:01.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:01.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:01.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:01.789 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:01.789 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:01.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:01.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:01.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:01.921 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:01.922 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 58 -lt 100 ']' 2022-03-21T14:10:01.922 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 58 + 1 2022-03-21T14:10:01.954 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=59 2022-03-21T14:10:01.954 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:01.995 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:01.995 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:01.998 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:01.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:01.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:01 smithi002 ceph-mon[17167]: pgmap v366: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:02.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:01 smithi016 ceph-mon[22010]: pgmap v366: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:02.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:01 smithi079 ceph-mon[23158]: pgmap v366: 97 pgs: 97 active+clean; 954 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:02.572 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:02.573 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:02.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:02.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:02.703 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:02.704 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:02.704 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 59 -lt 100 ']' 2022-03-21T14:10:02.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 59 + 1 2022-03-21T14:10:02.737 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=60 2022-03-21T14:10:02.737 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:02.755 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:02.756 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:02.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:02.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:02.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:03.232 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:03.232 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:03.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:03.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:03.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:03.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:03.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 60 -lt 100 ']' 2022-03-21T14:10:03.330 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 60 + 1 2022-03-21T14:10:03.362 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=61 2022-03-21T14:10:03.362 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:03.386 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:03.387 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:03.389 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:03.390 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:03.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:03.780 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:03.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:03.886 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:03.887 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:03.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:03.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:03.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 61 -lt 100 ']' 2022-03-21T14:10:03.896 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 61 + 1 2022-03-21T14:10:03.929 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=62 2022-03-21T14:10:03.929 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:03.952 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:03.953 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:03.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:03.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:03.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:03 smithi002 ceph-mon[17167]: pgmap v367: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:03 smithi016 ceph-mon[22010]: pgmap v367: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:04.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:03 smithi079 ceph-mon[23158]: pgmap v367: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:04.330 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:04.331 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:04.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:04.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:04.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:04.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:04.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 62 -lt 100 ']' 2022-03-21T14:10:04.521 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 62 + 1 2022-03-21T14:10:04.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=63 2022-03-21T14:10:04.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:04.602 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:04.602 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:04.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:04.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:04.607 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:05.122 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:05.122 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:05.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:05.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:05.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:05.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:05.236 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 63 -lt 100 ']' 2022-03-21T14:10:05.237 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 63 + 1 2022-03-21T14:10:05.270 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=64 2022-03-21T14:10:05.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:05.294 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:05.295 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:05.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:05.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:05.299 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:05.705 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:05.705 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:05.827 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:05.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:05.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:05.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:05.837 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 64 -lt 100 ']' 2022-03-21T14:10:05.837 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 64 + 1 2022-03-21T14:10:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:05 smithi002 ceph-mon[17167]: pgmap v368: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 4 op/s 2022-03-21T14:10:05.870 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=65 2022-03-21T14:10:05.870 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:05.893 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:05.894 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:05.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:05.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:05.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:05 smithi016 ceph-mon[22010]: pgmap v368: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 4 op/s 2022-03-21T14:10:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:05 smithi079 ceph-mon[23158]: pgmap v368: 97 pgs: 97 active+clean; 1008 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 4 op/s 2022-03-21T14:10:06.255 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:06.255 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:06.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:06.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:06.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:06.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:06.362 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 65 -lt 100 ']' 2022-03-21T14:10:06.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 65 + 1 2022-03-21T14:10:06.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=66 2022-03-21T14:10:06.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:06.411 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:06.412 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:06.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:06.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:06.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:06.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:06.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:06.906 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:06.906 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:10:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:10:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:10:06.919 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:06.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:06.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 66 -lt 100 ']' 2022-03-21T14:10:06.920 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 66 + 1 2022-03-21T14:10:06.961 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=67 2022-03-21T14:10:06.961 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:10:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:10:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:10:06.981 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:06.981 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:06.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:06.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:06.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:10:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:10:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:10:07.413 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:07.413 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:07.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:07.523 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:07.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:07.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:07.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 67 -lt 100 ']' 2022-03-21T14:10:07.536 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 67 + 1 2022-03-21T14:10:07.568 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=68 2022-03-21T14:10:07.568 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:07.600 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:07.601 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:07.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:07.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:07.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:08.046 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:08.047 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:07 smithi002 ceph-mon[17167]: pgmap v369: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:08.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:08.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:08.161 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:08.161 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:08.161 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 68 -lt 100 ']' 2022-03-21T14:10:08.162 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 68 + 1 2022-03-21T14:10:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:07 smithi016 ceph-mon[22010]: pgmap v369: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:07 smithi079 ceph-mon[23158]: pgmap v369: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 4 op/s 2022-03-21T14:10:08.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=69 2022-03-21T14:10:08.228 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:08.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:08.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:08.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:08.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:08.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:09.238 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:09.238 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:09.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:09.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:09.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:09.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:09.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 69 -lt 100 ']' 2022-03-21T14:10:09.462 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 69 + 1 2022-03-21T14:10:09.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=70 2022-03-21T14:10:09.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:09.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:09.553 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:09.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:09.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:09.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:10.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:09 smithi002 ceph-mon[17167]: pgmap v370: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:09 smithi016 ceph-mon[22010]: pgmap v370: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:09 smithi079 ceph-mon[23158]: pgmap v370: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:10.338 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:10.338 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:10.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:10.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:10.569 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:10.570 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:10.570 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 70 -lt 100 ']' 2022-03-21T14:10:10.570 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 70 + 1 2022-03-21T14:10:10.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=71 2022-03-21T14:10:10.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:10.652 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:10.652 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:10.656 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:10.656 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:10.657 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:11.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:11.379 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:11.552 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:11.552 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:11.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:11.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:11.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 71 -lt 100 ']' 2022-03-21T14:10:11.561 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 71 + 1 2022-03-21T14:10:11.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=72 2022-03-21T14:10:11.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:11.716 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:11.717 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:11.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:11.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:11.721 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:12.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:11 smithi002 ceph-mon[17167]: pgmap v371: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:11 smithi016 ceph-mon[22010]: pgmap v371: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:11 smithi079 ceph-mon[23158]: pgmap v371: 97 pgs: 97 active+clean; 1.1 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:12.337 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:12.338 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:12.418 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:12.418 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:12.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:12.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:12.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 72 -lt 100 ']' 2022-03-21T14:10:12.445 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 72 + 1 2022-03-21T14:10:12.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=73 2022-03-21T14:10:12.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:12.546 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:12.547 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:12.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:12.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:12.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:12.987 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:12.988 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:13.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:13.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:13.093 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:13.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:13.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 73 -lt 100 ']' 2022-03-21T14:10:13.095 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 73 + 1 2022-03-21T14:10:13.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=74 2022-03-21T14:10:13.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:13.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:13.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:13.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:13.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:13.150 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:13.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:13.496 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:13.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:13.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:13.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:13.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:13.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 74 -lt 100 ']' 2022-03-21T14:10:13.594 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 74 + 1 2022-03-21T14:10:13.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=75 2022-03-21T14:10:13.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:13.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:13.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:13.648 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:13.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:13.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:13 smithi002 ceph-mon[17167]: pgmap v372: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:10:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:13 smithi016 ceph-mon[22010]: pgmap v372: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:10:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:13 smithi079 ceph-mon[23158]: pgmap v372: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:10:14.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:14.221 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:14.318 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:14.318 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:14.326 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:14.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:14.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 75 -lt 100 ']' 2022-03-21T14:10:14.328 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 75 + 1 2022-03-21T14:10:14.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=76 2022-03-21T14:10:14.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:14.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:14.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:14.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:14.403 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:14.403 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:14.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:14.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:14.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:14.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:14.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:14.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:14.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 76 -lt 100 ']' 2022-03-21T14:10:14.919 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 76 + 1 2022-03-21T14:10:14.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=77 2022-03-21T14:10:14.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:14.971 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:14.971 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:14.974 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:14.975 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:14.975 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:15.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:15.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:15.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:15.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:15.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:15.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:15.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 77 -lt 100 ']' 2022-03-21T14:10:15.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 77 + 1 2022-03-21T14:10:15.518 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=78 2022-03-21T14:10:15.518 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:15.536 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:15.536 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:15.540 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:15.540 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:15.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:15.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:15 smithi002 ceph-mon[17167]: pgmap v373: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2022-03-21T14:10:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:15 smithi016 ceph-mon[22010]: pgmap v373: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2022-03-21T14:10:15.954 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:15.954 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:15 smithi079 ceph-mon[23158]: pgmap v373: 97 pgs: 97 active+clean; 1.1 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2022-03-21T14:10:16.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:16.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:16.101 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:16.102 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:16.102 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 78 -lt 100 ']' 2022-03-21T14:10:16.102 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 78 + 1 2022-03-21T14:10:16.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=79 2022-03-21T14:10:16.161 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:16.184 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:16.185 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:16.188 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:16.188 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:16.189 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:16.570 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:16.571 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:16.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:16.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:16.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:16.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:16.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 79 -lt 100 ']' 2022-03-21T14:10:16.669 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 79 + 1 2022-03-21T14:10:16.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=80 2022-03-21T14:10:16.702 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:16.719 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:16.720 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:16.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:16.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:16.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:17.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:17.146 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:17.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:17.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:17.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:17.260 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:17.260 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 80 -lt 100 ']' 2022-03-21T14:10:17.260 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 80 + 1 2022-03-21T14:10:17.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=81 2022-03-21T14:10:17.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:17.319 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:17.319 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:17.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:17.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:17.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:17.778 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:17.779 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:17 smithi002 ceph-mon[17167]: pgmap v374: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:17.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:17.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:17.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:17.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:17.885 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 81 -lt 100 ']' 2022-03-21T14:10:17.886 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 81 + 1 2022-03-21T14:10:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:17 smithi016 ceph-mon[22010]: pgmap v374: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:17.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=82 2022-03-21T14:10:17.919 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:17.943 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:17.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:17.947 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:17.948 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:17.948 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:17 smithi079 ceph-mon[23158]: pgmap v374: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:18.428 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:18.429 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:18.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:18.551 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:18.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:18.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:18.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 82 -lt 100 ']' 2022-03-21T14:10:18.560 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 82 + 1 2022-03-21T14:10:18.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=83 2022-03-21T14:10:18.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:18.617 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:18.618 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:18.621 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:18.622 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:18.622 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:19.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:19.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:19.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:19.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:19.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:19.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:19.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 83 -lt 100 ']' 2022-03-21T14:10:19.119 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 83 + 1 2022-03-21T14:10:19.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=84 2022-03-21T14:10:19.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:19.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:19.169 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:19.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:19.173 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:19.173 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:19.577 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:19.578 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:19.650 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:19.651 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:19.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:19.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:19.659 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 84 -lt 100 ']' 2022-03-21T14:10:19.660 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 84 + 1 2022-03-21T14:10:19.693 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=85 2022-03-21T14:10:19.693 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:19.710 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:19.711 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:19.714 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:19.714 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:19.715 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:19 smithi002 ceph-mon[17167]: pgmap v375: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:19.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:19 smithi016 ceph-mon[22010]: pgmap v375: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:19.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:19 smithi079 ceph-mon[23158]: pgmap v375: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:20.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:20.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:20.167 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:20.167 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:20.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:20.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:20.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 85 -lt 100 ']' 2022-03-21T14:10:20.177 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 85 + 1 2022-03-21T14:10:20.209 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=86 2022-03-21T14:10:20.209 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:20.227 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:20.228 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:20.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:20.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:20.232 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:20.711 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:20.711 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:20.792 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:20.792 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:20.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:20.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:20.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 86 -lt 100 ']' 2022-03-21T14:10:20.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 86 + 1 2022-03-21T14:10:20.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=87 2022-03-21T14:10:20.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:20.858 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:20.858 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:20.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:20.862 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:20.863 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:10:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:10:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:10:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:10:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:10:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:10:21.235 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:21.236 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:21.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:21.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:21.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:21.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:21.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 87 -lt 100 ']' 2022-03-21T14:10:21.335 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 87 + 1 2022-03-21T14:10:21.367 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=88 2022-03-21T14:10:21.367 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:21.391 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:21.391 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:21.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:21.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:21.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:21.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:21.753 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:21.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:21.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:21.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 88 -lt 100 ']' 2022-03-21T14:10:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 88 + 1 2022-03-21T14:10:21.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=89 2022-03-21T14:10:21.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:21 smithi016 ceph-mon[22010]: pgmap v376: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:21.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:21.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:21.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:21.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:21.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:21 smithi079 ceph-mon[23158]: pgmap v376: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:21 smithi002 ceph-mon[17167]: pgmap v376: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:22.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:22.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:22.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:22.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:22.442 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:22.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:22.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 89 -lt 100 ']' 2022-03-21T14:10:22.443 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 89 + 1 2022-03-21T14:10:22.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=90 2022-03-21T14:10:22.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:22.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:22.526 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:22.529 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:22.529 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:22.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:23.127 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:23.127 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:23.241 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:23.242 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:23.253 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:23.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:23.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 90 -lt 100 ']' 2022-03-21T14:10:23.254 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 90 + 1 2022-03-21T14:10:23.284 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=91 2022-03-21T14:10:23.284 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:23.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:23.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:23.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:23.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:23.313 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:23.661 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:23.661 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:23.750 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:23.750 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:23.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:23.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:23.759 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 91 -lt 100 ']' 2022-03-21T14:10:23.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 91 + 1 2022-03-21T14:10:23.792 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=92 2022-03-21T14:10:23.792 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:23.810 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:23.811 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:23.814 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:23.815 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:23.815 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:23 smithi002 ceph-mon[17167]: pgmap v377: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:23 smithi016 ceph-mon[22010]: pgmap v377: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:23 smithi079 ceph-mon[23158]: pgmap v377: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:24.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:24.194 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:24.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:24.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:24.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:24.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:24.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 92 -lt 100 ']' 2022-03-21T14:10:24.292 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 92 + 1 2022-03-21T14:10:24.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=93 2022-03-21T14:10:24.326 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:24.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:24.356 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:24.359 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:24.359 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:24.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:24.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:24.836 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:24.949 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:24.950 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:24.958 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:24.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:24.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 93 -lt 100 ']' 2022-03-21T14:10:24.959 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 93 + 1 2022-03-21T14:10:24.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=94 2022-03-21T14:10:24.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:25.015 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:25.016 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:25.019 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:25.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:25.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:25.410 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:25.411 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:25.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:25.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:25.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:25.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:25.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 94 -lt 100 ']' 2022-03-21T14:10:25.526 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 94 + 1 2022-03-21T14:10:25.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=95 2022-03-21T14:10:25.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:25.582 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:25.583 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:25.586 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:25.587 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:25.587 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:25 smithi016 ceph-mon[22010]: pgmap v378: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:25.920 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:25 smithi002 ceph-mon[17167]: pgmap v378: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:25.960 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:25.960 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:25 smithi079 ceph-mon[23158]: pgmap v378: 97 pgs: 97 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 3 op/s 2022-03-21T14:10:26.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:26.074 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:26.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:26.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:26.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 95 -lt 100 ']' 2022-03-21T14:10:26.096 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 95 + 1 2022-03-21T14:10:26.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=96 2022-03-21T14:10:26.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:26.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:26.144 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:26.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:26.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:26.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:26.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:26.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:26.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:26.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:26.599 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:26.600 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:26.600 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 96 -lt 100 ']' 2022-03-21T14:10:26.600 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 96 + 1 2022-03-21T14:10:26.633 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=97 2022-03-21T14:10:26.633 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:26.657 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:26.657 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:26.661 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:26.661 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:26.662 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:27.035 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:27.035 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:27.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:27.150 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:27.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:27.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:27.159 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 97 -lt 100 ']' 2022-03-21T14:10:27.159 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 97 + 1 2022-03-21T14:10:27.191 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=98 2022-03-21T14:10:27.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:27.222 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:27.222 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:27.225 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:27.226 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:27.226 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:27.635 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:27.635 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:27.733 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:27.733 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:27.741 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:27.742 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:27.742 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 98 -lt 100 ']' 2022-03-21T14:10:27.743 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 98 + 1 2022-03-21T14:10:27.775 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=99 2022-03-21T14:10:27.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:27.799 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:27.799 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:27.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:27.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:27.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:27 smithi002 ceph-mon[17167]: pgmap v379: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:27 smithi016 ceph-mon[22010]: pgmap v379: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:27 smithi079 ceph-mon[23158]: pgmap v379: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:10:28.260 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:28.260 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:28.374 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:28.375 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:28.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:28.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:28.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 99 -lt 100 ']' 2022-03-21T14:10:28.384 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 99 + 1 2022-03-21T14:10:28.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=100 2022-03-21T14:10:28.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:28.440 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:28.441 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:28.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2022-03-21T14:10:28.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2022-03-21T14:10:28.445 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2022-03-21T14:10:28.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2022-03-21T14:10:28.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2022-03-21T14:10:28.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2022-03-21T14:10:28.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:28.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:28.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2022-03-21T14:10:28.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 100 -lt 100 ']' 2022-03-21T14:10:28.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2022-03-21T14:10:29.047 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2022-03-21T14:10:29.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2022-03-21T14:10:29.077 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2022-03-21T14:10:29.077 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_access d1 2022-03-21T14:10:29.125 INFO:tasks.workunit.client.0.smithi002.stderr:d1: system.posix_acl_access: No such attribute 2022-03-21T14:10:29.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2022-03-21T14:10:29.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2022-03-21T14:10:29.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_default . 2022-03-21T14:10:29.128 INFO:tasks.workunit.client.0.smithi002.stderr:.: system.posix_acl_default: No such attribute 2022-03-21T14:10:29.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2022-03-21T14:10:29.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2022-03-21T14:10:29.133 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd .. 2022-03-21T14:10:29.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir testdir 2022-03-21T14:10:29.144 INFO:tasks.workunit.client.0.smithi002.stdout:OK 2022-03-21T14:10:29.145 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo OK 2022-03-21T14:10:29.146 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-03-21T14:10:29.147 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-03-21T14:10:29.712 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2022-03-21T14:10:29.713 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2022-03-21T14:10:29.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2022-03-21T14:10:29.824 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.73298 2022-03-21T14:10:29.824 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2022-03-21T14:10:29.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 400 test_chmod.73298 2022-03-21T14:10:29.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.73298 -r-------- 2022-03-21T14:10:29.839 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.73298 2022-03-21T14:10:29.839 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.73298 2022-03-21T14:10:29.850 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Mar 21 14:10 test_chmod.73298' 2022-03-21T14:10:29.851 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.851 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-r-------- 2022-03-21T14:10:29.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Mar 21 14:10 test_chmod.73298 2022-03-21T14:10:29.853 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2022-03-21T14:10:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:29 smithi002 ceph-mon[17167]: pgmap v380: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:29 smithi016 ceph-mon[22010]: pgmap v380: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:29.925 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r-------- '!=' -r--------. 2022-03-21T14:10:29.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Mar 21 14:10 test_chmod.73298 2022-03-21T14:10:29.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2022-03-21T14:10:29.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r--------. '!=' -r--------. 2022-03-21T14:10:29.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2022-03-21T14:10:29.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2022-03-21T14:10:29.937 INFO:tasks.workunit.client.0.smithi002.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.73298: Permission denied 2022-03-21T14:10:29.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 1 = 0 2022-03-21T14:10:29.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2022-03-21T14:10:29.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 600 test_chmod.73298 2022-03-21T14:10:29.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2022-03-21T14:10:29.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.73298 -rw------- 2022-03-21T14:10:29.939 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.73298 2022-03-21T14:10:29.939 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.73298 2022-03-21T14:10:29.939 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Mar 21 14:10 test_chmod.73298' 2022-03-21T14:10:29.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-rw------- 2022-03-21T14:10:29.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Mar 21 14:10 test_chmod.73298 2022-03-21T14:10:29.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2022-03-21T14:10:29.948 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw------- '!=' -rw-------. 2022-03-21T14:10:29.950 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Mar 21 14:10 test_chmod.73298 2022-03-21T14:10:29.950 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2022-03-21T14:10:29.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw-------. '!=' -rw-------. 2022-03-21T14:10:29.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2022-03-21T14:10:29.955 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2022-03-21T14:10:29.956 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-03-21T14:10:29.957 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-03-21T14:10:29.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:29 smithi079 ceph-mon[23158]: pgmap v380: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:30.043 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2022-03-21T14:10:30.044 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2022-03-21T14:10:31.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:31 smithi016 ceph-mon[22010]: pgmap v381: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:31 smithi079 ceph-mon[23158]: pgmap v381: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:31 smithi002 ceph-mon[17167]: pgmap v381: 97 pgs: 97 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 3 op/s 2022-03-21T14:10:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:33 smithi016 ceph-mon[22010]: pgmap v382: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 4 op/s 2022-03-21T14:10:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:33 smithi079 ceph-mon[23158]: pgmap v382: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 4 op/s 2022-03-21T14:10:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:33 smithi002 ceph-mon[17167]: pgmap v382: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 4 op/s 2022-03-21T14:10:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:35 smithi016 ceph-mon[22010]: pgmap v383: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 4 op/s 2022-03-21T14:10:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:35 smithi079 ceph-mon[23158]: pgmap v383: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 4 op/s 2022-03-21T14:10:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:35 smithi002 ceph-mon[17167]: pgmap v383: 97 pgs: 97 active+clean; 1.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 4 op/s 2022-03-21T14:10:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:37 smithi016 ceph-mon[22010]: pgmap v384: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:10:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:37 smithi079 ceph-mon[23158]: pgmap v384: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:10:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:37 smithi002 ceph-mon[17167]: pgmap v384: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:10:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:39 smithi079 ceph-mon[23158]: pgmap v385: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:39 smithi002 ceph-mon[17167]: pgmap v385: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:40.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:39 smithi016 ceph-mon[22010]: pgmap v385: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:41 smithi079 ceph-mon[23158]: pgmap v386: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:41 smithi002 ceph-mon[17167]: pgmap v386: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:42.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:41 smithi016 ceph-mon[22010]: pgmap v386: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:10:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:43 smithi079 ceph-mon[23158]: pgmap v387: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 5 op/s 2022-03-21T14:10:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:43 smithi002 ceph-mon[17167]: pgmap v387: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 5 op/s 2022-03-21T14:10:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:43 smithi016 ceph-mon[22010]: pgmap v387: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 5 op/s 2022-03-21T14:10:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:45 smithi079 ceph-mon[23158]: pgmap v388: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2022-03-21T14:10:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:45 smithi002 ceph-mon[17167]: pgmap v388: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2022-03-21T14:10:46.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:45 smithi016 ceph-mon[22010]: pgmap v388: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2022-03-21T14:10:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:47 smithi079 ceph-mon[23158]: pgmap v389: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2022-03-21T14:10:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:47 smithi002 ceph-mon[17167]: pgmap v389: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2022-03-21T14:10:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:47 smithi016 ceph-mon[22010]: pgmap v389: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2022-03-21T14:10:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:49 smithi079 ceph-mon[23158]: pgmap v390: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-03-21T14:10:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:49 smithi002 ceph-mon[17167]: pgmap v390: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-03-21T14:10:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:49 smithi016 ceph-mon[22010]: pgmap v390: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-03-21T14:10:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:51 smithi079 ceph-mon[23158]: pgmap v391: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2022-03-21T14:10:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:51 smithi002 ceph-mon[17167]: pgmap v391: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2022-03-21T14:10:52.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:51 smithi016 ceph-mon[22010]: pgmap v391: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2022-03-21T14:10:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:53 smithi079 ceph-mon[23158]: pgmap v392: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 938 B/s wr, 3 op/s 2022-03-21T14:10:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:53 smithi002 ceph-mon[17167]: pgmap v392: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 938 B/s wr, 3 op/s 2022-03-21T14:10:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:53 smithi016 ceph-mon[22010]: pgmap v392: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 938 B/s wr, 3 op/s 2022-03-21T14:10:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:55 smithi079 ceph-mon[23158]: pgmap v393: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:10:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:55 smithi002 ceph-mon[17167]: pgmap v393: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:10:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:55 smithi016 ceph-mon[22010]: pgmap v393: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2022-03-21T14:10:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:57 smithi079 ceph-mon[23158]: pgmap v394: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2022-03-21T14:10:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:57 smithi002 ceph-mon[17167]: pgmap v394: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2022-03-21T14:10:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:57 smithi016 ceph-mon[22010]: pgmap v394: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2022-03-21T14:10:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:10:59 smithi079 ceph-mon[23158]: pgmap v395: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-03-21T14:11:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:10:59 smithi002 ceph-mon[17167]: pgmap v395: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-03-21T14:11:00.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:10:59 smithi016 ceph-mon[22010]: pgmap v395: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-03-21T14:11:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:01 smithi079 ceph-mon[23158]: pgmap v396: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:01 smithi002 ceph-mon[17167]: pgmap v396: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:02.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:01 smithi016 ceph-mon[22010]: pgmap v396: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:03 smithi079 ceph-mon[23158]: pgmap v397: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:03 smithi002 ceph-mon[17167]: pgmap v397: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:03 smithi016 ceph-mon[22010]: pgmap v397: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:05 smithi079 ceph-mon[23158]: pgmap v398: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:05 smithi002 ceph-mon[17167]: pgmap v398: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:06.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:05 smithi016 ceph-mon[22010]: pgmap v398: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:07 smithi079 ceph-mon[23158]: pgmap v399: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:07 smithi002 ceph-mon[17167]: pgmap v399: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:07 smithi016 ceph-mon[22010]: pgmap v399: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:09 smithi079 ceph-mon[23158]: pgmap v400: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:09 smithi002 ceph-mon[17167]: pgmap v400: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:09 smithi016 ceph-mon[22010]: pgmap v400: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:11 smithi079 ceph-mon[23158]: pgmap v401: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:11 smithi002 ceph-mon[17167]: pgmap v401: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:11 smithi016 ceph-mon[22010]: pgmap v401: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:11:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:11:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:11:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:11:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:11:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:11:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:11:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:11:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:11:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:14 smithi002 ceph-mon[17167]: pgmap v402: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:14 smithi016 ceph-mon[22010]: pgmap v402: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:14 smithi079 ceph-mon[23158]: pgmap v402: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:15 smithi079 ceph-mon[23158]: pgmap v403: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:15 smithi002 ceph-mon[17167]: pgmap v403: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:16.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:15 smithi016 ceph-mon[22010]: pgmap v403: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:11:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:17 smithi079 ceph-mon[23158]: pgmap v404: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:18.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:17 smithi002 ceph-mon[17167]: pgmap v404: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:17 smithi016 ceph-mon[22010]: pgmap v404: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:19.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:19 smithi079 ceph-mon[23158]: pgmap v405: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:20.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:19 smithi002 ceph-mon[17167]: pgmap v405: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:19 smithi016 ceph-mon[22010]: pgmap v405: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:11:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:11:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:11:21.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:11:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:11:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:11:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:21 smithi079 ceph-mon[23158]: pgmap v406: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:21 smithi002 ceph-mon[17167]: pgmap v406: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:21 smithi016 ceph-mon[22010]: pgmap v406: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:23 smithi079 ceph-mon[23158]: pgmap v407: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:24.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:23 smithi002 ceph-mon[17167]: pgmap v407: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:23 smithi016 ceph-mon[22010]: pgmap v407: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:25 smithi079 ceph-mon[23158]: pgmap v408: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:26.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:25 smithi002 ceph-mon[17167]: pgmap v408: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:26.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:25 smithi016 ceph-mon[22010]: pgmap v408: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:27 smithi079 ceph-mon[23158]: pgmap v409: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:27 smithi002 ceph-mon[17167]: pgmap v409: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:27 smithi016 ceph-mon[22010]: pgmap v409: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:11:29.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:29 smithi079 ceph-mon[23158]: pgmap v410: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:30.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:29 smithi002 ceph-mon[17167]: pgmap v410: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:30.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:29 smithi016 ceph-mon[22010]: pgmap v410: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:30.902 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2022-03-21T14:11:30.909 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-03-21T14:11:30.910 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-03-21T14:11:30.993 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2022-03-21T14:11:30.994 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2022-03-21T14:11:31.084 INFO:tasks.workunit.client.0.smithi002.stdout:creating folder hierarchy 2022-03-21T14:11:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:31 smithi079 ceph-mon[23158]: pgmap v411: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:31 smithi002 ceph-mon[17167]: pgmap v411: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:32.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:31 smithi016 ceph-mon[22010]: pgmap v411: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:11:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:33 smithi002 ceph-mon[17167]: pgmap v412: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:11:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:33 smithi016 ceph-mon[22010]: pgmap v412: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:11:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:33 smithi079 ceph-mon[23158]: pgmap v412: 97 pgs: 97 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:11:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:35 smithi002 ceph-mon[17167]: pgmap v413: 97 pgs: 97 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:11:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:35 smithi016 ceph-mon[22010]: pgmap v413: 97 pgs: 97 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:11:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:35 smithi079 ceph-mon[23158]: pgmap v413: 97 pgs: 97 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 5 op/s 2022-03-21T14:11:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:37 smithi002 ceph-mon[17167]: pgmap v414: 97 pgs: 97 active+clean; 4.6 MiB data, 97 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 300 KiB/s wr, 10 op/s 2022-03-21T14:11:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:37 smithi016 ceph-mon[22010]: pgmap v414: 97 pgs: 97 active+clean; 4.6 MiB data, 97 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 300 KiB/s wr, 10 op/s 2022-03-21T14:11:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:37 smithi079 ceph-mon[23158]: pgmap v414: 97 pgs: 97 active+clean; 4.6 MiB data, 97 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 300 KiB/s wr, 10 op/s 2022-03-21T14:11:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:39 smithi002 ceph-mon[17167]: pgmap v415: 97 pgs: 97 active+clean; 13 MiB data, 108 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2022-03-21T14:11:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:39 smithi016 ceph-mon[22010]: pgmap v415: 97 pgs: 97 active+clean; 13 MiB data, 108 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2022-03-21T14:11:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:39 smithi079 ceph-mon[23158]: pgmap v415: 97 pgs: 97 active+clean; 13 MiB data, 108 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2022-03-21T14:11:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:41 smithi002 ceph-mon[17167]: pgmap v416: 97 pgs: 97 active+clean; 13 MiB data, 125 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 39 op/s 2022-03-21T14:11:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:41 smithi016 ceph-mon[22010]: pgmap v416: 97 pgs: 97 active+clean; 13 MiB data, 125 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 39 op/s 2022-03-21T14:11:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:41 smithi079 ceph-mon[23158]: pgmap v416: 97 pgs: 97 active+clean; 13 MiB data, 125 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 39 op/s 2022-03-21T14:11:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:43 smithi002 ceph-mon[17167]: pgmap v417: 97 pgs: 97 active+clean; 16 MiB data, 139 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 62 op/s 2022-03-21T14:11:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:43 smithi016 ceph-mon[22010]: pgmap v417: 97 pgs: 97 active+clean; 16 MiB data, 139 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 62 op/s 2022-03-21T14:11:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:43 smithi079 ceph-mon[23158]: pgmap v417: 97 pgs: 97 active+clean; 16 MiB data, 139 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 62 op/s 2022-03-21T14:11:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:45 smithi002 ceph-mon[17167]: pgmap v418: 97 pgs: 97 active+clean; 19 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2022-03-21T14:11:46.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:45 smithi016 ceph-mon[22010]: pgmap v418: 97 pgs: 97 active+clean; 19 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2022-03-21T14:11:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:45 smithi079 ceph-mon[23158]: pgmap v418: 97 pgs: 97 active+clean; 19 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2022-03-21T14:11:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:47 smithi002 ceph-mon[17167]: pgmap v419: 97 pgs: 97 active+clean; 25 MiB data, 172 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 69 op/s 2022-03-21T14:11:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:47 smithi016 ceph-mon[22010]: pgmap v419: 97 pgs: 97 active+clean; 25 MiB data, 172 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 69 op/s 2022-03-21T14:11:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:47 smithi079 ceph-mon[23158]: pgmap v419: 97 pgs: 97 active+clean; 25 MiB data, 172 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 69 op/s 2022-03-21T14:11:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:49 smithi002 ceph-mon[17167]: pgmap v420: 97 pgs: 97 active+clean; 29 MiB data, 179 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 65 op/s 2022-03-21T14:11:50.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:49 smithi016 ceph-mon[22010]: pgmap v420: 97 pgs: 97 active+clean; 29 MiB data, 179 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 65 op/s 2022-03-21T14:11:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:49 smithi079 ceph-mon[23158]: pgmap v420: 97 pgs: 97 active+clean; 29 MiB data, 179 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 65 op/s 2022-03-21T14:11:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:51 smithi002 ceph-mon[17167]: pgmap v421: 97 pgs: 97 active+clean; 31 MiB data, 187 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 48 op/s 2022-03-21T14:11:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:51 smithi016 ceph-mon[22010]: pgmap v421: 97 pgs: 97 active+clean; 31 MiB data, 187 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 48 op/s 2022-03-21T14:11:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:51 smithi079 ceph-mon[23158]: pgmap v421: 97 pgs: 97 active+clean; 31 MiB data, 187 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 48 op/s 2022-03-21T14:11:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:53 smithi002 ceph-mon[17167]: pgmap v422: 97 pgs: 97 active+clean; 35 MiB data, 199 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 39 op/s 2022-03-21T14:11:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:53 smithi016 ceph-mon[22010]: pgmap v422: 97 pgs: 97 active+clean; 35 MiB data, 199 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 39 op/s 2022-03-21T14:11:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:53 smithi079 ceph-mon[23158]: pgmap v422: 97 pgs: 97 active+clean; 35 MiB data, 199 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 39 op/s 2022-03-21T14:11:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:55 smithi002 ceph-mon[17167]: pgmap v423: 97 pgs: 97 active+clean; 35 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-03-21T14:11:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:55 smithi016 ceph-mon[22010]: pgmap v423: 97 pgs: 97 active+clean; 35 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-03-21T14:11:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:55 smithi079 ceph-mon[23158]: pgmap v423: 97 pgs: 97 active+clean; 35 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2022-03-21T14:11:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:57 smithi002 ceph-mon[17167]: pgmap v424: 97 pgs: 97 active+clean; 41 MiB data, 217 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2022-03-21T14:11:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:57 smithi016 ceph-mon[22010]: pgmap v424: 97 pgs: 97 active+clean; 41 MiB data, 217 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2022-03-21T14:11:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:57 smithi079 ceph-mon[23158]: pgmap v424: 97 pgs: 97 active+clean; 41 MiB data, 217 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2022-03-21T14:12:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:11:59 smithi002 ceph-mon[17167]: pgmap v425: 97 pgs: 97 active+clean; 41 MiB data, 219 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2022-03-21T14:12:00.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:11:59 smithi016 ceph-mon[22010]: pgmap v425: 97 pgs: 97 active+clean; 41 MiB data, 219 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2022-03-21T14:12:00.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:11:59 smithi079 ceph-mon[23158]: pgmap v425: 97 pgs: 97 active+clean; 41 MiB data, 219 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2022-03-21T14:12:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:01 smithi002 ceph-mon[17167]: pgmap v426: 97 pgs: 97 active+clean; 48 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2022-03-21T14:12:02.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:01 smithi016 ceph-mon[22010]: pgmap v426: 97 pgs: 97 active+clean; 48 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2022-03-21T14:12:02.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:01 smithi079 ceph-mon[23158]: pgmap v426: 97 pgs: 97 active+clean; 48 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2022-03-21T14:12:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:03 smithi002 ceph-mon[17167]: pgmap v427: 97 pgs: 97 active+clean; 49 MiB data, 248 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 71 op/s 2022-03-21T14:12:04.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:03 smithi016 ceph-mon[22010]: pgmap v427: 97 pgs: 97 active+clean; 49 MiB data, 248 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 71 op/s 2022-03-21T14:12:04.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:03 smithi079 ceph-mon[23158]: pgmap v427: 97 pgs: 97 active+clean; 49 MiB data, 248 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 71 op/s 2022-03-21T14:12:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:05 smithi002 ceph-mon[17167]: pgmap v428: 97 pgs: 97 active+clean; 49 MiB data, 259 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 110 op/s 2022-03-21T14:12:06.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:05 smithi016 ceph-mon[22010]: pgmap v428: 97 pgs: 97 active+clean; 49 MiB data, 259 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 110 op/s 2022-03-21T14:12:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:05 smithi079 ceph-mon[23158]: pgmap v428: 97 pgs: 97 active+clean; 49 MiB data, 259 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 110 op/s 2022-03-21T14:12:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:07 smithi002 ceph-mon[17167]: pgmap v429: 97 pgs: 97 active+clean; 53 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 155 op/s 2022-03-21T14:12:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:07 smithi016 ceph-mon[22010]: pgmap v429: 97 pgs: 97 active+clean; 53 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 155 op/s 2022-03-21T14:12:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:07 smithi079 ceph-mon[23158]: pgmap v429: 97 pgs: 97 active+clean; 53 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 155 op/s 2022-03-21T14:12:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:09 smithi002 ceph-mon[17167]: pgmap v430: 97 pgs: 97 active+clean; 57 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2022-03-21T14:12:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:09 smithi016 ceph-mon[22010]: pgmap v430: 97 pgs: 97 active+clean; 57 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2022-03-21T14:12:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:09 smithi079 ceph-mon[23158]: pgmap v430: 97 pgs: 97 active+clean; 57 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 191 op/s 2022-03-21T14:12:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:11 smithi002 ceph-mon[17167]: pgmap v431: 97 pgs: 97 active+clean; 57 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2022-03-21T14:12:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:11 smithi016 ceph-mon[22010]: pgmap v431: 97 pgs: 97 active+clean; 57 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2022-03-21T14:12:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:11 smithi079 ceph-mon[23158]: pgmap v431: 97 pgs: 97 active+clean; 57 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2022-03-21T14:12:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:13 smithi002 ceph-mon[17167]: pgmap v432: 97 pgs: 97 active+clean; 61 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2022-03-21T14:12:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:13 smithi016 ceph-mon[22010]: pgmap v432: 97 pgs: 97 active+clean; 61 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2022-03-21T14:12:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:13 smithi079 ceph-mon[23158]: pgmap v432: 97 pgs: 97 active+clean; 61 MiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2022-03-21T14:12:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:15 smithi079 ceph-mon[23158]: pgmap v433: 97 pgs: 97 active+clean; 62 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 147 op/s 2022-03-21T14:12:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:12:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:12:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:12:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:15 smithi002 ceph-mon[17167]: pgmap v433: 97 pgs: 97 active+clean; 62 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 147 op/s 2022-03-21T14:12:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:12:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:12:16.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:12:16.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:15 smithi016 ceph-mon[22010]: pgmap v433: 97 pgs: 97 active+clean; 62 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 147 op/s 2022-03-21T14:12:16.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:12:16.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:12:16.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:12:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:17 smithi002 ceph-mon[17167]: pgmap v434: 97 pgs: 97 active+clean; 62 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 105 op/s 2022-03-21T14:12:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:12:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:17 smithi016 ceph-mon[22010]: pgmap v434: 97 pgs: 97 active+clean; 62 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 105 op/s 2022-03-21T14:12:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:12:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:17 smithi079 ceph-mon[23158]: pgmap v434: 97 pgs: 97 active+clean; 62 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 105 op/s 2022-03-21T14:12:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:12:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:19 smithi002 ceph-mon[17167]: pgmap v435: 97 pgs: 97 active+clean; 65 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 61 op/s 2022-03-21T14:12:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:19 smithi016 ceph-mon[22010]: pgmap v435: 97 pgs: 97 active+clean; 65 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 61 op/s 2022-03-21T14:12:19.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:19 smithi079 ceph-mon[23158]: pgmap v435: 97 pgs: 97 active+clean; 65 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 61 op/s 2022-03-21T14:12:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:12:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:12:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:12:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:12:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:12:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:12:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:21 smithi002 ceph-mon[17167]: pgmap v436: 97 pgs: 97 active+clean; 69 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2022-03-21T14:12:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:21 smithi016 ceph-mon[22010]: pgmap v436: 97 pgs: 97 active+clean; 69 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2022-03-21T14:12:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:21 smithi079 ceph-mon[23158]: pgmap v436: 97 pgs: 97 active+clean; 69 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2022-03-21T14:12:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:23 smithi002 ceph-mon[17167]: pgmap v437: 97 pgs: 97 active+clean; 69 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.0 MiB/s wr, 15 op/s 2022-03-21T14:12:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:23 smithi016 ceph-mon[22010]: pgmap v437: 97 pgs: 97 active+clean; 69 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.0 MiB/s wr, 15 op/s 2022-03-21T14:12:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:23 smithi079 ceph-mon[23158]: pgmap v437: 97 pgs: 97 active+clean; 69 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.0 MiB/s wr, 15 op/s 2022-03-21T14:12:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:25 smithi002 ceph-mon[17167]: pgmap v438: 97 pgs: 97 active+clean; 73 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2022-03-21T14:12:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:25 smithi016 ceph-mon[22010]: pgmap v438: 97 pgs: 97 active+clean; 73 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2022-03-21T14:12:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:25 smithi079 ceph-mon[23158]: pgmap v438: 97 pgs: 97 active+clean; 73 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2022-03-21T14:12:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:27 smithi002 ceph-mon[17167]: pgmap v439: 97 pgs: 97 active+clean; 73 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:27 smithi016 ceph-mon[22010]: pgmap v439: 97 pgs: 97 active+clean; 73 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:27 smithi079 ceph-mon[23158]: pgmap v439: 97 pgs: 97 active+clean; 73 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:29 smithi002 ceph-mon[17167]: pgmap v440: 97 pgs: 97 active+clean; 73 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:29.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:29 smithi016 ceph-mon[22010]: pgmap v440: 97 pgs: 97 active+clean; 73 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:29.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:29 smithi079 ceph-mon[23158]: pgmap v440: 97 pgs: 97 active+clean; 73 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 889 KiB/s wr, 6 op/s 2022-03-21T14:12:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:31 smithi002 ceph-mon[17167]: pgmap v441: 97 pgs: 97 active+clean; 77 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:31.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:31 smithi016 ceph-mon[22010]: pgmap v441: 97 pgs: 97 active+clean; 77 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:31 smithi079 ceph-mon[23158]: pgmap v441: 97 pgs: 97 active+clean; 77 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:33 smithi016 ceph-mon[22010]: pgmap v442: 97 pgs: 97 active+clean; 81 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:33 smithi079 ceph-mon[23158]: pgmap v442: 97 pgs: 97 active+clean; 81 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:33 smithi002 ceph-mon[17167]: pgmap v442: 97 pgs: 97 active+clean; 81 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:35 smithi016 ceph-mon[22010]: pgmap v443: 97 pgs: 97 active+clean; 81 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2022-03-21T14:12:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:35 smithi079 ceph-mon[23158]: pgmap v443: 97 pgs: 97 active+clean; 81 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2022-03-21T14:12:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:35 smithi002 ceph-mon[17167]: pgmap v443: 97 pgs: 97 active+clean; 81 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2022-03-21T14:12:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:37 smithi016 ceph-mon[22010]: pgmap v444: 97 pgs: 97 active+clean; 81 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 672 KiB/s wr, 6 op/s 2022-03-21T14:12:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:37 smithi079 ceph-mon[23158]: pgmap v444: 97 pgs: 97 active+clean; 81 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 672 KiB/s wr, 6 op/s 2022-03-21T14:12:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:37 smithi002 ceph-mon[17167]: pgmap v444: 97 pgs: 97 active+clean; 81 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 672 KiB/s wr, 6 op/s 2022-03-21T14:12:39.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:39 smithi016 ceph-mon[22010]: pgmap v445: 97 pgs: 97 active+clean; 85 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1013 KiB/s wr, 7 op/s 2022-03-21T14:12:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:39 smithi079 ceph-mon[23158]: pgmap v445: 97 pgs: 97 active+clean; 85 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1013 KiB/s wr, 7 op/s 2022-03-21T14:12:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:39 smithi002 ceph-mon[17167]: pgmap v445: 97 pgs: 97 active+clean; 85 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1013 KiB/s wr, 7 op/s 2022-03-21T14:12:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:41 smithi016 ceph-mon[22010]: pgmap v446: 97 pgs: 97 active+clean; 89 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2022-03-21T14:12:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:41 smithi079 ceph-mon[23158]: pgmap v446: 97 pgs: 97 active+clean; 89 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2022-03-21T14:12:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:41 smithi002 ceph-mon[17167]: pgmap v446: 97 pgs: 97 active+clean; 89 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2022-03-21T14:12:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:43 smithi016 ceph-mon[22010]: pgmap v447: 97 pgs: 97 active+clean; 91 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2022-03-21T14:12:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:43 smithi079 ceph-mon[23158]: pgmap v447: 97 pgs: 97 active+clean; 91 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2022-03-21T14:12:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:43 smithi002 ceph-mon[17167]: pgmap v447: 97 pgs: 97 active+clean; 91 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2022-03-21T14:12:45.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:45 smithi016 ceph-mon[22010]: pgmap v448: 97 pgs: 97 active+clean; 91 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 931 KiB/s wr, 7 op/s 2022-03-21T14:12:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:45 smithi079 ceph-mon[23158]: pgmap v448: 97 pgs: 97 active+clean; 91 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 931 KiB/s wr, 7 op/s 2022-03-21T14:12:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:45 smithi002 ceph-mon[17167]: pgmap v448: 97 pgs: 97 active+clean; 91 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 931 KiB/s wr, 7 op/s 2022-03-21T14:12:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:47 smithi016 ceph-mon[22010]: pgmap v449: 97 pgs: 97 active+clean; 97 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:47 smithi079 ceph-mon[23158]: pgmap v449: 97 pgs: 97 active+clean; 97 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:47 smithi002 ceph-mon[17167]: pgmap v449: 97 pgs: 97 active+clean; 97 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:49 smithi016 ceph-mon[22010]: pgmap v450: 97 pgs: 97 active+clean; 97 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:49 smithi079 ceph-mon[23158]: pgmap v450: 97 pgs: 97 active+clean; 97 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:49 smithi002 ceph-mon[17167]: pgmap v450: 97 pgs: 97 active+clean; 97 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2022-03-21T14:12:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:51 smithi016 ceph-mon[22010]: pgmap v451: 97 pgs: 97 active+clean; 97 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:51 smithi079 ceph-mon[23158]: pgmap v451: 97 pgs: 97 active+clean; 97 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:51 smithi002 ceph-mon[17167]: pgmap v451: 97 pgs: 97 active+clean; 97 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 6 op/s 2022-03-21T14:12:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:53 smithi016 ceph-mon[22010]: pgmap v452: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 964 KiB/s wr, 8 op/s 2022-03-21T14:12:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:53 smithi079 ceph-mon[23158]: pgmap v452: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 964 KiB/s wr, 8 op/s 2022-03-21T14:12:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:53 smithi002 ceph-mon[17167]: pgmap v452: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 964 KiB/s wr, 8 op/s 2022-03-21T14:12:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:55 smithi016 ceph-mon[22010]: pgmap v453: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 6 op/s 2022-03-21T14:12:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:55 smithi079 ceph-mon[23158]: pgmap v453: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 6 op/s 2022-03-21T14:12:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:55 smithi002 ceph-mon[17167]: pgmap v453: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 6 op/s 2022-03-21T14:12:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:57 smithi016 ceph-mon[22010]: pgmap v454: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 7 op/s 2022-03-21T14:12:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:57 smithi079 ceph-mon[23158]: pgmap v454: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 7 op/s 2022-03-21T14:12:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:57 smithi002 ceph-mon[17167]: pgmap v454: 97 pgs: 97 active+clean; 100 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 716 KiB/s wr, 7 op/s 2022-03-21T14:12:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:12:59 smithi016 ceph-mon[22010]: pgmap v455: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:12:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:12:59 smithi079 ceph-mon[23158]: pgmap v455: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:13:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:12:59 smithi002 ceph-mon[17167]: pgmap v455: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:13:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:01 smithi079 ceph-mon[23158]: pgmap v456: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:13:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:01 smithi002 ceph-mon[17167]: pgmap v456: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:13:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:01 smithi016 ceph-mon[22010]: pgmap v456: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 5 op/s 2022-03-21T14:13:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:03 smithi079 ceph-mon[23158]: pgmap v457: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 296 KiB/s wr, 6 op/s 2022-03-21T14:13:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:03 smithi002 ceph-mon[17167]: pgmap v457: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 296 KiB/s wr, 6 op/s 2022-03-21T14:13:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:03 smithi016 ceph-mon[22010]: pgmap v457: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 296 KiB/s wr, 6 op/s 2022-03-21T14:13:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:05 smithi079 ceph-mon[23158]: pgmap v458: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:05 smithi002 ceph-mon[17167]: pgmap v458: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:06.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:05 smithi016 ceph-mon[22010]: pgmap v458: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:07 smithi079 ceph-mon[23158]: pgmap v459: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:07 smithi002 ceph-mon[17167]: pgmap v459: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:07 smithi016 ceph-mon[22010]: pgmap v459: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 3 op/s 2022-03-21T14:13:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:09 smithi079 ceph-mon[23158]: pgmap v460: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 2 op/s 2022-03-21T14:13:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:09 smithi002 ceph-mon[17167]: pgmap v460: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 2 op/s 2022-03-21T14:13:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:09 smithi016 ceph-mon[22010]: pgmap v460: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 2 op/s 2022-03-21T14:13:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:11 smithi079 ceph-mon[23158]: pgmap v461: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-03-21T14:13:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:11 smithi002 ceph-mon[17167]: pgmap v461: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-03-21T14:13:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:11 smithi016 ceph-mon[22010]: pgmap v461: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-03-21T14:13:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:13 smithi079 ceph-mon[23158]: pgmap v462: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:13:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:13 smithi002 ceph-mon[17167]: pgmap v462: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:13:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:13 smithi016 ceph-mon[22010]: pgmap v462: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2022-03-21T14:13:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:15 smithi079 ceph-mon[23158]: pgmap v463: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:15 smithi002 ceph-mon[17167]: pgmap v463: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:16.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:15 smithi016 ceph-mon[22010]: pgmap v463: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:13:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:13:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:13:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:13:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:13:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:13:16.956 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:13:16.957 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:13:16.957 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:13:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:18 smithi016 ceph-mon[22010]: pgmap v464: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:13:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:18 smithi079 ceph-mon[23158]: pgmap v464: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:13:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:18 smithi002 ceph-mon[17167]: pgmap v464: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:13:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:20 smithi016 ceph-mon[22010]: pgmap v465: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:20 smithi079 ceph-mon[23158]: pgmap v465: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:20 smithi002 ceph-mon[17167]: pgmap v465: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:13:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:13:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:13:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:13:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:13:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:13:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:22 smithi016 ceph-mon[22010]: pgmap v466: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:22 smithi079 ceph-mon[23158]: pgmap v466: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:22 smithi002 ceph-mon[17167]: pgmap v466: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2022-03-21T14:13:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:24 smithi016 ceph-mon[22010]: pgmap v467: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:24 smithi079 ceph-mon[23158]: pgmap v467: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:24 smithi002 ceph-mon[17167]: pgmap v467: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2022-03-21T14:13:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:26 smithi016 ceph-mon[22010]: pgmap v468: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:26 smithi079 ceph-mon[23158]: pgmap v468: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:26 smithi002 ceph-mon[17167]: pgmap v468: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:28 smithi016 ceph-mon[22010]: pgmap v469: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:13:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:28 smithi079 ceph-mon[23158]: pgmap v469: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:13:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:28 smithi002 ceph-mon[17167]: pgmap v469: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T14:13:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:30 smithi016 ceph-mon[22010]: pgmap v470: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:30 smithi079 ceph-mon[23158]: pgmap v470: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:30 smithi002 ceph-mon[17167]: pgmap v470: 97 pgs: 97 active+clean; 100 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:32 smithi016 ceph-mon[22010]: pgmap v471: 97 pgs: 97 active+clean; 100 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:32 smithi079 ceph-mon[23158]: pgmap v471: 97 pgs: 97 active+clean; 100 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:32 smithi002 ceph-mon[17167]: pgmap v471: 97 pgs: 97 active+clean; 100 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-03-21T14:13:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:34 smithi016 ceph-mon[22010]: pgmap v472: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 3 op/s 2022-03-21T14:13:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:34 smithi079 ceph-mon[23158]: pgmap v472: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 3 op/s 2022-03-21T14:13:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:34 smithi002 ceph-mon[17167]: pgmap v472: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 3 op/s 2022-03-21T14:13:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:36 smithi016 ceph-mon[22010]: pgmap v473: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 232 KiB/s wr, 2 op/s 2022-03-21T14:13:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:36 smithi079 ceph-mon[23158]: pgmap v473: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 232 KiB/s wr, 2 op/s 2022-03-21T14:13:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:36 smithi002 ceph-mon[17167]: pgmap v473: 97 pgs: 97 active+clean; 103 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 232 KiB/s wr, 2 op/s 2022-03-21T14:13:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:38 smithi016 ceph-mon[22010]: pgmap v474: 97 pgs: 97 active+clean; 105 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 395 KiB/s wr, 4 op/s 2022-03-21T14:13:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:38 smithi079 ceph-mon[23158]: pgmap v474: 97 pgs: 97 active+clean; 105 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 395 KiB/s wr, 4 op/s 2022-03-21T14:13:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:38 smithi002 ceph-mon[17167]: pgmap v474: 97 pgs: 97 active+clean; 105 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 395 KiB/s wr, 4 op/s 2022-03-21T14:13:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:40 smithi016 ceph-mon[22010]: pgmap v475: 97 pgs: 97 active+clean; 105 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 3 op/s 2022-03-21T14:13:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:40 smithi079 ceph-mon[23158]: pgmap v475: 97 pgs: 97 active+clean; 105 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 3 op/s 2022-03-21T14:13:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:40 smithi002 ceph-mon[17167]: pgmap v475: 97 pgs: 97 active+clean; 105 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 3 op/s 2022-03-21T14:13:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:42 smithi079 ceph-mon[23158]: pgmap v476: 97 pgs: 97 active+clean; 107 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 620 KiB/s wr, 4 op/s 2022-03-21T14:13:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:42 smithi002 ceph-mon[17167]: pgmap v476: 97 pgs: 97 active+clean; 107 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 620 KiB/s wr, 4 op/s 2022-03-21T14:13:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:42 smithi016 ceph-mon[22010]: pgmap v476: 97 pgs: 97 active+clean; 107 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 620 KiB/s wr, 4 op/s 2022-03-21T14:13:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:44 smithi079 ceph-mon[23158]: pgmap v477: 97 pgs: 97 active+clean; 107 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 640 KiB/s wr, 9 op/s 2022-03-21T14:13:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:44 smithi002 ceph-mon[17167]: pgmap v477: 97 pgs: 97 active+clean; 107 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 640 KiB/s wr, 9 op/s 2022-03-21T14:13:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:44 smithi016 ceph-mon[22010]: pgmap v477: 97 pgs: 97 active+clean; 107 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 640 KiB/s wr, 9 op/s 2022-03-21T14:13:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:46 smithi079 ceph-mon[23158]: pgmap v478: 97 pgs: 97 active+clean; 109 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 603 KiB/s wr, 19 op/s 2022-03-21T14:13:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:46 smithi002 ceph-mon[17167]: pgmap v478: 97 pgs: 97 active+clean; 109 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 603 KiB/s wr, 19 op/s 2022-03-21T14:13:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:46 smithi016 ceph-mon[22010]: pgmap v478: 97 pgs: 97 active+clean; 109 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 603 KiB/s wr, 19 op/s 2022-03-21T14:13:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:48 smithi079 ceph-mon[23158]: pgmap v479: 97 pgs: 97 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2022-03-21T14:13:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:48 smithi002 ceph-mon[17167]: pgmap v479: 97 pgs: 97 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2022-03-21T14:13:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:48 smithi016 ceph-mon[22010]: pgmap v479: 97 pgs: 97 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2022-03-21T14:13:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:50 smithi079 ceph-mon[23158]: pgmap v480: 97 pgs: 97 active+clean; 113 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2022-03-21T14:13:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:50 smithi002 ceph-mon[17167]: pgmap v480: 97 pgs: 97 active+clean; 113 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2022-03-21T14:13:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:50 smithi016 ceph-mon[22010]: pgmap v480: 97 pgs: 97 active+clean; 113 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2022-03-21T14:13:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:52 smithi079 ceph-mon[23158]: pgmap v481: 97 pgs: 97 active+clean; 113 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2022-03-21T14:13:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:52 smithi002 ceph-mon[17167]: pgmap v481: 97 pgs: 97 active+clean; 113 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2022-03-21T14:13:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:52 smithi016 ceph-mon[22010]: pgmap v481: 97 pgs: 97 active+clean; 113 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2022-03-21T14:13:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:54 smithi079 ceph-mon[23158]: pgmap v482: 97 pgs: 97 active+clean; 117 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2022-03-21T14:13:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:54 smithi002 ceph-mon[17167]: pgmap v482: 97 pgs: 97 active+clean; 117 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2022-03-21T14:13:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:54 smithi016 ceph-mon[22010]: pgmap v482: 97 pgs: 97 active+clean; 117 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2022-03-21T14:13:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:55 smithi002 ceph-mon[17167]: pgmap v483: 97 pgs: 97 active+clean; 117 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2022-03-21T14:13:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:55 smithi016 ceph-mon[22010]: pgmap v483: 97 pgs: 97 active+clean; 117 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2022-03-21T14:13:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:55 smithi079 ceph-mon[23158]: pgmap v483: 97 pgs: 97 active+clean; 117 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 1.3 MiB/s wr, 101 op/s 2022-03-21T14:13:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:57 smithi002 ceph-mon[17167]: pgmap v484: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2022-03-21T14:13:58.073 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:57 smithi079 ceph-mon[23158]: pgmap v484: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2022-03-21T14:13:58.074 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:57 smithi016 ceph-mon[22010]: pgmap v484: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2022-03-21T14:13:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:13:59 smithi016 ceph-mon[22010]: pgmap v485: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 526 KiB/s wr, 47 op/s 2022-03-21T14:13:59.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:13:59 smithi079 ceph-mon[23158]: pgmap v485: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 526 KiB/s wr, 47 op/s 2022-03-21T14:14:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:13:59 smithi002 ceph-mon[17167]: pgmap v485: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 526 KiB/s wr, 47 op/s 2022-03-21T14:14:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:01 smithi016 ceph-mon[22010]: pgmap v486: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 418 KiB/s wr, 25 op/s 2022-03-21T14:14:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:01 smithi079 ceph-mon[23158]: pgmap v486: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 418 KiB/s wr, 25 op/s 2022-03-21T14:14:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:01 smithi002 ceph-mon[17167]: pgmap v486: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 418 KiB/s wr, 25 op/s 2022-03-21T14:14:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:03 smithi016 ceph-mon[22010]: pgmap v487: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 366 KiB/s wr, 15 op/s 2022-03-21T14:14:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:03 smithi079 ceph-mon[23158]: pgmap v487: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 366 KiB/s wr, 15 op/s 2022-03-21T14:14:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:03 smithi002 ceph-mon[17167]: pgmap v487: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 366 KiB/s wr, 15 op/s 2022-03-21T14:14:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:05 smithi016 ceph-mon[22010]: pgmap v488: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:14:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:05 smithi079 ceph-mon[23158]: pgmap v488: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:14:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:05 smithi002 ceph-mon[17167]: pgmap v488: 97 pgs: 97 active+clean; 117 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 14 KiB/s wr, 4 op/s 2022-03-21T14:14:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:07 smithi016 ceph-mon[22010]: pgmap v489: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 293 KiB/s wr, 4 op/s 2022-03-21T14:14:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:07 smithi079 ceph-mon[23158]: pgmap v489: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 293 KiB/s wr, 4 op/s 2022-03-21T14:14:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:07 smithi002 ceph-mon[17167]: pgmap v489: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 293 KiB/s wr, 4 op/s 2022-03-21T14:14:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:09 smithi016 ceph-mon[22010]: pgmap v490: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:09 smithi079 ceph-mon[23158]: pgmap v490: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:09 smithi002 ceph-mon[17167]: pgmap v490: 97 pgs: 97 active+clean; 121 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:11 smithi016 ceph-mon[22010]: pgmap v491: 97 pgs: 97 active+clean; 121 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:11 smithi079 ceph-mon[23158]: pgmap v491: 97 pgs: 97 active+clean; 121 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:11 smithi002 ceph-mon[17167]: pgmap v491: 97 pgs: 97 active+clean; 121 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 292 KiB/s wr, 3 op/s 2022-03-21T14:14:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:13 smithi016 ceph-mon[22010]: pgmap v492: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 4 op/s 2022-03-21T14:14:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:13 smithi079 ceph-mon[23158]: pgmap v492: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 4 op/s 2022-03-21T14:14:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:13 smithi002 ceph-mon[17167]: pgmap v492: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 4 op/s 2022-03-21T14:14:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:15 smithi016 ceph-mon[22010]: pgmap v493: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 3 op/s 2022-03-21T14:14:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:15 smithi079 ceph-mon[23158]: pgmap v493: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 3 op/s 2022-03-21T14:14:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:15 smithi002 ceph-mon[17167]: pgmap v493: 97 pgs: 97 active+clean; 125 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 633 KiB/s wr, 3 op/s 2022-03-21T14:14:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:17 smithi079 ceph-mon[23158]: pgmap v494: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 662 KiB/s wr, 4 op/s 2022-03-21T14:14:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:14:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:14:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:14:17.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:17 smithi016 ceph-mon[22010]: pgmap v494: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 662 KiB/s wr, 4 op/s 2022-03-21T14:14:17.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:14:17.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:14:17.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:14:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:17 smithi002 ceph-mon[17167]: pgmap v494: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 662 KiB/s wr, 4 op/s 2022-03-21T14:14:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:14:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:14:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:14:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:14:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:14:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:14:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:20 smithi002 ceph-mon[17167]: pgmap v495: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:20 smithi016 ceph-mon[22010]: pgmap v495: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:20 smithi079 ceph-mon[23158]: pgmap v495: 97 pgs: 97 active+clean; 125 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:14:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:14:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:14:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:14:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:14:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:14:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:22 smithi002 ceph-mon[17167]: pgmap v496: 97 pgs: 97 active+clean; 125 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:22 smithi016 ceph-mon[22010]: pgmap v496: 97 pgs: 97 active+clean; 125 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:22 smithi079 ceph-mon[23158]: pgmap v496: 97 pgs: 97 active+clean; 125 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 371 KiB/s wr, 3 op/s 2022-03-21T14:14:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:24 smithi002 ceph-mon[17167]: pgmap v497: 97 pgs: 97 active+clean; 129 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 683 KiB/s wr, 4 op/s 2022-03-21T14:14:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:24 smithi016 ceph-mon[22010]: pgmap v497: 97 pgs: 97 active+clean; 129 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 683 KiB/s wr, 4 op/s 2022-03-21T14:14:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:24 smithi079 ceph-mon[23158]: pgmap v497: 97 pgs: 97 active+clean; 129 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 683 KiB/s wr, 4 op/s 2022-03-21T14:14:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:26 smithi002 ceph-mon[17167]: pgmap v498: 97 pgs: 97 active+clean; 131 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 589 KiB/s wr, 3 op/s 2022-03-21T14:14:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:26 smithi016 ceph-mon[22010]: pgmap v498: 97 pgs: 97 active+clean; 131 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 589 KiB/s wr, 3 op/s 2022-03-21T14:14:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:26 smithi079 ceph-mon[23158]: pgmap v498: 97 pgs: 97 active+clean; 131 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 589 KiB/s wr, 3 op/s 2022-03-21T14:14:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:28 smithi002 ceph-mon[17167]: pgmap v499: 97 pgs: 97 active+clean; 131 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 589 KiB/s wr, 4 op/s 2022-03-21T14:14:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:28 smithi016 ceph-mon[22010]: pgmap v499: 97 pgs: 97 active+clean; 131 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 589 KiB/s wr, 4 op/s 2022-03-21T14:14:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:28 smithi079 ceph-mon[23158]: pgmap v499: 97 pgs: 97 active+clean; 131 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 589 KiB/s wr, 4 op/s 2022-03-21T14:14:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:30 smithi002 ceph-mon[17167]: pgmap v500: 97 pgs: 97 active+clean; 133 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:30 smithi016 ceph-mon[22010]: pgmap v500: 97 pgs: 97 active+clean; 133 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:30 smithi079 ceph-mon[23158]: pgmap v500: 97 pgs: 97 active+clean; 133 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:32 smithi002 ceph-mon[17167]: pgmap v501: 97 pgs: 97 active+clean; 133 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:32 smithi016 ceph-mon[22010]: pgmap v501: 97 pgs: 97 active+clean; 133 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:32 smithi079 ceph-mon[23158]: pgmap v501: 97 pgs: 97 active+clean; 133 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 653 KiB/s wr, 3 op/s 2022-03-21T14:14:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:34 smithi002 ceph-mon[17167]: pgmap v502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 4 op/s 2022-03-21T14:14:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:34 smithi016 ceph-mon[22010]: pgmap v502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 4 op/s 2022-03-21T14:14:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:34 smithi079 ceph-mon[23158]: pgmap v502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 4 op/s 2022-03-21T14:14:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:36 smithi002 ceph-mon[17167]: pgmap v503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 3 op/s 2022-03-21T14:14:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:36 smithi016 ceph-mon[22010]: pgmap v503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 3 op/s 2022-03-21T14:14:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:36 smithi079 ceph-mon[23158]: pgmap v503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 3 op/s 2022-03-21T14:14:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:38 smithi002 ceph-mon[17167]: pgmap v504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:38 smithi016 ceph-mon[22010]: pgmap v504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:38 smithi079 ceph-mon[23158]: pgmap v504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:40 smithi002 ceph-mon[17167]: pgmap v505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:40 smithi016 ceph-mon[22010]: pgmap v505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:40 smithi079 ceph-mon[23158]: pgmap v505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 4 op/s 2022-03-21T14:14:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: pgmap v506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 3 op/s 2022-03-21T14:14:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 184.852710 secs 2022-03-21T14:14:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: slow request 184.747940 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: slow request 184.536756 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: slow request 184.849349 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: slow request 184.800212 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:42 smithi002 ceph-mon[17167]: slow request 184.774952 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: pgmap v506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 3 op/s 2022-03-21T14:14:42.907 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 184.852710 secs 2022-03-21T14:14:42.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: slow request 184.747940 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: slow request 184.536756 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: slow request 184.849349 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: slow request 184.800212 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:42 smithi016 ceph-mon[22010]: slow request 184.774952 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: pgmap v506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 3 op/s 2022-03-21T14:14:42.911 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 184.852710 secs 2022-03-21T14:14:42.911 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: slow request 184.747940 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.912 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: slow request 184.536756 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.912 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: slow request 184.849349 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.912 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: slow request 184.800212 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:42.914 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:42 smithi079 ceph-mon[23158]: slow request 184.774952 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:43 smithi002 ceph-mon[17167]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-03-21T14:14:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:43 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:14:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:43 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:14:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:43 smithi016 ceph-mon[22010]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-03-21T14:14:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:43 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:14:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:43 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:14:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:43 smithi079 ceph-mon[23158]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-03-21T14:14:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:43 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:14:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:43 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:14:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:44 smithi002 ceph-mon[17167]: pgmap v507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:14:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:44 smithi016 ceph-mon[22010]: pgmap v507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:14:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:44 smithi079 ceph-mon[23158]: pgmap v507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 4 op/s 2022-03-21T14:14:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:46 smithi002 ceph-mon[17167]: pgmap v508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:46 smithi016 ceph-mon[22010]: pgmap v508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:46 smithi079 ceph-mon[23158]: pgmap v508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 189.852786 secs 2022-03-21T14:14:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: slow request 189.744748 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: slow request 189.824079 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: slow request 189.741789 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: slow request 189.845985 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:47 smithi016 ceph-mon[22010]: slow request 189.797010 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 189.852786 secs 2022-03-21T14:14:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: slow request 189.744748 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: slow request 189.824079 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: slow request 189.741789 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: slow request 189.845985 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:47 smithi079 ceph-mon[23158]: slow request 189.797010 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 189.852786 secs 2022-03-21T14:14:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: slow request 189.744748 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: slow request 189.824079 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: slow request 189.741789 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: slow request 189.845985 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:47 smithi002 ceph-mon[17167]: slow request 189.797010 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:48 smithi016 ceph-mon[22010]: pgmap v509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:48 smithi079 ceph-mon[23158]: pgmap v509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:48 smithi002 ceph-mon[17167]: pgmap v509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:50 smithi016 ceph-mon[22010]: pgmap v510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:50 smithi079 ceph-mon[23158]: pgmap v510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:50 smithi002 ceph-mon[17167]: pgmap v510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: pgmap v511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 194.852868 secs 2022-03-21T14:14:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: slow request 194.738852 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: slow request 194.821436 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: slow request 194.771011 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: slow request 194.736106 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:52 smithi016 ceph-mon[22010]: slow request 194.842889 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: pgmap v511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 194.852868 secs 2022-03-21T14:14:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: slow request 194.738852 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: slow request 194.821436 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: slow request 194.771011 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: slow request 194.736106 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:52 smithi079 ceph-mon[23158]: slow request 194.842889 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: pgmap v511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 194.852868 secs 2022-03-21T14:14:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: slow request 194.738852 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: slow request 194.821436 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: slow request 194.771011 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: slow request 194.736106 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:52 smithi002 ceph-mon[17167]: slow request 194.842889 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:54 smithi016 ceph-mon[22010]: pgmap v512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:54 smithi079 ceph-mon[23158]: pgmap v512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:54 smithi002 ceph-mon[17167]: pgmap v512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:56 smithi016 ceph-mon[22010]: pgmap v513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:56 smithi079 ceph-mon[23158]: pgmap v513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:56 smithi002 ceph-mon[17167]: pgmap v513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:14:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 199.852981 secs 2022-03-21T14:14:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: slow request 199.818739 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: slow request 199.793886 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: slow request 199.766974 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: slow request 199.840086 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:57 smithi016 ceph-mon[22010]: slow request 199.815672 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 199.852981 secs 2022-03-21T14:14:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: slow request 199.818739 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: slow request 199.793886 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: slow request 199.766974 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: slow request 199.840086 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:57 smithi079 ceph-mon[23158]: slow request 199.815672 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 199.852981 secs 2022-03-21T14:14:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: slow request 199.818739 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: slow request 199.793886 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: slow request 199.766974 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: slow request 199.840086 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:57 smithi002 ceph-mon[17167]: slow request 199.815672 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:14:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:14:58 smithi016 ceph-mon[22010]: pgmap v514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:14:58 smithi079 ceph-mon[23158]: pgmap v514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:14:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:14:58 smithi002 ceph-mon[17167]: pgmap v514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:00 smithi016 ceph-mon[22010]: pgmap v515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:00 smithi079 ceph-mon[23158]: pgmap v515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:00 smithi002 ceph-mon[17167]: pgmap v515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: pgmap v516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 204.853129 secs 2022-03-21T14:15:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: slow request 204.791230 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: slow request 204.764385 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: slow request 204.788267 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: slow request 204.836848 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:02 smithi016 ceph-mon[22010]: slow request 204.813145 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: pgmap v516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 204.853129 secs 2022-03-21T14:15:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: slow request 204.791230 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: slow request 204.764385 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: slow request 204.788267 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: slow request 204.836848 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:02 smithi079 ceph-mon[23158]: slow request 204.813145 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: pgmap v516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 204.853129 secs 2022-03-21T14:15:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: slow request 204.791230 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: slow request 204.764385 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: slow request 204.788267 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: slow request 204.836848 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:02 smithi002 ceph-mon[17167]: slow request 204.813145 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:04 smithi016 ceph-mon[22010]: pgmap v517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:04 smithi079 ceph-mon[23158]: pgmap v517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:04 smithi002 ceph-mon[17167]: pgmap v517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:06 smithi016 ceph-mon[22010]: pgmap v518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:06 smithi079 ceph-mon[23158]: pgmap v518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:06 smithi002 ceph-mon[17167]: pgmap v518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 209.853245 secs 2022-03-21T14:15:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: slow request 209.761308 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: slow request 209.810413 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: slow request 209.785308 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: slow request 209.833903 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:07 smithi016 ceph-mon[22010]: slow request 209.757546 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 209.853245 secs 2022-03-21T14:15:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: slow request 209.761308 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: slow request 209.810413 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: slow request 209.785308 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: slow request 209.833903 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:07 smithi079 ceph-mon[23158]: slow request 209.757546 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 209.853245 secs 2022-03-21T14:15:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: slow request 209.761308 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: slow request 209.810413 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: slow request 209.785308 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: slow request 209.833903 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:07 smithi002 ceph-mon[17167]: slow request 209.757546 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:08 smithi079 ceph-mon[23158]: pgmap v519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:08 smithi002 ceph-mon[17167]: pgmap v519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:08 smithi016 ceph-mon[22010]: pgmap v519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:10 smithi079 ceph-mon[23158]: pgmap v520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:10 smithi002 ceph-mon[17167]: pgmap v520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:10 smithi016 ceph-mon[22010]: pgmap v520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: pgmap v521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 214.853364 secs 2022-03-21T14:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: slow request 214.737257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: slow request 214.754662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: slow request 214.830563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: slow request 214.807161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:12 smithi079 ceph-mon[23158]: slow request 214.782104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: pgmap v521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 214.853364 secs 2022-03-21T14:15:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: slow request 214.737257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:15:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: slow request 214.754662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: slow request 214.830563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: slow request 214.807161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:12 smithi002 ceph-mon[17167]: slow request 214.782104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: pgmap v521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 214.853364 secs 2022-03-21T14:15:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: slow request 214.737257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:15:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: slow request 214.754662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: slow request 214.830563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: slow request 214.807161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:12 smithi016 ceph-mon[22010]: slow request 214.782104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:14 smithi079 ceph-mon[23158]: pgmap v522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:14 smithi002 ceph-mon[17167]: pgmap v522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:14 smithi016 ceph-mon[22010]: pgmap v522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:16 smithi079 ceph-mon[23158]: pgmap v523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:16 smithi002 ceph-mon[17167]: pgmap v523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:16 smithi016 ceph-mon[22010]: pgmap v523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 219.853498 secs 2022-03-21T14:15:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: slow request 219.751474 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: slow request 219.853497 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: slow request 219.804274 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: slow request 219.779324 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:17 smithi079 ceph-mon[23158]: slow request 219.827771 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 219.853498 secs 2022-03-21T14:15:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: slow request 219.751474 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: slow request 219.853497 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: slow request 219.804274 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: slow request 219.779324 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:17 smithi002 ceph-mon[17167]: slow request 219.827771 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 219.853498 secs 2022-03-21T14:15:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: slow request 219.751474 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: slow request 219.853497 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: slow request 219.804274 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: slow request 219.779324 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:17 smithi016 ceph-mon[22010]: slow request 219.827771 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:15:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:18 smithi002 ceph-mon[17167]: pgmap v524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:15:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:15:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:15:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:18 smithi016 ceph-mon[22010]: pgmap v524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:15:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:15:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:15:18.691 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:18 smithi079 ceph-mon[23158]: pgmap v524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:15:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:15:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:15:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:20 smithi016 ceph-mon[22010]: pgmap v525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:15:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:20 smithi079 ceph-mon[23158]: pgmap v525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:15:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:20 smithi002 ceph-mon[17167]: pgmap v525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:15:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:15:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:15:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:15:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:15:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:15:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:15:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:22 smithi016 ceph-mon[22010]: pgmap v526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:22 smithi079 ceph-mon[23158]: pgmap v526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:22 smithi002 ceph-mon[17167]: pgmap v526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 224.853612 secs 2022-03-21T14:15:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 224.853612 secs 2022-03-21T14:15:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 224.853612 secs 2022-03-21T14:15:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:24 smithi016 ceph-mon[22010]: pgmap v527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:24 smithi079 ceph-mon[23158]: pgmap v527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:24 smithi002 ceph-mon[17167]: pgmap v527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:26 smithi016 ceph-mon[22010]: pgmap v528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:26 smithi079 ceph-mon[23158]: pgmap v528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:26 smithi002 ceph-mon[17167]: pgmap v528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:28 smithi016 ceph-mon[22010]: pgmap v529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 229.853731 secs 2022-03-21T14:15:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:28 smithi079 ceph-mon[23158]: pgmap v529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 229.853731 secs 2022-03-21T14:15:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:28 smithi002 ceph-mon[17167]: pgmap v529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 229.853731 secs 2022-03-21T14:15:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:30 smithi016 ceph-mon[22010]: pgmap v530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:30 smithi079 ceph-mon[23158]: pgmap v530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:30 smithi002 ceph-mon[17167]: pgmap v530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:32 smithi016 ceph-mon[22010]: pgmap v531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:32 smithi079 ceph-mon[23158]: pgmap v531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:32 smithi002 ceph-mon[17167]: pgmap v531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:33.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 234.853844 secs 2022-03-21T14:15:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 234.853844 secs 2022-03-21T14:15:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 234.853844 secs 2022-03-21T14:15:34.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:34 smithi016 ceph-mon[22010]: pgmap v532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:34 smithi079 ceph-mon[23158]: pgmap v532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:34 smithi002 ceph-mon[17167]: pgmap v532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:36.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:36 smithi016 ceph-mon[22010]: pgmap v533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:36 smithi079 ceph-mon[23158]: pgmap v533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:36 smithi002 ceph-mon[17167]: pgmap v533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:38 smithi079 ceph-mon[23158]: pgmap v534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 239.853926 secs 2022-03-21T14:15:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:38 smithi002 ceph-mon[17167]: pgmap v534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 239.853926 secs 2022-03-21T14:15:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:38 smithi016 ceph-mon[22010]: pgmap v534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 239.853926 secs 2022-03-21T14:15:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:40 smithi079 ceph-mon[23158]: pgmap v535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:40 smithi002 ceph-mon[17167]: pgmap v535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:40 smithi016 ceph-mon[22010]: pgmap v535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:42 smithi079 ceph-mon[23158]: pgmap v536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:42 smithi002 ceph-mon[17167]: pgmap v536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:42 smithi016 ceph-mon[22010]: pgmap v536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 244.854052 secs 2022-03-21T14:15:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 244.854052 secs 2022-03-21T14:15:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 244.854052 secs 2022-03-21T14:15:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:44 smithi079 ceph-mon[23158]: pgmap v537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:44 smithi002 ceph-mon[17167]: pgmap v537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:44 smithi016 ceph-mon[22010]: pgmap v537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:46 smithi079 ceph-mon[23158]: pgmap v538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:46 smithi002 ceph-mon[17167]: pgmap v538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:46 smithi016 ceph-mon[22010]: pgmap v538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:48 smithi079 ceph-mon[23158]: pgmap v539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 249.854159 secs 2022-03-21T14:15:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:48 smithi002 ceph-mon[17167]: pgmap v539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 249.854159 secs 2022-03-21T14:15:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:48 smithi016 ceph-mon[22010]: pgmap v539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 249.854159 secs 2022-03-21T14:15:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:50 smithi079 ceph-mon[23158]: pgmap v540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:50 smithi002 ceph-mon[17167]: pgmap v540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:50 smithi016 ceph-mon[22010]: pgmap v540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:52 smithi079 ceph-mon[23158]: pgmap v541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:52 smithi002 ceph-mon[17167]: pgmap v541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:52 smithi016 ceph-mon[22010]: pgmap v541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 254.854311 secs 2022-03-21T14:15:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 254.854311 secs 2022-03-21T14:15:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 254.854311 secs 2022-03-21T14:15:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:54 smithi079 ceph-mon[23158]: pgmap v542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:54 smithi002 ceph-mon[17167]: pgmap v542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:54 smithi016 ceph-mon[22010]: pgmap v542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:56 smithi079 ceph-mon[23158]: pgmap v543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:56 smithi002 ceph-mon[17167]: pgmap v543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:56 smithi016 ceph-mon[22010]: pgmap v543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:15:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:58 smithi079 ceph-mon[23158]: pgmap v544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:15:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 259.854437 secs 2022-03-21T14:15:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:58 smithi002 ceph-mon[17167]: pgmap v544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:15:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 259.854437 secs 2022-03-21T14:15:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:58 smithi016 ceph-mon[22010]: pgmap v544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:15:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:15:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 259.854437 secs 2022-03-21T14:16:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:00 smithi079 ceph-mon[23158]: pgmap v545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:00 smithi002 ceph-mon[17167]: pgmap v545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:00 smithi016 ceph-mon[22010]: pgmap v545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:02 smithi079 ceph-mon[23158]: pgmap v546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:02 smithi002 ceph-mon[17167]: pgmap v546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:02 smithi016 ceph-mon[22010]: pgmap v546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 264.854581 secs 2022-03-21T14:16:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 264.854581 secs 2022-03-21T14:16:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 264.854581 secs 2022-03-21T14:16:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:04 smithi002 ceph-mon[17167]: pgmap v547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:04 smithi016 ceph-mon[22010]: pgmap v547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:04 smithi079 ceph-mon[23158]: pgmap v547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:06 smithi002 ceph-mon[17167]: pgmap v548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:06 smithi016 ceph-mon[22010]: pgmap v548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:06 smithi079 ceph-mon[23158]: pgmap v548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:07.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 269.854672 secs 2022-03-21T14:16:07.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 269.854672 secs 2022-03-21T14:16:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 269.854672 secs 2022-03-21T14:16:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:08 smithi002 ceph-mon[17167]: pgmap v549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:08 smithi016 ceph-mon[22010]: pgmap v549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:08 smithi079 ceph-mon[23158]: pgmap v549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:10 smithi002 ceph-mon[17167]: pgmap v550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:10 smithi016 ceph-mon[22010]: pgmap v550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:10 smithi079 ceph-mon[23158]: pgmap v550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:12 smithi002 ceph-mon[17167]: pgmap v551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 274.854769 secs 2022-03-21T14:16:12.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:12 smithi016 ceph-mon[22010]: pgmap v551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 274.854769 secs 2022-03-21T14:16:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:12 smithi079 ceph-mon[23158]: pgmap v551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 274.854769 secs 2022-03-21T14:16:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:14 smithi002 ceph-mon[17167]: pgmap v552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:14 smithi016 ceph-mon[22010]: pgmap v552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:14 smithi079 ceph-mon[23158]: pgmap v552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:16 smithi002 ceph-mon[17167]: pgmap v553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:16 smithi016 ceph-mon[22010]: pgmap v553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:16 smithi079 ceph-mon[23158]: pgmap v553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 279.854883 secs 2022-03-21T14:16:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 279.854883 secs 2022-03-21T14:16:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 279.854883 secs 2022-03-21T14:16:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:18 smithi002 ceph-mon[17167]: pgmap v554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:18 smithi016 ceph-mon[22010]: pgmap v554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:18 smithi079 ceph-mon[23158]: pgmap v554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: pgmap v555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:16:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: pgmap v555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:19.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:16:19.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:16:19.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:16:19.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.973 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.973 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.973 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.973 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.973 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.974 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: pgmap v555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:16:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:16:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:16:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:16:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:16:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:16:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:16:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:16:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:16:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:16:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:16:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:16:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:16:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:16:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:16:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:16:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:16:21.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:16:21.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:16:21.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:16:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:16:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:16:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:16:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:16:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:16:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:16:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:16:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:16:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:16:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:16:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:21 smithi002 ceph-mon[17167]: pgmap v556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:21 smithi016 ceph-mon[22010]: pgmap v556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:21 smithi079 ceph-mon[23158]: pgmap v556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 284.854999 secs 2022-03-21T14:16:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 284.854999 secs 2022-03-21T14:16:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 284.854999 secs 2022-03-21T14:16:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:23 smithi002 ceph-mon[17167]: pgmap v557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:23 smithi016 ceph-mon[22010]: pgmap v557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:23 smithi079 ceph-mon[23158]: pgmap v557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:25 smithi002 ceph-mon[17167]: pgmap v558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:25 smithi016 ceph-mon[22010]: pgmap v558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:25 smithi079 ceph-mon[23158]: pgmap v558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:28 smithi002 ceph-mon[17167]: pgmap v559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 289.855109 secs 2022-03-21T14:16:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:28 smithi016 ceph-mon[22010]: pgmap v559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 289.855109 secs 2022-03-21T14:16:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:28 smithi079 ceph-mon[23158]: pgmap v559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 289.855109 secs 2022-03-21T14:16:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:30 smithi002 ceph-mon[17167]: pgmap v560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:30 smithi016 ceph-mon[22010]: pgmap v560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:30 smithi079 ceph-mon[23158]: pgmap v560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:32 smithi002 ceph-mon[17167]: pgmap v561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:32 smithi016 ceph-mon[22010]: pgmap v561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:32 smithi079 ceph-mon[23158]: pgmap v561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 294.855230 secs 2022-03-21T14:16:33.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 294.855230 secs 2022-03-21T14:16:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 294.855230 secs 2022-03-21T14:16:34.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:34 smithi002 ceph-mon[17167]: pgmap v562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:34.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:34 smithi016 ceph-mon[22010]: pgmap v562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:34 smithi079 ceph-mon[23158]: pgmap v562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:36.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:36 smithi002 ceph-mon[17167]: pgmap v563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:36 smithi016 ceph-mon[22010]: pgmap v563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:36 smithi079 ceph-mon[23158]: pgmap v563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:38 smithi002 ceph-mon[17167]: pgmap v564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 299.855350 secs 2022-03-21T14:16:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:38 smithi016 ceph-mon[22010]: pgmap v564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 299.855350 secs 2022-03-21T14:16:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:38 smithi079 ceph-mon[23158]: pgmap v564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 299.855350 secs 2022-03-21T14:16:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:40 smithi002 ceph-mon[17167]: pgmap v565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:40 smithi016 ceph-mon[22010]: pgmap v565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:40 smithi079 ceph-mon[23158]: pgmap v565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:42.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:42 smithi002 ceph-mon[17167]: pgmap v566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:42.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:42 smithi016 ceph-mon[22010]: pgmap v566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:42 smithi079 ceph-mon[23158]: pgmap v566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 304.855471 secs 2022-03-21T14:16:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:43 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 304.740678 seconds ago 2022-03-21T14:16:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 304.855471 secs 2022-03-21T14:16:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:43 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 304.740678 seconds ago 2022-03-21T14:16:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 304.855471 secs 2022-03-21T14:16:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:43 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 304.740678 seconds ago 2022-03-21T14:16:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:44 smithi002 ceph-mon[17167]: pgmap v567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:44 smithi002 ceph-mon[17167]: Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2022-03-21T14:16:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:44 smithi002 ceph-mon[17167]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:16:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:44 smithi002 ceph-mon[17167]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:16:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:44 smithi016 ceph-mon[22010]: pgmap v567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:44 smithi016 ceph-mon[22010]: Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2022-03-21T14:16:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:44 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:16:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:44 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:16:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:44 smithi079 ceph-mon[23158]: pgmap v567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:44 smithi079 ceph-mon[23158]: Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2022-03-21T14:16:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:44 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T14:16:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:44 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T14:16:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:46 smithi002 ceph-mon[17167]: pgmap v568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:46 smithi016 ceph-mon[22010]: pgmap v568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:46 smithi079 ceph-mon[23158]: pgmap v568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:48 smithi002 ceph-mon[17167]: pgmap v569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 309.855612 secs 2022-03-21T14:16:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:48 smithi016 ceph-mon[22010]: pgmap v569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 309.855612 secs 2022-03-21T14:16:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:48 smithi079 ceph-mon[23158]: pgmap v569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 309.855612 secs 2022-03-21T14:16:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:50 smithi002 ceph-mon[17167]: pgmap v570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:50 smithi016 ceph-mon[22010]: pgmap v570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:50 smithi079 ceph-mon[23158]: pgmap v570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:52 smithi002 ceph-mon[17167]: pgmap v571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:52.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:52 smithi016 ceph-mon[22010]: pgmap v571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:52 smithi079 ceph-mon[23158]: pgmap v571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 314.855735 secs 2022-03-21T14:16:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 314.855735 secs 2022-03-21T14:16:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 314.855735 secs 2022-03-21T14:16:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:54 smithi002 ceph-mon[17167]: pgmap v572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:54.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:54 smithi016 ceph-mon[22010]: pgmap v572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:54 smithi079 ceph-mon[23158]: pgmap v572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:56 smithi002 ceph-mon[17167]: pgmap v573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:56.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:56 smithi016 ceph-mon[22010]: pgmap v573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:56 smithi079 ceph-mon[23158]: pgmap v573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:16:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:58 smithi002 ceph-mon[17167]: pgmap v574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:16:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 319.855857 secs 2022-03-21T14:16:58.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:58 smithi016 ceph-mon[22010]: pgmap v574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:16:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 319.855857 secs 2022-03-21T14:16:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:58 smithi079 ceph-mon[23158]: pgmap v574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:16:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:16:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 319.855857 secs 2022-03-21T14:17:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:00 smithi002 ceph-mon[17167]: pgmap v575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:00 smithi016 ceph-mon[22010]: pgmap v575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:00 smithi079 ceph-mon[23158]: pgmap v575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:02 smithi002 ceph-mon[17167]: pgmap v576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:02 smithi016 ceph-mon[22010]: pgmap v576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:02 smithi079 ceph-mon[23158]: pgmap v576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 324.855974 secs 2022-03-21T14:17:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 324.855974 secs 2022-03-21T14:17:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 324.855974 secs 2022-03-21T14:17:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:04 smithi002 ceph-mon[17167]: pgmap v577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:04 smithi016 ceph-mon[22010]: pgmap v577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:04 smithi079 ceph-mon[23158]: pgmap v577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:06 smithi002 ceph-mon[17167]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:06 smithi016 ceph-mon[22010]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:06 smithi079 ceph-mon[23158]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:08 smithi002 ceph-mon[17167]: pgmap v579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 329.856089 secs 2022-03-21T14:17:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:08 smithi016 ceph-mon[22010]: pgmap v579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 329.856089 secs 2022-03-21T14:17:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:08 smithi079 ceph-mon[23158]: pgmap v579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 329.856089 secs 2022-03-21T14:17:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:10 smithi002 ceph-mon[17167]: pgmap v580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:10.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:10 smithi016 ceph-mon[22010]: pgmap v580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:10 smithi079 ceph-mon[23158]: pgmap v580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:12 smithi002 ceph-mon[17167]: pgmap v581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:12 smithi016 ceph-mon[22010]: pgmap v581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:12 smithi079 ceph-mon[23158]: pgmap v581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 334.856227 secs 2022-03-21T14:17:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 334.856227 secs 2022-03-21T14:17:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 334.856227 secs 2022-03-21T14:17:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:14 smithi002 ceph-mon[17167]: pgmap v582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:14 smithi016 ceph-mon[22010]: pgmap v582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:14 smithi079 ceph-mon[23158]: pgmap v582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:16 smithi002 ceph-mon[17167]: pgmap v583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:16 smithi016 ceph-mon[22010]: pgmap v583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:16 smithi079 ceph-mon[23158]: pgmap v583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:18 smithi002 ceph-mon[17167]: pgmap v584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 339.856346 secs 2022-03-21T14:17:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:18 smithi016 ceph-mon[22010]: pgmap v584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 339.856346 secs 2022-03-21T14:17:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:18 smithi079 ceph-mon[23158]: pgmap v584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 339.856346 secs 2022-03-21T14:17:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:20 smithi002 ceph-mon[17167]: pgmap v585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:17:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:17:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:17:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:20 smithi016 ceph-mon[22010]: pgmap v585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:17:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:17:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:17:20.431 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:20 smithi079 ceph-mon[23158]: pgmap v585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:20.431 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:17:20.431 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:17:20.432 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:17:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:17:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:17:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:17:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:17:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:17:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:17:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:17:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:17:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:17:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:22 smithi016 ceph-mon[22010]: pgmap v586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:22 smithi079 ceph-mon[23158]: pgmap v586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:22 smithi002 ceph-mon[17167]: pgmap v586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 344.856412 secs 2022-03-21T14:17:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 344.856412 secs 2022-03-21T14:17:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 344.856412 secs 2022-03-21T14:17:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:24 smithi016 ceph-mon[22010]: pgmap v587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:24 smithi079 ceph-mon[23158]: pgmap v587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:24 smithi002 ceph-mon[17167]: pgmap v587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:26 smithi016 ceph-mon[22010]: pgmap v588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:26 smithi079 ceph-mon[23158]: pgmap v588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:26 smithi002 ceph-mon[17167]: pgmap v588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:28 smithi016 ceph-mon[22010]: pgmap v589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 349.856550 secs 2022-03-21T14:17:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:28 smithi079 ceph-mon[23158]: pgmap v589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 349.856550 secs 2022-03-21T14:17:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:28 smithi002 ceph-mon[17167]: pgmap v589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 349.856550 secs 2022-03-21T14:17:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:30 smithi016 ceph-mon[22010]: pgmap v590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:30 smithi079 ceph-mon[23158]: pgmap v590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:30 smithi002 ceph-mon[17167]: pgmap v590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:32 smithi016 ceph-mon[22010]: pgmap v591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:32 smithi079 ceph-mon[23158]: pgmap v591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:32 smithi002 ceph-mon[17167]: pgmap v591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:33.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 354.856648 secs 2022-03-21T14:17:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 354.856648 secs 2022-03-21T14:17:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 354.856648 secs 2022-03-21T14:17:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:34 smithi016 ceph-mon[22010]: pgmap v592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:34 smithi079 ceph-mon[23158]: pgmap v592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:34 smithi002 ceph-mon[17167]: pgmap v592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:36.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:36 smithi016 ceph-mon[22010]: pgmap v593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:36 smithi079 ceph-mon[23158]: pgmap v593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:36 smithi002 ceph-mon[17167]: pgmap v593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:38 smithi016 ceph-mon[22010]: pgmap v594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 359.856731 secs 2022-03-21T14:17:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:38 smithi079 ceph-mon[23158]: pgmap v594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 359.856731 secs 2022-03-21T14:17:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:38 smithi002 ceph-mon[17167]: pgmap v594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 359.856731 secs 2022-03-21T14:17:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:40 smithi079 ceph-mon[23158]: pgmap v595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:40 smithi002 ceph-mon[17167]: pgmap v595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:40 smithi016 ceph-mon[22010]: pgmap v595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:42 smithi079 ceph-mon[23158]: pgmap v596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:42 smithi002 ceph-mon[17167]: pgmap v596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:42 smithi016 ceph-mon[22010]: pgmap v596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:17:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 364.856845 secs 2022-03-21T14:17:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: slow request 364.752074 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: slow request 364.540890 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: slow request 364.853484 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: slow request 364.804346 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:43 smithi079 ceph-mon[23158]: slow request 364.779086 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 364.856845 secs 2022-03-21T14:17:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: slow request 364.752074 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: slow request 364.540890 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: slow request 364.853484 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: slow request 364.804346 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:43 smithi002 ceph-mon[17167]: slow request 364.779086 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 364.856845 secs 2022-03-21T14:17:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: slow request 364.752074 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: slow request 364.540890 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: slow request 364.853484 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: slow request 364.804346 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:43 smithi016 ceph-mon[22010]: slow request 364.779086 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:44 smithi079 ceph-mon[23158]: pgmap v597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:44 smithi002 ceph-mon[17167]: pgmap v597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:44 smithi016 ceph-mon[22010]: pgmap v597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:46 smithi079 ceph-mon[23158]: pgmap v598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:46 smithi002 ceph-mon[17167]: pgmap v598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:46 smithi016 ceph-mon[22010]: pgmap v598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: pgmap v599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 369.856937 secs 2022-03-21T14:17:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: slow request 369.748899 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: slow request 369.828231 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: slow request 369.745941 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: slow request 369.850137 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:48 smithi079 ceph-mon[23158]: slow request 369.801161 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: pgmap v599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 369.856937 secs 2022-03-21T14:17:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: slow request 369.748899 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: slow request 369.828231 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: slow request 369.745941 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: slow request 369.850137 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:48 smithi002 ceph-mon[17167]: slow request 369.801161 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: pgmap v599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 369.856937 secs 2022-03-21T14:17:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: slow request 369.748899 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: slow request 369.828231 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: slow request 369.745941 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: slow request 369.850137 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:48 smithi016 ceph-mon[22010]: slow request 369.801161 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:50 smithi079 ceph-mon[23158]: pgmap v600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:50 smithi002 ceph-mon[17167]: pgmap v600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:50 smithi016 ceph-mon[22010]: pgmap v600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:52 smithi079 ceph-mon[23158]: pgmap v601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:52 smithi002 ceph-mon[17167]: pgmap v601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:52 smithi016 ceph-mon[22010]: pgmap v601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 374.857056 secs 2022-03-21T14:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: slow request 374.743041 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: slow request 374.825625 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: slow request 374.775200 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: slow request 374.740295 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:53 smithi079 ceph-mon[23158]: slow request 374.847078 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 374.857056 secs 2022-03-21T14:17:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: slow request 374.743041 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: slow request 374.825625 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: slow request 374.775200 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: slow request 374.740295 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:53 smithi002 ceph-mon[17167]: slow request 374.847078 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 374.857056 secs 2022-03-21T14:17:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: slow request 374.743041 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: slow request 374.825625 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: slow request 374.775200 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: slow request 374.740295 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:53 smithi016 ceph-mon[22010]: slow request 374.847078 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:54 smithi079 ceph-mon[23158]: pgmap v602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:54 smithi002 ceph-mon[17167]: pgmap v602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:54 smithi016 ceph-mon[22010]: pgmap v602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:56 smithi079 ceph-mon[23158]: pgmap v603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:56 smithi002 ceph-mon[17167]: pgmap v603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:56.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:56 smithi016 ceph-mon[22010]: pgmap v603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:17:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: pgmap v604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 379.857152 secs 2022-03-21T14:17:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: slow request 379.822910 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: slow request 379.798057 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: slow request 379.771144 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: slow request 379.844256 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:17:58 smithi079 ceph-mon[23158]: slow request 379.819843 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: pgmap v604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 379.857152 secs 2022-03-21T14:17:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: slow request 379.822910 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: slow request 379.798057 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: slow request 379.771144 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: slow request 379.844256 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:17:58 smithi002 ceph-mon[17167]: slow request 379.819843 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: pgmap v604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:17:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 379.857152 secs 2022-03-21T14:17:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: slow request 379.822910 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: slow request 379.798057 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: slow request 379.771144 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: slow request 379.844256 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:17:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:17:58 smithi016 ceph-mon[22010]: slow request 379.819843 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:00 smithi079 ceph-mon[23158]: pgmap v605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:00 smithi002 ceph-mon[17167]: pgmap v605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:00 smithi016 ceph-mon[22010]: pgmap v605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:02 smithi002 ceph-mon[17167]: pgmap v606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:02 smithi016 ceph-mon[22010]: pgmap v606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:02 smithi079 ceph-mon[23158]: pgmap v606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 384.857274 secs 2022-03-21T14:18:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: slow request 384.795374 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: slow request 384.768529 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: slow request 384.792412 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: slow request 384.840992 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:03 smithi002 ceph-mon[17167]: slow request 384.817289 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 384.857274 secs 2022-03-21T14:18:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: slow request 384.795374 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: slow request 384.768529 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: slow request 384.792412 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: slow request 384.840992 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:03 smithi016 ceph-mon[22010]: slow request 384.817289 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 384.857274 secs 2022-03-21T14:18:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: slow request 384.795374 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: slow request 384.768529 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: slow request 384.792412 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: slow request 384.840992 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:03 smithi079 ceph-mon[23158]: slow request 384.817289 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:04 smithi002 ceph-mon[17167]: pgmap v607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:04 smithi016 ceph-mon[22010]: pgmap v607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:04 smithi079 ceph-mon[23158]: pgmap v607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:06 smithi002 ceph-mon[17167]: pgmap v608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:06 smithi016 ceph-mon[22010]: pgmap v608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:06 smithi079 ceph-mon[23158]: pgmap v608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: pgmap v609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 389.857357 secs 2022-03-21T14:18:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: slow request 389.765419 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: slow request 389.814524 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: slow request 389.789419 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: slow request 389.838015 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:08 smithi002 ceph-mon[17167]: slow request 389.761657 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: pgmap v609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 389.857357 secs 2022-03-21T14:18:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: slow request 389.765419 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: slow request 389.814524 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: slow request 389.789419 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: slow request 389.838015 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:08 smithi016 ceph-mon[22010]: slow request 389.761657 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: pgmap v609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 389.857357 secs 2022-03-21T14:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: slow request 389.765419 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: slow request 389.814524 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: slow request 389.789419 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: slow request 389.838015 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:08 smithi079 ceph-mon[23158]: slow request 389.761657 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:10 smithi002 ceph-mon[17167]: pgmap v610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:10 smithi016 ceph-mon[22010]: pgmap v610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:10 smithi079 ceph-mon[23158]: pgmap v610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:12 smithi002 ceph-mon[17167]: pgmap v611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:12 smithi016 ceph-mon[22010]: pgmap v611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:12 smithi079 ceph-mon[23158]: pgmap v611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 394.857447 secs 2022-03-21T14:18:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: slow request 394.741340 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:18:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: slow request 394.758745 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: slow request 394.834646 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: slow request 394.811244 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:13 smithi002 ceph-mon[17167]: slow request 394.786186 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 394.857447 secs 2022-03-21T14:18:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: slow request 394.741340 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:18:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: slow request 394.758745 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: slow request 394.834646 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: slow request 394.811244 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:13 smithi016 ceph-mon[22010]: slow request 394.786186 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 394.857447 secs 2022-03-21T14:18:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: slow request 394.741340 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:18:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: slow request 394.758745 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: slow request 394.834646 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: slow request 394.811244 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:13 smithi079 ceph-mon[23158]: slow request 394.786186 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:14 smithi002 ceph-mon[17167]: pgmap v612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:14 smithi016 ceph-mon[22010]: pgmap v612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:14 smithi079 ceph-mon[23158]: pgmap v612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:16 smithi002 ceph-mon[17167]: pgmap v613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:16 smithi016 ceph-mon[22010]: pgmap v613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:16 smithi079 ceph-mon[23158]: pgmap v613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 399.857565 secs 2022-03-21T14:18:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: slow request 399.755540 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: slow request 399.857564 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: slow request 399.808341 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: slow request 399.783391 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:17 smithi002 ceph-mon[17167]: slow request 399.831838 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 399.857565 secs 2022-03-21T14:18:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: slow request 399.755540 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: slow request 399.857564 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: slow request 399.808341 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: slow request 399.783391 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:17 smithi016 ceph-mon[22010]: slow request 399.831838 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 399.857565 secs 2022-03-21T14:18:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: slow request 399.755540 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: slow request 399.857564 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: slow request 399.808341 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: slow request 399.783391 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:17 smithi079 ceph-mon[23158]: slow request 399.831838 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:18:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:18 smithi002 ceph-mon[17167]: pgmap v614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:18 smithi016 ceph-mon[22010]: pgmap v614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:18 smithi079 ceph-mon[23158]: pgmap v614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:20 smithi002 ceph-mon[17167]: pgmap v615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:20 smithi016 ceph-mon[22010]: pgmap v615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:20 smithi079 ceph-mon[23158]: pgmap v615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:18:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:18:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:18:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:18:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:18:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:18:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:18:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:18:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:18:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:18:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:18:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:18:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:22 smithi002 ceph-mon[17167]: pgmap v616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:18:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 404.857687 secs 2022-03-21T14:18:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:22 smithi016 ceph-mon[22010]: pgmap v616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:18:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 404.857687 secs 2022-03-21T14:18:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:22 smithi079 ceph-mon[23158]: pgmap v616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:18:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 404.857687 secs 2022-03-21T14:18:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:24 smithi002 ceph-mon[17167]: pgmap v617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:24 smithi016 ceph-mon[22010]: pgmap v617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:24 smithi079 ceph-mon[23158]: pgmap v617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:26 smithi002 ceph-mon[17167]: pgmap v618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:26 smithi016 ceph-mon[22010]: pgmap v618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:26 smithi079 ceph-mon[23158]: pgmap v618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 409.857753 secs 2022-03-21T14:18:27.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 409.857753 secs 2022-03-21T14:18:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 409.857753 secs 2022-03-21T14:18:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:28 smithi002 ceph-mon[17167]: pgmap v619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:28 smithi016 ceph-mon[22010]: pgmap v619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:28 smithi079 ceph-mon[23158]: pgmap v619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:30 smithi002 ceph-mon[17167]: pgmap v620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:30 smithi016 ceph-mon[22010]: pgmap v620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:30 smithi079 ceph-mon[23158]: pgmap v620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:32 smithi002 ceph-mon[17167]: pgmap v621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 414.857880 secs 2022-03-21T14:18:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:32 smithi016 ceph-mon[22010]: pgmap v621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 414.857880 secs 2022-03-21T14:18:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:32 smithi079 ceph-mon[23158]: pgmap v621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 414.857880 secs 2022-03-21T14:18:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:34 smithi002 ceph-mon[17167]: pgmap v622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:34 smithi016 ceph-mon[22010]: pgmap v622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:34 smithi079 ceph-mon[23158]: pgmap v622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:36 smithi002 ceph-mon[17167]: pgmap v623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:36 smithi016 ceph-mon[22010]: pgmap v623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:36 smithi079 ceph-mon[23158]: pgmap v623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 419.857995 secs 2022-03-21T14:18:37.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 419.857995 secs 2022-03-21T14:18:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 419.857995 secs 2022-03-21T14:18:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:38 smithi002 ceph-mon[17167]: pgmap v624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:38 smithi016 ceph-mon[22010]: pgmap v624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:38 smithi079 ceph-mon[23158]: pgmap v624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:40 smithi002 ceph-mon[17167]: pgmap v625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:40 smithi016 ceph-mon[22010]: pgmap v625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:40 smithi079 ceph-mon[23158]: pgmap v625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:42 smithi002 ceph-mon[17167]: pgmap v626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 424.858145 secs 2022-03-21T14:18:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:42 smithi016 ceph-mon[22010]: pgmap v626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 424.858145 secs 2022-03-21T14:18:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:42 smithi079 ceph-mon[23158]: pgmap v626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 424.858145 secs 2022-03-21T14:18:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:44 smithi002 ceph-mon[17167]: pgmap v627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:44 smithi016 ceph-mon[22010]: pgmap v627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:44 smithi079 ceph-mon[23158]: pgmap v627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:46 smithi002 ceph-mon[17167]: pgmap v628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:46 smithi016 ceph-mon[22010]: pgmap v628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:46 smithi079 ceph-mon[23158]: pgmap v628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:47.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 429.858245 secs 2022-03-21T14:18:47.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 429.858245 secs 2022-03-21T14:18:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 429.858245 secs 2022-03-21T14:18:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:48 smithi002 ceph-mon[17167]: pgmap v629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:48 smithi016 ceph-mon[22010]: pgmap v629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:48 smithi079 ceph-mon[23158]: pgmap v629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:50 smithi002 ceph-mon[17167]: pgmap v630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:50 smithi016 ceph-mon[22010]: pgmap v630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:50 smithi079 ceph-mon[23158]: pgmap v630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:52 smithi002 ceph-mon[17167]: pgmap v631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 434.858361 secs 2022-03-21T14:18:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:52 smithi016 ceph-mon[22010]: pgmap v631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 434.858361 secs 2022-03-21T14:18:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:52 smithi079 ceph-mon[23158]: pgmap v631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 434.858361 secs 2022-03-21T14:18:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:54 smithi002 ceph-mon[17167]: pgmap v632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:54 smithi016 ceph-mon[22010]: pgmap v632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:54 smithi079 ceph-mon[23158]: pgmap v632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:56 smithi002 ceph-mon[17167]: pgmap v633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:56.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:56 smithi016 ceph-mon[22010]: pgmap v633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:56 smithi079 ceph-mon[23158]: pgmap v633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:18:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 439.858474 secs 2022-03-21T14:18:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 439.858474 secs 2022-03-21T14:18:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 439.858474 secs 2022-03-21T14:18:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:18:58 smithi002 ceph-mon[17167]: pgmap v634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:18:58 smithi016 ceph-mon[22010]: pgmap v634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:18:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:18:58 smithi079 ceph-mon[23158]: pgmap v634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:00 smithi002 ceph-mon[17167]: pgmap v635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:00 smithi016 ceph-mon[22010]: pgmap v635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:00 smithi079 ceph-mon[23158]: pgmap v635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:02 smithi002 ceph-mon[17167]: pgmap v636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 444.858593 secs 2022-03-21T14:19:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:02 smithi016 ceph-mon[22010]: pgmap v636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 444.858593 secs 2022-03-21T14:19:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:02 smithi079 ceph-mon[23158]: pgmap v636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 444.858593 secs 2022-03-21T14:19:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:04 smithi016 ceph-mon[22010]: pgmap v637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:04 smithi079 ceph-mon[23158]: pgmap v637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:04 smithi002 ceph-mon[17167]: pgmap v637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:06 smithi016 ceph-mon[22010]: pgmap v638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:06 smithi079 ceph-mon[23158]: pgmap v638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:06 smithi002 ceph-mon[17167]: pgmap v638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 449.858711 secs 2022-03-21T14:19:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 449.858711 secs 2022-03-21T14:19:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 449.858711 secs 2022-03-21T14:19:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:08 smithi016 ceph-mon[22010]: pgmap v639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:08 smithi079 ceph-mon[23158]: pgmap v639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:08 smithi002 ceph-mon[17167]: pgmap v639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:10 smithi016 ceph-mon[22010]: pgmap v640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:10 smithi079 ceph-mon[23158]: pgmap v640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:10 smithi002 ceph-mon[17167]: pgmap v640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:12 smithi016 ceph-mon[22010]: pgmap v641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 454.858831 secs 2022-03-21T14:19:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:12 smithi079 ceph-mon[23158]: pgmap v641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 454.858831 secs 2022-03-21T14:19:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:12 smithi002 ceph-mon[17167]: pgmap v641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 454.858831 secs 2022-03-21T14:19:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:14 smithi016 ceph-mon[22010]: pgmap v642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:14 smithi079 ceph-mon[23158]: pgmap v642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:14 smithi002 ceph-mon[17167]: pgmap v642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:16 smithi016 ceph-mon[22010]: pgmap v643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:16 smithi079 ceph-mon[23158]: pgmap v643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:16 smithi002 ceph-mon[17167]: pgmap v643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 459.858944 secs 2022-03-21T14:19:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 459.858944 secs 2022-03-21T14:19:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 459.858944 secs 2022-03-21T14:19:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:18 smithi016 ceph-mon[22010]: pgmap v644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:18 smithi079 ceph-mon[23158]: pgmap v644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:18 smithi002 ceph-mon[17167]: pgmap v644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:20 smithi016 ceph-mon[22010]: pgmap v645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:20 smithi079 ceph-mon[23158]: pgmap v645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:20 smithi002 ceph-mon[17167]: pgmap v645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:19:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:19:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:19:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:19:21.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:19:21.793 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:19:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:22 smithi016 ceph-mon[22010]: pgmap v646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:19:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:19:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:19:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 464.859036 secs 2022-03-21T14:19:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:22 smithi079 ceph-mon[23158]: pgmap v646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:19:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:19:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:19:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 464.859036 secs 2022-03-21T14:19:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:22 smithi002 ceph-mon[17167]: pgmap v646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:19:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:19:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:19:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 464.859036 secs 2022-03-21T14:19:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:24 smithi016 ceph-mon[22010]: pgmap v647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:24.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:24 smithi079 ceph-mon[23158]: pgmap v647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:24.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:24.671 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:24 smithi002 ceph-mon[17167]: pgmap v647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:26.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:26 smithi002 ceph-mon[17167]: pgmap v648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:26 smithi079 ceph-mon[23158]: pgmap v648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:26 smithi016 ceph-mon[22010]: pgmap v648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:27 smithi002 ceph-mon[17167]: pgmap v649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 469.859131 secs 2022-03-21T14:19:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:27 smithi079 ceph-mon[23158]: pgmap v649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 469.859131 secs 2022-03-21T14:19:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:27 smithi016 ceph-mon[22010]: pgmap v649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 469.859131 secs 2022-03-21T14:19:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:19:30.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:29 smithi002 ceph-mon[17167]: pgmap v650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:29 smithi016 ceph-mon[22010]: pgmap v650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:30.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:29 smithi079 ceph-mon[23158]: pgmap v650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:31 smithi002 ceph-mon[17167]: pgmap v651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:32.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:31 smithi016 ceph-mon[22010]: pgmap v651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:31 smithi079 ceph-mon[23158]: pgmap v651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 474.859243 secs 2022-03-21T14:19:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 474.859243 secs 2022-03-21T14:19:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 474.859243 secs 2022-03-21T14:19:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:33 smithi002 ceph-mon[17167]: pgmap v652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:33 smithi016 ceph-mon[22010]: pgmap v652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:33 smithi079 ceph-mon[23158]: pgmap v652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:36.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:35 smithi002 ceph-mon[17167]: pgmap v653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:36.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:35 smithi016 ceph-mon[22010]: pgmap v653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:35 smithi079 ceph-mon[23158]: pgmap v653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:37 smithi002 ceph-mon[17167]: pgmap v654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 479.859361 secs 2022-03-21T14:19:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:37 smithi016 ceph-mon[22010]: pgmap v654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 479.859361 secs 2022-03-21T14:19:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:37 smithi079 ceph-mon[23158]: pgmap v654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 479.859361 secs 2022-03-21T14:19:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:39 smithi002 ceph-mon[17167]: pgmap v655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:40.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:39 smithi016 ceph-mon[22010]: pgmap v655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:39 smithi079 ceph-mon[23158]: pgmap v655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:41 smithi002 ceph-mon[17167]: pgmap v656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:41 smithi016 ceph-mon[22010]: pgmap v656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:41 smithi079 ceph-mon[23158]: pgmap v656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 484.859486 secs 2022-03-21T14:19:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 484.859486 secs 2022-03-21T14:19:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 484.859486 secs 2022-03-21T14:19:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:43 smithi002 ceph-mon[17167]: pgmap v657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:43 smithi016 ceph-mon[22010]: pgmap v657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:43 smithi079 ceph-mon[23158]: pgmap v657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:45 smithi002 ceph-mon[17167]: pgmap v658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:45 smithi016 ceph-mon[22010]: pgmap v658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:45 smithi079 ceph-mon[23158]: pgmap v658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:47 smithi002 ceph-mon[17167]: pgmap v659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 489.859577 secs 2022-03-21T14:19:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:47 smithi016 ceph-mon[22010]: pgmap v659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 489.859577 secs 2022-03-21T14:19:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:47 smithi079 ceph-mon[23158]: pgmap v659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 489.859577 secs 2022-03-21T14:19:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:49 smithi002 ceph-mon[17167]: pgmap v660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:50.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:49 smithi016 ceph-mon[22010]: pgmap v660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:49 smithi079 ceph-mon[23158]: pgmap v660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:51 smithi002 ceph-mon[17167]: pgmap v661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:52.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:51 smithi016 ceph-mon[22010]: pgmap v661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:51 smithi079 ceph-mon[23158]: pgmap v661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 494.859697 secs 2022-03-21T14:19:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 494.859697 secs 2022-03-21T14:19:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 494.859697 secs 2022-03-21T14:19:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:53 smithi002 ceph-mon[17167]: pgmap v662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:53 smithi016 ceph-mon[22010]: pgmap v662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:53 smithi079 ceph-mon[23158]: pgmap v662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:55 smithi002 ceph-mon[17167]: pgmap v663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:56.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:55 smithi016 ceph-mon[22010]: pgmap v663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:55 smithi079 ceph-mon[23158]: pgmap v663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:19:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:57 smithi002 ceph-mon[17167]: pgmap v664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 499.859820 secs 2022-03-21T14:19:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:57 smithi016 ceph-mon[22010]: pgmap v664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 499.859820 secs 2022-03-21T14:19:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:57 smithi079 ceph-mon[23158]: pgmap v664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:19:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 499.859820 secs 2022-03-21T14:20:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:19:59 smithi002 ceph-mon[17167]: pgmap v665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:19:59 smithi016 ceph-mon[22010]: pgmap v665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:00.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:19:59 smithi079 ceph-mon[23158]: pgmap v665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:00 smithi002 ceph-mon[17167]: Health detail: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:00 smithi002 ceph-mon[17167]: [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release 2022-03-21T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:00 smithi002 ceph-mon[17167]: mds.l(mds.0): Client smithi002:0 failing to respond to capability release client_id: 14964 2022-03-21T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:00 smithi002 ceph-mon[17167]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:00 smithi002 ceph-mon[17167]: mds.l(mds.0): 40 slow requests are blocked > 180 secs 2022-03-21T14:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:00 smithi016 ceph-mon[22010]: [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release 2022-03-21T14:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:00 smithi016 ceph-mon[22010]: mds.l(mds.0): Client smithi002:0 failing to respond to capability release client_id: 14964 2022-03-21T14:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:00 smithi016 ceph-mon[22010]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T14:20:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:00 smithi016 ceph-mon[22010]: mds.l(mds.0): 40 slow requests are blocked > 180 secs 2022-03-21T14:20:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:20:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:00 smithi079 ceph-mon[23158]: [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release 2022-03-21T14:20:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:00 smithi079 ceph-mon[23158]: mds.l(mds.0): Client smithi002:0 failing to respond to capability release client_id: 14964 2022-03-21T14:20:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:00 smithi079 ceph-mon[23158]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T14:20:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:00 smithi079 ceph-mon[23158]: mds.l(mds.0): 40 slow requests are blocked > 180 secs 2022-03-21T14:20:02.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:01 smithi002 ceph-mon[17167]: pgmap v666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:02.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:01 smithi016 ceph-mon[22010]: pgmap v666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:02.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:01 smithi079 ceph-mon[23158]: pgmap v666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 504.859919 secs 2022-03-21T14:20:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 504.859919 secs 2022-03-21T14:20:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 504.859919 secs 2022-03-21T14:20:04.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:03 smithi002 ceph-mon[17167]: pgmap v667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:03 smithi016 ceph-mon[22010]: pgmap v667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:04.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:03 smithi079 ceph-mon[23158]: pgmap v667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:06.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:05 smithi002 ceph-mon[17167]: pgmap v668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:06.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:05 smithi016 ceph-mon[22010]: pgmap v668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:05 smithi079 ceph-mon[23158]: pgmap v668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:07 smithi002 ceph-mon[17167]: pgmap v669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 509.860058 secs 2022-03-21T14:20:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:07 smithi016 ceph-mon[22010]: pgmap v669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 509.860058 secs 2022-03-21T14:20:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:07 smithi079 ceph-mon[23158]: pgmap v669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 509.860058 secs 2022-03-21T14:20:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:09 smithi002 ceph-mon[17167]: pgmap v670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:09 smithi016 ceph-mon[22010]: pgmap v670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:09 smithi079 ceph-mon[23158]: pgmap v670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:12.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:11 smithi002 ceph-mon[17167]: pgmap v671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:12.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:11 smithi016 ceph-mon[22010]: pgmap v671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:11 smithi079 ceph-mon[23158]: pgmap v671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 514.860170 secs 2022-03-21T14:20:13.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 514.860170 secs 2022-03-21T14:20:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 514.860170 secs 2022-03-21T14:20:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:13 smithi002 ceph-mon[17167]: pgmap v672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:13 smithi016 ceph-mon[22010]: pgmap v672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:13 smithi079 ceph-mon[23158]: pgmap v672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:15 smithi002 ceph-mon[17167]: pgmap v673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:16.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:15 smithi016 ceph-mon[22010]: pgmap v673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:15 smithi079 ceph-mon[23158]: pgmap v673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:17 smithi002 ceph-mon[17167]: pgmap v674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 519.860306 secs 2022-03-21T14:20:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:17 smithi016 ceph-mon[22010]: pgmap v674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 519.860306 secs 2022-03-21T14:20:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:17 smithi079 ceph-mon[23158]: pgmap v674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 519.860306 secs 2022-03-21T14:20:20.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:19 smithi002 ceph-mon[17167]: pgmap v675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:19 smithi016 ceph-mon[22010]: pgmap v675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:19 smithi079 ceph-mon[23158]: pgmap v675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:20:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:20:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:20:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:20:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:20:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:20:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:21 smithi002 ceph-mon[17167]: pgmap v676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:22.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:21 smithi016 ceph-mon[22010]: pgmap v676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:21 smithi079 ceph-mon[23158]: pgmap v676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 524.860421 secs 2022-03-21T14:20:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 524.860421 secs 2022-03-21T14:20:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 524.860421 secs 2022-03-21T14:20:24.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:23 smithi016 ceph-mon[22010]: pgmap v677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:23 smithi079 ceph-mon[23158]: pgmap v677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:23 smithi002 ceph-mon[17167]: pgmap v677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:26.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:25 smithi016 ceph-mon[22010]: pgmap v678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:25 smithi079 ceph-mon[23158]: pgmap v678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:25 smithi002 ceph-mon[17167]: pgmap v678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:27 smithi016 ceph-mon[22010]: pgmap v679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 529.860570 secs 2022-03-21T14:20:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:20:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:20:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:20:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:27 smithi079 ceph-mon[23158]: pgmap v679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 529.860570 secs 2022-03-21T14:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:20:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:27 smithi002 ceph-mon[17167]: pgmap v679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 529.860570 secs 2022-03-21T14:20:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:20:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:20:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:20:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:20:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:20:29.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:20:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:30 smithi016 ceph-mon[22010]: pgmap v680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:30 smithi079 ceph-mon[23158]: pgmap v680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:30 smithi002 ceph-mon[17167]: pgmap v680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:32 smithi016 ceph-mon[22010]: pgmap v681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 534.860694 secs 2022-03-21T14:20:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:32 smithi079 ceph-mon[23158]: pgmap v681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 534.860694 secs 2022-03-21T14:20:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:32 smithi002 ceph-mon[17167]: pgmap v681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 534.860694 secs 2022-03-21T14:20:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:34 smithi079 ceph-mon[23158]: pgmap v682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:34 smithi002 ceph-mon[17167]: pgmap v682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:34 smithi016 ceph-mon[22010]: pgmap v682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:36 smithi079 ceph-mon[23158]: pgmap v683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:36 smithi002 ceph-mon[17167]: pgmap v683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:36 smithi016 ceph-mon[22010]: pgmap v683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 539.860814 secs 2022-03-21T14:20:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 539.860814 secs 2022-03-21T14:20:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 539.860814 secs 2022-03-21T14:20:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:38 smithi079 ceph-mon[23158]: pgmap v684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:38 smithi002 ceph-mon[17167]: pgmap v684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:38 smithi016 ceph-mon[22010]: pgmap v684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:40 smithi079 ceph-mon[23158]: pgmap v685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:40 smithi002 ceph-mon[17167]: pgmap v685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:40 smithi016 ceph-mon[22010]: pgmap v685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:42 smithi079 ceph-mon[23158]: pgmap v686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 544.860932 secs 2022-03-21T14:20:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:42 smithi002 ceph-mon[17167]: pgmap v686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 544.860932 secs 2022-03-21T14:20:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:42 smithi016 ceph-mon[22010]: pgmap v686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 544.860932 secs 2022-03-21T14:20:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:44 smithi079 ceph-mon[23158]: pgmap v687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:44 smithi002 ceph-mon[17167]: pgmap v687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:44 smithi016 ceph-mon[22010]: pgmap v687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:46 smithi079 ceph-mon[23158]: pgmap v688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:46 smithi002 ceph-mon[17167]: pgmap v688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:46 smithi016 ceph-mon[22010]: pgmap v688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 549.861011 secs 2022-03-21T14:20:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 549.861011 secs 2022-03-21T14:20:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 549.861011 secs 2022-03-21T14:20:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:48 smithi079 ceph-mon[23158]: pgmap v689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:48 smithi002 ceph-mon[17167]: pgmap v689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:48 smithi016 ceph-mon[22010]: pgmap v689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:50 smithi079 ceph-mon[23158]: pgmap v690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:50 smithi002 ceph-mon[17167]: pgmap v690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:50 smithi016 ceph-mon[22010]: pgmap v690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:52 smithi079 ceph-mon[23158]: pgmap v691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 554.861139 secs 2022-03-21T14:20:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:52 smithi002 ceph-mon[17167]: pgmap v691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 554.861139 secs 2022-03-21T14:20:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:52 smithi016 ceph-mon[22010]: pgmap v691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 554.861139 secs 2022-03-21T14:20:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:54 smithi079 ceph-mon[23158]: pgmap v692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:54 smithi002 ceph-mon[17167]: pgmap v692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:54 smithi016 ceph-mon[22010]: pgmap v692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:56 smithi079 ceph-mon[23158]: pgmap v693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:56 smithi002 ceph-mon[17167]: pgmap v693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:56 smithi016 ceph-mon[22010]: pgmap v693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:20:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 559.861252 secs 2022-03-21T14:20:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 559.861252 secs 2022-03-21T14:20:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 559.861252 secs 2022-03-21T14:20:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:20:58 smithi002 ceph-mon[17167]: pgmap v694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:20:58 smithi016 ceph-mon[22010]: pgmap v694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:20:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:20:58 smithi079 ceph-mon[23158]: pgmap v694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:00 smithi002 ceph-mon[17167]: pgmap v695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:00 smithi016 ceph-mon[22010]: pgmap v695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:00 smithi079 ceph-mon[23158]: pgmap v695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:02 smithi002 ceph-mon[17167]: pgmap v696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 564.861391 secs 2022-03-21T14:21:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:02 smithi016 ceph-mon[22010]: pgmap v696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 564.861391 secs 2022-03-21T14:21:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:02 smithi079 ceph-mon[23158]: pgmap v696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 564.861391 secs 2022-03-21T14:21:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:04 smithi002 ceph-mon[17167]: pgmap v697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:04 smithi016 ceph-mon[22010]: pgmap v697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:04 smithi079 ceph-mon[23158]: pgmap v697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:06 smithi002 ceph-mon[17167]: pgmap v698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:06 smithi016 ceph-mon[22010]: pgmap v698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:06 smithi079 ceph-mon[23158]: pgmap v698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 569.861518 secs 2022-03-21T14:21:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 569.861518 secs 2022-03-21T14:21:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 569.861518 secs 2022-03-21T14:21:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:08 smithi002 ceph-mon[17167]: pgmap v699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:08 smithi016 ceph-mon[22010]: pgmap v699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:08 smithi079 ceph-mon[23158]: pgmap v699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:10 smithi002 ceph-mon[17167]: pgmap v700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:10 smithi016 ceph-mon[22010]: pgmap v700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:10 smithi079 ceph-mon[23158]: pgmap v700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:12 smithi002 ceph-mon[17167]: pgmap v701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 574.861630 secs 2022-03-21T14:21:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:12 smithi016 ceph-mon[22010]: pgmap v701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 574.861630 secs 2022-03-21T14:21:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:12 smithi079 ceph-mon[23158]: pgmap v701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 574.861630 secs 2022-03-21T14:21:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:14 smithi002 ceph-mon[17167]: pgmap v702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:14 smithi016 ceph-mon[22010]: pgmap v702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:14 smithi079 ceph-mon[23158]: pgmap v702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:16 smithi002 ceph-mon[17167]: pgmap v703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:16 smithi016 ceph-mon[22010]: pgmap v703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:16 smithi079 ceph-mon[23158]: pgmap v703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 579.861725 secs 2022-03-21T14:21:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 579.861725 secs 2022-03-21T14:21:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 579.861725 secs 2022-03-21T14:21:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:18 smithi002 ceph-mon[17167]: pgmap v704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:18 smithi016 ceph-mon[22010]: pgmap v704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:18 smithi079 ceph-mon[23158]: pgmap v704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:20 smithi002 ceph-mon[17167]: pgmap v705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:21:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:21:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:20 smithi016 ceph-mon[22010]: pgmap v705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:21:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:21:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:20 smithi079 ceph-mon[23158]: pgmap v705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:21:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:21:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:22 smithi002 ceph-mon[17167]: pgmap v706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 584.861844 secs 2022-03-21T14:21:22.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:22 smithi016 ceph-mon[22010]: pgmap v706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 584.861844 secs 2022-03-21T14:21:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:22 smithi079 ceph-mon[23158]: pgmap v706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 584.861844 secs 2022-03-21T14:21:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:23 smithi002 ceph-mon[17167]: pgmap v707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:23 smithi016 ceph-mon[22010]: pgmap v707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:23 smithi079 ceph-mon[23158]: pgmap v707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:26.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:25 smithi002 ceph-mon[17167]: pgmap v708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:25 smithi016 ceph-mon[22010]: pgmap v708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:25 smithi079 ceph-mon[23158]: pgmap v708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:27 smithi002 ceph-mon[17167]: pgmap v709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 589.861971 secs 2022-03-21T14:21:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:27 smithi016 ceph-mon[22010]: pgmap v709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 589.861971 secs 2022-03-21T14:21:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:27 smithi079 ceph-mon[23158]: pgmap v709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 589.861971 secs 2022-03-21T14:21:29.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:21:29.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:21:29.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:21:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:21:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:21:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:21:29.214 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:21:29.214 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:21:29.215 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:21:30.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:29 smithi002 ceph-mon[17167]: pgmap v710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:29 smithi016 ceph-mon[22010]: pgmap v710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:29 smithi079 ceph-mon[23158]: pgmap v710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:21:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:31 smithi016 ceph-mon[22010]: pgmap v711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:31 smithi079 ceph-mon[23158]: pgmap v711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:31 smithi002 ceph-mon[17167]: pgmap v711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 594.862084 secs 2022-03-21T14:21:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 594.862084 secs 2022-03-21T14:21:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 594.862084 secs 2022-03-21T14:21:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:33 smithi016 ceph-mon[22010]: pgmap v712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:33 smithi079 ceph-mon[23158]: pgmap v712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:34.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:33 smithi002 ceph-mon[17167]: pgmap v712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:35 smithi079 ceph-mon[23158]: pgmap v713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:35 smithi002 ceph-mon[17167]: pgmap v713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:36.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:35 smithi016 ceph-mon[22010]: pgmap v713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:37 smithi079 ceph-mon[23158]: pgmap v714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 599.862190 secs 2022-03-21T14:21:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:37 smithi002 ceph-mon[17167]: pgmap v714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 599.862190 secs 2022-03-21T14:21:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:37 smithi016 ceph-mon[22010]: pgmap v714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 599.862190 secs 2022-03-21T14:21:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:39 smithi079 ceph-mon[23158]: pgmap v715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:39 smithi002 ceph-mon[17167]: pgmap v715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:39 smithi016 ceph-mon[22010]: pgmap v715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:41 smithi079 ceph-mon[23158]: pgmap v716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:42.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:41 smithi002 ceph-mon[17167]: pgmap v716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:42.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:41 smithi016 ceph-mon[22010]: pgmap v716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 604.862290 secs 2022-03-21T14:21:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:42 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 604.747389 seconds ago 2022-03-21T14:21:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 604.862290 secs 2022-03-21T14:21:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:42 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 604.747389 seconds ago 2022-03-21T14:21:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 604.862290 secs 2022-03-21T14:21:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:42 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 604.747389 seconds ago 2022-03-21T14:21:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:43 smithi079 ceph-mon[23158]: pgmap v717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:43 smithi002 ceph-mon[17167]: pgmap v717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:43 smithi016 ceph-mon[22010]: pgmap v717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:45 smithi079 ceph-mon[23158]: pgmap v718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:45 smithi002 ceph-mon[17167]: pgmap v718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:45 smithi016 ceph-mon[22010]: pgmap v718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:47 smithi079 ceph-mon[23158]: pgmap v719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 609.862380 secs 2022-03-21T14:21:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:47 smithi002 ceph-mon[17167]: pgmap v719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 609.862380 secs 2022-03-21T14:21:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:47 smithi016 ceph-mon[22010]: pgmap v719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 609.862380 secs 2022-03-21T14:21:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:49 smithi079 ceph-mon[23158]: pgmap v720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:49 smithi002 ceph-mon[17167]: pgmap v720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:49 smithi016 ceph-mon[22010]: pgmap v720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:51 smithi079 ceph-mon[23158]: pgmap v721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:51 smithi002 ceph-mon[17167]: pgmap v721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:51 smithi016 ceph-mon[22010]: pgmap v721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 614.862503 secs 2022-03-21T14:21:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 614.862503 secs 2022-03-21T14:21:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 614.862503 secs 2022-03-21T14:21:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:53 smithi079 ceph-mon[23158]: pgmap v722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:53 smithi002 ceph-mon[17167]: pgmap v722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:53 smithi016 ceph-mon[22010]: pgmap v722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:55 smithi079 ceph-mon[23158]: pgmap v723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:55 smithi002 ceph-mon[17167]: pgmap v723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:56.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:55 smithi016 ceph-mon[22010]: pgmap v723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:21:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:57 smithi079 ceph-mon[23158]: pgmap v724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 619.862622 secs 2022-03-21T14:21:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:57 smithi002 ceph-mon[17167]: pgmap v724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 619.862622 secs 2022-03-21T14:21:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:57 smithi016 ceph-mon[22010]: pgmap v724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:21:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 619.862622 secs 2022-03-21T14:22:00.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:21:59 smithi079 ceph-mon[23158]: pgmap v725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:21:59 smithi002 ceph-mon[17167]: pgmap v725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:00.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:21:59 smithi016 ceph-mon[22010]: pgmap v725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:01 smithi002 ceph-mon[17167]: pgmap v726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:02.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:01 smithi016 ceph-mon[22010]: pgmap v726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:01 smithi079 ceph-mon[23158]: pgmap v726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 624.862739 secs 2022-03-21T14:22:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 624.862739 secs 2022-03-21T14:22:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 624.862739 secs 2022-03-21T14:22:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:03 smithi002 ceph-mon[17167]: pgmap v727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:03 smithi016 ceph-mon[22010]: pgmap v727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:03 smithi079 ceph-mon[23158]: pgmap v727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:05 smithi002 ceph-mon[17167]: pgmap v728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:05 smithi016 ceph-mon[22010]: pgmap v728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:05 smithi079 ceph-mon[23158]: pgmap v728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:07 smithi002 ceph-mon[17167]: pgmap v729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 629.862864 secs 2022-03-21T14:22:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:07 smithi016 ceph-mon[22010]: pgmap v729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 629.862864 secs 2022-03-21T14:22:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:07 smithi079 ceph-mon[23158]: pgmap v729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 629.862864 secs 2022-03-21T14:22:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:09 smithi002 ceph-mon[17167]: pgmap v730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:10.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:09 smithi016 ceph-mon[22010]: pgmap v730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:09 smithi079 ceph-mon[23158]: pgmap v730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:11 smithi002 ceph-mon[17167]: pgmap v731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:11 smithi016 ceph-mon[22010]: pgmap v731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:11 smithi079 ceph-mon[23158]: pgmap v731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 634.862985 secs 2022-03-21T14:22:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 634.862985 secs 2022-03-21T14:22:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 634.862985 secs 2022-03-21T14:22:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:14 smithi002 ceph-mon[17167]: pgmap v732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:14 smithi016 ceph-mon[22010]: pgmap v732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:14 smithi079 ceph-mon[23158]: pgmap v732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:16 smithi002 ceph-mon[17167]: pgmap v733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:16 smithi016 ceph-mon[22010]: pgmap v733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:16 smithi079 ceph-mon[23158]: pgmap v733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:18 smithi002 ceph-mon[17167]: pgmap v734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 639.863081 secs 2022-03-21T14:22:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:18 smithi016 ceph-mon[22010]: pgmap v734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 639.863081 secs 2022-03-21T14:22:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:18 smithi079 ceph-mon[23158]: pgmap v734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 639.863081 secs 2022-03-21T14:22:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:20 smithi002 ceph-mon[17167]: pgmap v735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:20 smithi016 ceph-mon[22010]: pgmap v735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:20 smithi079 ceph-mon[23158]: pgmap v735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:22:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:22:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:22:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:22:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:22:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:22:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:22 smithi002 ceph-mon[17167]: pgmap v736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:22 smithi016 ceph-mon[22010]: pgmap v736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:22 smithi079 ceph-mon[23158]: pgmap v736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 644.863205 secs 2022-03-21T14:22:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 644.863205 secs 2022-03-21T14:22:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 644.863205 secs 2022-03-21T14:22:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:24 smithi002 ceph-mon[17167]: pgmap v737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:24 smithi016 ceph-mon[22010]: pgmap v737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:24 smithi079 ceph-mon[23158]: pgmap v737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:26 smithi002 ceph-mon[17167]: pgmap v738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:26 smithi016 ceph-mon[22010]: pgmap v738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:26 smithi079 ceph-mon[23158]: pgmap v738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:28 smithi002 ceph-mon[17167]: pgmap v739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 649.863292 secs 2022-03-21T14:22:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:28 smithi016 ceph-mon[22010]: pgmap v739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 649.863292 secs 2022-03-21T14:22:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:28 smithi079 ceph-mon[23158]: pgmap v739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 649.863292 secs 2022-03-21T14:22:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:30 smithi079 ceph-mon[23158]: pgmap v740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:22:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:22:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:22:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:30 smithi002 ceph-mon[17167]: pgmap v740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:22:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:22:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:22:30.368 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:30 smithi016 ceph-mon[22010]: pgmap v740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:30.368 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:22:30.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:22:30.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:22:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:31 smithi002 ceph-mon[17167]: pgmap v741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:22:32.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:31 smithi016 ceph-mon[22010]: pgmap v741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:22:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:31 smithi079 ceph-mon[23158]: pgmap v741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:22:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 654.863413 secs 2022-03-21T14:22:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 654.863413 secs 2022-03-21T14:22:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 654.863413 secs 2022-03-21T14:22:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:33 smithi002 ceph-mon[17167]: pgmap v742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:33 smithi016 ceph-mon[22010]: pgmap v742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:33 smithi079 ceph-mon[23158]: pgmap v742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:36.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:35 smithi002 ceph-mon[17167]: pgmap v743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:36.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:35 smithi016 ceph-mon[22010]: pgmap v743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:36.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:35 smithi079 ceph-mon[23158]: pgmap v743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:37 smithi002 ceph-mon[17167]: pgmap v744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 659.863526 secs 2022-03-21T14:22:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:37 smithi016 ceph-mon[22010]: pgmap v744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 659.863526 secs 2022-03-21T14:22:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:37 smithi079 ceph-mon[23158]: pgmap v744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 659.863526 secs 2022-03-21T14:22:40.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:39 smithi002 ceph-mon[17167]: pgmap v745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:40.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:39 smithi016 ceph-mon[22010]: pgmap v745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:39 smithi079 ceph-mon[23158]: pgmap v745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:42.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:41 smithi002 ceph-mon[17167]: pgmap v746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:42.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:41 smithi016 ceph-mon[22010]: pgmap v746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:41 smithi079 ceph-mon[23158]: pgmap v746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 664.863653 secs 2022-03-21T14:22:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 664.863653 secs 2022-03-21T14:22:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 664.863653 secs 2022-03-21T14:22:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:43 smithi002 ceph-mon[17167]: pgmap v747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:43 smithi016 ceph-mon[22010]: pgmap v747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:43 smithi079 ceph-mon[23158]: pgmap v747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:46.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:45 smithi002 ceph-mon[17167]: pgmap v748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:46.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:45 smithi016 ceph-mon[22010]: pgmap v748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:45 smithi079 ceph-mon[23158]: pgmap v748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:47 smithi002 ceph-mon[17167]: pgmap v749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 669.863771 secs 2022-03-21T14:22:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:47 smithi016 ceph-mon[22010]: pgmap v749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 669.863771 secs 2022-03-21T14:22:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:47 smithi079 ceph-mon[23158]: pgmap v749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 669.863771 secs 2022-03-21T14:22:50.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:49 smithi002 ceph-mon[17167]: pgmap v750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:50.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:49 smithi016 ceph-mon[22010]: pgmap v750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:49 smithi079 ceph-mon[23158]: pgmap v750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:51 smithi002 ceph-mon[17167]: pgmap v751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:51 smithi016 ceph-mon[22010]: pgmap v751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:51 smithi079 ceph-mon[23158]: pgmap v751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 674.863888 secs 2022-03-21T14:22:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 674.863888 secs 2022-03-21T14:22:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 674.863888 secs 2022-03-21T14:22:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:53 smithi002 ceph-mon[17167]: pgmap v752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:53 smithi016 ceph-mon[22010]: pgmap v752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:53 smithi079 ceph-mon[23158]: pgmap v752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:56.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:55 smithi002 ceph-mon[17167]: pgmap v753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:56.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:55 smithi016 ceph-mon[22010]: pgmap v753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:55 smithi079 ceph-mon[23158]: pgmap v753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:22:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:57 smithi002 ceph-mon[17167]: pgmap v754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 679.864008 secs 2022-03-21T14:22:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:57 smithi016 ceph-mon[22010]: pgmap v754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 679.864008 secs 2022-03-21T14:22:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:57 smithi079 ceph-mon[23158]: pgmap v754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:22:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 679.864008 secs 2022-03-21T14:23:00.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:22:59 smithi002 ceph-mon[17167]: pgmap v755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:00.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:22:59 smithi016 ceph-mon[22010]: pgmap v755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:00.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:22:59 smithi079 ceph-mon[23158]: pgmap v755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:01 smithi016 ceph-mon[22010]: pgmap v756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:02.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:01 smithi079 ceph-mon[23158]: pgmap v756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:01 smithi002 ceph-mon[17167]: pgmap v756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 684.864127 secs 2022-03-21T14:23:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 684.864127 secs 2022-03-21T14:23:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 684.864127 secs 2022-03-21T14:23:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:03 smithi016 ceph-mon[22010]: pgmap v757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:04.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:03 smithi079 ceph-mon[23158]: pgmap v757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:03 smithi002 ceph-mon[17167]: pgmap v757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:05 smithi016 ceph-mon[22010]: pgmap v758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:05 smithi079 ceph-mon[23158]: pgmap v758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:05 smithi002 ceph-mon[17167]: pgmap v758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:07 smithi016 ceph-mon[22010]: pgmap v759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 689.864262 secs 2022-03-21T14:23:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:07 smithi079 ceph-mon[23158]: pgmap v759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 689.864262 secs 2022-03-21T14:23:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:07 smithi002 ceph-mon[17167]: pgmap v759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 689.864262 secs 2022-03-21T14:23:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:09 smithi016 ceph-mon[22010]: pgmap v760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:10.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:09 smithi079 ceph-mon[23158]: pgmap v760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:09 smithi002 ceph-mon[17167]: pgmap v760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:11 smithi016 ceph-mon[22010]: pgmap v761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:12.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:11 smithi079 ceph-mon[23158]: pgmap v761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:11 smithi002 ceph-mon[17167]: pgmap v761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:13.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 694.864399 secs 2022-03-21T14:23:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 694.864399 secs 2022-03-21T14:23:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 694.864399 secs 2022-03-21T14:23:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:13 smithi016 ceph-mon[22010]: pgmap v762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:13 smithi079 ceph-mon[23158]: pgmap v762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:13 smithi002 ceph-mon[17167]: pgmap v762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:15 smithi016 ceph-mon[22010]: pgmap v763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:15 smithi079 ceph-mon[23158]: pgmap v763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:15 smithi002 ceph-mon[17167]: pgmap v763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:17 smithi016 ceph-mon[22010]: pgmap v764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 699.864517 secs 2022-03-21T14:23:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:17 smithi079 ceph-mon[23158]: pgmap v764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 699.864517 secs 2022-03-21T14:23:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:17 smithi002 ceph-mon[17167]: pgmap v764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 699.864517 secs 2022-03-21T14:23:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:19 smithi016 ceph-mon[22010]: pgmap v765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:19 smithi079 ceph-mon[23158]: pgmap v765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:19 smithi002 ceph-mon[17167]: pgmap v765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:23:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:23:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:23:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:23:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:23:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:23:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:21 smithi016 ceph-mon[22010]: pgmap v766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:21 smithi079 ceph-mon[23158]: pgmap v766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:21 smithi002 ceph-mon[17167]: pgmap v766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 704.864561 secs 2022-03-21T14:23:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 704.864561 secs 2022-03-21T14:23:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 704.864561 secs 2022-03-21T14:23:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:23 smithi079 ceph-mon[23158]: pgmap v767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:23 smithi002 ceph-mon[17167]: pgmap v767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:23 smithi016 ceph-mon[22010]: pgmap v767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:25 smithi079 ceph-mon[23158]: pgmap v768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:25 smithi002 ceph-mon[17167]: pgmap v768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:25 smithi016 ceph-mon[22010]: pgmap v768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:27 smithi079 ceph-mon[23158]: pgmap v769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 709.864615 secs 2022-03-21T14:23:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:27 smithi002 ceph-mon[17167]: pgmap v769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 709.864615 secs 2022-03-21T14:23:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:27 smithi016 ceph-mon[22010]: pgmap v769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 709.864615 secs 2022-03-21T14:23:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:29 smithi079 ceph-mon[23158]: pgmap v770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:29 smithi002 ceph-mon[17167]: pgmap v770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:29 smithi016 ceph-mon[22010]: pgmap v770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:23:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:23:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:23:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:23:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:23:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:23:31.200 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:23:31.200 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:23:31.201 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:23:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:31 smithi079 ceph-mon[23158]: pgmap v771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:23:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:31 smithi002 ceph-mon[17167]: pgmap v771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:23:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:31 smithi016 ceph-mon[22010]: pgmap v771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:23:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 714.864697 secs 2022-03-21T14:23:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 714.864697 secs 2022-03-21T14:23:33.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 714.864697 secs 2022-03-21T14:23:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:33 smithi079 ceph-mon[23158]: pgmap v772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:34.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:33 smithi002 ceph-mon[17167]: pgmap v772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:34.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:33 smithi016 ceph-mon[22010]: pgmap v772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:36.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:35 smithi079 ceph-mon[23158]: pgmap v773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:36.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:35 smithi002 ceph-mon[17167]: pgmap v773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:35 smithi016 ceph-mon[22010]: pgmap v773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:37 smithi079 ceph-mon[23158]: pgmap v774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 719.864792 secs 2022-03-21T14:23:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:37 smithi002 ceph-mon[17167]: pgmap v774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 719.864792 secs 2022-03-21T14:23:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:37 smithi016 ceph-mon[22010]: pgmap v774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 719.864792 secs 2022-03-21T14:23:40.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:39 smithi079 ceph-mon[23158]: pgmap v775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:39 smithi002 ceph-mon[17167]: pgmap v775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:39 smithi016 ceph-mon[22010]: pgmap v775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:42.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:41 smithi079 ceph-mon[23158]: pgmap v776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:42.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:41 smithi002 ceph-mon[17167]: pgmap v776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:42.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:41 smithi016 ceph-mon[22010]: pgmap v776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 724.864879 secs 2022-03-21T14:23:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: slow request 724.760108 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: slow request 724.548924 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: slow request 724.861518 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: slow request 724.812380 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:42 smithi079 ceph-mon[23158]: slow request 724.787120 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 724.864879 secs 2022-03-21T14:23:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: slow request 724.760108 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: slow request 724.548924 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: slow request 724.861518 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: slow request 724.812380 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:42 smithi002 ceph-mon[17167]: slow request 724.787120 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 724.864879 secs 2022-03-21T14:23:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: slow request 724.760108 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: slow request 724.548924 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: slow request 724.861518 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: slow request 724.812380 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:42 smithi016 ceph-mon[22010]: slow request 724.787120 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:43 smithi079 ceph-mon[23158]: pgmap v777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:43 smithi002 ceph-mon[17167]: pgmap v777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:44.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:43 smithi016 ceph-mon[22010]: pgmap v777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:45 smithi002 ceph-mon[17167]: pgmap v778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:45 smithi016 ceph-mon[22010]: pgmap v778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:45 smithi079 ceph-mon[23158]: pgmap v778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: pgmap v779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 729.864979 secs 2022-03-21T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: slow request 729.756941 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: slow request 729.836273 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: slow request 729.753982 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: slow request 729.858178 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:47 smithi002 ceph-mon[17167]: slow request 729.809203 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: pgmap v779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 729.864979 secs 2022-03-21T14:23:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: slow request 729.756941 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: slow request 729.836273 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: slow request 729.753982 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: slow request 729.858178 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:47 smithi016 ceph-mon[22010]: slow request 729.809203 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: pgmap v779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 729.864979 secs 2022-03-21T14:23:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: slow request 729.756941 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: slow request 729.836273 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: slow request 729.753982 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: slow request 729.858178 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:47 smithi079 ceph-mon[23158]: slow request 729.809203 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:49 smithi002 ceph-mon[17167]: pgmap v780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:49 smithi016 ceph-mon[22010]: pgmap v780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:49 smithi079 ceph-mon[23158]: pgmap v780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:51 smithi002 ceph-mon[17167]: pgmap v781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:52.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:51 smithi016 ceph-mon[22010]: pgmap v781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:51 smithi079 ceph-mon[23158]: pgmap v781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 734.865100 secs 2022-03-21T14:23:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: slow request 734.751085 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: slow request 734.833669 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: slow request 734.783244 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: slow request 734.748339 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:52 smithi002 ceph-mon[17167]: slow request 734.855122 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 734.865100 secs 2022-03-21T14:23:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: slow request 734.751085 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: slow request 734.833669 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: slow request 734.783244 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: slow request 734.748339 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:52 smithi016 ceph-mon[22010]: slow request 734.855122 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 734.865100 secs 2022-03-21T14:23:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: slow request 734.751085 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: slow request 734.833669 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: slow request 734.783244 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: slow request 734.748339 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:52 smithi079 ceph-mon[23158]: slow request 734.855122 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:53 smithi002 ceph-mon[17167]: pgmap v782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:54.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:53 smithi016 ceph-mon[22010]: pgmap v782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:53 smithi079 ceph-mon[23158]: pgmap v782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:56 smithi002 ceph-mon[17167]: pgmap v783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:56 smithi016 ceph-mon[22010]: pgmap v783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:56 smithi079 ceph-mon[23158]: pgmap v783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:23:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: pgmap v784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 739.865221 secs 2022-03-21T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: slow request 739.830979 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: slow request 739.806126 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: slow request 739.779214 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: slow request 739.852326 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:23:58 smithi002 ceph-mon[17167]: slow request 739.827912 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: pgmap v784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 739.865221 secs 2022-03-21T14:23:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: slow request 739.830979 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: slow request 739.806126 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: slow request 739.779214 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: slow request 739.852326 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:23:58 smithi016 ceph-mon[22010]: slow request 739.827912 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: pgmap v784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:23:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 739.865221 secs 2022-03-21T14:23:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: slow request 739.830979 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: slow request 739.806126 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: slow request 739.779214 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: slow request 739.852326 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:23:58 smithi079 ceph-mon[23158]: slow request 739.827912 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:00 smithi002 ceph-mon[17167]: pgmap v785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:00 smithi016 ceph-mon[22010]: pgmap v785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:00 smithi079 ceph-mon[23158]: pgmap v785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:02 smithi002 ceph-mon[17167]: pgmap v786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:02 smithi016 ceph-mon[22010]: pgmap v786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:02 smithi079 ceph-mon[23158]: pgmap v786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 744.865362 secs 2022-03-21T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: slow request 744.803462 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: slow request 744.776617 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: slow request 744.800500 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: slow request 744.849080 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:03 smithi002 ceph-mon[17167]: slow request 744.825377 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 744.865362 secs 2022-03-21T14:24:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: slow request 744.803462 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: slow request 744.776617 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: slow request 744.800500 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: slow request 744.849080 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:03 smithi016 ceph-mon[22010]: slow request 744.825377 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 744.865362 secs 2022-03-21T14:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: slow request 744.803462 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: slow request 744.776617 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: slow request 744.800500 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: slow request 744.849080 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:03 smithi079 ceph-mon[23158]: slow request 744.825377 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:04 smithi002 ceph-mon[17167]: pgmap v787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:04 smithi016 ceph-mon[22010]: pgmap v787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:04 smithi079 ceph-mon[23158]: pgmap v787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:06 smithi002 ceph-mon[17167]: pgmap v788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:06 smithi016 ceph-mon[22010]: pgmap v788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:06 smithi079 ceph-mon[23158]: pgmap v788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: pgmap v789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 749.865504 secs 2022-03-21T14:24:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: slow request 749.773567 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: slow request 749.822672 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: slow request 749.797567 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: slow request 749.846162 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:08 smithi002 ceph-mon[17167]: slow request 749.769805 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: pgmap v789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 749.865504 secs 2022-03-21T14:24:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: slow request 749.773567 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: slow request 749.822672 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: slow request 749.797567 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: slow request 749.846162 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:08 smithi016 ceph-mon[22010]: slow request 749.769805 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: pgmap v789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 749.865504 secs 2022-03-21T14:24:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: slow request 749.773567 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: slow request 749.822672 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: slow request 749.797567 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: slow request 749.846162 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:08 smithi079 ceph-mon[23158]: slow request 749.769805 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:10 smithi002 ceph-mon[17167]: pgmap v790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:10.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:10 smithi016 ceph-mon[22010]: pgmap v790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:10 smithi079 ceph-mon[23158]: pgmap v790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:12 smithi002 ceph-mon[17167]: pgmap v791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:12 smithi016 ceph-mon[22010]: pgmap v791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:12 smithi079 ceph-mon[23158]: pgmap v791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 754.865622 secs 2022-03-21T14:24:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: slow request 754.749516 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:24:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: slow request 754.766920 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: slow request 754.842821 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: slow request 754.819420 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:13 smithi002 ceph-mon[17167]: slow request 754.794362 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 754.865622 secs 2022-03-21T14:24:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: slow request 754.749516 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:24:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: slow request 754.766920 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: slow request 754.842821 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: slow request 754.819420 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:13 smithi016 ceph-mon[22010]: slow request 754.794362 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 754.865622 secs 2022-03-21T14:24:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: slow request 754.749516 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:24:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: slow request 754.766920 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: slow request 754.842821 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: slow request 754.819420 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:13 smithi079 ceph-mon[23158]: slow request 754.794362 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:14 smithi002 ceph-mon[17167]: pgmap v792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:14 smithi016 ceph-mon[22010]: pgmap v792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:14 smithi079 ceph-mon[23158]: pgmap v792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:16 smithi002 ceph-mon[17167]: pgmap v793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:16 smithi016 ceph-mon[22010]: pgmap v793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:16 smithi079 ceph-mon[23158]: pgmap v793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: pgmap v794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 759.865741 secs 2022-03-21T14:24:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: slow request 759.763717 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: slow request 759.865740 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: slow request 759.816518 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: slow request 759.791567 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:18 smithi002 ceph-mon[17167]: slow request 759.840015 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: pgmap v794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 759.865741 secs 2022-03-21T14:24:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: slow request 759.763717 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: slow request 759.865740 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: slow request 759.816518 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: slow request 759.791567 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:18 smithi016 ceph-mon[22010]: slow request 759.840015 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: pgmap v794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 759.865741 secs 2022-03-21T14:24:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: slow request 759.763717 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: slow request 759.865740 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: slow request 759.816518 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: slow request 759.791567 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:18 smithi079 ceph-mon[23158]: slow request 759.840015 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:24:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:20 smithi002 ceph-mon[17167]: pgmap v795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:20 smithi016 ceph-mon[22010]: pgmap v795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:20 smithi079 ceph-mon[23158]: pgmap v795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:24:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:24:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:24:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:24:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:24:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:24:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:22 smithi002 ceph-mon[17167]: pgmap v796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:22 smithi016 ceph-mon[22010]: pgmap v796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:22 smithi079 ceph-mon[23158]: pgmap v796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 764.865867 secs 2022-03-21T14:24:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 764.865867 secs 2022-03-21T14:24:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 764.865867 secs 2022-03-21T14:24:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:24 smithi002 ceph-mon[17167]: pgmap v797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:24 smithi016 ceph-mon[22010]: pgmap v797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:24 smithi079 ceph-mon[23158]: pgmap v797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:26 smithi002 ceph-mon[17167]: pgmap v798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:26 smithi016 ceph-mon[22010]: pgmap v798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:26 smithi079 ceph-mon[23158]: pgmap v798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:28 smithi002 ceph-mon[17167]: pgmap v799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 769.865979 secs 2022-03-21T14:24:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:28 smithi016 ceph-mon[22010]: pgmap v799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 769.865979 secs 2022-03-21T14:24:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:28 smithi079 ceph-mon[23158]: pgmap v799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 769.865979 secs 2022-03-21T14:24:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:30 smithi002 ceph-mon[17167]: pgmap v800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:30 smithi016 ceph-mon[22010]: pgmap v800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:30 smithi079 ceph-mon[23158]: pgmap v800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:32 smithi002 ceph-mon[17167]: pgmap v801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:24:32.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:24:32.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:24:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:32 smithi016 ceph-mon[22010]: pgmap v801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:24:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:24:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:24:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:32 smithi079 ceph-mon[23158]: pgmap v801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:24:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:24:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:24:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 774.866079 secs 2022-03-21T14:24:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:24:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 774.866079 secs 2022-03-21T14:24:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:24:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 774.866079 secs 2022-03-21T14:24:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:24:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:34 smithi002 ceph-mon[17167]: pgmap v802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:34 smithi016 ceph-mon[22010]: pgmap v802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:34 smithi079 ceph-mon[23158]: pgmap v802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:36 smithi002 ceph-mon[17167]: pgmap v803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:36 smithi016 ceph-mon[22010]: pgmap v803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:36 smithi079 ceph-mon[23158]: pgmap v803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 779.866198 secs 2022-03-21T14:24:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 779.866198 secs 2022-03-21T14:24:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 779.866198 secs 2022-03-21T14:24:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:38 smithi002 ceph-mon[17167]: pgmap v804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:38 smithi016 ceph-mon[22010]: pgmap v804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:38 smithi079 ceph-mon[23158]: pgmap v804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:40 smithi002 ceph-mon[17167]: pgmap v805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:40 smithi016 ceph-mon[22010]: pgmap v805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:40 smithi079 ceph-mon[23158]: pgmap v805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:42 smithi002 ceph-mon[17167]: pgmap v806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 784.866350 secs 2022-03-21T14:24:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:42 smithi016 ceph-mon[22010]: pgmap v806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 784.866350 secs 2022-03-21T14:24:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:42 smithi079 ceph-mon[23158]: pgmap v806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 784.866350 secs 2022-03-21T14:24:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:44 smithi002 ceph-mon[17167]: pgmap v807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:44 smithi016 ceph-mon[22010]: pgmap v807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:44 smithi079 ceph-mon[23158]: pgmap v807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:46 smithi002 ceph-mon[17167]: pgmap v808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:46 smithi016 ceph-mon[22010]: pgmap v808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:46 smithi079 ceph-mon[23158]: pgmap v808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 789.866498 secs 2022-03-21T14:24:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 789.866498 secs 2022-03-21T14:24:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 789.866498 secs 2022-03-21T14:24:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:48 smithi002 ceph-mon[17167]: pgmap v809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:48 smithi016 ceph-mon[22010]: pgmap v809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:48 smithi079 ceph-mon[23158]: pgmap v809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:50 smithi002 ceph-mon[17167]: pgmap v810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:50 smithi016 ceph-mon[22010]: pgmap v810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:50 smithi079 ceph-mon[23158]: pgmap v810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:52 smithi002 ceph-mon[17167]: pgmap v811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 794.866633 secs 2022-03-21T14:24:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:52 smithi016 ceph-mon[22010]: pgmap v811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 794.866633 secs 2022-03-21T14:24:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:52 smithi079 ceph-mon[23158]: pgmap v811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 794.866633 secs 2022-03-21T14:24:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:54 smithi002 ceph-mon[17167]: pgmap v812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:54 smithi016 ceph-mon[22010]: pgmap v812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:54 smithi079 ceph-mon[23158]: pgmap v812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:56 smithi002 ceph-mon[17167]: pgmap v813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:56 smithi016 ceph-mon[22010]: pgmap v813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:56 smithi079 ceph-mon[23158]: pgmap v813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:24:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 799.866761 secs 2022-03-21T14:24:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 799.866761 secs 2022-03-21T14:24:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 799.866761 secs 2022-03-21T14:24:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:24:58 smithi002 ceph-mon[17167]: pgmap v814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:24:58 smithi016 ceph-mon[22010]: pgmap v814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:24:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:24:58 smithi079 ceph-mon[23158]: pgmap v814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:00 smithi002 ceph-mon[17167]: pgmap v815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:00 smithi016 ceph-mon[22010]: pgmap v815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:00 smithi079 ceph-mon[23158]: pgmap v815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:02 smithi002 ceph-mon[17167]: pgmap v816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 804.866906 secs 2022-03-21T14:25:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:02 smithi016 ceph-mon[22010]: pgmap v816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 804.866906 secs 2022-03-21T14:25:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:02 smithi079 ceph-mon[23158]: pgmap v816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 804.866906 secs 2022-03-21T14:25:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:04 smithi002 ceph-mon[17167]: pgmap v817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:04 smithi016 ceph-mon[22010]: pgmap v817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:04 smithi079 ceph-mon[23158]: pgmap v817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:06 smithi002 ceph-mon[17167]: pgmap v818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:06 smithi016 ceph-mon[22010]: pgmap v818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:06 smithi079 ceph-mon[23158]: pgmap v818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 809.867027 secs 2022-03-21T14:25:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 809.867027 secs 2022-03-21T14:25:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 809.867027 secs 2022-03-21T14:25:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:08 smithi002 ceph-mon[17167]: pgmap v819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:08 smithi016 ceph-mon[22010]: pgmap v819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:08 smithi079 ceph-mon[23158]: pgmap v819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:10 smithi002 ceph-mon[17167]: pgmap v820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:10 smithi016 ceph-mon[22010]: pgmap v820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:10 smithi079 ceph-mon[23158]: pgmap v820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:12 smithi002 ceph-mon[17167]: pgmap v821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 814.867145 secs 2022-03-21T14:25:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:12 smithi016 ceph-mon[22010]: pgmap v821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 814.867145 secs 2022-03-21T14:25:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:12 smithi079 ceph-mon[23158]: pgmap v821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 814.867145 secs 2022-03-21T14:25:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:13 smithi002 ceph-mon[17167]: pgmap v822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:13 smithi016 ceph-mon[22010]: pgmap v822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:13 smithi079 ceph-mon[23158]: pgmap v822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:15 smithi016 ceph-mon[22010]: pgmap v823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:15 smithi079 ceph-mon[23158]: pgmap v823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:15 smithi002 ceph-mon[17167]: pgmap v823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:17 smithi016 ceph-mon[22010]: pgmap v824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 819.867185 secs 2022-03-21T14:25:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:17 smithi079 ceph-mon[23158]: pgmap v824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 819.867185 secs 2022-03-21T14:25:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:17 smithi002 ceph-mon[17167]: pgmap v824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 819.867185 secs 2022-03-21T14:25:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:19 smithi016 ceph-mon[22010]: pgmap v825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:19 smithi079 ceph-mon[23158]: pgmap v825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:19 smithi002 ceph-mon[17167]: pgmap v825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:25:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:25:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:25:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:25:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:25:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:25:22.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:21 smithi016 ceph-mon[22010]: pgmap v826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:21 smithi079 ceph-mon[23158]: pgmap v826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:21 smithi002 ceph-mon[17167]: pgmap v826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 824.867304 secs 2022-03-21T14:25:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 824.867304 secs 2022-03-21T14:25:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 824.867304 secs 2022-03-21T14:25:24.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:23 smithi016 ceph-mon[22010]: pgmap v827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:23 smithi079 ceph-mon[23158]: pgmap v827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:23 smithi002 ceph-mon[17167]: pgmap v827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:26.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:25 smithi016 ceph-mon[22010]: pgmap v828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:26.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:25 smithi079 ceph-mon[23158]: pgmap v828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:25 smithi002 ceph-mon[17167]: pgmap v828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:27 smithi016 ceph-mon[22010]: pgmap v829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 829.867434 secs 2022-03-21T14:25:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:27 smithi079 ceph-mon[23158]: pgmap v829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 829.867434 secs 2022-03-21T14:25:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:27 smithi002 ceph-mon[17167]: pgmap v829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 829.867434 secs 2022-03-21T14:25:30.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:29 smithi016 ceph-mon[22010]: pgmap v830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:29 smithi079 ceph-mon[23158]: pgmap v830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:29 smithi002 ceph-mon[17167]: pgmap v830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:31 smithi016 ceph-mon[22010]: pgmap v831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:31 smithi079 ceph-mon[23158]: pgmap v831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:31 smithi002 ceph-mon[17167]: pgmap v831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 834.867538 secs 2022-03-21T14:25:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:25:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:25:33.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:25:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 834.867538 secs 2022-03-21T14:25:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:25:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:25:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:25:33.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 834.867538 secs 2022-03-21T14:25:33.321 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:25:33.321 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:25:33.321 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:25:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:34 smithi002 ceph-mon[17167]: pgmap v832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:25:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:34 smithi016 ceph-mon[22010]: pgmap v832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:25:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:34 smithi079 ceph-mon[23158]: pgmap v832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:25:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:36 smithi002 ceph-mon[17167]: pgmap v833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:36 smithi016 ceph-mon[22010]: pgmap v833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:36 smithi079 ceph-mon[23158]: pgmap v833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 839.867691 secs 2022-03-21T14:25:37.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 839.867691 secs 2022-03-21T14:25:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 839.867691 secs 2022-03-21T14:25:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:38 smithi002 ceph-mon[17167]: pgmap v834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:38 smithi016 ceph-mon[22010]: pgmap v834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:38 smithi079 ceph-mon[23158]: pgmap v834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:40 smithi002 ceph-mon[17167]: pgmap v835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:40 smithi016 ceph-mon[22010]: pgmap v835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:40 smithi079 ceph-mon[23158]: pgmap v835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:42 smithi002 ceph-mon[17167]: pgmap v836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 844.867814 secs 2022-03-21T14:25:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:42 smithi016 ceph-mon[22010]: pgmap v836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 844.867814 secs 2022-03-21T14:25:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:42 smithi079 ceph-mon[23158]: pgmap v836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 844.867814 secs 2022-03-21T14:25:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:44 smithi002 ceph-mon[17167]: pgmap v837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:44 smithi016 ceph-mon[22010]: pgmap v837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:44 smithi079 ceph-mon[23158]: pgmap v837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:46 smithi002 ceph-mon[17167]: pgmap v838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:46 smithi016 ceph-mon[22010]: pgmap v838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:46 smithi079 ceph-mon[23158]: pgmap v838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:47.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 849.867949 secs 2022-03-21T14:25:47.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 849.867949 secs 2022-03-21T14:25:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 849.867949 secs 2022-03-21T14:25:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:48 smithi016 ceph-mon[22010]: pgmap v839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:48 smithi079 ceph-mon[23158]: pgmap v839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:48 smithi002 ceph-mon[17167]: pgmap v839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:50 smithi016 ceph-mon[22010]: pgmap v840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:50 smithi079 ceph-mon[23158]: pgmap v840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:50 smithi002 ceph-mon[17167]: pgmap v840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:52 smithi016 ceph-mon[22010]: pgmap v841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 854.868071 secs 2022-03-21T14:25:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:52 smithi079 ceph-mon[23158]: pgmap v841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 854.868071 secs 2022-03-21T14:25:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:52 smithi002 ceph-mon[17167]: pgmap v841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 854.868071 secs 2022-03-21T14:25:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:54 smithi016 ceph-mon[22010]: pgmap v842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:54 smithi079 ceph-mon[23158]: pgmap v842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:54 smithi002 ceph-mon[17167]: pgmap v842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:56 smithi016 ceph-mon[22010]: pgmap v843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:56 smithi079 ceph-mon[23158]: pgmap v843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:56 smithi002 ceph-mon[17167]: pgmap v843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:25:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 859.868184 secs 2022-03-21T14:25:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 859.868184 secs 2022-03-21T14:25:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 859.868184 secs 2022-03-21T14:25:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:25:58 smithi016 ceph-mon[22010]: pgmap v844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:25:58 smithi079 ceph-mon[23158]: pgmap v844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:25:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:25:58 smithi002 ceph-mon[17167]: pgmap v844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:00 smithi016 ceph-mon[22010]: pgmap v845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:00 smithi079 ceph-mon[23158]: pgmap v845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:00 smithi002 ceph-mon[17167]: pgmap v845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:02 smithi016 ceph-mon[22010]: pgmap v846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 864.868323 secs 2022-03-21T14:26:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:02 smithi079 ceph-mon[23158]: pgmap v846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 864.868323 secs 2022-03-21T14:26:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:02 smithi002 ceph-mon[17167]: pgmap v846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 864.868323 secs 2022-03-21T14:26:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:04 smithi016 ceph-mon[22010]: pgmap v847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:04 smithi079 ceph-mon[23158]: pgmap v847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:04 smithi002 ceph-mon[17167]: pgmap v847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:06 smithi016 ceph-mon[22010]: pgmap v848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:06 smithi079 ceph-mon[23158]: pgmap v848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:06 smithi002 ceph-mon[17167]: pgmap v848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:07.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 869.868443 secs 2022-03-21T14:26:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 869.868443 secs 2022-03-21T14:26:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 869.868443 secs 2022-03-21T14:26:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:08 smithi016 ceph-mon[22010]: pgmap v849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:08 smithi079 ceph-mon[23158]: pgmap v849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:08 smithi002 ceph-mon[17167]: pgmap v849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:10 smithi016 ceph-mon[22010]: pgmap v850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:10 smithi079 ceph-mon[23158]: pgmap v850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:10 smithi002 ceph-mon[17167]: pgmap v850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:12 smithi079 ceph-mon[23158]: pgmap v851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 874.868577 secs 2022-03-21T14:26:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:12 smithi002 ceph-mon[17167]: pgmap v851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 874.868577 secs 2022-03-21T14:26:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:12 smithi016 ceph-mon[22010]: pgmap v851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:26:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 874.868577 secs 2022-03-21T14:26:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:14 smithi079 ceph-mon[23158]: pgmap v852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:14 smithi002 ceph-mon[17167]: pgmap v852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:14 smithi016 ceph-mon[22010]: pgmap v852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:16 smithi079 ceph-mon[23158]: pgmap v853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:16 smithi002 ceph-mon[17167]: pgmap v853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:16 smithi016 ceph-mon[22010]: pgmap v853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 879.868690 secs 2022-03-21T14:26:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 879.868690 secs 2022-03-21T14:26:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 879.868690 secs 2022-03-21T14:26:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:18 smithi079 ceph-mon[23158]: pgmap v854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:18 smithi002 ceph-mon[17167]: pgmap v854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:18 smithi016 ceph-mon[22010]: pgmap v854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:20 smithi079 ceph-mon[23158]: pgmap v855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:20 smithi002 ceph-mon[17167]: pgmap v855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:20 smithi016 ceph-mon[22010]: pgmap v855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:26:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:26:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:26:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:26:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:26:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:26:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:22 smithi079 ceph-mon[23158]: pgmap v856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 884.868815 secs 2022-03-21T14:26:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:22 smithi002 ceph-mon[17167]: pgmap v856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 884.868815 secs 2022-03-21T14:26:22.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:22 smithi016 ceph-mon[22010]: pgmap v856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 884.868815 secs 2022-03-21T14:26:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:24 smithi079 ceph-mon[23158]: pgmap v857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:24 smithi002 ceph-mon[17167]: pgmap v857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:24.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:24 smithi016 ceph-mon[22010]: pgmap v857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:26 smithi079 ceph-mon[23158]: pgmap v858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:26 smithi002 ceph-mon[17167]: pgmap v858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:26 smithi016 ceph-mon[22010]: pgmap v858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 889.868928 secs 2022-03-21T14:26:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 889.868928 secs 2022-03-21T14:26:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 889.868928 secs 2022-03-21T14:26:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:28 smithi079 ceph-mon[23158]: pgmap v859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:28 smithi002 ceph-mon[17167]: pgmap v859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:28 smithi016 ceph-mon[22010]: pgmap v859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:30 smithi079 ceph-mon[23158]: pgmap v860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:30 smithi002 ceph-mon[17167]: pgmap v860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:30 smithi016 ceph-mon[22010]: pgmap v860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:32 smithi079 ceph-mon[23158]: pgmap v861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 894.869042 secs 2022-03-21T14:26:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:32 smithi002 ceph-mon[17167]: pgmap v861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 894.869042 secs 2022-03-21T14:26:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:32 smithi016 ceph-mon[22010]: pgmap v861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 894.869042 secs 2022-03-21T14:26:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:26:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:26:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:26:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:26:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:26:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:26:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:26:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:26:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:26:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: pgmap v862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:26:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: pgmap v862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:26:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: pgmap v862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:26:34.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:26:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:26:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:26:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:26:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:26:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:26:35.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:35 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:26:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:26:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:26:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:26:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:26:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:26:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:35 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:26:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:26:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:26:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:26:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:26:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:26:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:35 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:26:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:36 smithi002 ceph-mon[17167]: pgmap v863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:36 smithi016 ceph-mon[22010]: pgmap v863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:36 smithi079 ceph-mon[23158]: pgmap v863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 899.869158 secs 2022-03-21T14:26:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 899.869158 secs 2022-03-21T14:26:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 899.869158 secs 2022-03-21T14:26:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:38 smithi002 ceph-mon[17167]: pgmap v864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:38 smithi016 ceph-mon[22010]: pgmap v864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:38 smithi079 ceph-mon[23158]: pgmap v864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:40 smithi002 ceph-mon[17167]: pgmap v865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:40 smithi016 ceph-mon[22010]: pgmap v865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:40 smithi079 ceph-mon[23158]: pgmap v865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:42 smithi002 ceph-mon[17167]: pgmap v866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 904.869294 secs 2022-03-21T14:26:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:42 smithi016 ceph-mon[22010]: pgmap v866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 904.869294 secs 2022-03-21T14:26:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:42 smithi079 ceph-mon[23158]: pgmap v866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 904.869294 secs 2022-03-21T14:26:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:44 smithi002 ceph-mon[17167]: pgmap v867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:44 smithi016 ceph-mon[22010]: pgmap v867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:44 smithi079 ceph-mon[23158]: pgmap v867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:46 smithi002 ceph-mon[17167]: pgmap v868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:46 smithi016 ceph-mon[22010]: pgmap v868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:46 smithi079 ceph-mon[23158]: pgmap v868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 909.869402 secs 2022-03-21T14:26:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 909.869402 secs 2022-03-21T14:26:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 909.869402 secs 2022-03-21T14:26:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:48 smithi002 ceph-mon[17167]: pgmap v869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:48 smithi016 ceph-mon[22010]: pgmap v869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:48 smithi079 ceph-mon[23158]: pgmap v869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:50 smithi002 ceph-mon[17167]: pgmap v870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:50 smithi016 ceph-mon[22010]: pgmap v870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:50 smithi079 ceph-mon[23158]: pgmap v870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:52 smithi002 ceph-mon[17167]: pgmap v871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 914.869499 secs 2022-03-21T14:26:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:52 smithi016 ceph-mon[22010]: pgmap v871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 914.869499 secs 2022-03-21T14:26:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:52 smithi079 ceph-mon[23158]: pgmap v871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 914.869499 secs 2022-03-21T14:26:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:54 smithi002 ceph-mon[17167]: pgmap v872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:54 smithi016 ceph-mon[22010]: pgmap v872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:54 smithi079 ceph-mon[23158]: pgmap v872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:56 smithi002 ceph-mon[17167]: pgmap v873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:56 smithi016 ceph-mon[22010]: pgmap v873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:56 smithi079 ceph-mon[23158]: pgmap v873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:26:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 919.869626 secs 2022-03-21T14:26:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 919.869626 secs 2022-03-21T14:26:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 919.869626 secs 2022-03-21T14:26:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:26:58 smithi002 ceph-mon[17167]: pgmap v874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:26:58 smithi016 ceph-mon[22010]: pgmap v874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:26:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:26:58 smithi079 ceph-mon[23158]: pgmap v874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:00 smithi002 ceph-mon[17167]: pgmap v875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:00 smithi016 ceph-mon[22010]: pgmap v875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:00 smithi079 ceph-mon[23158]: pgmap v875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:02 smithi002 ceph-mon[17167]: pgmap v876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 924.869716 secs 2022-03-21T14:27:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:02 smithi016 ceph-mon[22010]: pgmap v876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 924.869716 secs 2022-03-21T14:27:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:02 smithi079 ceph-mon[23158]: pgmap v876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 924.869716 secs 2022-03-21T14:27:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:04 smithi002 ceph-mon[17167]: pgmap v877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:04 smithi016 ceph-mon[22010]: pgmap v877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:04 smithi079 ceph-mon[23158]: pgmap v877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:06 smithi002 ceph-mon[17167]: pgmap v878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:27:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:06 smithi016 ceph-mon[22010]: pgmap v878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:27:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:06 smithi079 ceph-mon[23158]: pgmap v878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:27:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 929.869837 secs 2022-03-21T14:27:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 929.869837 secs 2022-03-21T14:27:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 929.869837 secs 2022-03-21T14:27:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:08 smithi002 ceph-mon[17167]: pgmap v879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:08 smithi016 ceph-mon[22010]: pgmap v879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:08 smithi079 ceph-mon[23158]: pgmap v879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:10 smithi002 ceph-mon[17167]: pgmap v880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:10 smithi016 ceph-mon[22010]: pgmap v880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:10 smithi079 ceph-mon[23158]: pgmap v880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:12 smithi002 ceph-mon[17167]: pgmap v881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 934.869938 secs 2022-03-21T14:27:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:12 smithi016 ceph-mon[22010]: pgmap v881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 934.869938 secs 2022-03-21T14:27:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:12 smithi079 ceph-mon[23158]: pgmap v881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 934.869938 secs 2022-03-21T14:27:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:14 smithi002 ceph-mon[17167]: pgmap v882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:14 smithi016 ceph-mon[22010]: pgmap v882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:14 smithi079 ceph-mon[23158]: pgmap v882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:16 smithi002 ceph-mon[17167]: pgmap v883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:16 smithi016 ceph-mon[22010]: pgmap v883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:16 smithi079 ceph-mon[23158]: pgmap v883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 939.870054 secs 2022-03-21T14:27:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 939.870054 secs 2022-03-21T14:27:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 939.870054 secs 2022-03-21T14:27:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:18 smithi002 ceph-mon[17167]: pgmap v884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:18 smithi016 ceph-mon[22010]: pgmap v884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:18 smithi079 ceph-mon[23158]: pgmap v884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:20 smithi002 ceph-mon[17167]: pgmap v885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:27:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:27:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:20 smithi016 ceph-mon[22010]: pgmap v885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:27:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:27:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:20 smithi079 ceph-mon[23158]: pgmap v885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:27:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:27:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:22 smithi002 ceph-mon[17167]: pgmap v886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 944.870188 secs 2022-03-21T14:27:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:22 smithi016 ceph-mon[22010]: pgmap v886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 944.870188 secs 2022-03-21T14:27:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:22 smithi079 ceph-mon[23158]: pgmap v886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 944.870188 secs 2022-03-21T14:27:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:24 smithi002 ceph-mon[17167]: pgmap v887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:24.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:24 smithi016 ceph-mon[22010]: pgmap v887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:24 smithi079 ceph-mon[23158]: pgmap v887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:26 smithi002 ceph-mon[17167]: pgmap v888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:26 smithi016 ceph-mon[22010]: pgmap v888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:26 smithi079 ceph-mon[23158]: pgmap v888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 949.870309 secs 2022-03-21T14:27:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 949.870309 secs 2022-03-21T14:27:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 949.870309 secs 2022-03-21T14:27:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:28 smithi002 ceph-mon[17167]: pgmap v889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:28 smithi016 ceph-mon[22010]: pgmap v889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:28 smithi079 ceph-mon[23158]: pgmap v889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:30.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:29 smithi079 ceph-mon[23158]: pgmap v890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:29 smithi002 ceph-mon[17167]: pgmap v890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:29 smithi016 ceph-mon[22010]: pgmap v890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:32.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:31 smithi079 ceph-mon[23158]: pgmap v891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:31 smithi002 ceph-mon[17167]: pgmap v891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:31 smithi016 ceph-mon[22010]: pgmap v891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 954.870424 secs 2022-03-21T14:27:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 954.870424 secs 2022-03-21T14:27:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 954.870424 secs 2022-03-21T14:27:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:33 smithi079 ceph-mon[23158]: pgmap v892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:34.238 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:33 smithi016 ceph-mon[22010]: pgmap v892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:34.273 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:33 smithi002 ceph-mon[17167]: pgmap v892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:27:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:27:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:27:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:27:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:27:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:27:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:27:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:27:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:27:36.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:36 smithi002 ceph-mon[17167]: pgmap v893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:27:36.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:36 smithi016 ceph-mon[22010]: pgmap v893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:27:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:36 smithi079 ceph-mon[23158]: pgmap v893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:27:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:38 smithi002 ceph-mon[17167]: pgmap v894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 959.870505 secs 2022-03-21T14:27:38.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:38 smithi016 ceph-mon[22010]: pgmap v894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 959.870505 secs 2022-03-21T14:27:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:38 smithi079 ceph-mon[23158]: pgmap v894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 959.870505 secs 2022-03-21T14:27:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:40 smithi002 ceph-mon[17167]: pgmap v895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:40 smithi016 ceph-mon[22010]: pgmap v895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:40 smithi079 ceph-mon[23158]: pgmap v895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:42.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:42 smithi002 ceph-mon[17167]: pgmap v896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:42.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:42 smithi016 ceph-mon[22010]: pgmap v896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:42 smithi079 ceph-mon[23158]: pgmap v896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 964.870628 secs 2022-03-21T14:27:43.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 964.870628 secs 2022-03-21T14:27:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 964.870628 secs 2022-03-21T14:27:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:44 smithi002 ceph-mon[17167]: pgmap v897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:44.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:44 smithi016 ceph-mon[22010]: pgmap v897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:44 smithi079 ceph-mon[23158]: pgmap v897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:46 smithi002 ceph-mon[17167]: pgmap v898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:46 smithi016 ceph-mon[22010]: pgmap v898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:46 smithi079 ceph-mon[23158]: pgmap v898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:48 smithi002 ceph-mon[17167]: pgmap v899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 969.870761 secs 2022-03-21T14:27:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:48 smithi016 ceph-mon[22010]: pgmap v899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 969.870761 secs 2022-03-21T14:27:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:48 smithi079 ceph-mon[23158]: pgmap v899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 969.870761 secs 2022-03-21T14:27:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:50 smithi002 ceph-mon[17167]: pgmap v900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:50 smithi016 ceph-mon[22010]: pgmap v900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:50 smithi079 ceph-mon[23158]: pgmap v900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:52 smithi002 ceph-mon[17167]: pgmap v901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:52 smithi016 ceph-mon[22010]: pgmap v901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:52 smithi079 ceph-mon[23158]: pgmap v901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 974.870880 secs 2022-03-21T14:27:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 974.870880 secs 2022-03-21T14:27:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 974.870880 secs 2022-03-21T14:27:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:54 smithi002 ceph-mon[17167]: pgmap v902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:54.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:54 smithi016 ceph-mon[22010]: pgmap v902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:54 smithi079 ceph-mon[23158]: pgmap v902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:56.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:56 smithi016 ceph-mon[22010]: pgmap v903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:56 smithi079 ceph-mon[23158]: pgmap v903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:56 smithi002 ceph-mon[17167]: pgmap v903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:27:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:58 smithi016 ceph-mon[22010]: pgmap v904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:27:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 979.871004 secs 2022-03-21T14:27:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:58 smithi079 ceph-mon[23158]: pgmap v904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:27:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 979.871004 secs 2022-03-21T14:27:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:58 smithi002 ceph-mon[17167]: pgmap v904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:27:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:27:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 979.871004 secs 2022-03-21T14:28:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:00 smithi016 ceph-mon[22010]: pgmap v905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:00 smithi079 ceph-mon[23158]: pgmap v905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:00 smithi002 ceph-mon[17167]: pgmap v905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:02 smithi016 ceph-mon[22010]: pgmap v906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:02 smithi079 ceph-mon[23158]: pgmap v906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:02 smithi002 ceph-mon[17167]: pgmap v906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 984.871130 secs 2022-03-21T14:28:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 984.871130 secs 2022-03-21T14:28:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 984.871130 secs 2022-03-21T14:28:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:04 smithi016 ceph-mon[22010]: pgmap v907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:04 smithi079 ceph-mon[23158]: pgmap v907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:04 smithi002 ceph-mon[17167]: pgmap v907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:06 smithi016 ceph-mon[22010]: pgmap v908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:06 smithi079 ceph-mon[23158]: pgmap v908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:06 smithi002 ceph-mon[17167]: pgmap v908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:08 smithi016 ceph-mon[22010]: pgmap v909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 989.871269 secs 2022-03-21T14:28:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:08 smithi079 ceph-mon[23158]: pgmap v909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 989.871269 secs 2022-03-21T14:28:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:08 smithi002 ceph-mon[17167]: pgmap v909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 989.871269 secs 2022-03-21T14:28:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:10 smithi016 ceph-mon[22010]: pgmap v910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:10 smithi079 ceph-mon[23158]: pgmap v910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:10 smithi002 ceph-mon[17167]: pgmap v910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:12 smithi016 ceph-mon[22010]: pgmap v911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:12 smithi079 ceph-mon[23158]: pgmap v911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:12 smithi002 ceph-mon[17167]: pgmap v911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 994.871384 secs 2022-03-21T14:28:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 994.871384 secs 2022-03-21T14:28:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 994.871384 secs 2022-03-21T14:28:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:14 smithi016 ceph-mon[22010]: pgmap v912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:14 smithi079 ceph-mon[23158]: pgmap v912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:14 smithi002 ceph-mon[17167]: pgmap v912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:16 smithi016 ceph-mon[22010]: pgmap v913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:16 smithi079 ceph-mon[23158]: pgmap v913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:16 smithi002 ceph-mon[17167]: pgmap v913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:18 smithi079 ceph-mon[23158]: pgmap v914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 999.871500 secs 2022-03-21T14:28:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:18 smithi002 ceph-mon[17167]: pgmap v914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 999.871500 secs 2022-03-21T14:28:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:18 smithi016 ceph-mon[22010]: pgmap v914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 999.871500 secs 2022-03-21T14:28:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:20 smithi079 ceph-mon[23158]: pgmap v915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:20 smithi002 ceph-mon[17167]: pgmap v915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:20 smithi016 ceph-mon[22010]: pgmap v915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:28:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:28:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:28:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:28:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:28:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:28:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:22 smithi079 ceph-mon[23158]: pgmap v916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:22 smithi002 ceph-mon[17167]: pgmap v916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:22 smithi016 ceph-mon[22010]: pgmap v916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1004.871649 secs 2022-03-21T14:28:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1004.871649 secs 2022-03-21T14:28:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1004.871649 secs 2022-03-21T14:28:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:24 smithi079 ceph-mon[23158]: pgmap v917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:24 smithi002 ceph-mon[17167]: pgmap v917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:24 smithi016 ceph-mon[22010]: pgmap v917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:26 smithi079 ceph-mon[23158]: pgmap v918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:26 smithi002 ceph-mon[17167]: pgmap v918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:26 smithi016 ceph-mon[22010]: pgmap v918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:28 smithi079 ceph-mon[23158]: pgmap v919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1009.871712 secs 2022-03-21T14:28:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:28 smithi002 ceph-mon[17167]: pgmap v919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1009.871712 secs 2022-03-21T14:28:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:28 smithi016 ceph-mon[22010]: pgmap v919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1009.871712 secs 2022-03-21T14:28:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:30 smithi079 ceph-mon[23158]: pgmap v920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:30 smithi002 ceph-mon[17167]: pgmap v920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:30 smithi016 ceph-mon[22010]: pgmap v920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:32 smithi079 ceph-mon[23158]: pgmap v921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:32 smithi002 ceph-mon[17167]: pgmap v921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:32 smithi016 ceph-mon[22010]: pgmap v921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1014.871832 secs 2022-03-21T14:28:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1014.871832 secs 2022-03-21T14:28:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1014.871832 secs 2022-03-21T14:28:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:34 smithi079 ceph-mon[23158]: pgmap v922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:34 smithi002 ceph-mon[17167]: pgmap v922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:34 smithi016 ceph-mon[22010]: pgmap v922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:35.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:28:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:28:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:28:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:28:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:28:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:28:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:28:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:28:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:28:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:36 smithi079 ceph-mon[23158]: pgmap v923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:28:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:36 smithi002 ceph-mon[17167]: pgmap v923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:28:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:36 smithi016 ceph-mon[22010]: pgmap v923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:28:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:38 smithi079 ceph-mon[23158]: pgmap v924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1019.871944 secs 2022-03-21T14:28:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:38 smithi002 ceph-mon[17167]: pgmap v924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1019.871944 secs 2022-03-21T14:28:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:38 smithi016 ceph-mon[22010]: pgmap v924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1019.871944 secs 2022-03-21T14:28:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:40 smithi079 ceph-mon[23158]: pgmap v925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:40 smithi002 ceph-mon[17167]: pgmap v925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:40 smithi016 ceph-mon[22010]: pgmap v925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:42 smithi002 ceph-mon[17167]: pgmap v926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:42 smithi016 ceph-mon[22010]: pgmap v926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:42 smithi079 ceph-mon[23158]: pgmap v926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1024.872060 secs 2022-03-21T14:28:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1024.872060 secs 2022-03-21T14:28:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1024.872060 secs 2022-03-21T14:28:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:44 smithi002 ceph-mon[17167]: pgmap v927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:44 smithi016 ceph-mon[22010]: pgmap v927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:44 smithi079 ceph-mon[23158]: pgmap v927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:46 smithi002 ceph-mon[17167]: pgmap v928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:46 smithi016 ceph-mon[22010]: pgmap v928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:46 smithi079 ceph-mon[23158]: pgmap v928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:48 smithi002 ceph-mon[17167]: pgmap v929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1029.872181 secs 2022-03-21T14:28:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:48 smithi016 ceph-mon[22010]: pgmap v929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1029.872181 secs 2022-03-21T14:28:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:48 smithi079 ceph-mon[23158]: pgmap v929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1029.872181 secs 2022-03-21T14:28:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:50 smithi002 ceph-mon[17167]: pgmap v930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:50 smithi016 ceph-mon[22010]: pgmap v930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:50 smithi079 ceph-mon[23158]: pgmap v930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:52 smithi002 ceph-mon[17167]: pgmap v931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:52 smithi016 ceph-mon[22010]: pgmap v931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:52 smithi079 ceph-mon[23158]: pgmap v931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1034.872282 secs 2022-03-21T14:28:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1034.872282 secs 2022-03-21T14:28:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1034.872282 secs 2022-03-21T14:28:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:54 smithi002 ceph-mon[17167]: pgmap v932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:54 smithi016 ceph-mon[22010]: pgmap v932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:54 smithi079 ceph-mon[23158]: pgmap v932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:56 smithi002 ceph-mon[17167]: pgmap v933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:56 smithi016 ceph-mon[22010]: pgmap v933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:56 smithi079 ceph-mon[23158]: pgmap v933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:28:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:58 smithi002 ceph-mon[17167]: pgmap v934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:28:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1039.872418 secs 2022-03-21T14:28:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:58 smithi016 ceph-mon[22010]: pgmap v934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:28:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1039.872418 secs 2022-03-21T14:28:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:58 smithi079 ceph-mon[23158]: pgmap v934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:28:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:28:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1039.872418 secs 2022-03-21T14:29:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:00 smithi002 ceph-mon[17167]: pgmap v935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:00 smithi016 ceph-mon[22010]: pgmap v935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:00 smithi079 ceph-mon[23158]: pgmap v935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:02 smithi002 ceph-mon[17167]: pgmap v936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:02 smithi016 ceph-mon[22010]: pgmap v936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:02 smithi079 ceph-mon[23158]: pgmap v936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1044.872574 secs 2022-03-21T14:29:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1044.872574 secs 2022-03-21T14:29:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1044.872574 secs 2022-03-21T14:29:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:04 smithi002 ceph-mon[17167]: pgmap v937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:04 smithi016 ceph-mon[22010]: pgmap v937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:04 smithi079 ceph-mon[23158]: pgmap v937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:06 smithi002 ceph-mon[17167]: pgmap v938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:06 smithi016 ceph-mon[22010]: pgmap v938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:06 smithi079 ceph-mon[23158]: pgmap v938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:08 smithi002 ceph-mon[17167]: pgmap v939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1049.872700 secs 2022-03-21T14:29:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:08 smithi016 ceph-mon[22010]: pgmap v939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1049.872700 secs 2022-03-21T14:29:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:08 smithi079 ceph-mon[23158]: pgmap v939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1049.872700 secs 2022-03-21T14:29:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:10 smithi002 ceph-mon[17167]: pgmap v940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:10 smithi016 ceph-mon[22010]: pgmap v940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:10 smithi079 ceph-mon[23158]: pgmap v940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:12 smithi002 ceph-mon[17167]: pgmap v941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:12.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:12 smithi016 ceph-mon[22010]: pgmap v941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:12 smithi079 ceph-mon[23158]: pgmap v941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1054.872837 secs 2022-03-21T14:29:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1054.872837 secs 2022-03-21T14:29:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1054.872837 secs 2022-03-21T14:29:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:14 smithi002 ceph-mon[17167]: pgmap v942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:14 smithi016 ceph-mon[22010]: pgmap v942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:14 smithi079 ceph-mon[23158]: pgmap v942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:16 smithi002 ceph-mon[17167]: pgmap v943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:16 smithi016 ceph-mon[22010]: pgmap v943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:16 smithi079 ceph-mon[23158]: pgmap v943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:18 smithi002 ceph-mon[17167]: pgmap v944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1059.872945 secs 2022-03-21T14:29:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:18 smithi016 ceph-mon[22010]: pgmap v944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1059.872945 secs 2022-03-21T14:29:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:18 smithi079 ceph-mon[23158]: pgmap v944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1059.872945 secs 2022-03-21T14:29:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:20 smithi002 ceph-mon[17167]: pgmap v945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:20 smithi016 ceph-mon[22010]: pgmap v945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:20 smithi079 ceph-mon[23158]: pgmap v945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:29:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:29:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:29:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:29:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:29:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:29:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:22 smithi002 ceph-mon[17167]: pgmap v946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:22 smithi016 ceph-mon[22010]: pgmap v946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:22 smithi079 ceph-mon[23158]: pgmap v946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1064.873079 secs 2022-03-21T14:29:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1064.873079 secs 2022-03-21T14:29:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1064.873079 secs 2022-03-21T14:29:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:24 smithi002 ceph-mon[17167]: pgmap v947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:24 smithi016 ceph-mon[22010]: pgmap v947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:24 smithi079 ceph-mon[23158]: pgmap v947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:26 smithi002 ceph-mon[17167]: pgmap v948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:26 smithi016 ceph-mon[22010]: pgmap v948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:26 smithi079 ceph-mon[23158]: pgmap v948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:28 smithi002 ceph-mon[17167]: pgmap v949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1069.873155 secs 2022-03-21T14:29:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:28 smithi016 ceph-mon[22010]: pgmap v949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1069.873155 secs 2022-03-21T14:29:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:28 smithi079 ceph-mon[23158]: pgmap v949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1069.873155 secs 2022-03-21T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:30 smithi002 ceph-mon[17167]: pgmap v950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:30 smithi016 ceph-mon[22010]: pgmap v950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:30 smithi079 ceph-mon[23158]: pgmap v950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:32 smithi002 ceph-mon[17167]: pgmap v951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:32 smithi016 ceph-mon[22010]: pgmap v951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:32 smithi079 ceph-mon[23158]: pgmap v951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1074.873271 secs 2022-03-21T14:29:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1074.873271 secs 2022-03-21T14:29:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1074.873271 secs 2022-03-21T14:29:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:34 smithi002 ceph-mon[17167]: pgmap v952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:34 smithi016 ceph-mon[22010]: pgmap v952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:34 smithi079 ceph-mon[23158]: pgmap v952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:36 smithi002 ceph-mon[17167]: pgmap v953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:29:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:29:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:29:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:36 smithi016 ceph-mon[22010]: pgmap v953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:29:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:29:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:29:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:36 smithi079 ceph-mon[23158]: pgmap v953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:29:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:29:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:29:38.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:38 smithi016 ceph-mon[22010]: pgmap v954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:38.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1079.873354 secs 2022-03-21T14:29:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:38 smithi002 ceph-mon[17167]: pgmap v954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1079.873354 secs 2022-03-21T14:29:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:38 smithi079 ceph-mon[23158]: pgmap v954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1079.873354 secs 2022-03-21T14:29:39.788 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:39.788 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:39.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:40 smithi002 ceph-mon[17167]: pgmap v955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:40 smithi016 ceph-mon[22010]: pgmap v955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:40 smithi079 ceph-mon[23158]: pgmap v955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:29:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:42 smithi016 ceph-mon[22010]: pgmap v956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1084.873472 secs 2022-03-21T14:29:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:42 smithi079 ceph-mon[23158]: pgmap v956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1084.873472 secs 2022-03-21T14:29:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:42 smithi002 ceph-mon[17167]: pgmap v956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1084.873472 secs 2022-03-21T14:29:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:43 smithi079 ceph-mon[23158]: pgmap v957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:43 smithi002 ceph-mon[17167]: pgmap v957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:43 smithi016 ceph-mon[22010]: pgmap v957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:46.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:45 smithi079 ceph-mon[23158]: pgmap v958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:45 smithi002 ceph-mon[17167]: pgmap v958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:45 smithi016 ceph-mon[22010]: pgmap v958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:47 smithi079 ceph-mon[23158]: pgmap v959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1089.873587 secs 2022-03-21T14:29:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:47 smithi002 ceph-mon[17167]: pgmap v959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1089.873587 secs 2022-03-21T14:29:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:47 smithi016 ceph-mon[22010]: pgmap v959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1089.873587 secs 2022-03-21T14:29:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:49 smithi002 ceph-mon[17167]: pgmap v960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:49 smithi016 ceph-mon[22010]: pgmap v960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:49 smithi079 ceph-mon[23158]: pgmap v960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:51 smithi002 ceph-mon[17167]: pgmap v961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:52.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:51 smithi016 ceph-mon[22010]: pgmap v961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:51 smithi079 ceph-mon[23158]: pgmap v961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1094.873705 secs 2022-03-21T14:29:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1094.873705 secs 2022-03-21T14:29:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1094.873705 secs 2022-03-21T14:29:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:53 smithi002 ceph-mon[17167]: pgmap v962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:54.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:53 smithi016 ceph-mon[22010]: pgmap v962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:53 smithi079 ceph-mon[23158]: pgmap v962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:55 smithi002 ceph-mon[17167]: pgmap v963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:56.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:55 smithi016 ceph-mon[22010]: pgmap v963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:55 smithi079 ceph-mon[23158]: pgmap v963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:29:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:57 smithi002 ceph-mon[17167]: pgmap v964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1099.873801 secs 2022-03-21T14:29:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:57 smithi016 ceph-mon[22010]: pgmap v964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1099.873801 secs 2022-03-21T14:29:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:57 smithi079 ceph-mon[23158]: pgmap v964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:29:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1099.873801 secs 2022-03-21T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:29:59 smithi002 ceph-mon[17167]: pgmap v965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:00.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:29:59 smithi016 ceph-mon[22010]: pgmap v965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:29:59 smithi079 ceph-mon[23158]: pgmap v965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:30:01.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:30:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:30:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:02 smithi002 ceph-mon[17167]: pgmap v966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:02 smithi016 ceph-mon[22010]: pgmap v966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:02 smithi079 ceph-mon[23158]: pgmap v966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1104.873954 secs 2022-03-21T14:30:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1104.873954 secs 2022-03-21T14:30:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1104.873954 secs 2022-03-21T14:30:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:04 smithi002 ceph-mon[17167]: pgmap v967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:04 smithi016 ceph-mon[22010]: pgmap v967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:04 smithi079 ceph-mon[23158]: pgmap v967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:06 smithi002 ceph-mon[17167]: pgmap v968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:06 smithi016 ceph-mon[22010]: pgmap v968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:06 smithi079 ceph-mon[23158]: pgmap v968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:08 smithi002 ceph-mon[17167]: pgmap v969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1109.874054 secs 2022-03-21T14:30:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:08 smithi016 ceph-mon[22010]: pgmap v969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1109.874054 secs 2022-03-21T14:30:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:08 smithi079 ceph-mon[23158]: pgmap v969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1109.874054 secs 2022-03-21T14:30:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:10 smithi002 ceph-mon[17167]: pgmap v970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:10 smithi016 ceph-mon[22010]: pgmap v970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:10 smithi079 ceph-mon[23158]: pgmap v970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:12 smithi002 ceph-mon[17167]: pgmap v971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:12 smithi016 ceph-mon[22010]: pgmap v971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:12 smithi079 ceph-mon[23158]: pgmap v971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1114.874191 secs 2022-03-21T14:30:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1114.874191 secs 2022-03-21T14:30:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1114.874191 secs 2022-03-21T14:30:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:14 smithi002 ceph-mon[17167]: pgmap v972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:14 smithi016 ceph-mon[22010]: pgmap v972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:14 smithi079 ceph-mon[23158]: pgmap v972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:16.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:16 smithi002 ceph-mon[17167]: pgmap v973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:16 smithi016 ceph-mon[22010]: pgmap v973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:16 smithi079 ceph-mon[23158]: pgmap v973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:18 smithi002 ceph-mon[17167]: pgmap v974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1119.874323 secs 2022-03-21T14:30:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:18 smithi016 ceph-mon[22010]: pgmap v974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1119.874323 secs 2022-03-21T14:30:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:18 smithi079 ceph-mon[23158]: pgmap v974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1119.874323 secs 2022-03-21T14:30:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:20 smithi002 ceph-mon[17167]: pgmap v975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:20 smithi016 ceph-mon[22010]: pgmap v975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:20 smithi079 ceph-mon[23158]: pgmap v975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:30:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:30:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:30:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:30:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:30:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:30:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:22 smithi002 ceph-mon[17167]: pgmap v976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:22 smithi016 ceph-mon[22010]: pgmap v976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:22 smithi079 ceph-mon[23158]: pgmap v976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1124.874404 secs 2022-03-21T14:30:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1124.874404 secs 2022-03-21T14:30:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1124.874404 secs 2022-03-21T14:30:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:24 smithi002 ceph-mon[17167]: pgmap v977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:24 smithi016 ceph-mon[22010]: pgmap v977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:24 smithi079 ceph-mon[23158]: pgmap v977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:26 smithi002 ceph-mon[17167]: pgmap v978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:26 smithi016 ceph-mon[22010]: pgmap v978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:26 smithi079 ceph-mon[23158]: pgmap v978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:28 smithi002 ceph-mon[17167]: pgmap v979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1129.874520 secs 2022-03-21T14:30:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:28 smithi016 ceph-mon[22010]: pgmap v979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1129.874520 secs 2022-03-21T14:30:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:28 smithi079 ceph-mon[23158]: pgmap v979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1129.874520 secs 2022-03-21T14:30:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:30 smithi002 ceph-mon[17167]: pgmap v980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:30 smithi016 ceph-mon[22010]: pgmap v980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:30 smithi079 ceph-mon[23158]: pgmap v980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:32 smithi002 ceph-mon[17167]: pgmap v981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:32 smithi016 ceph-mon[22010]: pgmap v981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:32 smithi079 ceph-mon[23158]: pgmap v981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1134.874639 secs 2022-03-21T14:30:33.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1134.874639 secs 2022-03-21T14:30:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1134.874639 secs 2022-03-21T14:30:34.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:34 smithi002 ceph-mon[17167]: pgmap v982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:34.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:34 smithi016 ceph-mon[22010]: pgmap v982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:34 smithi079 ceph-mon[23158]: pgmap v982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:36.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:36 smithi002 ceph-mon[17167]: pgmap v983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:36.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:36 smithi016 ceph-mon[22010]: pgmap v983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:36 smithi079 ceph-mon[23158]: pgmap v983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:38 smithi002 ceph-mon[17167]: pgmap v984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1139.874754 secs 2022-03-21T14:30:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:38 smithi016 ceph-mon[22010]: pgmap v984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1139.874754 secs 2022-03-21T14:30:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:38 smithi079 ceph-mon[23158]: pgmap v984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1139.874754 secs 2022-03-21T14:30:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:40 smithi002 ceph-mon[17167]: pgmap v985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:40 smithi016 ceph-mon[22010]: pgmap v985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:40 smithi079 ceph-mon[23158]: pgmap v985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:30:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:30:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:30:41.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:30:41.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:30:41.389 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:30:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:30:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:30:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:30:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:42 smithi016 ceph-mon[22010]: pgmap v986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1144.874867 secs 2022-03-21T14:30:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:42 smithi079 ceph-mon[23158]: pgmap v986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1144.874867 secs 2022-03-21T14:30:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:42 smithi002 ceph-mon[17167]: pgmap v986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:30:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1144.874867 secs 2022-03-21T14:30:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:44 smithi016 ceph-mon[22010]: pgmap v987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:44 smithi079 ceph-mon[23158]: pgmap v987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:44 smithi002 ceph-mon[17167]: pgmap v987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:46 smithi016 ceph-mon[22010]: pgmap v988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:46 smithi079 ceph-mon[23158]: pgmap v988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:46 smithi002 ceph-mon[17167]: pgmap v988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1149.874980 secs 2022-03-21T14:30:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1149.874980 secs 2022-03-21T14:30:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1149.874980 secs 2022-03-21T14:30:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:48 smithi016 ceph-mon[22010]: pgmap v989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:48 smithi079 ceph-mon[23158]: pgmap v989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:48 smithi002 ceph-mon[17167]: pgmap v989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:50 smithi079 ceph-mon[23158]: pgmap v990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:50 smithi002 ceph-mon[17167]: pgmap v990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:50 smithi016 ceph-mon[22010]: pgmap v990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:52 smithi079 ceph-mon[23158]: pgmap v991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1154.875098 secs 2022-03-21T14:30:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:52 smithi002 ceph-mon[17167]: pgmap v991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1154.875098 secs 2022-03-21T14:30:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:52 smithi016 ceph-mon[22010]: pgmap v991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1154.875098 secs 2022-03-21T14:30:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:54 smithi079 ceph-mon[23158]: pgmap v992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:54 smithi002 ceph-mon[17167]: pgmap v992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:54 smithi016 ceph-mon[22010]: pgmap v992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:56 smithi079 ceph-mon[23158]: pgmap v993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:56 smithi002 ceph-mon[17167]: pgmap v993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:56 smithi016 ceph-mon[22010]: pgmap v993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:30:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1159.875206 secs 2022-03-21T14:30:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1159.875206 secs 2022-03-21T14:30:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1159.875206 secs 2022-03-21T14:30:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:30:58 smithi079 ceph-mon[23158]: pgmap v994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:30:58 smithi002 ceph-mon[17167]: pgmap v994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:30:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:30:58 smithi016 ceph-mon[22010]: pgmap v994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:00 smithi079 ceph-mon[23158]: pgmap v995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:00 smithi002 ceph-mon[17167]: pgmap v995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:00 smithi016 ceph-mon[22010]: pgmap v995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:02 smithi079 ceph-mon[23158]: pgmap v996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1164.875340 secs 2022-03-21T14:31:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:02 smithi002 ceph-mon[17167]: pgmap v996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1164.875340 secs 2022-03-21T14:31:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:02 smithi016 ceph-mon[22010]: pgmap v996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1164.875340 secs 2022-03-21T14:31:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:04 smithi079 ceph-mon[23158]: pgmap v997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:04 smithi002 ceph-mon[17167]: pgmap v997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:04 smithi016 ceph-mon[22010]: pgmap v997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:06 smithi079 ceph-mon[23158]: pgmap v998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:06 smithi002 ceph-mon[17167]: pgmap v998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:06 smithi016 ceph-mon[22010]: pgmap v998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1169.875431 secs 2022-03-21T14:31:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1169.875431 secs 2022-03-21T14:31:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1169.875431 secs 2022-03-21T14:31:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:08 smithi079 ceph-mon[23158]: pgmap v999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:08 smithi002 ceph-mon[17167]: pgmap v999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:08 smithi016 ceph-mon[22010]: pgmap v999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:10 smithi079 ceph-mon[23158]: pgmap v1000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:10 smithi002 ceph-mon[17167]: pgmap v1000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:10 smithi016 ceph-mon[22010]: pgmap v1000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:12 smithi079 ceph-mon[23158]: pgmap v1001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1174.875530 secs 2022-03-21T14:31:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:12 smithi002 ceph-mon[17167]: pgmap v1001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1174.875530 secs 2022-03-21T14:31:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:12 smithi016 ceph-mon[22010]: pgmap v1001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1174.875530 secs 2022-03-21T14:31:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:14 smithi079 ceph-mon[23158]: pgmap v1002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:14 smithi002 ceph-mon[17167]: pgmap v1002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:14 smithi016 ceph-mon[22010]: pgmap v1002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:16 smithi002 ceph-mon[17167]: pgmap v1003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:16 smithi016 ceph-mon[22010]: pgmap v1003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:16 smithi079 ceph-mon[23158]: pgmap v1003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1179.875637 secs 2022-03-21T14:31:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1179.875637 secs 2022-03-21T14:31:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1179.875637 secs 2022-03-21T14:31:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:18 smithi002 ceph-mon[17167]: pgmap v1004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:18 smithi016 ceph-mon[22010]: pgmap v1004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:18 smithi079 ceph-mon[23158]: pgmap v1004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:20 smithi002 ceph-mon[17167]: pgmap v1005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:20 smithi016 ceph-mon[22010]: pgmap v1005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:20 smithi079 ceph-mon[23158]: pgmap v1005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:31:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:31:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:31:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:31:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:31:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:31:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:22 smithi002 ceph-mon[17167]: pgmap v1006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1184.875749 secs 2022-03-21T14:31:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:22 smithi016 ceph-mon[22010]: pgmap v1006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1184.875749 secs 2022-03-21T14:31:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:22 smithi079 ceph-mon[23158]: pgmap v1006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1184.875749 secs 2022-03-21T14:31:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:24 smithi002 ceph-mon[17167]: pgmap v1007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:24 smithi016 ceph-mon[22010]: pgmap v1007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:24 smithi079 ceph-mon[23158]: pgmap v1007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:26 smithi002 ceph-mon[17167]: pgmap v1008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:26 smithi016 ceph-mon[22010]: pgmap v1008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:26 smithi079 ceph-mon[23158]: pgmap v1008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1189.875851 secs 2022-03-21T14:31:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1189.875851 secs 2022-03-21T14:31:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1189.875851 secs 2022-03-21T14:31:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:28 smithi002 ceph-mon[17167]: pgmap v1009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:28 smithi016 ceph-mon[22010]: pgmap v1009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:28 smithi079 ceph-mon[23158]: pgmap v1009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:30 smithi002 ceph-mon[17167]: pgmap v1010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:30 smithi016 ceph-mon[22010]: pgmap v1010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:30 smithi079 ceph-mon[23158]: pgmap v1010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:32 smithi002 ceph-mon[17167]: pgmap v1011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:31:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1194.875976 secs 2022-03-21T14:31:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:32 smithi016 ceph-mon[22010]: pgmap v1011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:31:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1194.875976 secs 2022-03-21T14:31:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:32 smithi079 ceph-mon[23158]: pgmap v1011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:31:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1194.875976 secs 2022-03-21T14:31:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:34 smithi002 ceph-mon[17167]: pgmap v1012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:34 smithi016 ceph-mon[22010]: pgmap v1012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:34 smithi079 ceph-mon[23158]: pgmap v1012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:36 smithi002 ceph-mon[17167]: pgmap v1013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:36 smithi016 ceph-mon[22010]: pgmap v1013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:36 smithi079 ceph-mon[23158]: pgmap v1013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1199.876117 secs 2022-03-21T14:31:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1199.876117 secs 2022-03-21T14:31:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1199.876117 secs 2022-03-21T14:31:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:38 smithi002 ceph-mon[17167]: pgmap v1014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:38 smithi016 ceph-mon[22010]: pgmap v1014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:38 smithi079 ceph-mon[23158]: pgmap v1014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:40 smithi002 ceph-mon[17167]: pgmap v1015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:40 smithi016 ceph-mon[22010]: pgmap v1015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:40 smithi079 ceph-mon[23158]: pgmap v1015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: pgmap v1016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:31:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:31:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1204.876186 secs 2022-03-21T14:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 1204.761297 seconds ago 2022-03-21T14:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: pgmap v1016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:31:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:31:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:31:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1204.876186 secs 2022-03-21T14:31:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 1204.761297 seconds ago 2022-03-21T14:31:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:42.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: pgmap v1016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:42.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:31:42.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:31:42.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:31:42.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1204.876186 secs 2022-03-21T14:31:42.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 1204.761297 seconds ago 2022-03-21T14:31:42.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:42.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:43 smithi002 ceph-mon[17167]: pgmap v1017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:43 smithi079 ceph-mon[23158]: pgmap v1017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:43 smithi016 ceph-mon[22010]: pgmap v1017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:31:46.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:46 smithi002 ceph-mon[17167]: pgmap v1018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:46.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:46 smithi016 ceph-mon[22010]: pgmap v1018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:46 smithi079 ceph-mon[23158]: pgmap v1018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:48 smithi002 ceph-mon[17167]: pgmap v1019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1209.876302 secs 2022-03-21T14:31:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:48 smithi016 ceph-mon[22010]: pgmap v1019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1209.876302 secs 2022-03-21T14:31:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:48 smithi079 ceph-mon[23158]: pgmap v1019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1209.876302 secs 2022-03-21T14:31:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:50 smithi002 ceph-mon[17167]: pgmap v1020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:50 smithi016 ceph-mon[22010]: pgmap v1020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:50.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:50 smithi079 ceph-mon[23158]: pgmap v1020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:52.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:52 smithi002 ceph-mon[17167]: pgmap v1021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:52 smithi016 ceph-mon[22010]: pgmap v1021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:52.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:52 smithi079 ceph-mon[23158]: pgmap v1021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1214.876440 secs 2022-03-21T14:31:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1214.876440 secs 2022-03-21T14:31:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1214.876440 secs 2022-03-21T14:31:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:54 smithi002 ceph-mon[17167]: pgmap v1022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:54 smithi016 ceph-mon[22010]: pgmap v1022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:54 smithi079 ceph-mon[23158]: pgmap v1022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:56 smithi002 ceph-mon[17167]: pgmap v1023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:56 smithi016 ceph-mon[22010]: pgmap v1023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:56 smithi079 ceph-mon[23158]: pgmap v1023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:31:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:58 smithi002 ceph-mon[17167]: pgmap v1024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:31:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1219.876577 secs 2022-03-21T14:31:58.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:58 smithi016 ceph-mon[22010]: pgmap v1024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:31:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1219.876577 secs 2022-03-21T14:31:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:58 smithi079 ceph-mon[23158]: pgmap v1024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:31:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:31:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1219.876577 secs 2022-03-21T14:32:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:00 smithi002 ceph-mon[17167]: pgmap v1025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:00.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:00 smithi016 ceph-mon[22010]: pgmap v1025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:00 smithi079 ceph-mon[23158]: pgmap v1025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:02.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:02 smithi002 ceph-mon[17167]: pgmap v1026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:02.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:02 smithi016 ceph-mon[22010]: pgmap v1026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:02 smithi079 ceph-mon[23158]: pgmap v1026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1224.876695 secs 2022-03-21T14:32:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1224.876695 secs 2022-03-21T14:32:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1224.876695 secs 2022-03-21T14:32:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:04 smithi002 ceph-mon[17167]: pgmap v1027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:04.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:04 smithi016 ceph-mon[22010]: pgmap v1027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:04 smithi079 ceph-mon[23158]: pgmap v1027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:06 smithi002 ceph-mon[17167]: pgmap v1028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:06 smithi016 ceph-mon[22010]: pgmap v1028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:06 smithi079 ceph-mon[23158]: pgmap v1028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:08 smithi002 ceph-mon[17167]: pgmap v1029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1229.876807 secs 2022-03-21T14:32:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:08 smithi016 ceph-mon[22010]: pgmap v1029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1229.876807 secs 2022-03-21T14:32:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:08 smithi079 ceph-mon[23158]: pgmap v1029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1229.876807 secs 2022-03-21T14:32:10.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:10 smithi002 ceph-mon[17167]: pgmap v1030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:10.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:10 smithi016 ceph-mon[22010]: pgmap v1030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:10 smithi079 ceph-mon[23158]: pgmap v1030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:12 smithi002 ceph-mon[17167]: pgmap v1031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:12 smithi016 ceph-mon[22010]: pgmap v1031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:12 smithi079 ceph-mon[23158]: pgmap v1031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1234.876925 secs 2022-03-21T14:32:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1234.876925 secs 2022-03-21T14:32:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1234.876925 secs 2022-03-21T14:32:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:14 smithi002 ceph-mon[17167]: pgmap v1032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:14.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:14 smithi016 ceph-mon[22010]: pgmap v1032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:14 smithi079 ceph-mon[23158]: pgmap v1032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:16.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:16 smithi002 ceph-mon[17167]: pgmap v1033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:16 smithi016 ceph-mon[22010]: pgmap v1033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:16 smithi079 ceph-mon[23158]: pgmap v1033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:18 smithi002 ceph-mon[17167]: pgmap v1034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1239.877059 secs 2022-03-21T14:32:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:18 smithi016 ceph-mon[22010]: pgmap v1034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1239.877059 secs 2022-03-21T14:32:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:18 smithi079 ceph-mon[23158]: pgmap v1034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1239.877059 secs 2022-03-21T14:32:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:20 smithi002 ceph-mon[17167]: pgmap v1035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:20 smithi016 ceph-mon[22010]: pgmap v1035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:20 smithi079 ceph-mon[23158]: pgmap v1035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:32:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:32:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:32:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:32:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:32:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:32:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:22 smithi002 ceph-mon[17167]: pgmap v1036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:22 smithi016 ceph-mon[22010]: pgmap v1036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:22 smithi079 ceph-mon[23158]: pgmap v1036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1244.877140 secs 2022-03-21T14:32:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1244.877140 secs 2022-03-21T14:32:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1244.877140 secs 2022-03-21T14:32:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:24 smithi002 ceph-mon[17167]: pgmap v1037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:24 smithi016 ceph-mon[22010]: pgmap v1037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:24 smithi079 ceph-mon[23158]: pgmap v1037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:26 smithi002 ceph-mon[17167]: pgmap v1038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:26 smithi016 ceph-mon[22010]: pgmap v1038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:26 smithi079 ceph-mon[23158]: pgmap v1038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:28 smithi002 ceph-mon[17167]: pgmap v1039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1249.877257 secs 2022-03-21T14:32:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:28 smithi016 ceph-mon[22010]: pgmap v1039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1249.877257 secs 2022-03-21T14:32:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:28 smithi079 ceph-mon[23158]: pgmap v1039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1249.877257 secs 2022-03-21T14:32:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:30 smithi002 ceph-mon[17167]: pgmap v1040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:30 smithi016 ceph-mon[22010]: pgmap v1040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:30 smithi079 ceph-mon[23158]: pgmap v1040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:32.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:32 smithi002 ceph-mon[17167]: pgmap v1041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:32 smithi016 ceph-mon[22010]: pgmap v1041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:32 smithi079 ceph-mon[23158]: pgmap v1041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1254.877374 secs 2022-03-21T14:32:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1254.877374 secs 2022-03-21T14:32:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1254.877374 secs 2022-03-21T14:32:34.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:34 smithi016 ceph-mon[22010]: pgmap v1042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:34 smithi079 ceph-mon[23158]: pgmap v1042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:34 smithi002 ceph-mon[17167]: pgmap v1042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:36 smithi016 ceph-mon[22010]: pgmap v1043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:36 smithi079 ceph-mon[23158]: pgmap v1043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:36 smithi002 ceph-mon[17167]: pgmap v1043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:38 smithi016 ceph-mon[22010]: pgmap v1044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1259.877457 secs 2022-03-21T14:32:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:38 smithi079 ceph-mon[23158]: pgmap v1044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1259.877457 secs 2022-03-21T14:32:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:38 smithi002 ceph-mon[17167]: pgmap v1044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1259.877457 secs 2022-03-21T14:32:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:40 smithi016 ceph-mon[22010]: pgmap v1045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:40 smithi079 ceph-mon[23158]: pgmap v1045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:40 smithi002 ceph-mon[17167]: pgmap v1045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:42 smithi016 ceph-mon[22010]: pgmap v1046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:42 smithi079 ceph-mon[23158]: pgmap v1046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:42 smithi002 ceph-mon[17167]: pgmap v1046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:43.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1264.877617 secs 2022-03-21T14:32:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1264.877617 secs 2022-03-21T14:32:43.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1264.877617 secs 2022-03-21T14:32:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:44 smithi079 ceph-mon[23158]: pgmap v1047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:32:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:32:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:32:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:44 smithi016 ceph-mon[22010]: pgmap v1047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:32:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:32:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:32:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:44 smithi002 ceph-mon[17167]: pgmap v1047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:32:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:32:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:32:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:32:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:32:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:32:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:46 smithi016 ceph-mon[22010]: pgmap v1048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:46 smithi079 ceph-mon[23158]: pgmap v1048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:46 smithi002 ceph-mon[17167]: pgmap v1048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:48 smithi016 ceph-mon[22010]: pgmap v1049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1269.877752 secs 2022-03-21T14:32:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:48 smithi079 ceph-mon[23158]: pgmap v1049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1269.877752 secs 2022-03-21T14:32:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:48 smithi002 ceph-mon[17167]: pgmap v1049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1269.877752 secs 2022-03-21T14:32:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:50 smithi016 ceph-mon[22010]: pgmap v1050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:50 smithi079 ceph-mon[23158]: pgmap v1050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:50 smithi002 ceph-mon[17167]: pgmap v1050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:52 smithi016 ceph-mon[22010]: pgmap v1051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1274.877868 secs 2022-03-21T14:32:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:52 smithi079 ceph-mon[23158]: pgmap v1051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1274.877868 secs 2022-03-21T14:32:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:52 smithi002 ceph-mon[17167]: pgmap v1051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1274.877868 secs 2022-03-21T14:32:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:54 smithi016 ceph-mon[22010]: pgmap v1052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:54 smithi079 ceph-mon[23158]: pgmap v1052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:54 smithi002 ceph-mon[17167]: pgmap v1052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:56 smithi016 ceph-mon[22010]: pgmap v1053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:56 smithi079 ceph-mon[23158]: pgmap v1053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:56 smithi002 ceph-mon[17167]: pgmap v1053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:32:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1279.877987 secs 2022-03-21T14:32:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1279.877987 secs 2022-03-21T14:32:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1279.877987 secs 2022-03-21T14:32:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:32:58 smithi016 ceph-mon[22010]: pgmap v1054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:32:58 smithi079 ceph-mon[23158]: pgmap v1054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:32:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:32:58 smithi002 ceph-mon[17167]: pgmap v1054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:00 smithi016 ceph-mon[22010]: pgmap v1055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:00 smithi079 ceph-mon[23158]: pgmap v1055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:00 smithi002 ceph-mon[17167]: pgmap v1055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:02 smithi016 ceph-mon[22010]: pgmap v1056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1284.878052 secs 2022-03-21T14:33:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:02 smithi079 ceph-mon[23158]: pgmap v1056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1284.878052 secs 2022-03-21T14:33:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:02 smithi002 ceph-mon[17167]: pgmap v1056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1284.878052 secs 2022-03-21T14:33:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:04 smithi016 ceph-mon[22010]: pgmap v1057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:04 smithi079 ceph-mon[23158]: pgmap v1057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:04 smithi002 ceph-mon[17167]: pgmap v1057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:06 smithi079 ceph-mon[23158]: pgmap v1058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:06 smithi002 ceph-mon[17167]: pgmap v1058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:06 smithi016 ceph-mon[22010]: pgmap v1058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1289.878165 secs 2022-03-21T14:33:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1289.878165 secs 2022-03-21T14:33:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1289.878165 secs 2022-03-21T14:33:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:08 smithi079 ceph-mon[23158]: pgmap v1059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:08 smithi002 ceph-mon[17167]: pgmap v1059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:08 smithi016 ceph-mon[22010]: pgmap v1059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:10 smithi079 ceph-mon[23158]: pgmap v1060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:10 smithi002 ceph-mon[17167]: pgmap v1060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:10 smithi016 ceph-mon[22010]: pgmap v1060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:12 smithi079 ceph-mon[23158]: pgmap v1061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1294.878278 secs 2022-03-21T14:33:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:12 smithi002 ceph-mon[17167]: pgmap v1061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1294.878278 secs 2022-03-21T14:33:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:12 smithi016 ceph-mon[22010]: pgmap v1061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1294.878278 secs 2022-03-21T14:33:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:14 smithi079 ceph-mon[23158]: pgmap v1062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:14 smithi002 ceph-mon[17167]: pgmap v1062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:14 smithi016 ceph-mon[22010]: pgmap v1062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:16 smithi079 ceph-mon[23158]: pgmap v1063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:16 smithi002 ceph-mon[17167]: pgmap v1063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:16 smithi016 ceph-mon[22010]: pgmap v1063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1299.878426 secs 2022-03-21T14:33:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1299.878426 secs 2022-03-21T14:33:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1299.878426 secs 2022-03-21T14:33:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:18 smithi079 ceph-mon[23158]: pgmap v1064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:18 smithi002 ceph-mon[17167]: pgmap v1064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:18 smithi016 ceph-mon[22010]: pgmap v1064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:20 smithi079 ceph-mon[23158]: pgmap v1065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:20 smithi002 ceph-mon[17167]: pgmap v1065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:20 smithi016 ceph-mon[22010]: pgmap v1065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:33:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:33:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:33:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:33:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:33:22.243 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:33:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:22 smithi079 ceph-mon[23158]: pgmap v1066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1304.878571 secs 2022-03-21T14:33:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:22 smithi002 ceph-mon[17167]: pgmap v1066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1304.878571 secs 2022-03-21T14:33:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:22 smithi016 ceph-mon[22010]: pgmap v1066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1304.878571 secs 2022-03-21T14:33:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:24 smithi079 ceph-mon[23158]: pgmap v1067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:24 smithi002 ceph-mon[17167]: pgmap v1067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:24 smithi016 ceph-mon[22010]: pgmap v1067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:26 smithi079 ceph-mon[23158]: pgmap v1068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:26 smithi002 ceph-mon[17167]: pgmap v1068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:26 smithi016 ceph-mon[22010]: pgmap v1068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1309.878681 secs 2022-03-21T14:33:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1309.878681 secs 2022-03-21T14:33:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1309.878681 secs 2022-03-21T14:33:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:28 smithi002 ceph-mon[17167]: pgmap v1069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:28 smithi016 ceph-mon[22010]: pgmap v1069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:28 smithi079 ceph-mon[23158]: pgmap v1069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:30 smithi002 ceph-mon[17167]: pgmap v1070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:30 smithi016 ceph-mon[22010]: pgmap v1070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:30 smithi079 ceph-mon[23158]: pgmap v1070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:32 smithi002 ceph-mon[17167]: pgmap v1071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1314.878796 secs 2022-03-21T14:33:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:32 smithi016 ceph-mon[22010]: pgmap v1071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1314.878796 secs 2022-03-21T14:33:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:32 smithi079 ceph-mon[23158]: pgmap v1071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1314.878796 secs 2022-03-21T14:33:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:34 smithi002 ceph-mon[17167]: pgmap v1072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:34 smithi016 ceph-mon[22010]: pgmap v1072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:34 smithi079 ceph-mon[23158]: pgmap v1072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:36 smithi002 ceph-mon[17167]: pgmap v1073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:36 smithi016 ceph-mon[22010]: pgmap v1073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:36 smithi079 ceph-mon[23158]: pgmap v1073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1319.878912 secs 2022-03-21T14:33:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1319.878912 secs 2022-03-21T14:33:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1319.878912 secs 2022-03-21T14:33:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:38 smithi002 ceph-mon[17167]: pgmap v1074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:38 smithi016 ceph-mon[22010]: pgmap v1074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:38 smithi079 ceph-mon[23158]: pgmap v1074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:40 smithi002 ceph-mon[17167]: pgmap v1075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:40 smithi016 ceph-mon[22010]: pgmap v1075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:40 smithi079 ceph-mon[23158]: pgmap v1075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:42 smithi002 ceph-mon[17167]: pgmap v1076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1324.878974 secs 2022-03-21T14:33:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:42 smithi016 ceph-mon[22010]: pgmap v1076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1324.878974 secs 2022-03-21T14:33:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:42 smithi079 ceph-mon[23158]: pgmap v1076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1324.878974 secs 2022-03-21T14:33:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:44 smithi016 ceph-mon[22010]: pgmap v1077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:33:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:33:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:33:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:44 smithi079 ceph-mon[23158]: pgmap v1077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:33:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:33:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:33:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:44 smithi002 ceph-mon[17167]: pgmap v1077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:33:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:33:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:33:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:46 smithi002 ceph-mon[17167]: pgmap v1078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:33:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:46 smithi016 ceph-mon[22010]: pgmap v1078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:33:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:46 smithi079 ceph-mon[23158]: pgmap v1078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:33:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:48 smithi002 ceph-mon[17167]: pgmap v1079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1329.879090 secs 2022-03-21T14:33:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:48 smithi016 ceph-mon[22010]: pgmap v1079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1329.879090 secs 2022-03-21T14:33:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:48 smithi079 ceph-mon[23158]: pgmap v1079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1329.879090 secs 2022-03-21T14:33:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:50 smithi002 ceph-mon[17167]: pgmap v1080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:50 smithi016 ceph-mon[22010]: pgmap v1080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:50 smithi079 ceph-mon[23158]: pgmap v1080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:52 smithi002 ceph-mon[17167]: pgmap v1081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:52.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:52 smithi079 ceph-mon[23158]: pgmap v1081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:52.983 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:52 smithi016 ceph-mon[22010]: pgmap v1081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1334.879207 secs 2022-03-21T14:33:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1334.879207 secs 2022-03-21T14:33:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1334.879207 secs 2022-03-21T14:33:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:54 smithi002 ceph-mon[17167]: pgmap v1082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:54 smithi016 ceph-mon[22010]: pgmap v1082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:54 smithi079 ceph-mon[23158]: pgmap v1082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:56 smithi002 ceph-mon[17167]: pgmap v1083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:56 smithi016 ceph-mon[22010]: pgmap v1083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:56 smithi079 ceph-mon[23158]: pgmap v1083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:33:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:58 smithi002 ceph-mon[17167]: pgmap v1084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:33:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1339.879315 secs 2022-03-21T14:33:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:58 smithi016 ceph-mon[22010]: pgmap v1084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:33:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1339.879315 secs 2022-03-21T14:33:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:58 smithi079 ceph-mon[23158]: pgmap v1084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:33:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:33:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1339.879315 secs 2022-03-21T14:34:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:00 smithi002 ceph-mon[17167]: pgmap v1085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:00 smithi016 ceph-mon[22010]: pgmap v1085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:00 smithi079 ceph-mon[23158]: pgmap v1085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:02 smithi002 ceph-mon[17167]: pgmap v1086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:02 smithi016 ceph-mon[22010]: pgmap v1086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:02 smithi079 ceph-mon[23158]: pgmap v1086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1344.879428 secs 2022-03-21T14:34:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1344.879428 secs 2022-03-21T14:34:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1344.879428 secs 2022-03-21T14:34:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:04 smithi002 ceph-mon[17167]: pgmap v1087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:04 smithi016 ceph-mon[22010]: pgmap v1087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:04 smithi079 ceph-mon[23158]: pgmap v1087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:06 smithi002 ceph-mon[17167]: pgmap v1088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:06 smithi016 ceph-mon[22010]: pgmap v1088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:06 smithi079 ceph-mon[23158]: pgmap v1088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:08 smithi002 ceph-mon[17167]: pgmap v1089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1349.879581 secs 2022-03-21T14:34:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:08 smithi016 ceph-mon[22010]: pgmap v1089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1349.879581 secs 2022-03-21T14:34:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:08 smithi079 ceph-mon[23158]: pgmap v1089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1349.879581 secs 2022-03-21T14:34:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:10 smithi002 ceph-mon[17167]: pgmap v1090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:10 smithi016 ceph-mon[22010]: pgmap v1090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:10 smithi079 ceph-mon[23158]: pgmap v1090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:12 smithi002 ceph-mon[17167]: pgmap v1091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:12.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:12 smithi016 ceph-mon[22010]: pgmap v1091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:12 smithi079 ceph-mon[23158]: pgmap v1091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1354.879692 secs 2022-03-21T14:34:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1354.879692 secs 2022-03-21T14:34:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1354.879692 secs 2022-03-21T14:34:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:14 smithi002 ceph-mon[17167]: pgmap v1092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:14 smithi016 ceph-mon[22010]: pgmap v1092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:14 smithi079 ceph-mon[23158]: pgmap v1092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:16 smithi002 ceph-mon[17167]: pgmap v1093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:16 smithi016 ceph-mon[22010]: pgmap v1093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:16 smithi079 ceph-mon[23158]: pgmap v1093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:18 smithi002 ceph-mon[17167]: pgmap v1094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1359.879799 secs 2022-03-21T14:34:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:18 smithi016 ceph-mon[22010]: pgmap v1094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1359.879799 secs 2022-03-21T14:34:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:18 smithi079 ceph-mon[23158]: pgmap v1094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1359.879799 secs 2022-03-21T14:34:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:20 smithi002 ceph-mon[17167]: pgmap v1095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:20 smithi016 ceph-mon[22010]: pgmap v1095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:20 smithi079 ceph-mon[23158]: pgmap v1095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:34:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:34:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:34:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:34:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:34:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:34:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:22 smithi002 ceph-mon[17167]: pgmap v1096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:22 smithi016 ceph-mon[22010]: pgmap v1096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:22 smithi079 ceph-mon[23158]: pgmap v1096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1364.879866 secs 2022-03-21T14:34:23.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1364.879866 secs 2022-03-21T14:34:23.765 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1364.879866 secs 2022-03-21T14:34:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:24 smithi002 ceph-mon[17167]: pgmap v1097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:24 smithi016 ceph-mon[22010]: pgmap v1097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:24 smithi079 ceph-mon[23158]: pgmap v1097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:26 smithi002 ceph-mon[17167]: pgmap v1098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:26 smithi016 ceph-mon[22010]: pgmap v1098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:26 smithi079 ceph-mon[23158]: pgmap v1098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:28 smithi002 ceph-mon[17167]: pgmap v1099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1369.880003 secs 2022-03-21T14:34:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:28 smithi016 ceph-mon[22010]: pgmap v1099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1369.880003 secs 2022-03-21T14:34:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:28 smithi079 ceph-mon[23158]: pgmap v1099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1369.880003 secs 2022-03-21T14:34:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:30 smithi002 ceph-mon[17167]: pgmap v1100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:30 smithi016 ceph-mon[22010]: pgmap v1100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:30 smithi079 ceph-mon[23158]: pgmap v1100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:32 smithi002 ceph-mon[17167]: pgmap v1101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:32 smithi016 ceph-mon[22010]: pgmap v1101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:32 smithi079 ceph-mon[23158]: pgmap v1101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1374.880122 secs 2022-03-21T14:34:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1374.880122 secs 2022-03-21T14:34:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1374.880122 secs 2022-03-21T14:34:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:34 smithi002 ceph-mon[17167]: pgmap v1102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:34 smithi016 ceph-mon[22010]: pgmap v1102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:34 smithi079 ceph-mon[23158]: pgmap v1102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:36 smithi002 ceph-mon[17167]: pgmap v1103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:36 smithi016 ceph-mon[22010]: pgmap v1103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:36 smithi079 ceph-mon[23158]: pgmap v1103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:34:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:38 smithi002 ceph-mon[17167]: pgmap v1104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1379.880237 secs 2022-03-21T14:34:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:38 smithi016 ceph-mon[22010]: pgmap v1104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1379.880237 secs 2022-03-21T14:34:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:38 smithi079 ceph-mon[23158]: pgmap v1104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1379.880237 secs 2022-03-21T14:34:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:40 smithi002 ceph-mon[17167]: pgmap v1105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:40 smithi016 ceph-mon[22010]: pgmap v1105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:40 smithi079 ceph-mon[23158]: pgmap v1105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:42 smithi002 ceph-mon[17167]: pgmap v1106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:42 smithi016 ceph-mon[22010]: pgmap v1106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:42 smithi079 ceph-mon[23158]: pgmap v1106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1384.880350 secs 2022-03-21T14:34:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1384.880350 secs 2022-03-21T14:34:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1384.880350 secs 2022-03-21T14:34:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:44 smithi002 ceph-mon[17167]: pgmap v1107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:44 smithi016 ceph-mon[22010]: pgmap v1107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:44 smithi079 ceph-mon[23158]: pgmap v1107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:34:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:34:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:45 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:34:45.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:34:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:34:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:45 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:34:45.682 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:34:45.682 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:34:45.683 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:45 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:34:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:46 smithi016 ceph-mon[22010]: pgmap v1108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:34:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:46 smithi079 ceph-mon[23158]: pgmap v1108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:34:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:46 smithi002 ceph-mon[17167]: pgmap v1108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:34:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:34:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:48 smithi016 ceph-mon[22010]: pgmap v1109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1389.880481 secs 2022-03-21T14:34:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:48 smithi079 ceph-mon[23158]: pgmap v1109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1389.880481 secs 2022-03-21T14:34:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:48 smithi002 ceph-mon[17167]: pgmap v1109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1389.880481 secs 2022-03-21T14:34:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:50 smithi016 ceph-mon[22010]: pgmap v1110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:50 smithi079 ceph-mon[23158]: pgmap v1110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:50 smithi002 ceph-mon[17167]: pgmap v1110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:52 smithi016 ceph-mon[22010]: pgmap v1111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:52 smithi079 ceph-mon[23158]: pgmap v1111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:52 smithi002 ceph-mon[17167]: pgmap v1111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1394.880599 secs 2022-03-21T14:34:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1394.880599 secs 2022-03-21T14:34:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1394.880599 secs 2022-03-21T14:34:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:54 smithi016 ceph-mon[22010]: pgmap v1112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:54 smithi079 ceph-mon[23158]: pgmap v1112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:54 smithi002 ceph-mon[17167]: pgmap v1112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:56 smithi016 ceph-mon[22010]: pgmap v1113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:56 smithi079 ceph-mon[23158]: pgmap v1113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:56 smithi002 ceph-mon[17167]: pgmap v1113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T14:34:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:58 smithi016 ceph-mon[22010]: pgmap v1114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:34:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1399.880731 secs 2022-03-21T14:34:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:58 smithi079 ceph-mon[23158]: pgmap v1114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:34:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1399.880731 secs 2022-03-21T14:34:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:58 smithi002 ceph-mon[17167]: pgmap v1114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:34:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:34:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1399.880731 secs 2022-03-21T14:35:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:00 smithi016 ceph-mon[22010]: pgmap v1115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:00 smithi079 ceph-mon[23158]: pgmap v1115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:00 smithi002 ceph-mon[17167]: pgmap v1115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:02 smithi016 ceph-mon[22010]: pgmap v1116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:02 smithi079 ceph-mon[23158]: pgmap v1116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:02 smithi002 ceph-mon[17167]: pgmap v1116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1404.880808 secs 2022-03-21T14:35:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1404.880808 secs 2022-03-21T14:35:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1404.880808 secs 2022-03-21T14:35:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:04 smithi016 ceph-mon[22010]: pgmap v1117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:04 smithi079 ceph-mon[23158]: pgmap v1117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:04 smithi002 ceph-mon[17167]: pgmap v1117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:06 smithi079 ceph-mon[23158]: pgmap v1118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:06 smithi002 ceph-mon[17167]: pgmap v1118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:06 smithi016 ceph-mon[22010]: pgmap v1118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:07.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1409.880898 secs 2022-03-21T14:35:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1409.880898 secs 2022-03-21T14:35:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1409.880898 secs 2022-03-21T14:35:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:08 smithi079 ceph-mon[23158]: pgmap v1119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:08 smithi002 ceph-mon[17167]: pgmap v1119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:08 smithi016 ceph-mon[22010]: pgmap v1119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:10 smithi079 ceph-mon[23158]: pgmap v1120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:10 smithi002 ceph-mon[17167]: pgmap v1120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:10 smithi016 ceph-mon[22010]: pgmap v1120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:12 smithi079 ceph-mon[23158]: pgmap v1121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1414.881021 secs 2022-03-21T14:35:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:12 smithi002 ceph-mon[17167]: pgmap v1121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1414.881021 secs 2022-03-21T14:35:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:12 smithi016 ceph-mon[22010]: pgmap v1121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1414.881021 secs 2022-03-21T14:35:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:14 smithi079 ceph-mon[23158]: pgmap v1122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:14 smithi002 ceph-mon[17167]: pgmap v1122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:14 smithi016 ceph-mon[22010]: pgmap v1122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:16 smithi079 ceph-mon[23158]: pgmap v1123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:16 smithi002 ceph-mon[17167]: pgmap v1123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:16 smithi016 ceph-mon[22010]: pgmap v1123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1419.881137 secs 2022-03-21T14:35:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1419.881137 secs 2022-03-21T14:35:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1419.881137 secs 2022-03-21T14:35:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:18 smithi079 ceph-mon[23158]: pgmap v1124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:18 smithi002 ceph-mon[17167]: pgmap v1124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:18 smithi016 ceph-mon[22010]: pgmap v1124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:20 smithi079 ceph-mon[23158]: pgmap v1125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:20 smithi002 ceph-mon[17167]: pgmap v1125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:20 smithi016 ceph-mon[22010]: pgmap v1125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:35:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:35:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:35:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:35:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:35:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:35:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:22 smithi079 ceph-mon[23158]: pgmap v1126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1424.881246 secs 2022-03-21T14:35:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:22 smithi002 ceph-mon[17167]: pgmap v1126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1424.881246 secs 2022-03-21T14:35:22.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:22 smithi016 ceph-mon[22010]: pgmap v1126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1424.881246 secs 2022-03-21T14:35:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:24 smithi079 ceph-mon[23158]: pgmap v1127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:24 smithi002 ceph-mon[17167]: pgmap v1127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:25.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:24 smithi016 ceph-mon[22010]: pgmap v1127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:26 smithi079 ceph-mon[23158]: pgmap v1128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:26 smithi002 ceph-mon[17167]: pgmap v1128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:26 smithi016 ceph-mon[22010]: pgmap v1128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1429.881358 secs 2022-03-21T14:35:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1429.881358 secs 2022-03-21T14:35:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1429.881358 secs 2022-03-21T14:35:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:28 smithi079 ceph-mon[23158]: pgmap v1129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:28 smithi002 ceph-mon[17167]: pgmap v1129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:28 smithi016 ceph-mon[22010]: pgmap v1129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:30 smithi002 ceph-mon[17167]: pgmap v1130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:30 smithi016 ceph-mon[22010]: pgmap v1130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:30 smithi079 ceph-mon[23158]: pgmap v1130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:32 smithi002 ceph-mon[17167]: pgmap v1131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1434.881475 secs 2022-03-21T14:35:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:32 smithi016 ceph-mon[22010]: pgmap v1131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1434.881475 secs 2022-03-21T14:35:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:32 smithi079 ceph-mon[23158]: pgmap v1131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1434.881475 secs 2022-03-21T14:35:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:34 smithi002 ceph-mon[17167]: pgmap v1132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:34 smithi016 ceph-mon[22010]: pgmap v1132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:34 smithi079 ceph-mon[23158]: pgmap v1132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:36 smithi002 ceph-mon[17167]: pgmap v1133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:36 smithi016 ceph-mon[22010]: pgmap v1133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:36 smithi079 ceph-mon[23158]: pgmap v1133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1439.881583 secs 2022-03-21T14:35:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1439.881583 secs 2022-03-21T14:35:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1439.881583 secs 2022-03-21T14:35:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:38 smithi002 ceph-mon[17167]: pgmap v1134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:38 smithi016 ceph-mon[22010]: pgmap v1134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:38 smithi079 ceph-mon[23158]: pgmap v1134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:40 smithi002 ceph-mon[17167]: pgmap v1135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:41.006 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:40 smithi079 ceph-mon[23158]: pgmap v1135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:41.006 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:40 smithi016 ceph-mon[22010]: pgmap v1135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: pgmap v1136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1444.881668 secs 2022-03-21T14:35:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: slow request 1444.776897 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: slow request 1444.565713 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: slow request 1444.878307 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: slow request 1444.829169 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:42 smithi002 ceph-mon[17167]: slow request 1444.803909 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: pgmap v1136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1444.881668 secs 2022-03-21T14:35:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: slow request 1444.776897 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: slow request 1444.565713 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: slow request 1444.878307 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: slow request 1444.829169 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:42 smithi016 ceph-mon[22010]: slow request 1444.803909 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: pgmap v1136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1444.881668 secs 2022-03-21T14:35:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: slow request 1444.776897 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: slow request 1444.565713 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: slow request 1444.878307 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: slow request 1444.829169 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:42 smithi079 ceph-mon[23158]: slow request 1444.803909 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:44 smithi002 ceph-mon[17167]: pgmap v1137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:44 smithi016 ceph-mon[22010]: pgmap v1137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:44 smithi079 ceph-mon[23158]: pgmap v1137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:46 smithi002 ceph-mon[17167]: pgmap v1138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:35:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:35:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:35:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:46 smithi016 ceph-mon[22010]: pgmap v1138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:35:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:35:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:35:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:46 smithi079 ceph-mon[23158]: pgmap v1138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:35:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:35:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:35:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1449.881810 secs 2022-03-21T14:35:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: slow request 1449.773772 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: slow request 1449.853104 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: slow request 1449.770813 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: slow request 1449.875009 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:47 smithi016 ceph-mon[22010]: slow request 1449.826034 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1449.881810 secs 2022-03-21T14:35:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: slow request 1449.773772 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: slow request 1449.853104 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: slow request 1449.770813 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: slow request 1449.875009 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:47 smithi002 ceph-mon[17167]: slow request 1449.826034 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1449.881810 secs 2022-03-21T14:35:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: slow request 1449.773772 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: slow request 1449.853104 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: slow request 1449.770813 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: slow request 1449.875009 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:47 smithi079 ceph-mon[23158]: slow request 1449.826034 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:48.848 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:48 smithi002 ceph-mon[17167]: pgmap v1139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:48 smithi016 ceph-mon[22010]: pgmap v1139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:48 smithi079 ceph-mon[23158]: pgmap v1139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:50 smithi079 ceph-mon[23158]: pgmap v1140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:50 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T14:35:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:50 smithi002 ceph-mon[17167]: pgmap v1140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:50 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T14:35:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:50 smithi016 ceph-mon[22010]: pgmap v1140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:50 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T14:35:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: pgmap v1141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1454.881889 secs 2022-03-21T14:35:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: slow request 1454.767873 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: slow request 1454.850458 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: slow request 1454.800032 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: slow request 1454.765127 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:52 smithi079 ceph-mon[23158]: slow request 1454.871910 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: pgmap v1141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:52.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1454.881889 secs 2022-03-21T14:35:52.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: slow request 1454.767873 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: slow request 1454.850458 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: slow request 1454.800032 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: slow request 1454.765127 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:52 smithi002 ceph-mon[17167]: slow request 1454.871910 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: pgmap v1141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1454.881889 secs 2022-03-21T14:35:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: slow request 1454.767873 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: slow request 1454.850458 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: slow request 1454.800032 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: slow request 1454.765127 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:52 smithi016 ceph-mon[22010]: slow request 1454.871910 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:54.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:54 smithi002 ceph-mon[17167]: pgmap v1142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:54 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T14:35:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:54.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:54 smithi016 ceph-mon[22010]: pgmap v1142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:54 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T14:35:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:54 smithi079 ceph-mon[23158]: pgmap v1142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:54 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T14:35:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:35:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:56 smithi002 ceph-mon[17167]: pgmap v1143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:56 smithi016 ceph-mon[22010]: pgmap v1143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:56 smithi079 ceph-mon[23158]: pgmap v1143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:35:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: pgmap v1144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1459.881980 secs 2022-03-21T14:35:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: slow request 1459.847738 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: slow request 1459.822885 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: slow request 1459.795973 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: slow request 1459.869085 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:35:58 smithi002 ceph-mon[17167]: slow request 1459.844671 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: pgmap v1144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1459.881980 secs 2022-03-21T14:35:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: slow request 1459.847738 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: slow request 1459.822885 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: slow request 1459.795973 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: slow request 1459.869085 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:35:58 smithi016 ceph-mon[22010]: slow request 1459.844671 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: pgmap v1144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:35:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1459.881980 secs 2022-03-21T14:35:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: slow request 1459.847738 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: slow request 1459.822885 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: slow request 1459.795973 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: slow request 1459.869085 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:35:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:35:58 smithi079 ceph-mon[23158]: slow request 1459.844671 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:00 smithi002 ceph-mon[17167]: pgmap v1145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:00 smithi016 ceph-mon[22010]: pgmap v1145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:00 smithi079 ceph-mon[23158]: pgmap v1145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:02.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:02 smithi002 ceph-mon[17167]: pgmap v1146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:02.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:02 smithi016 ceph-mon[22010]: pgmap v1146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:02 smithi079 ceph-mon[23158]: pgmap v1146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1464.882095 secs 2022-03-21T14:36:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: slow request 1464.820196 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: slow request 1464.793351 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: slow request 1464.817233 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: slow request 1464.865814 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:03 smithi016 ceph-mon[22010]: slow request 1464.842111 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1464.882095 secs 2022-03-21T14:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: slow request 1464.820196 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: slow request 1464.793351 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: slow request 1464.817233 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: slow request 1464.865814 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:03 smithi079 ceph-mon[23158]: slow request 1464.842111 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1464.882095 secs 2022-03-21T14:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: slow request 1464.820196 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: slow request 1464.793351 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: slow request 1464.817233 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: slow request 1464.865814 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:03 smithi002 ceph-mon[17167]: slow request 1464.842111 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:04 smithi016 ceph-mon[22010]: pgmap v1147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:04 smithi079 ceph-mon[23158]: pgmap v1147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:04 smithi002 ceph-mon[17167]: pgmap v1147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:06 smithi016 ceph-mon[22010]: pgmap v1148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:06 smithi079 ceph-mon[23158]: pgmap v1148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:06 smithi002 ceph-mon[17167]: pgmap v1148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: pgmap v1149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1469.882237 secs 2022-03-21T14:36:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: slow request 1469.790299 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: slow request 1469.839404 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: slow request 1469.814300 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: slow request 1469.862895 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:08 smithi016 ceph-mon[22010]: slow request 1469.786538 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: pgmap v1149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1469.882237 secs 2022-03-21T14:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: slow request 1469.790299 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: slow request 1469.839404 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: slow request 1469.814300 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: slow request 1469.862895 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:08 smithi079 ceph-mon[23158]: slow request 1469.786538 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: pgmap v1149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1469.882237 secs 2022-03-21T14:36:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: slow request 1469.790299 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: slow request 1469.839404 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: slow request 1469.814300 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: slow request 1469.862895 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:08 smithi002 ceph-mon[17167]: slow request 1469.786538 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:10 smithi016 ceph-mon[22010]: pgmap v1150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:10 smithi079 ceph-mon[23158]: pgmap v1150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:10 smithi002 ceph-mon[17167]: pgmap v1150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:12 smithi016 ceph-mon[22010]: pgmap v1151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:12 smithi079 ceph-mon[23158]: pgmap v1151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:12 smithi002 ceph-mon[17167]: pgmap v1151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1474.882364 secs 2022-03-21T14:36:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: slow request 1474.766257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:36:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: slow request 1474.783662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: slow request 1474.859563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: slow request 1474.836161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:13 smithi016 ceph-mon[22010]: slow request 1474.811104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1474.882364 secs 2022-03-21T14:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: slow request 1474.766257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: slow request 1474.783662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: slow request 1474.859563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: slow request 1474.836161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:13 smithi079 ceph-mon[23158]: slow request 1474.811104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1474.882364 secs 2022-03-21T14:36:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: slow request 1474.766257 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T14:36:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: slow request 1474.783662 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: slow request 1474.859563 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: slow request 1474.836161 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:13 smithi002 ceph-mon[17167]: slow request 1474.811104 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:14 smithi016 ceph-mon[22010]: pgmap v1152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:14 smithi079 ceph-mon[23158]: pgmap v1152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:14 smithi002 ceph-mon[17167]: pgmap v1152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:16 smithi016 ceph-mon[22010]: pgmap v1153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:16 smithi079 ceph-mon[23158]: pgmap v1153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:16 smithi002 ceph-mon[17167]: pgmap v1153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: pgmap v1154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 1479.882476 secs 2022-03-21T14:36:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: slow request 1479.780452 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: slow request 1479.882475 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: slow request 1479.833253 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: slow request 1479.808302 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:18 smithi016 ceph-mon[22010]: slow request 1479.856750 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: pgmap v1154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 1479.882476 secs 2022-03-21T14:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: slow request 1479.780452 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: slow request 1479.882475 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: slow request 1479.833253 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: slow request 1479.808302 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:18 smithi079 ceph-mon[23158]: slow request 1479.856750 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: pgmap v1154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 1479.882476 secs 2022-03-21T14:36:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: slow request 1479.780452 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: slow request 1479.882475 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: slow request 1479.833253 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: slow request 1479.808302 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:18 smithi002 ceph-mon[17167]: slow request 1479.856750 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:36:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:20 smithi016 ceph-mon[22010]: pgmap v1155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:20 smithi079 ceph-mon[23158]: pgmap v1155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:20 smithi002 ceph-mon[17167]: pgmap v1155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:36:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:36:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:36:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:36:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:36:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:36:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:22 smithi016 ceph-mon[22010]: pgmap v1156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:22 smithi079 ceph-mon[23158]: pgmap v1156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:22 smithi002 ceph-mon[17167]: pgmap v1156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1484.882575 secs 2022-03-21T14:36:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1484.882575 secs 2022-03-21T14:36:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1484.882575 secs 2022-03-21T14:36:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:24 smithi079 ceph-mon[23158]: pgmap v1157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:24 smithi002 ceph-mon[17167]: pgmap v1157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:24 smithi016 ceph-mon[22010]: pgmap v1157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:26 smithi079 ceph-mon[23158]: pgmap v1158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:26 smithi002 ceph-mon[17167]: pgmap v1158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:26 smithi016 ceph-mon[22010]: pgmap v1158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:28 smithi079 ceph-mon[23158]: pgmap v1159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1489.882690 secs 2022-03-21T14:36:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:28 smithi002 ceph-mon[17167]: pgmap v1159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1489.882690 secs 2022-03-21T14:36:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:28 smithi016 ceph-mon[22010]: pgmap v1159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1489.882690 secs 2022-03-21T14:36:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:30 smithi079 ceph-mon[23158]: pgmap v1160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:30 smithi002 ceph-mon[17167]: pgmap v1160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:30 smithi016 ceph-mon[22010]: pgmap v1160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:32 smithi079 ceph-mon[23158]: pgmap v1161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:32 smithi002 ceph-mon[17167]: pgmap v1161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:32 smithi016 ceph-mon[22010]: pgmap v1161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1494.882808 secs 2022-03-21T14:36:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1494.882808 secs 2022-03-21T14:36:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1494.882808 secs 2022-03-21T14:36:34.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:34 smithi079 ceph-mon[23158]: pgmap v1162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:34 smithi002 ceph-mon[17167]: pgmap v1162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:34 smithi016 ceph-mon[22010]: pgmap v1162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:36.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:36 smithi079 ceph-mon[23158]: pgmap v1163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:36 smithi002 ceph-mon[17167]: pgmap v1163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:36 smithi016 ceph-mon[22010]: pgmap v1163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:38 smithi079 ceph-mon[23158]: pgmap v1164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1499.882871 secs 2022-03-21T14:36:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:38 smithi002 ceph-mon[17167]: pgmap v1164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1499.882871 secs 2022-03-21T14:36:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:38 smithi016 ceph-mon[22010]: pgmap v1164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1499.882871 secs 2022-03-21T14:36:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:40 smithi079 ceph-mon[23158]: pgmap v1165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:40 smithi002 ceph-mon[17167]: pgmap v1165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:40 smithi016 ceph-mon[22010]: pgmap v1165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:42.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:42 smithi079 ceph-mon[23158]: pgmap v1166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:42 smithi002 ceph-mon[17167]: pgmap v1166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:42 smithi016 ceph-mon[22010]: pgmap v1166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1504.882988 secs 2022-03-21T14:36:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1504.882988 secs 2022-03-21T14:36:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1504.882988 secs 2022-03-21T14:36:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:44 smithi079 ceph-mon[23158]: pgmap v1167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:44 smithi002 ceph-mon[17167]: pgmap v1167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:44 smithi016 ceph-mon[22010]: pgmap v1167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:46.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:46 smithi079 ceph-mon[23158]: pgmap v1168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:46 smithi002 ceph-mon[17167]: pgmap v1168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:46 smithi016 ceph-mon[22010]: pgmap v1168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:48 smithi079 ceph-mon[23158]: pgmap v1169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1509.883094 secs 2022-03-21T14:36:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:48 smithi002 ceph-mon[17167]: pgmap v1169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1509.883094 secs 2022-03-21T14:36:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:48 smithi016 ceph-mon[22010]: pgmap v1169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1509.883094 secs 2022-03-21T14:36:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:50 smithi002 ceph-mon[17167]: pgmap v1170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:50 smithi016 ceph-mon[22010]: pgmap v1170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:50 smithi079 ceph-mon[23158]: pgmap v1170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:52 smithi002 ceph-mon[17167]: pgmap v1171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:52 smithi016 ceph-mon[22010]: pgmap v1171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:52 smithi079 ceph-mon[23158]: pgmap v1171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1514.883217 secs 2022-03-21T14:36:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:36:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:36:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:36:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1514.883217 secs 2022-03-21T14:36:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:36:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1514.883217 secs 2022-03-21T14:36:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:36:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:36:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:36:54.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: pgmap v1172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: pgmap v1172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:36:54.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:36:54.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: pgmap v1172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:36:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:36:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:55 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:36:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:55 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:36:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:55 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:36:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:55 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:36:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:55 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:36:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:55 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:36:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: pgmap v1173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: pgmap v1173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: pgmap v1173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:36:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:36:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:36:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:57 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T14:36:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:57 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:36:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:57 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:36:57.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:57 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T14:36:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:57 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:36:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:57 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:36:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:57 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T14:36:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:57 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:36:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:57 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:36:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:58 smithi002 ceph-mon[17167]: pgmap v1174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:36:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1519.883327 secs 2022-03-21T14:36:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:58 smithi016 ceph-mon[22010]: pgmap v1174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:36:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1519.883327 secs 2022-03-21T14:36:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:58 smithi079 ceph-mon[23158]: pgmap v1174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:36:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:36:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1519.883327 secs 2022-03-21T14:37:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:00 smithi002 ceph-mon[17167]: pgmap v1175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:00 smithi016 ceph-mon[22010]: pgmap v1175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:00 smithi079 ceph-mon[23158]: pgmap v1175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:02 smithi002 ceph-mon[17167]: pgmap v1176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:02 smithi016 ceph-mon[22010]: pgmap v1176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:02 smithi079 ceph-mon[23158]: pgmap v1176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1524.883425 secs 2022-03-21T14:37:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1524.883425 secs 2022-03-21T14:37:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1524.883425 secs 2022-03-21T14:37:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:04 smithi002 ceph-mon[17167]: pgmap v1177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:04 smithi016 ceph-mon[22010]: pgmap v1177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:04 smithi079 ceph-mon[23158]: pgmap v1177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:06 smithi002 ceph-mon[17167]: pgmap v1178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:06 smithi016 ceph-mon[22010]: pgmap v1178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:06 smithi079 ceph-mon[23158]: pgmap v1178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:08 smithi002 ceph-mon[17167]: pgmap v1179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1529.883519 secs 2022-03-21T14:37:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:08 smithi016 ceph-mon[22010]: pgmap v1179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1529.883519 secs 2022-03-21T14:37:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:08 smithi079 ceph-mon[23158]: pgmap v1179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1529.883519 secs 2022-03-21T14:37:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:10 smithi002 ceph-mon[17167]: pgmap v1180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:10 smithi016 ceph-mon[22010]: pgmap v1180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:10 smithi079 ceph-mon[23158]: pgmap v1180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:12 smithi002 ceph-mon[17167]: pgmap v1181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:12.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:12 smithi016 ceph-mon[22010]: pgmap v1181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:12 smithi079 ceph-mon[23158]: pgmap v1181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1534.883614 secs 2022-03-21T14:37:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1534.883614 secs 2022-03-21T14:37:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1534.883614 secs 2022-03-21T14:37:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:14 smithi002 ceph-mon[17167]: pgmap v1182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:14 smithi016 ceph-mon[22010]: pgmap v1182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:14 smithi079 ceph-mon[23158]: pgmap v1182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:16 smithi002 ceph-mon[17167]: pgmap v1183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:16 smithi016 ceph-mon[22010]: pgmap v1183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:16 smithi079 ceph-mon[23158]: pgmap v1183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:18 smithi002 ceph-mon[17167]: pgmap v1184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1539.883730 secs 2022-03-21T14:37:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:18 smithi016 ceph-mon[22010]: pgmap v1184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1539.883730 secs 2022-03-21T14:37:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:18 smithi079 ceph-mon[23158]: pgmap v1184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1539.883730 secs 2022-03-21T14:37:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:20 smithi002 ceph-mon[17167]: pgmap v1185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:20 smithi016 ceph-mon[22010]: pgmap v1185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:20 smithi079 ceph-mon[23158]: pgmap v1185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:37:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:37:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:37:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:37:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:37:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:37:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:22 smithi002 ceph-mon[17167]: pgmap v1186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:22 smithi016 ceph-mon[22010]: pgmap v1186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:22 smithi079 ceph-mon[23158]: pgmap v1186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1544.883880 secs 2022-03-21T14:37:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1544.883880 secs 2022-03-21T14:37:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1544.883880 secs 2022-03-21T14:37:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:24 smithi002 ceph-mon[17167]: pgmap v1187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:24 smithi016 ceph-mon[22010]: pgmap v1187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:24 smithi079 ceph-mon[23158]: pgmap v1187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:26 smithi002 ceph-mon[17167]: pgmap v1188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:26 smithi016 ceph-mon[22010]: pgmap v1188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:26 smithi079 ceph-mon[23158]: pgmap v1188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:28 smithi002 ceph-mon[17167]: pgmap v1189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1549.884000 secs 2022-03-21T14:37:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:28 smithi016 ceph-mon[22010]: pgmap v1189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1549.884000 secs 2022-03-21T14:37:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:28 smithi079 ceph-mon[23158]: pgmap v1189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1549.884000 secs 2022-03-21T14:37:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:30 smithi002 ceph-mon[17167]: pgmap v1190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:30 smithi016 ceph-mon[22010]: pgmap v1190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:30 smithi079 ceph-mon[23158]: pgmap v1190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:32 smithi002 ceph-mon[17167]: pgmap v1191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:32 smithi016 ceph-mon[22010]: pgmap v1191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:32 smithi079 ceph-mon[23158]: pgmap v1191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1554.884136 secs 2022-03-21T14:37:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1554.884136 secs 2022-03-21T14:37:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1554.884136 secs 2022-03-21T14:37:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:34 smithi002 ceph-mon[17167]: pgmap v1192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:34 smithi016 ceph-mon[22010]: pgmap v1192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:34 smithi079 ceph-mon[23158]: pgmap v1192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:36 smithi002 ceph-mon[17167]: pgmap v1193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:36 smithi016 ceph-mon[22010]: pgmap v1193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:36 smithi079 ceph-mon[23158]: pgmap v1193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:38 smithi002 ceph-mon[17167]: pgmap v1194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1559.884244 secs 2022-03-21T14:37:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:38 smithi016 ceph-mon[22010]: pgmap v1194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1559.884244 secs 2022-03-21T14:37:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:38 smithi079 ceph-mon[23158]: pgmap v1194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1559.884244 secs 2022-03-21T14:37:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:40 smithi002 ceph-mon[17167]: pgmap v1195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:40 smithi016 ceph-mon[22010]: pgmap v1195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:40 smithi079 ceph-mon[23158]: pgmap v1195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:42 smithi002 ceph-mon[17167]: pgmap v1196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:42 smithi016 ceph-mon[22010]: pgmap v1196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:42 smithi079 ceph-mon[23158]: pgmap v1196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1564.884374 secs 2022-03-21T14:37:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1564.884374 secs 2022-03-21T14:37:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1564.884374 secs 2022-03-21T14:37:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:44 smithi002 ceph-mon[17167]: pgmap v1197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:44 smithi016 ceph-mon[22010]: pgmap v1197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:44 smithi079 ceph-mon[23158]: pgmap v1197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:46 smithi002 ceph-mon[17167]: pgmap v1198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:46 smithi016 ceph-mon[22010]: pgmap v1198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:46 smithi079 ceph-mon[23158]: pgmap v1198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:48 smithi016 ceph-mon[22010]: pgmap v1199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1569.884464 secs 2022-03-21T14:37:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:48 smithi079 ceph-mon[23158]: pgmap v1199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1569.884464 secs 2022-03-21T14:37:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:48 smithi002 ceph-mon[17167]: pgmap v1199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1569.884464 secs 2022-03-21T14:37:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:50 smithi016 ceph-mon[22010]: pgmap v1200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:50 smithi079 ceph-mon[23158]: pgmap v1200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:50 smithi002 ceph-mon[17167]: pgmap v1200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:52 smithi016 ceph-mon[22010]: pgmap v1201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:52 smithi079 ceph-mon[23158]: pgmap v1201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:52 smithi002 ceph-mon[17167]: pgmap v1201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1574.884576 secs 2022-03-21T14:37:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1574.884576 secs 2022-03-21T14:37:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1574.884576 secs 2022-03-21T14:37:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:54 smithi016 ceph-mon[22010]: pgmap v1202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:54 smithi079 ceph-mon[23158]: pgmap v1202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:54 smithi002 ceph-mon[17167]: pgmap v1202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:56 smithi002 ceph-mon[17167]: pgmap v1203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:37:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:37:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:37:56.631 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:56 smithi079 ceph-mon[23158]: pgmap v1203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:56.632 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:37:56.632 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:37:56.632 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:37:56.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:56 smithi016 ceph-mon[22010]: pgmap v1203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:37:56.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:37:56.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:37:56.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:37:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:58 smithi002 ceph-mon[17167]: pgmap v1204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:37:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:37:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1579.884685 secs 2022-03-21T14:37:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:58 smithi016 ceph-mon[22010]: pgmap v1204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:37:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:37:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1579.884685 secs 2022-03-21T14:37:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:58 smithi079 ceph-mon[23158]: pgmap v1204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:37:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:37:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:37:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1579.884685 secs 2022-03-21T14:38:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:00 smithi016 ceph-mon[22010]: pgmap v1205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:00 smithi079 ceph-mon[23158]: pgmap v1205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:00 smithi002 ceph-mon[17167]: pgmap v1205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:02 smithi016 ceph-mon[22010]: pgmap v1206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:02.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:02 smithi079 ceph-mon[23158]: pgmap v1206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:02 smithi002 ceph-mon[17167]: pgmap v1206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1584.884762 secs 2022-03-21T14:38:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1584.884762 secs 2022-03-21T14:38:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1584.884762 secs 2022-03-21T14:38:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:04 smithi016 ceph-mon[22010]: pgmap v1207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:04.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:04 smithi079 ceph-mon[23158]: pgmap v1207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:04 smithi002 ceph-mon[17167]: pgmap v1207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:06.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:06 smithi016 ceph-mon[22010]: pgmap v1208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:06.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:06 smithi079 ceph-mon[23158]: pgmap v1208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:06 smithi002 ceph-mon[17167]: pgmap v1208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:08 smithi016 ceph-mon[22010]: pgmap v1209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1589.884858 secs 2022-03-21T14:38:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:08 smithi079 ceph-mon[23158]: pgmap v1209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1589.884858 secs 2022-03-21T14:38:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:08 smithi002 ceph-mon[17167]: pgmap v1209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1589.884858 secs 2022-03-21T14:38:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:10 smithi079 ceph-mon[23158]: pgmap v1210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:10 smithi002 ceph-mon[17167]: pgmap v1210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:10 smithi016 ceph-mon[22010]: pgmap v1210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:12 smithi079 ceph-mon[23158]: pgmap v1211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:12 smithi002 ceph-mon[17167]: pgmap v1211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:12 smithi016 ceph-mon[22010]: pgmap v1211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1594.884961 secs 2022-03-21T14:38:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1594.884961 secs 2022-03-21T14:38:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1594.884961 secs 2022-03-21T14:38:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:14 smithi079 ceph-mon[23158]: pgmap v1212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:14 smithi002 ceph-mon[17167]: pgmap v1212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:14 smithi016 ceph-mon[22010]: pgmap v1212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:16 smithi079 ceph-mon[23158]: pgmap v1213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:16 smithi002 ceph-mon[17167]: pgmap v1213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:16 smithi016 ceph-mon[22010]: pgmap v1213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:18 smithi079 ceph-mon[23158]: pgmap v1214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1599.885057 secs 2022-03-21T14:38:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:18 smithi002 ceph-mon[17167]: pgmap v1214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1599.885057 secs 2022-03-21T14:38:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:18 smithi016 ceph-mon[22010]: pgmap v1214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1599.885057 secs 2022-03-21T14:38:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:20 smithi079 ceph-mon[23158]: pgmap v1215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:20 smithi002 ceph-mon[17167]: pgmap v1215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:20 smithi016 ceph-mon[22010]: pgmap v1215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:38:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:38:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:38:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:38:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:38:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:38:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:22 smithi079 ceph-mon[23158]: pgmap v1216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:22 smithi002 ceph-mon[17167]: pgmap v1216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:22 smithi016 ceph-mon[22010]: pgmap v1216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1604.885193 secs 2022-03-21T14:38:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1604.885193 secs 2022-03-21T14:38:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1604.885193 secs 2022-03-21T14:38:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:24 smithi079 ceph-mon[23158]: pgmap v1217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:24 smithi002 ceph-mon[17167]: pgmap v1217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:24 smithi016 ceph-mon[22010]: pgmap v1217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:26 smithi079 ceph-mon[23158]: pgmap v1218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:26 smithi002 ceph-mon[17167]: pgmap v1218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:26 smithi016 ceph-mon[22010]: pgmap v1218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:28 smithi079 ceph-mon[23158]: pgmap v1219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1609.885307 secs 2022-03-21T14:38:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:28 smithi002 ceph-mon[17167]: pgmap v1219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1609.885307 secs 2022-03-21T14:38:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:28 smithi016 ceph-mon[22010]: pgmap v1219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1609.885307 secs 2022-03-21T14:38:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:30 smithi079 ceph-mon[23158]: pgmap v1220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:30 smithi002 ceph-mon[17167]: pgmap v1220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:30 smithi016 ceph-mon[22010]: pgmap v1220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:32.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:32 smithi079 ceph-mon[23158]: pgmap v1221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:32 smithi002 ceph-mon[17167]: pgmap v1221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:32 smithi016 ceph-mon[22010]: pgmap v1221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1614.885441 secs 2022-03-21T14:38:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1614.885441 secs 2022-03-21T14:38:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1614.885441 secs 2022-03-21T14:38:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:34 smithi002 ceph-mon[17167]: pgmap v1222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:34 smithi016 ceph-mon[22010]: pgmap v1222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:34 smithi079 ceph-mon[23158]: pgmap v1222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:36 smithi002 ceph-mon[17167]: pgmap v1223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:36 smithi016 ceph-mon[22010]: pgmap v1223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:36 smithi079 ceph-mon[23158]: pgmap v1223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:38 smithi002 ceph-mon[17167]: pgmap v1224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1619.885532 secs 2022-03-21T14:38:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:38 smithi016 ceph-mon[22010]: pgmap v1224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1619.885532 secs 2022-03-21T14:38:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:38 smithi079 ceph-mon[23158]: pgmap v1224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1619.885532 secs 2022-03-21T14:38:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:40 smithi002 ceph-mon[17167]: pgmap v1225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:40 smithi016 ceph-mon[22010]: pgmap v1225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:40 smithi079 ceph-mon[23158]: pgmap v1225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:42 smithi002 ceph-mon[17167]: pgmap v1226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:42 smithi016 ceph-mon[22010]: pgmap v1226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:42 smithi079 ceph-mon[23158]: pgmap v1226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1624.885620 secs 2022-03-21T14:38:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1624.885620 secs 2022-03-21T14:38:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1624.885620 secs 2022-03-21T14:38:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:44 smithi002 ceph-mon[17167]: pgmap v1227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:44 smithi016 ceph-mon[22010]: pgmap v1227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:44 smithi079 ceph-mon[23158]: pgmap v1227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:46 smithi002 ceph-mon[17167]: pgmap v1228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:46 smithi016 ceph-mon[22010]: pgmap v1228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:46 smithi079 ceph-mon[23158]: pgmap v1228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:48 smithi002 ceph-mon[17167]: pgmap v1229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1629.885695 secs 2022-03-21T14:38:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:48 smithi016 ceph-mon[22010]: pgmap v1229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1629.885695 secs 2022-03-21T14:38:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:48 smithi079 ceph-mon[23158]: pgmap v1229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1629.885695 secs 2022-03-21T14:38:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:50 smithi002 ceph-mon[17167]: pgmap v1230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:50 smithi016 ceph-mon[22010]: pgmap v1230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:50 smithi079 ceph-mon[23158]: pgmap v1230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:52 smithi002 ceph-mon[17167]: pgmap v1231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:52 smithi016 ceph-mon[22010]: pgmap v1231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:52 smithi079 ceph-mon[23158]: pgmap v1231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1634.885772 secs 2022-03-21T14:38:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1634.885772 secs 2022-03-21T14:38:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1634.885772 secs 2022-03-21T14:38:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:54 smithi002 ceph-mon[17167]: pgmap v1232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:54 smithi016 ceph-mon[22010]: pgmap v1232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:54 smithi079 ceph-mon[23158]: pgmap v1232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:56 smithi002 ceph-mon[17167]: pgmap v1233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:56.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:56 smithi016 ceph-mon[22010]: pgmap v1233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:56 smithi079 ceph-mon[23158]: pgmap v1233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:38:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:38:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:38:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:38:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:38:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:38:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:38:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:38:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:38:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:38:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:58 smithi002 ceph-mon[17167]: pgmap v1234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1639.885836 secs 2022-03-21T14:38:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:38:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:38:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:58 smithi016 ceph-mon[22010]: pgmap v1234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1639.885836 secs 2022-03-21T14:38:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:38:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:38:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:58 smithi079 ceph-mon[23158]: pgmap v1234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:38:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1639.885836 secs 2022-03-21T14:38:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:38:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:39:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:00 smithi002 ceph-mon[17167]: pgmap v1235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:00 smithi016 ceph-mon[22010]: pgmap v1235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:00 smithi079 ceph-mon[23158]: pgmap v1235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:02 smithi002 ceph-mon[17167]: pgmap v1236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:02 smithi016 ceph-mon[22010]: pgmap v1236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:02 smithi079 ceph-mon[23158]: pgmap v1236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1644.885933 secs 2022-03-21T14:39:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1644.885933 secs 2022-03-21T14:39:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1644.885933 secs 2022-03-21T14:39:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:04 smithi002 ceph-mon[17167]: pgmap v1237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:04 smithi016 ceph-mon[22010]: pgmap v1237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:04 smithi079 ceph-mon[23158]: pgmap v1237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:06 smithi002 ceph-mon[17167]: pgmap v1238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:06 smithi016 ceph-mon[22010]: pgmap v1238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:06 smithi079 ceph-mon[23158]: pgmap v1238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:08 smithi002 ceph-mon[17167]: pgmap v1239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1649.886056 secs 2022-03-21T14:39:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:08 smithi016 ceph-mon[22010]: pgmap v1239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1649.886056 secs 2022-03-21T14:39:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:08 smithi079 ceph-mon[23158]: pgmap v1239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1649.886056 secs 2022-03-21T14:39:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:10 smithi002 ceph-mon[17167]: pgmap v1240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:10 smithi016 ceph-mon[22010]: pgmap v1240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:10 smithi079 ceph-mon[23158]: pgmap v1240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:12 smithi002 ceph-mon[17167]: pgmap v1241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:12.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:12 smithi016 ceph-mon[22010]: pgmap v1241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:12 smithi079 ceph-mon[23158]: pgmap v1241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1654.886194 secs 2022-03-21T14:39:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1654.886194 secs 2022-03-21T14:39:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1654.886194 secs 2022-03-21T14:39:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:14 smithi002 ceph-mon[17167]: pgmap v1242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:14 smithi016 ceph-mon[22010]: pgmap v1242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:14 smithi079 ceph-mon[23158]: pgmap v1242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:16 smithi002 ceph-mon[17167]: pgmap v1243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:16 smithi016 ceph-mon[22010]: pgmap v1243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:16 smithi079 ceph-mon[23158]: pgmap v1243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:18 smithi002 ceph-mon[17167]: pgmap v1244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1659.886244 secs 2022-03-21T14:39:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:18 smithi016 ceph-mon[22010]: pgmap v1244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1659.886244 secs 2022-03-21T14:39:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:18 smithi079 ceph-mon[23158]: pgmap v1244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1659.886244 secs 2022-03-21T14:39:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:20 smithi002 ceph-mon[17167]: pgmap v1245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:20 smithi016 ceph-mon[22010]: pgmap v1245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:20 smithi079 ceph-mon[23158]: pgmap v1245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:39:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:39:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:39:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:39:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:39:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:39:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:22 smithi002 ceph-mon[17167]: pgmap v1246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:22 smithi016 ceph-mon[22010]: pgmap v1246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:22 smithi079 ceph-mon[23158]: pgmap v1246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1664.886393 secs 2022-03-21T14:39:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1664.886393 secs 2022-03-21T14:39:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1664.886393 secs 2022-03-21T14:39:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:24 smithi002 ceph-mon[17167]: pgmap v1247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:24 smithi016 ceph-mon[22010]: pgmap v1247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:24 smithi079 ceph-mon[23158]: pgmap v1247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:26 smithi002 ceph-mon[17167]: pgmap v1248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:26 smithi016 ceph-mon[22010]: pgmap v1248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:26 smithi079 ceph-mon[23158]: pgmap v1248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:28 smithi002 ceph-mon[17167]: pgmap v1249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1669.886503 secs 2022-03-21T14:39:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:28 smithi016 ceph-mon[22010]: pgmap v1249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1669.886503 secs 2022-03-21T14:39:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:28 smithi079 ceph-mon[23158]: pgmap v1249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1669.886503 secs 2022-03-21T14:39:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:30 smithi002 ceph-mon[17167]: pgmap v1250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:30 smithi016 ceph-mon[22010]: pgmap v1250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:30 smithi079 ceph-mon[23158]: pgmap v1250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:32 smithi002 ceph-mon[17167]: pgmap v1251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:32 smithi016 ceph-mon[22010]: pgmap v1251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:32 smithi079 ceph-mon[23158]: pgmap v1251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1674.886635 secs 2022-03-21T14:39:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1674.886635 secs 2022-03-21T14:39:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1674.886635 secs 2022-03-21T14:39:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:34 smithi016 ceph-mon[22010]: pgmap v1252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:34 smithi079 ceph-mon[23158]: pgmap v1252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:34 smithi002 ceph-mon[17167]: pgmap v1252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:36 smithi016 ceph-mon[22010]: pgmap v1253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:36 smithi079 ceph-mon[23158]: pgmap v1253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:36.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:36 smithi002 ceph-mon[17167]: pgmap v1253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:38 smithi016 ceph-mon[22010]: pgmap v1254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1679.886707 secs 2022-03-21T14:39:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:38 smithi079 ceph-mon[23158]: pgmap v1254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1679.886707 secs 2022-03-21T14:39:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:38 smithi002 ceph-mon[17167]: pgmap v1254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1679.886707 secs 2022-03-21T14:39:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:40 smithi016 ceph-mon[22010]: pgmap v1255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:40 smithi079 ceph-mon[23158]: pgmap v1255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:40 smithi002 ceph-mon[17167]: pgmap v1255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:42 smithi016 ceph-mon[22010]: pgmap v1256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:42 smithi079 ceph-mon[23158]: pgmap v1256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:42 smithi002 ceph-mon[17167]: pgmap v1256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1684.886775 secs 2022-03-21T14:39:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1684.886775 secs 2022-03-21T14:39:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1684.886775 secs 2022-03-21T14:39:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:44 smithi016 ceph-mon[22010]: pgmap v1257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:44 smithi079 ceph-mon[23158]: pgmap v1257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:44 smithi002 ceph-mon[17167]: pgmap v1257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:46 smithi016 ceph-mon[22010]: pgmap v1258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:46 smithi079 ceph-mon[23158]: pgmap v1258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:46 smithi002 ceph-mon[17167]: pgmap v1258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:48 smithi016 ceph-mon[22010]: pgmap v1259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1689.886879 secs 2022-03-21T14:39:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:48 smithi079 ceph-mon[23158]: pgmap v1259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1689.886879 secs 2022-03-21T14:39:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:48 smithi002 ceph-mon[17167]: pgmap v1259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1689.886879 secs 2022-03-21T14:39:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:50 smithi016 ceph-mon[22010]: pgmap v1260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:50 smithi079 ceph-mon[23158]: pgmap v1260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:50 smithi002 ceph-mon[17167]: pgmap v1260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:52 smithi016 ceph-mon[22010]: pgmap v1261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:52 smithi079 ceph-mon[23158]: pgmap v1261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:52 smithi002 ceph-mon[17167]: pgmap v1261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1694.886988 secs 2022-03-21T14:39:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1694.886988 secs 2022-03-21T14:39:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1694.886988 secs 2022-03-21T14:39:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:54 smithi016 ceph-mon[22010]: pgmap v1262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:54 smithi079 ceph-mon[23158]: pgmap v1262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:54 smithi002 ceph-mon[17167]: pgmap v1262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:56 smithi079 ceph-mon[23158]: pgmap v1263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:56 smithi002 ceph-mon[17167]: pgmap v1263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:56 smithi016 ceph-mon[22010]: pgmap v1263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:39:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:58 smithi079 ceph-mon[23158]: pgmap v1264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1699.887103 secs 2022-03-21T14:39:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:39:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:39:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:39:58 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:39:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:58 smithi002 ceph-mon[17167]: pgmap v1264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1699.887103 secs 2022-03-21T14:39:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:39:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:39:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:39:58 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:39:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:58 smithi016 ceph-mon[22010]: pgmap v1264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:39:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1699.887103 secs 2022-03-21T14:39:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:39:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:39:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:39:58 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:40:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:00 smithi079 ceph-mon[23158]: pgmap v1265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:40:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:00 smithi002 ceph-mon[17167]: pgmap v1265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:40:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:00 smithi016 ceph-mon[22010]: pgmap v1265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:40:01.787 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:01.788 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:02 smithi002 ceph-mon[17167]: pgmap v1266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1704.887220 secs 2022-03-21T14:40:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:02 smithi016 ceph-mon[22010]: pgmap v1266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1704.887220 secs 2022-03-21T14:40:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:02 smithi079 ceph-mon[23158]: pgmap v1266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1704.887220 secs 2022-03-21T14:40:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:40:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:04 smithi002 ceph-mon[17167]: pgmap v1267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:04 smithi016 ceph-mon[22010]: pgmap v1267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:04 smithi079 ceph-mon[23158]: pgmap v1267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:06 smithi002 ceph-mon[17167]: pgmap v1268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:06 smithi016 ceph-mon[22010]: pgmap v1268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:06 smithi079 ceph-mon[23158]: pgmap v1268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1709.887339 secs 2022-03-21T14:40:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1709.887339 secs 2022-03-21T14:40:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1709.887339 secs 2022-03-21T14:40:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:08 smithi002 ceph-mon[17167]: pgmap v1269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:08 smithi016 ceph-mon[22010]: pgmap v1269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:08 smithi079 ceph-mon[23158]: pgmap v1269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:10 smithi002 ceph-mon[17167]: pgmap v1270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:10 smithi016 ceph-mon[22010]: pgmap v1270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:10 smithi079 ceph-mon[23158]: pgmap v1270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:12 smithi002 ceph-mon[17167]: pgmap v1271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1714.887455 secs 2022-03-21T14:40:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:12 smithi016 ceph-mon[22010]: pgmap v1271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1714.887455 secs 2022-03-21T14:40:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:12 smithi079 ceph-mon[23158]: pgmap v1271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1714.887455 secs 2022-03-21T14:40:14.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:13 smithi002 ceph-mon[17167]: pgmap v1272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:13 smithi016 ceph-mon[22010]: pgmap v1272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:14.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:13 smithi079 ceph-mon[23158]: pgmap v1272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:16.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:16 smithi079 ceph-mon[23158]: pgmap v1273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:16 smithi002 ceph-mon[17167]: pgmap v1273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:16 smithi016 ceph-mon[22010]: pgmap v1273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:18 smithi079 ceph-mon[23158]: pgmap v1274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1719.887517 secs 2022-03-21T14:40:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:18 smithi002 ceph-mon[17167]: pgmap v1274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1719.887517 secs 2022-03-21T14:40:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:18 smithi016 ceph-mon[22010]: pgmap v1274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1719.887517 secs 2022-03-21T14:40:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:20 smithi079 ceph-mon[23158]: pgmap v1275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:20 smithi002 ceph-mon[17167]: pgmap v1275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:20 smithi016 ceph-mon[22010]: pgmap v1275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:40:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:40:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:40:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:40:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:40:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:40:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:22 smithi079 ceph-mon[23158]: pgmap v1276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:22 smithi002 ceph-mon[17167]: pgmap v1276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:22 smithi016 ceph-mon[22010]: pgmap v1276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1724.887593 secs 2022-03-21T14:40:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1724.887593 secs 2022-03-21T14:40:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1724.887593 secs 2022-03-21T14:40:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:24 smithi079 ceph-mon[23158]: pgmap v1277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:24 smithi002 ceph-mon[17167]: pgmap v1277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:24 smithi016 ceph-mon[22010]: pgmap v1277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:26.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:26 smithi079 ceph-mon[23158]: pgmap v1278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:26 smithi002 ceph-mon[17167]: pgmap v1278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:26 smithi016 ceph-mon[22010]: pgmap v1278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:28 smithi002 ceph-mon[17167]: pgmap v1279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1729.887662 secs 2022-03-21T14:40:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:28 smithi016 ceph-mon[22010]: pgmap v1279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1729.887662 secs 2022-03-21T14:40:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:28 smithi079 ceph-mon[23158]: pgmap v1279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1729.887662 secs 2022-03-21T14:40:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:30 smithi002 ceph-mon[17167]: pgmap v1280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:30 smithi016 ceph-mon[22010]: pgmap v1280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:30 smithi079 ceph-mon[23158]: pgmap v1280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:32 smithi002 ceph-mon[17167]: pgmap v1281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:32 smithi016 ceph-mon[22010]: pgmap v1281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:32 smithi079 ceph-mon[23158]: pgmap v1281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1734.887739 secs 2022-03-21T14:40:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1734.887739 secs 2022-03-21T14:40:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1734.887739 secs 2022-03-21T14:40:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:34 smithi002 ceph-mon[17167]: pgmap v1282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:34 smithi016 ceph-mon[22010]: pgmap v1282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:34 smithi079 ceph-mon[23158]: pgmap v1282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:36 smithi002 ceph-mon[17167]: pgmap v1283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:36 smithi016 ceph-mon[22010]: pgmap v1283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:36 smithi079 ceph-mon[23158]: pgmap v1283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:38 smithi002 ceph-mon[17167]: pgmap v1284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1739.887796 secs 2022-03-21T14:40:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:38 smithi016 ceph-mon[22010]: pgmap v1284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1739.887796 secs 2022-03-21T14:40:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:38 smithi079 ceph-mon[23158]: pgmap v1284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1739.887796 secs 2022-03-21T14:40:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:40 smithi002 ceph-mon[17167]: pgmap v1285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:40 smithi016 ceph-mon[22010]: pgmap v1285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:40 smithi079 ceph-mon[23158]: pgmap v1285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:42 smithi002 ceph-mon[17167]: pgmap v1286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:42 smithi016 ceph-mon[22010]: pgmap v1286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:42 smithi079 ceph-mon[23158]: pgmap v1286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1744.887869 secs 2022-03-21T14:40:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1744.887869 secs 2022-03-21T14:40:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1744.887869 secs 2022-03-21T14:40:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:44 smithi002 ceph-mon[17167]: pgmap v1287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:44 smithi016 ceph-mon[22010]: pgmap v1287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:44 smithi079 ceph-mon[23158]: pgmap v1287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:46 smithi002 ceph-mon[17167]: pgmap v1288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:46 smithi016 ceph-mon[22010]: pgmap v1288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:46 smithi079 ceph-mon[23158]: pgmap v1288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:48 smithi002 ceph-mon[17167]: pgmap v1289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1749.887990 secs 2022-03-21T14:40:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:48 smithi016 ceph-mon[22010]: pgmap v1289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1749.887990 secs 2022-03-21T14:40:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:48 smithi079 ceph-mon[23158]: pgmap v1289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1749.887990 secs 2022-03-21T14:40:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:50 smithi002 ceph-mon[17167]: pgmap v1290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:50 smithi016 ceph-mon[22010]: pgmap v1290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:50 smithi079 ceph-mon[23158]: pgmap v1290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:52 smithi002 ceph-mon[17167]: pgmap v1291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:52 smithi016 ceph-mon[22010]: pgmap v1291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:52 smithi079 ceph-mon[23158]: pgmap v1291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1754.888149 secs 2022-03-21T14:40:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1754.888149 secs 2022-03-21T14:40:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1754.888149 secs 2022-03-21T14:40:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:54 smithi002 ceph-mon[17167]: pgmap v1292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:54 smithi016 ceph-mon[22010]: pgmap v1292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:54 smithi079 ceph-mon[23158]: pgmap v1292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:56 smithi002 ceph-mon[17167]: pgmap v1293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:56.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:56 smithi016 ceph-mon[22010]: pgmap v1293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:56 smithi079 ceph-mon[23158]: pgmap v1293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:40:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:58 smithi002 ceph-mon[17167]: pgmap v1294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:40:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1759.888291 secs 2022-03-21T14:40:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:58 smithi016 ceph-mon[22010]: pgmap v1294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:40:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1759.888291 secs 2022-03-21T14:40:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:58 smithi079 ceph-mon[23158]: pgmap v1294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:40:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:40:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1759.888291 secs 2022-03-21T14:41:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:00 smithi002 ceph-mon[17167]: pgmap v1295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:00 smithi016 ceph-mon[22010]: pgmap v1295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:00 smithi079 ceph-mon[23158]: pgmap v1295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:02 smithi079 ceph-mon[23158]: pgmap v1296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:02 smithi002 ceph-mon[17167]: pgmap v1296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:02.617 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:02 smithi016 ceph-mon[22010]: pgmap v1296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1764.888457 secs 2022-03-21T14:41:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:41:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:41:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:41:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1764.888457 secs 2022-03-21T14:41:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:41:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:41:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:41:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1764.888457 secs 2022-03-21T14:41:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:41:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:41:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:41:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:04 smithi079 ceph-mon[23158]: pgmap v1297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:41:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:04 smithi002 ceph-mon[17167]: pgmap v1297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:41:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:04 smithi016 ceph-mon[22010]: pgmap v1297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:41:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:06 smithi079 ceph-mon[23158]: pgmap v1298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:06 smithi002 ceph-mon[17167]: pgmap v1298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:06 smithi016 ceph-mon[22010]: pgmap v1298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:08 smithi079 ceph-mon[23158]: pgmap v1299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1769.888588 secs 2022-03-21T14:41:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:08 smithi002 ceph-mon[17167]: pgmap v1299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1769.888588 secs 2022-03-21T14:41:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:08 smithi016 ceph-mon[22010]: pgmap v1299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1769.888588 secs 2022-03-21T14:41:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:10 smithi079 ceph-mon[23158]: pgmap v1300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:10 smithi002 ceph-mon[17167]: pgmap v1300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:10 smithi016 ceph-mon[22010]: pgmap v1300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:12 smithi079 ceph-mon[23158]: pgmap v1301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1774.888706 secs 2022-03-21T14:41:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:12 smithi002 ceph-mon[17167]: pgmap v1301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1774.888706 secs 2022-03-21T14:41:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:12 smithi016 ceph-mon[22010]: pgmap v1301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1774.888706 secs 2022-03-21T14:41:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:14 smithi079 ceph-mon[23158]: pgmap v1302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:14 smithi002 ceph-mon[17167]: pgmap v1302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:14 smithi016 ceph-mon[22010]: pgmap v1302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:16 smithi079 ceph-mon[23158]: pgmap v1303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:16 smithi002 ceph-mon[17167]: pgmap v1303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:16 smithi016 ceph-mon[22010]: pgmap v1303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1779.888807 secs 2022-03-21T14:41:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1779.888807 secs 2022-03-21T14:41:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1779.888807 secs 2022-03-21T14:41:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:18 smithi002 ceph-mon[17167]: pgmap v1304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:18 smithi016 ceph-mon[22010]: pgmap v1304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:18 smithi079 ceph-mon[23158]: pgmap v1304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:20 smithi002 ceph-mon[17167]: pgmap v1305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:20 smithi016 ceph-mon[22010]: pgmap v1305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:20 smithi079 ceph-mon[23158]: pgmap v1305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:41:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:41:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:41:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:41:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:41:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:41:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:22 smithi002 ceph-mon[17167]: pgmap v1306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1784.888938 secs 2022-03-21T14:41:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:22 smithi016 ceph-mon[22010]: pgmap v1306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1784.888938 secs 2022-03-21T14:41:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:22 smithi079 ceph-mon[23158]: pgmap v1306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1784.888938 secs 2022-03-21T14:41:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:24 smithi002 ceph-mon[17167]: pgmap v1307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:24.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:24 smithi016 ceph-mon[22010]: pgmap v1307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:24 smithi079 ceph-mon[23158]: pgmap v1307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:26 smithi002 ceph-mon[17167]: pgmap v1308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:26 smithi016 ceph-mon[22010]: pgmap v1308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:26 smithi079 ceph-mon[23158]: pgmap v1308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1789.889063 secs 2022-03-21T14:41:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1789.889063 secs 2022-03-21T14:41:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1789.889063 secs 2022-03-21T14:41:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:28 smithi002 ceph-mon[17167]: pgmap v1309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:28 smithi016 ceph-mon[22010]: pgmap v1309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:28 smithi079 ceph-mon[23158]: pgmap v1309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:30 smithi002 ceph-mon[17167]: pgmap v1310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:30 smithi016 ceph-mon[22010]: pgmap v1310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:30 smithi079 ceph-mon[23158]: pgmap v1310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:32 smithi002 ceph-mon[17167]: pgmap v1311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1794.889188 secs 2022-03-21T14:41:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:32 smithi016 ceph-mon[22010]: pgmap v1311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1794.889188 secs 2022-03-21T14:41:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:32 smithi079 ceph-mon[23158]: pgmap v1311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1794.889188 secs 2022-03-21T14:41:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:34 smithi002 ceph-mon[17167]: pgmap v1312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:34 smithi016 ceph-mon[22010]: pgmap v1312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:34 smithi079 ceph-mon[23158]: pgmap v1312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:36 smithi002 ceph-mon[17167]: pgmap v1313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:36 smithi016 ceph-mon[22010]: pgmap v1313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:36 smithi079 ceph-mon[23158]: pgmap v1313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:41:37.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1799.889300 secs 2022-03-21T14:41:37.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1799.889300 secs 2022-03-21T14:41:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1799.889300 secs 2022-03-21T14:41:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:38 smithi002 ceph-mon[17167]: pgmap v1314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:38 smithi016 ceph-mon[22010]: pgmap v1314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:38 smithi079 ceph-mon[23158]: pgmap v1314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:40 smithi002 ceph-mon[17167]: pgmap v1315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:40 smithi016 ceph-mon[22010]: pgmap v1315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:40 smithi079 ceph-mon[23158]: pgmap v1315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:42.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:42 smithi002 ceph-mon[17167]: pgmap v1316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:42.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1804.889422 secs 2022-03-21T14:41:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:42 smithi016 ceph-mon[22010]: pgmap v1316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1804.889422 secs 2022-03-21T14:41:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:42 smithi079 ceph-mon[23158]: pgmap v1316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1804.889422 secs 2022-03-21T14:41:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:44 smithi002 ceph-mon[17167]: pgmap v1317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:44 smithi016 ceph-mon[22010]: pgmap v1317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:44 smithi079 ceph-mon[23158]: pgmap v1317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:46 smithi002 ceph-mon[17167]: pgmap v1318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:46 smithi016 ceph-mon[22010]: pgmap v1318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:46 smithi079 ceph-mon[23158]: pgmap v1318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:47.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1809.889533 secs 2022-03-21T14:41:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1809.889533 secs 2022-03-21T14:41:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1809.889533 secs 2022-03-21T14:41:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:48 smithi002 ceph-mon[17167]: pgmap v1319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:48 smithi016 ceph-mon[22010]: pgmap v1319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:48 smithi079 ceph-mon[23158]: pgmap v1319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:50 smithi002 ceph-mon[17167]: pgmap v1320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:50 smithi016 ceph-mon[22010]: pgmap v1320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:50 smithi079 ceph-mon[23158]: pgmap v1320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:52 smithi002 ceph-mon[17167]: pgmap v1321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1814.889642 secs 2022-03-21T14:41:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:52 smithi016 ceph-mon[22010]: pgmap v1321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1814.889642 secs 2022-03-21T14:41:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:52 smithi079 ceph-mon[23158]: pgmap v1321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1814.889642 secs 2022-03-21T14:41:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:54 smithi002 ceph-mon[17167]: pgmap v1322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:54 smithi016 ceph-mon[22010]: pgmap v1322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:54 smithi079 ceph-mon[23158]: pgmap v1322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:56 smithi002 ceph-mon[17167]: pgmap v1323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:56 smithi016 ceph-mon[22010]: pgmap v1323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:56 smithi079 ceph-mon[23158]: pgmap v1323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:41:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1819.889757 secs 2022-03-21T14:41:57.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1819.889757 secs 2022-03-21T14:41:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1819.889757 secs 2022-03-21T14:41:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:41:58 smithi002 ceph-mon[17167]: pgmap v1324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:41:58 smithi016 ceph-mon[22010]: pgmap v1324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:41:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:41:58 smithi079 ceph-mon[23158]: pgmap v1324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:00 smithi002 ceph-mon[17167]: pgmap v1325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:00 smithi016 ceph-mon[22010]: pgmap v1325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:00 smithi079 ceph-mon[23158]: pgmap v1325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:02 smithi002 ceph-mon[17167]: pgmap v1326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1824.889881 secs 2022-03-21T14:42:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:02 smithi016 ceph-mon[22010]: pgmap v1326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1824.889881 secs 2022-03-21T14:42:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:02 smithi079 ceph-mon[23158]: pgmap v1326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1824.889881 secs 2022-03-21T14:42:03.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:42:03.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:42:03.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:42:03.861 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:42:03.861 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:42:03.861 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:42:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:42:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:42:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:42:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:04 smithi002 ceph-mon[17167]: pgmap v1327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:04 smithi016 ceph-mon[22010]: pgmap v1327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:04 smithi079 ceph-mon[23158]: pgmap v1327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:05 smithi079 ceph-mon[23158]: pgmap v1328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:06.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:06.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:05 smithi002 ceph-mon[17167]: pgmap v1328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:42:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:05 smithi016 ceph-mon[22010]: pgmap v1328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:08 smithi016 ceph-mon[22010]: pgmap v1329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1829.889998 secs 2022-03-21T14:42:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:08 smithi079 ceph-mon[23158]: pgmap v1329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1829.889998 secs 2022-03-21T14:42:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:08 smithi002 ceph-mon[17167]: pgmap v1329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1829.889998 secs 2022-03-21T14:42:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:10 smithi002 ceph-mon[17167]: pgmap v1330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:10 smithi016 ceph-mon[22010]: pgmap v1330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:10 smithi079 ceph-mon[23158]: pgmap v1330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:12 smithi002 ceph-mon[17167]: pgmap v1331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:12 smithi016 ceph-mon[22010]: pgmap v1331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:12 smithi079 ceph-mon[23158]: pgmap v1331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1834.890115 secs 2022-03-21T14:42:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1834.890115 secs 2022-03-21T14:42:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1834.890115 secs 2022-03-21T14:42:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:14 smithi002 ceph-mon[17167]: pgmap v1332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:14.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:14 smithi016 ceph-mon[22010]: pgmap v1332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:14 smithi079 ceph-mon[23158]: pgmap v1332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:16 smithi002 ceph-mon[17167]: pgmap v1333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:16.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:16 smithi016 ceph-mon[22010]: pgmap v1333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:16 smithi079 ceph-mon[23158]: pgmap v1333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:18 smithi002 ceph-mon[17167]: pgmap v1334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1839.890181 secs 2022-03-21T14:42:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:18 smithi016 ceph-mon[22010]: pgmap v1334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1839.890181 secs 2022-03-21T14:42:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:18 smithi079 ceph-mon[23158]: pgmap v1334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1839.890181 secs 2022-03-21T14:42:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:20 smithi002 ceph-mon[17167]: pgmap v1335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:20 smithi016 ceph-mon[22010]: pgmap v1335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:20 smithi079 ceph-mon[23158]: pgmap v1335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:42:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:42:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:42:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:42:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:42:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:42:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:22 smithi002 ceph-mon[17167]: pgmap v1336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:22 smithi016 ceph-mon[22010]: pgmap v1336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:22 smithi079 ceph-mon[23158]: pgmap v1336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1844.890298 secs 2022-03-21T14:42:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1844.890298 secs 2022-03-21T14:42:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1844.890298 secs 2022-03-21T14:42:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:24 smithi002 ceph-mon[17167]: pgmap v1337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:24 smithi016 ceph-mon[22010]: pgmap v1337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:24 smithi079 ceph-mon[23158]: pgmap v1337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:26 smithi002 ceph-mon[17167]: pgmap v1338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:26.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:26 smithi016 ceph-mon[22010]: pgmap v1338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:26 smithi079 ceph-mon[23158]: pgmap v1338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:28 smithi002 ceph-mon[17167]: pgmap v1339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1849.890411 secs 2022-03-21T14:42:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:28 smithi016 ceph-mon[22010]: pgmap v1339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1849.890411 secs 2022-03-21T14:42:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:28 smithi079 ceph-mon[23158]: pgmap v1339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1849.890411 secs 2022-03-21T14:42:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:30 smithi002 ceph-mon[17167]: pgmap v1340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:30.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:30 smithi016 ceph-mon[22010]: pgmap v1340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:30 smithi079 ceph-mon[23158]: pgmap v1340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:32 smithi002 ceph-mon[17167]: pgmap v1341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:32 smithi016 ceph-mon[22010]: pgmap v1341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:32 smithi079 ceph-mon[23158]: pgmap v1341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1854.890566 secs 2022-03-21T14:42:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1854.890566 secs 2022-03-21T14:42:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1854.890566 secs 2022-03-21T14:42:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:34 smithi002 ceph-mon[17167]: pgmap v1342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:34 smithi016 ceph-mon[22010]: pgmap v1342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:34 smithi079 ceph-mon[23158]: pgmap v1342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:36 smithi002 ceph-mon[17167]: pgmap v1343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:36 smithi016 ceph-mon[22010]: pgmap v1343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:36 smithi079 ceph-mon[23158]: pgmap v1343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:38 smithi002 ceph-mon[17167]: pgmap v1344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1859.890688 secs 2022-03-21T14:42:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:38 smithi016 ceph-mon[22010]: pgmap v1344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1859.890688 secs 2022-03-21T14:42:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:38 smithi079 ceph-mon[23158]: pgmap v1344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1859.890688 secs 2022-03-21T14:42:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:40 smithi002 ceph-mon[17167]: pgmap v1345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:40 smithi016 ceph-mon[22010]: pgmap v1345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:40 smithi079 ceph-mon[23158]: pgmap v1345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:42 smithi002 ceph-mon[17167]: pgmap v1346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:42 smithi016 ceph-mon[22010]: pgmap v1346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:42 smithi079 ceph-mon[23158]: pgmap v1346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1864.890763 secs 2022-03-21T14:42:43.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1864.890763 secs 2022-03-21T14:42:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1864.890763 secs 2022-03-21T14:42:44.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:44 smithi002 ceph-mon[17167]: pgmap v1347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:44 smithi016 ceph-mon[22010]: pgmap v1347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:44 smithi079 ceph-mon[23158]: pgmap v1347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:46.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:46 smithi002 ceph-mon[17167]: pgmap v1348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:46 smithi016 ceph-mon[22010]: pgmap v1348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:46 smithi079 ceph-mon[23158]: pgmap v1348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:48 smithi002 ceph-mon[17167]: pgmap v1349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1869.890858 secs 2022-03-21T14:42:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:48 smithi016 ceph-mon[22010]: pgmap v1349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1869.890858 secs 2022-03-21T14:42:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:48 smithi079 ceph-mon[23158]: pgmap v1349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1869.890858 secs 2022-03-21T14:42:50.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:50 smithi002 ceph-mon[17167]: pgmap v1350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:50 smithi016 ceph-mon[22010]: pgmap v1350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:50 smithi079 ceph-mon[23158]: pgmap v1350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:52.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:52 smithi002 ceph-mon[17167]: pgmap v1351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:52 smithi016 ceph-mon[22010]: pgmap v1351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:52 smithi079 ceph-mon[23158]: pgmap v1351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1874.890962 secs 2022-03-21T14:42:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1874.890962 secs 2022-03-21T14:42:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1874.890962 secs 2022-03-21T14:42:54.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:54 smithi002 ceph-mon[17167]: pgmap v1352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:54.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:54 smithi016 ceph-mon[22010]: pgmap v1352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:54 smithi079 ceph-mon[23158]: pgmap v1352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:56 smithi002 ceph-mon[17167]: pgmap v1353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:56.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:56 smithi016 ceph-mon[22010]: pgmap v1353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:56 smithi079 ceph-mon[23158]: pgmap v1353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:42:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:58 smithi002 ceph-mon[17167]: pgmap v1354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:42:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1879.891072 secs 2022-03-21T14:42:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:58 smithi016 ceph-mon[22010]: pgmap v1354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:42:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1879.891072 secs 2022-03-21T14:42:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:58 smithi079 ceph-mon[23158]: pgmap v1354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:42:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:42:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1879.891072 secs 2022-03-21T14:43:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:00 smithi002 ceph-mon[17167]: pgmap v1355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:00 smithi016 ceph-mon[22010]: pgmap v1355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:00 smithi079 ceph-mon[23158]: pgmap v1355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:02.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:02 smithi002 ceph-mon[17167]: pgmap v1356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:02 smithi016 ceph-mon[22010]: pgmap v1356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:02 smithi079 ceph-mon[23158]: pgmap v1356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1884.891208 secs 2022-03-21T14:43:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1884.891208 secs 2022-03-21T14:43:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1884.891208 secs 2022-03-21T14:43:04.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:04 smithi002 ceph-mon[17167]: pgmap v1357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:04 smithi016 ceph-mon[22010]: pgmap v1357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:04 smithi079 ceph-mon[23158]: pgmap v1357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:43:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:43:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:43:05.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:43:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:43:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:43:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:43:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:43:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:43:06.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:06 smithi002 ceph-mon[17167]: pgmap v1358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:43:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:06 smithi016 ceph-mon[22010]: pgmap v1358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:43:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:06 smithi079 ceph-mon[23158]: pgmap v1358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:43:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:08 smithi016 ceph-mon[22010]: pgmap v1359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1889.891355 secs 2022-03-21T14:43:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:08 smithi079 ceph-mon[23158]: pgmap v1359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1889.891355 secs 2022-03-21T14:43:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:08 smithi002 ceph-mon[17167]: pgmap v1359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1889.891355 secs 2022-03-21T14:43:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:10 smithi016 ceph-mon[22010]: pgmap v1360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:10 smithi079 ceph-mon[23158]: pgmap v1360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:10 smithi002 ceph-mon[17167]: pgmap v1360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:12 smithi016 ceph-mon[22010]: pgmap v1361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:12 smithi079 ceph-mon[23158]: pgmap v1361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:12 smithi002 ceph-mon[17167]: pgmap v1361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1894.891453 secs 2022-03-21T14:43:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1894.891453 secs 2022-03-21T14:43:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1894.891453 secs 2022-03-21T14:43:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:14 smithi016 ceph-mon[22010]: pgmap v1362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:14 smithi079 ceph-mon[23158]: pgmap v1362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:14 smithi002 ceph-mon[17167]: pgmap v1362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:16 smithi016 ceph-mon[22010]: pgmap v1363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:16 smithi079 ceph-mon[23158]: pgmap v1363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:16 smithi002 ceph-mon[17167]: pgmap v1363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:18 smithi016 ceph-mon[22010]: pgmap v1364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1899.891556 secs 2022-03-21T14:43:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:18 smithi079 ceph-mon[23158]: pgmap v1364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1899.891556 secs 2022-03-21T14:43:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:18 smithi002 ceph-mon[17167]: pgmap v1364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1899.891556 secs 2022-03-21T14:43:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:20 smithi016 ceph-mon[22010]: pgmap v1365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:20 smithi079 ceph-mon[23158]: pgmap v1365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:20 smithi002 ceph-mon[17167]: pgmap v1365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:43:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:43:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:43:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:43:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:43:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:43:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:22 smithi016 ceph-mon[22010]: pgmap v1366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:22 smithi079 ceph-mon[23158]: pgmap v1366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:22 smithi002 ceph-mon[17167]: pgmap v1366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1904.891676 secs 2022-03-21T14:43:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1904.891676 secs 2022-03-21T14:43:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1904.891676 secs 2022-03-21T14:43:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:24 smithi016 ceph-mon[22010]: pgmap v1367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:24 smithi079 ceph-mon[23158]: pgmap v1367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:24 smithi002 ceph-mon[17167]: pgmap v1367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:26 smithi079 ceph-mon[23158]: pgmap v1368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:26 smithi002 ceph-mon[17167]: pgmap v1368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:26 smithi016 ceph-mon[22010]: pgmap v1368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:28 smithi079 ceph-mon[23158]: pgmap v1369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1909.891793 secs 2022-03-21T14:43:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:28 smithi002 ceph-mon[17167]: pgmap v1369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1909.891793 secs 2022-03-21T14:43:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:28 smithi016 ceph-mon[22010]: pgmap v1369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1909.891793 secs 2022-03-21T14:43:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:30 smithi079 ceph-mon[23158]: pgmap v1370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:30 smithi002 ceph-mon[17167]: pgmap v1370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:30 smithi016 ceph-mon[22010]: pgmap v1370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:32 smithi079 ceph-mon[23158]: pgmap v1371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:32 smithi002 ceph-mon[17167]: pgmap v1371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:32 smithi016 ceph-mon[22010]: pgmap v1371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1914.891932 secs 2022-03-21T14:43:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1914.891932 secs 2022-03-21T14:43:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1914.891932 secs 2022-03-21T14:43:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:34 smithi079 ceph-mon[23158]: pgmap v1372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:34 smithi002 ceph-mon[17167]: pgmap v1372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:34 smithi016 ceph-mon[22010]: pgmap v1372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:36 smithi079 ceph-mon[23158]: pgmap v1373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:36 smithi002 ceph-mon[17167]: pgmap v1373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:36 smithi016 ceph-mon[22010]: pgmap v1373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:38 smithi079 ceph-mon[23158]: pgmap v1374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1919.892044 secs 2022-03-21T14:43:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:38 smithi002 ceph-mon[17167]: pgmap v1374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1919.892044 secs 2022-03-21T14:43:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:38 smithi016 ceph-mon[22010]: pgmap v1374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1919.892044 secs 2022-03-21T14:43:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:40 smithi079 ceph-mon[23158]: pgmap v1375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:40 smithi002 ceph-mon[17167]: pgmap v1375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:40 smithi016 ceph-mon[22010]: pgmap v1375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:42 smithi079 ceph-mon[23158]: pgmap v1376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:42 smithi002 ceph-mon[17167]: pgmap v1376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:42 smithi016 ceph-mon[22010]: pgmap v1376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1924.892164 secs 2022-03-21T14:43:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1924.892164 secs 2022-03-21T14:43:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1924.892164 secs 2022-03-21T14:43:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:44 smithi079 ceph-mon[23158]: pgmap v1377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:44 smithi002 ceph-mon[17167]: pgmap v1377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:44 smithi016 ceph-mon[22010]: pgmap v1377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:46 smithi079 ceph-mon[23158]: pgmap v1378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:46 smithi002 ceph-mon[17167]: pgmap v1378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:46 smithi016 ceph-mon[22010]: pgmap v1378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:48 smithi079 ceph-mon[23158]: pgmap v1379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1929.892282 secs 2022-03-21T14:43:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:48 smithi002 ceph-mon[17167]: pgmap v1379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1929.892282 secs 2022-03-21T14:43:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:48 smithi016 ceph-mon[22010]: pgmap v1379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1929.892282 secs 2022-03-21T14:43:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:50 smithi079 ceph-mon[23158]: pgmap v1380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:50 smithi002 ceph-mon[17167]: pgmap v1380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:50 smithi016 ceph-mon[22010]: pgmap v1380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:52 smithi079 ceph-mon[23158]: pgmap v1381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:52 smithi002 ceph-mon[17167]: pgmap v1381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:52 smithi016 ceph-mon[22010]: pgmap v1381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:43:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1934.892424 secs 2022-03-21T14:43:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1934.892424 secs 2022-03-21T14:43:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1934.892424 secs 2022-03-21T14:43:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:54 smithi002 ceph-mon[17167]: pgmap v1382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:54 smithi016 ceph-mon[22010]: pgmap v1382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:54 smithi079 ceph-mon[23158]: pgmap v1382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:43:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:56 smithi002 ceph-mon[17167]: pgmap v1383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:56 smithi016 ceph-mon[22010]: pgmap v1383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:56 smithi079 ceph-mon[23158]: pgmap v1383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:43:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:58 smithi002 ceph-mon[17167]: pgmap v1384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:43:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1939.892500 secs 2022-03-21T14:43:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:58 smithi016 ceph-mon[22010]: pgmap v1384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:43:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1939.892500 secs 2022-03-21T14:43:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:58 smithi079 ceph-mon[23158]: pgmap v1384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:43:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:43:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1939.892500 secs 2022-03-21T14:44:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:00 smithi002 ceph-mon[17167]: pgmap v1385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:00 smithi016 ceph-mon[22010]: pgmap v1385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:00 smithi079 ceph-mon[23158]: pgmap v1385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:02 smithi002 ceph-mon[17167]: pgmap v1386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1944.892624 secs 2022-03-21T14:44:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:02 smithi016 ceph-mon[22010]: pgmap v1386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1944.892624 secs 2022-03-21T14:44:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:02 smithi079 ceph-mon[23158]: pgmap v1386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1944.892624 secs 2022-03-21T14:44:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:04 smithi002 ceph-mon[17167]: pgmap v1387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:04 smithi016 ceph-mon[22010]: pgmap v1387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:04 smithi079 ceph-mon[23158]: pgmap v1387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:06 smithi016 ceph-mon[22010]: pgmap v1388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:44:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:44:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:44:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:06 smithi002 ceph-mon[17167]: pgmap v1388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:44:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:44:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:44:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:06 smithi079 ceph-mon[23158]: pgmap v1388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:44:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:44:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:44:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:44:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1949.892720 secs 2022-03-21T14:44:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:44:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1949.892720 secs 2022-03-21T14:44:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:44:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1949.892720 secs 2022-03-21T14:44:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:08 smithi079 ceph-mon[23158]: pgmap v1389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:08 smithi002 ceph-mon[17167]: pgmap v1389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:08 smithi016 ceph-mon[22010]: pgmap v1389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:10 smithi079 ceph-mon[23158]: pgmap v1390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:10 smithi002 ceph-mon[17167]: pgmap v1390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:10 smithi016 ceph-mon[22010]: pgmap v1390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:12 smithi079 ceph-mon[23158]: pgmap v1391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1954.892838 secs 2022-03-21T14:44:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:12 smithi002 ceph-mon[17167]: pgmap v1391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1954.892838 secs 2022-03-21T14:44:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:12 smithi016 ceph-mon[22010]: pgmap v1391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1954.892838 secs 2022-03-21T14:44:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:14 smithi079 ceph-mon[23158]: pgmap v1392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:14 smithi002 ceph-mon[17167]: pgmap v1392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:14 smithi016 ceph-mon[22010]: pgmap v1392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:16 smithi079 ceph-mon[23158]: pgmap v1393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:16 smithi002 ceph-mon[17167]: pgmap v1393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:16 smithi016 ceph-mon[22010]: pgmap v1393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1959.892954 secs 2022-03-21T14:44:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1959.892954 secs 2022-03-21T14:44:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1959.892954 secs 2022-03-21T14:44:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:18 smithi079 ceph-mon[23158]: pgmap v1394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:18 smithi002 ceph-mon[17167]: pgmap v1394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:18 smithi016 ceph-mon[22010]: pgmap v1394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:20 smithi079 ceph-mon[23158]: pgmap v1395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:44:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:44:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:20 smithi002 ceph-mon[17167]: pgmap v1395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:44:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:44:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:20 smithi016 ceph-mon[22010]: pgmap v1395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:44:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:44:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:22 smithi079 ceph-mon[23158]: pgmap v1396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1964.893076 secs 2022-03-21T14:44:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:22 smithi002 ceph-mon[17167]: pgmap v1396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1964.893076 secs 2022-03-21T14:44:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:22 smithi016 ceph-mon[22010]: pgmap v1396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1964.893076 secs 2022-03-21T14:44:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:24 smithi079 ceph-mon[23158]: pgmap v1397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:24 smithi002 ceph-mon[17167]: pgmap v1397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:24 smithi016 ceph-mon[22010]: pgmap v1397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:26 smithi079 ceph-mon[23158]: pgmap v1398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:26 smithi002 ceph-mon[17167]: pgmap v1398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:26 smithi016 ceph-mon[22010]: pgmap v1398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1969.893188 secs 2022-03-21T14:44:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1969.893188 secs 2022-03-21T14:44:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1969.893188 secs 2022-03-21T14:44:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:28 smithi079 ceph-mon[23158]: pgmap v1399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:28 smithi002 ceph-mon[17167]: pgmap v1399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:28 smithi016 ceph-mon[22010]: pgmap v1399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:30 smithi079 ceph-mon[23158]: pgmap v1400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:30 smithi002 ceph-mon[17167]: pgmap v1400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:30 smithi016 ceph-mon[22010]: pgmap v1400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:32 smithi079 ceph-mon[23158]: pgmap v1401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1974.893298 secs 2022-03-21T14:44:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:32 smithi002 ceph-mon[17167]: pgmap v1401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1974.893298 secs 2022-03-21T14:44:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:32 smithi016 ceph-mon[22010]: pgmap v1401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1974.893298 secs 2022-03-21T14:44:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:34 smithi002 ceph-mon[17167]: pgmap v1402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:34 smithi016 ceph-mon[22010]: pgmap v1402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:34 smithi079 ceph-mon[23158]: pgmap v1402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:36 smithi002 ceph-mon[17167]: pgmap v1403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:36 smithi016 ceph-mon[22010]: pgmap v1403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:36 smithi079 ceph-mon[23158]: pgmap v1403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1979.893391 secs 2022-03-21T14:44:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1979.893391 secs 2022-03-21T14:44:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1979.893391 secs 2022-03-21T14:44:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:38 smithi002 ceph-mon[17167]: pgmap v1404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:38 smithi016 ceph-mon[22010]: pgmap v1404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:38 smithi079 ceph-mon[23158]: pgmap v1404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:40 smithi002 ceph-mon[17167]: pgmap v1405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:40 smithi016 ceph-mon[22010]: pgmap v1405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:40 smithi079 ceph-mon[23158]: pgmap v1405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:42 smithi002 ceph-mon[17167]: pgmap v1406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1984.893510 secs 2022-03-21T14:44:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:42 smithi016 ceph-mon[22010]: pgmap v1406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1984.893510 secs 2022-03-21T14:44:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:42 smithi079 ceph-mon[23158]: pgmap v1406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1984.893510 secs 2022-03-21T14:44:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:44 smithi002 ceph-mon[17167]: pgmap v1407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:44 smithi016 ceph-mon[22010]: pgmap v1407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:44 smithi079 ceph-mon[23158]: pgmap v1407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:46 smithi002 ceph-mon[17167]: pgmap v1408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:46 smithi016 ceph-mon[22010]: pgmap v1408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:46 smithi079 ceph-mon[23158]: pgmap v1408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1989.893631 secs 2022-03-21T14:44:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1989.893631 secs 2022-03-21T14:44:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1989.893631 secs 2022-03-21T14:44:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:48 smithi002 ceph-mon[17167]: pgmap v1409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:49.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:48 smithi016 ceph-mon[22010]: pgmap v1409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:48 smithi079 ceph-mon[23158]: pgmap v1409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:50 smithi002 ceph-mon[17167]: pgmap v1410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:50 smithi016 ceph-mon[22010]: pgmap v1410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:50 smithi079 ceph-mon[23158]: pgmap v1410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:52.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:52 smithi079 ceph-mon[23158]: pgmap v1411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:52.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1994.893749 secs 2022-03-21T14:44:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:52 smithi002 ceph-mon[17167]: pgmap v1411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1994.893749 secs 2022-03-21T14:44:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:52 smithi016 ceph-mon[22010]: pgmap v1411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:44:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1994.893749 secs 2022-03-21T14:44:55.019 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:54 smithi079 ceph-mon[23158]: pgmap v1412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:54 smithi002 ceph-mon[17167]: pgmap v1412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:54 smithi016 ceph-mon[22010]: pgmap v1412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:44:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:56 smithi079 ceph-mon[23158]: pgmap v1413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:56 smithi002 ceph-mon[17167]: pgmap v1413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:56 smithi016 ceph-mon[22010]: pgmap v1413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:44:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 1999.893871 secs 2022-03-21T14:44:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 1999.893871 secs 2022-03-21T14:44:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 1999.893871 secs 2022-03-21T14:44:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:44:58 smithi002 ceph-mon[17167]: pgmap v1414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:44:58 smithi016 ceph-mon[22010]: pgmap v1414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:44:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:44:58 smithi079 ceph-mon[23158]: pgmap v1414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:00 smithi002 ceph-mon[17167]: pgmap v1415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:00 smithi016 ceph-mon[22010]: pgmap v1415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:00 smithi079 ceph-mon[23158]: pgmap v1415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:02 smithi002 ceph-mon[17167]: pgmap v1416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2004.893970 secs 2022-03-21T14:45:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:02 smithi016 ceph-mon[22010]: pgmap v1416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2004.893970 secs 2022-03-21T14:45:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:02 smithi079 ceph-mon[23158]: pgmap v1416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2004.893970 secs 2022-03-21T14:45:05.087 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:04 smithi079 ceph-mon[23158]: pgmap v1417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:04 smithi002 ceph-mon[17167]: pgmap v1417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:04 smithi016 ceph-mon[22010]: pgmap v1417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:06 smithi079 ceph-mon[23158]: pgmap v1418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:45:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:45:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:45:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:06 smithi002 ceph-mon[17167]: pgmap v1418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:45:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:45:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:45:07.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:06 smithi016 ceph-mon[22010]: pgmap v1418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:07.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:45:07.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:45:07.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:45:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2009.894097 secs 2022-03-21T14:45:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:45:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2009.894097 secs 2022-03-21T14:45:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:45:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2009.894097 secs 2022-03-21T14:45:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:45:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:08 smithi002 ceph-mon[17167]: pgmap v1419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:08 smithi016 ceph-mon[22010]: pgmap v1419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:08 smithi079 ceph-mon[23158]: pgmap v1419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:10 smithi002 ceph-mon[17167]: pgmap v1420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:10 smithi016 ceph-mon[22010]: pgmap v1420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:10 smithi079 ceph-mon[23158]: pgmap v1420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:12 smithi002 ceph-mon[17167]: pgmap v1421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2014.894217 secs 2022-03-21T14:45:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:12 smithi016 ceph-mon[22010]: pgmap v1421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2014.894217 secs 2022-03-21T14:45:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:12 smithi079 ceph-mon[23158]: pgmap v1421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2014.894217 secs 2022-03-21T14:45:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:14 smithi002 ceph-mon[17167]: pgmap v1422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:14 smithi016 ceph-mon[22010]: pgmap v1422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:14 smithi079 ceph-mon[23158]: pgmap v1422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:15 smithi002 ceph-mon[17167]: pgmap v1423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:16.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:15 smithi016 ceph-mon[22010]: pgmap v1423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:15 smithi079 ceph-mon[23158]: pgmap v1423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:18 smithi079 ceph-mon[23158]: pgmap v1424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2019.894344 secs 2022-03-21T14:45:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:18 smithi002 ceph-mon[17167]: pgmap v1424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2019.894344 secs 2022-03-21T14:45:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:18 smithi016 ceph-mon[22010]: pgmap v1424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2019.894344 secs 2022-03-21T14:45:20.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:20 smithi002 ceph-mon[17167]: pgmap v1425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:20.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:20 smithi016 ceph-mon[22010]: pgmap v1425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:20 smithi079 ceph-mon[23158]: pgmap v1425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:45:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:45:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:45:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:45:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:45:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:45:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:22 smithi002 ceph-mon[17167]: pgmap v1426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:22 smithi016 ceph-mon[22010]: pgmap v1426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:22 smithi079 ceph-mon[23158]: pgmap v1426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2024.894444 secs 2022-03-21T14:45:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2024.894444 secs 2022-03-21T14:45:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2024.894444 secs 2022-03-21T14:45:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:24 smithi002 ceph-mon[17167]: pgmap v1427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:24 smithi016 ceph-mon[22010]: pgmap v1427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:24 smithi079 ceph-mon[23158]: pgmap v1427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:26.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:26 smithi002 ceph-mon[17167]: pgmap v1428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:26 smithi016 ceph-mon[22010]: pgmap v1428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:26 smithi079 ceph-mon[23158]: pgmap v1428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:28 smithi002 ceph-mon[17167]: pgmap v1429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2029.894580 secs 2022-03-21T14:45:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:28 smithi016 ceph-mon[22010]: pgmap v1429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2029.894580 secs 2022-03-21T14:45:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:28 smithi079 ceph-mon[23158]: pgmap v1429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2029.894580 secs 2022-03-21T14:45:30.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:30 smithi002 ceph-mon[17167]: pgmap v1430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:30 smithi016 ceph-mon[22010]: pgmap v1430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:30 smithi079 ceph-mon[23158]: pgmap v1430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:32 smithi002 ceph-mon[17167]: pgmap v1431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:32 smithi016 ceph-mon[22010]: pgmap v1431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:32 smithi079 ceph-mon[23158]: pgmap v1431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2034.894640 secs 2022-03-21T14:45:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2034.894640 secs 2022-03-21T14:45:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2034.894640 secs 2022-03-21T14:45:34.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:34 smithi002 ceph-mon[17167]: pgmap v1432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:34.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:34 smithi016 ceph-mon[22010]: pgmap v1432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:34 smithi079 ceph-mon[23158]: pgmap v1432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:36.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:36 smithi002 ceph-mon[17167]: pgmap v1433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:36.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:36 smithi016 ceph-mon[22010]: pgmap v1433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:36 smithi079 ceph-mon[23158]: pgmap v1433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:38 smithi002 ceph-mon[17167]: pgmap v1434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2039.894752 secs 2022-03-21T14:45:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:38 smithi016 ceph-mon[22010]: pgmap v1434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2039.894752 secs 2022-03-21T14:45:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:38 smithi079 ceph-mon[23158]: pgmap v1434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2039.894752 secs 2022-03-21T14:45:40.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:40 smithi002 ceph-mon[17167]: pgmap v1435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:40 smithi016 ceph-mon[22010]: pgmap v1435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:40 smithi079 ceph-mon[23158]: pgmap v1435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:42.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:42 smithi002 ceph-mon[17167]: pgmap v1436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:42 smithi016 ceph-mon[22010]: pgmap v1436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:42 smithi079 ceph-mon[23158]: pgmap v1436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2044.894863 secs 2022-03-21T14:45:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2044.894863 secs 2022-03-21T14:45:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2044.894863 secs 2022-03-21T14:45:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:44 smithi002 ceph-mon[17167]: pgmap v1437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:44.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:44 smithi016 ceph-mon[22010]: pgmap v1437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:44 smithi079 ceph-mon[23158]: pgmap v1437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:46 smithi016 ceph-mon[22010]: pgmap v1438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:46 smithi079 ceph-mon[23158]: pgmap v1438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:46 smithi002 ceph-mon[17167]: pgmap v1438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:48 smithi016 ceph-mon[22010]: pgmap v1439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2049.894978 secs 2022-03-21T14:45:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:48 smithi079 ceph-mon[23158]: pgmap v1439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2049.894978 secs 2022-03-21T14:45:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:48 smithi002 ceph-mon[17167]: pgmap v1439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2049.894978 secs 2022-03-21T14:45:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:50 smithi016 ceph-mon[22010]: pgmap v1440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:50 smithi079 ceph-mon[23158]: pgmap v1440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:50 smithi002 ceph-mon[17167]: pgmap v1440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:52 smithi016 ceph-mon[22010]: pgmap v1441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:52.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:52 smithi079 ceph-mon[23158]: pgmap v1441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:52 smithi002 ceph-mon[17167]: pgmap v1441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:45:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2054.895101 secs 2022-03-21T14:45:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2054.895101 secs 2022-03-21T14:45:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2054.895101 secs 2022-03-21T14:45:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:54 smithi016 ceph-mon[22010]: pgmap v1442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:54 smithi079 ceph-mon[23158]: pgmap v1442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:54 smithi002 ceph-mon[17167]: pgmap v1442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:45:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:56 smithi016 ceph-mon[22010]: pgmap v1443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:56.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:56 smithi079 ceph-mon[23158]: pgmap v1443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:56 smithi002 ceph-mon[17167]: pgmap v1443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:45:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:58 smithi016 ceph-mon[22010]: pgmap v1444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:45:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2059.895233 secs 2022-03-21T14:45:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:58 smithi079 ceph-mon[23158]: pgmap v1444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:45:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2059.895233 secs 2022-03-21T14:45:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:58 smithi002 ceph-mon[17167]: pgmap v1444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:45:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:45:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2059.895233 secs 2022-03-21T14:46:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:00 smithi016 ceph-mon[22010]: pgmap v1445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:00 smithi079 ceph-mon[23158]: pgmap v1445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:00 smithi002 ceph-mon[17167]: pgmap v1445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:02 smithi016 ceph-mon[22010]: pgmap v1446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:02.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:02 smithi079 ceph-mon[23158]: pgmap v1446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:02 smithi002 ceph-mon[17167]: pgmap v1446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2064.895360 secs 2022-03-21T14:46:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2064.895360 secs 2022-03-21T14:46:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2064.895360 secs 2022-03-21T14:46:04.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:04 smithi016 ceph-mon[22010]: pgmap v1447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:04 smithi079 ceph-mon[23158]: pgmap v1447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:04 smithi002 ceph-mon[17167]: pgmap v1447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:06.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:06 smithi016 ceph-mon[22010]: pgmap v1448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:06 smithi079 ceph-mon[23158]: pgmap v1448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:06 smithi002 ceph-mon[17167]: pgmap v1448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: pgmap v1449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2069.895490 secs 2022-03-21T14:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:46:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:46:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: pgmap v1449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2069.895490 secs 2022-03-21T14:46:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:46:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:46:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:46:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:46:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: pgmap v1449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2069.895490 secs 2022-03-21T14:46:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:46:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:46:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:46:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:46:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:10 smithi079 ceph-mon[23158]: pgmap v1450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:10 smithi002 ceph-mon[17167]: pgmap v1450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:10 smithi016 ceph-mon[22010]: pgmap v1450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:12 smithi079 ceph-mon[23158]: pgmap v1451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:12 smithi002 ceph-mon[17167]: pgmap v1451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:12 smithi016 ceph-mon[22010]: pgmap v1451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2074.895610 secs 2022-03-21T14:46:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2074.895610 secs 2022-03-21T14:46:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2074.895610 secs 2022-03-21T14:46:14.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:14 smithi079 ceph-mon[23158]: pgmap v1452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:14 smithi002 ceph-mon[17167]: pgmap v1452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:14 smithi016 ceph-mon[22010]: pgmap v1452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:16 smithi079 ceph-mon[23158]: pgmap v1453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:16 smithi002 ceph-mon[17167]: pgmap v1453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:16 smithi016 ceph-mon[22010]: pgmap v1453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:18 smithi079 ceph-mon[23158]: pgmap v1454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2079.895734 secs 2022-03-21T14:46:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:18 smithi002 ceph-mon[17167]: pgmap v1454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2079.895734 secs 2022-03-21T14:46:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:18 smithi016 ceph-mon[22010]: pgmap v1454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2079.895734 secs 2022-03-21T14:46:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:20 smithi079 ceph-mon[23158]: pgmap v1455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:20 smithi002 ceph-mon[17167]: pgmap v1455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:20 smithi016 ceph-mon[22010]: pgmap v1455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:46:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:46:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:46:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:46:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:46:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:46:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:22 smithi079 ceph-mon[23158]: pgmap v1456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:22 smithi002 ceph-mon[17167]: pgmap v1456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:22 smithi016 ceph-mon[22010]: pgmap v1456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2084.895855 secs 2022-03-21T14:46:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2084.895855 secs 2022-03-21T14:46:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2084.895855 secs 2022-03-21T14:46:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:24 smithi079 ceph-mon[23158]: pgmap v1457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:24 smithi002 ceph-mon[17167]: pgmap v1457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:24 smithi016 ceph-mon[22010]: pgmap v1457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:26 smithi079 ceph-mon[23158]: pgmap v1458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:26 smithi002 ceph-mon[17167]: pgmap v1458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:26 smithi016 ceph-mon[22010]: pgmap v1458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:28 smithi079 ceph-mon[23158]: pgmap v1459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2089.895978 secs 2022-03-21T14:46:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:28 smithi002 ceph-mon[17167]: pgmap v1459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2089.895978 secs 2022-03-21T14:46:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:28 smithi016 ceph-mon[22010]: pgmap v1459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2089.895978 secs 2022-03-21T14:46:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:30 smithi079 ceph-mon[23158]: pgmap v1460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:30 smithi002 ceph-mon[17167]: pgmap v1460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:30 smithi016 ceph-mon[22010]: pgmap v1460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:32 smithi079 ceph-mon[23158]: pgmap v1461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:32 smithi002 ceph-mon[17167]: pgmap v1461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:32 smithi016 ceph-mon[22010]: pgmap v1461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2094.896093 secs 2022-03-21T14:46:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2094.896093 secs 2022-03-21T14:46:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2094.896093 secs 2022-03-21T14:46:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:34 smithi002 ceph-mon[17167]: pgmap v1462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:34 smithi016 ceph-mon[22010]: pgmap v1462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:34 smithi079 ceph-mon[23158]: pgmap v1462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:36 smithi002 ceph-mon[17167]: pgmap v1463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:36 smithi016 ceph-mon[22010]: pgmap v1463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:36 smithi079 ceph-mon[23158]: pgmap v1463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:38 smithi002 ceph-mon[17167]: pgmap v1464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2099.896216 secs 2022-03-21T14:46:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:38 smithi016 ceph-mon[22010]: pgmap v1464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2099.896216 secs 2022-03-21T14:46:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:38 smithi079 ceph-mon[23158]: pgmap v1464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2099.896216 secs 2022-03-21T14:46:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:40 smithi002 ceph-mon[17167]: pgmap v1465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:40 smithi016 ceph-mon[22010]: pgmap v1465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:40 smithi079 ceph-mon[23158]: pgmap v1465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:42 smithi002 ceph-mon[17167]: pgmap v1466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:42 smithi016 ceph-mon[22010]: pgmap v1466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:42 smithi079 ceph-mon[23158]: pgmap v1466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2104.896352 secs 2022-03-21T14:46:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2104.896352 secs 2022-03-21T14:46:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2104.896352 secs 2022-03-21T14:46:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:44 smithi002 ceph-mon[17167]: pgmap v1467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:44 smithi016 ceph-mon[22010]: pgmap v1467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:44 smithi079 ceph-mon[23158]: pgmap v1467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:46 smithi002 ceph-mon[17167]: pgmap v1468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:46 smithi016 ceph-mon[22010]: pgmap v1468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:46 smithi079 ceph-mon[23158]: pgmap v1468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:48 smithi002 ceph-mon[17167]: pgmap v1469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2109.896487 secs 2022-03-21T14:46:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:48 smithi016 ceph-mon[22010]: pgmap v1469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2109.896487 secs 2022-03-21T14:46:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:48 smithi079 ceph-mon[23158]: pgmap v1469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2109.896487 secs 2022-03-21T14:46:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:50 smithi002 ceph-mon[17167]: pgmap v1470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:50 smithi016 ceph-mon[22010]: pgmap v1470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:50 smithi079 ceph-mon[23158]: pgmap v1470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:52 smithi002 ceph-mon[17167]: pgmap v1471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:52 smithi016 ceph-mon[22010]: pgmap v1471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:52 smithi079 ceph-mon[23158]: pgmap v1471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:46:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2114.896637 secs 2022-03-21T14:46:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2114.896637 secs 2022-03-21T14:46:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2114.896637 secs 2022-03-21T14:46:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:54 smithi002 ceph-mon[17167]: pgmap v1472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:54 smithi016 ceph-mon[22010]: pgmap v1472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:54 smithi079 ceph-mon[23158]: pgmap v1472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:46:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:56 smithi002 ceph-mon[17167]: pgmap v1473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:56 smithi016 ceph-mon[22010]: pgmap v1473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:56 smithi079 ceph-mon[23158]: pgmap v1473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:46:57.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2119.896752 secs 2022-03-21T14:46:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2119.896752 secs 2022-03-21T14:46:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2119.896752 secs 2022-03-21T14:46:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:46:58 smithi002 ceph-mon[17167]: pgmap v1474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:46:58 smithi016 ceph-mon[22010]: pgmap v1474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:46:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:46:58 smithi079 ceph-mon[23158]: pgmap v1474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:00 smithi002 ceph-mon[17167]: pgmap v1475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:00 smithi016 ceph-mon[22010]: pgmap v1475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:00 smithi079 ceph-mon[23158]: pgmap v1475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:02 smithi002 ceph-mon[17167]: pgmap v1476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2124.896873 secs 2022-03-21T14:47:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:02 smithi016 ceph-mon[22010]: pgmap v1476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2124.896873 secs 2022-03-21T14:47:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:02 smithi079 ceph-mon[23158]: pgmap v1476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2124.896873 secs 2022-03-21T14:47:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:04 smithi002 ceph-mon[17167]: pgmap v1477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:04 smithi016 ceph-mon[22010]: pgmap v1477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:04 smithi079 ceph-mon[23158]: pgmap v1477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:06 smithi002 ceph-mon[17167]: pgmap v1478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:06 smithi016 ceph-mon[22010]: pgmap v1478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:06 smithi079 ceph-mon[23158]: pgmap v1478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:07.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2129.896988 secs 2022-03-21T14:47:07.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2129.896988 secs 2022-03-21T14:47:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2129.896988 secs 2022-03-21T14:47:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:08 smithi079 ceph-mon[23158]: pgmap v1479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:47:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:47:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:47:08.863 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:08 smithi016 ceph-mon[22010]: pgmap v1479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:08.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:47:08.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:47:08.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:47:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:08 smithi002 ceph-mon[17167]: pgmap v1479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:47:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:47:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:47:09.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:47:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:47:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:47:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:47:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:47:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:47:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:47:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:47:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: pgmap v1480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:47:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:10 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:47:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:47:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:47:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:47:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:47:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: pgmap v1480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:47:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:10 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:47:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: pgmap v1480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:47:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:10 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:47:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:12 smithi002 ceph-mon[17167]: pgmap v1481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2134.897106 secs 2022-03-21T14:47:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:12 smithi016 ceph-mon[22010]: pgmap v1481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2134.897106 secs 2022-03-21T14:47:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:12 smithi079 ceph-mon[23158]: pgmap v1481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2134.897106 secs 2022-03-21T14:47:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:14 smithi002 ceph-mon[17167]: pgmap v1482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:14 smithi016 ceph-mon[22010]: pgmap v1482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:14 smithi079 ceph-mon[23158]: pgmap v1482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:16 smithi002 ceph-mon[17167]: pgmap v1483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:16 smithi016 ceph-mon[22010]: pgmap v1483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:16 smithi079 ceph-mon[23158]: pgmap v1483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2139.897233 secs 2022-03-21T14:47:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2139.897233 secs 2022-03-21T14:47:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2139.897233 secs 2022-03-21T14:47:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:18 smithi002 ceph-mon[17167]: pgmap v1484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:18.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:18 smithi016 ceph-mon[22010]: pgmap v1484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:18 smithi079 ceph-mon[23158]: pgmap v1484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:20 smithi002 ceph-mon[17167]: pgmap v1485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:20 smithi016 ceph-mon[22010]: pgmap v1485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:20 smithi079 ceph-mon[23158]: pgmap v1485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:47:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:47:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:47:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:47:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:47:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:47:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:22 smithi002 ceph-mon[17167]: pgmap v1486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2144.897318 secs 2022-03-21T14:47:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:22 smithi016 ceph-mon[22010]: pgmap v1486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2144.897318 secs 2022-03-21T14:47:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:22 smithi079 ceph-mon[23158]: pgmap v1486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2144.897318 secs 2022-03-21T14:47:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:24 smithi002 ceph-mon[17167]: pgmap v1487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:24 smithi016 ceph-mon[22010]: pgmap v1487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:24 smithi079 ceph-mon[23158]: pgmap v1487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:26 smithi002 ceph-mon[17167]: pgmap v1488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:26 smithi016 ceph-mon[22010]: pgmap v1488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:26 smithi079 ceph-mon[23158]: pgmap v1488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:27.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2149.897443 secs 2022-03-21T14:47:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2149.897443 secs 2022-03-21T14:47:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2149.897443 secs 2022-03-21T14:47:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:28 smithi002 ceph-mon[17167]: pgmap v1489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:28 smithi016 ceph-mon[22010]: pgmap v1489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:28 smithi079 ceph-mon[23158]: pgmap v1489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:30 smithi002 ceph-mon[17167]: pgmap v1490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:30 smithi016 ceph-mon[22010]: pgmap v1490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:30 smithi079 ceph-mon[23158]: pgmap v1490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:32 smithi002 ceph-mon[17167]: pgmap v1491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2154.897576 secs 2022-03-21T14:47:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:32 smithi016 ceph-mon[22010]: pgmap v1491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2154.897576 secs 2022-03-21T14:47:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:32 smithi079 ceph-mon[23158]: pgmap v1491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2154.897576 secs 2022-03-21T14:47:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:34 smithi016 ceph-mon[22010]: pgmap v1492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:34 smithi079 ceph-mon[23158]: pgmap v1492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:35.636 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:34 smithi002 ceph-mon[17167]: pgmap v1492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:36 smithi016 ceph-mon[22010]: pgmap v1493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:36 smithi079 ceph-mon[23158]: pgmap v1493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:36 smithi002 ceph-mon[17167]: pgmap v1493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2159.897698 secs 2022-03-21T14:47:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2159.897698 secs 2022-03-21T14:47:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2159.897698 secs 2022-03-21T14:47:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:38 smithi016 ceph-mon[22010]: pgmap v1494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:38 smithi079 ceph-mon[23158]: pgmap v1494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:38 smithi002 ceph-mon[17167]: pgmap v1494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:40 smithi016 ceph-mon[22010]: pgmap v1495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:40 smithi079 ceph-mon[23158]: pgmap v1495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:40 smithi002 ceph-mon[17167]: pgmap v1495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:42 smithi016 ceph-mon[22010]: pgmap v1496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2164.897811 secs 2022-03-21T14:47:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:42 smithi079 ceph-mon[23158]: pgmap v1496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2164.897811 secs 2022-03-21T14:47:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:42 smithi002 ceph-mon[17167]: pgmap v1496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2164.897811 secs 2022-03-21T14:47:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:44 smithi016 ceph-mon[22010]: pgmap v1497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:44 smithi079 ceph-mon[23158]: pgmap v1497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:44 smithi002 ceph-mon[17167]: pgmap v1497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:46 smithi016 ceph-mon[22010]: pgmap v1498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:46 smithi079 ceph-mon[23158]: pgmap v1498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:46 smithi002 ceph-mon[17167]: pgmap v1498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2169.897915 secs 2022-03-21T14:47:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2169.897915 secs 2022-03-21T14:47:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2169.897915 secs 2022-03-21T14:47:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:48 smithi016 ceph-mon[22010]: pgmap v1499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:48 smithi079 ceph-mon[23158]: pgmap v1499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:48 smithi002 ceph-mon[17167]: pgmap v1499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:50 smithi016 ceph-mon[22010]: pgmap v1500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:50 smithi079 ceph-mon[23158]: pgmap v1500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:50 smithi002 ceph-mon[17167]: pgmap v1500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:52 smithi016 ceph-mon[22010]: pgmap v1501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2174.898041 secs 2022-03-21T14:47:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:52 smithi079 ceph-mon[23158]: pgmap v1501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2174.898041 secs 2022-03-21T14:47:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:52 smithi002 ceph-mon[17167]: pgmap v1501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:47:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2174.898041 secs 2022-03-21T14:47:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:54 smithi079 ceph-mon[23158]: pgmap v1502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:54 smithi002 ceph-mon[17167]: pgmap v1502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:54 smithi016 ceph-mon[22010]: pgmap v1502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:47:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:56 smithi079 ceph-mon[23158]: pgmap v1503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:56 smithi002 ceph-mon[17167]: pgmap v1503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:56 smithi016 ceph-mon[22010]: pgmap v1503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:47:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2179.898153 secs 2022-03-21T14:47:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2179.898153 secs 2022-03-21T14:47:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2179.898153 secs 2022-03-21T14:47:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:47:58 smithi079 ceph-mon[23158]: pgmap v1504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:47:58 smithi002 ceph-mon[17167]: pgmap v1504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:47:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:47:58 smithi016 ceph-mon[22010]: pgmap v1504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:00 smithi079 ceph-mon[23158]: pgmap v1505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:01.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:00 smithi002 ceph-mon[17167]: pgmap v1505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:00 smithi016 ceph-mon[22010]: pgmap v1505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:02 smithi079 ceph-mon[23158]: pgmap v1506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2184.898251 secs 2022-03-21T14:48:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:02 smithi002 ceph-mon[17167]: pgmap v1506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2184.898251 secs 2022-03-21T14:48:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:02 smithi016 ceph-mon[22010]: pgmap v1506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2184.898251 secs 2022-03-21T14:48:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:04 smithi079 ceph-mon[23158]: pgmap v1507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:04 smithi002 ceph-mon[17167]: pgmap v1507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:04 smithi016 ceph-mon[22010]: pgmap v1507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:06 smithi079 ceph-mon[23158]: pgmap v1508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:06 smithi002 ceph-mon[17167]: pgmap v1508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:06 smithi016 ceph-mon[22010]: pgmap v1508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2189.898372 secs 2022-03-21T14:48:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2189.898372 secs 2022-03-21T14:48:08.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2189.898372 secs 2022-03-21T14:48:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:08 smithi079 ceph-mon[23158]: pgmap v1509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:08 smithi002 ceph-mon[17167]: pgmap v1509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:08 smithi016 ceph-mon[22010]: pgmap v1509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:48:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:48:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:48:10.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:48:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:48:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:48:10.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:48:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:48:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:48:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:10 smithi079 ceph-mon[23158]: pgmap v1510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:48:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:10 smithi002 ceph-mon[17167]: pgmap v1510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:48:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:10 smithi016 ceph-mon[22010]: pgmap v1510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:48:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:12 smithi079 ceph-mon[23158]: pgmap v1511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2194.898492 secs 2022-03-21T14:48:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:12 smithi002 ceph-mon[17167]: pgmap v1511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2194.898492 secs 2022-03-21T14:48:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:12 smithi016 ceph-mon[22010]: pgmap v1511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2194.898492 secs 2022-03-21T14:48:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:14 smithi079 ceph-mon[23158]: pgmap v1512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:14 smithi002 ceph-mon[17167]: pgmap v1512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:14 smithi016 ceph-mon[22010]: pgmap v1512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:16 smithi079 ceph-mon[23158]: pgmap v1513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:16 smithi002 ceph-mon[17167]: pgmap v1513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:16 smithi016 ceph-mon[22010]: pgmap v1513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2199.898612 secs 2022-03-21T14:48:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2199.898612 secs 2022-03-21T14:48:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2199.898612 secs 2022-03-21T14:48:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:18 smithi002 ceph-mon[17167]: pgmap v1514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:18 smithi016 ceph-mon[22010]: pgmap v1514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:18 smithi079 ceph-mon[23158]: pgmap v1514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:20 smithi002 ceph-mon[17167]: pgmap v1515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:48:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:48:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:20 smithi016 ceph-mon[22010]: pgmap v1515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:48:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:48:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:20 smithi079 ceph-mon[23158]: pgmap v1515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:48:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:48:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:22 smithi002 ceph-mon[17167]: pgmap v1516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2204.898687 secs 2022-03-21T14:48:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:22 smithi016 ceph-mon[22010]: pgmap v1516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2204.898687 secs 2022-03-21T14:48:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:22 smithi079 ceph-mon[23158]: pgmap v1516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2204.898687 secs 2022-03-21T14:48:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:24 smithi002 ceph-mon[17167]: pgmap v1517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:24 smithi016 ceph-mon[22010]: pgmap v1517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:24 smithi079 ceph-mon[23158]: pgmap v1517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:26 smithi002 ceph-mon[17167]: pgmap v1518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:26 smithi016 ceph-mon[22010]: pgmap v1518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:26 smithi079 ceph-mon[23158]: pgmap v1518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2209.898802 secs 2022-03-21T14:48:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2209.898802 secs 2022-03-21T14:48:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2209.898802 secs 2022-03-21T14:48:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:28 smithi002 ceph-mon[17167]: pgmap v1519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:28 smithi016 ceph-mon[22010]: pgmap v1519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:28 smithi079 ceph-mon[23158]: pgmap v1519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:30 smithi002 ceph-mon[17167]: pgmap v1520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:30 smithi016 ceph-mon[22010]: pgmap v1520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:30 smithi079 ceph-mon[23158]: pgmap v1520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:32 smithi002 ceph-mon[17167]: pgmap v1521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2214.898917 secs 2022-03-21T14:48:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:32 smithi016 ceph-mon[22010]: pgmap v1521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2214.898917 secs 2022-03-21T14:48:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:32 smithi079 ceph-mon[23158]: pgmap v1521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2214.898917 secs 2022-03-21T14:48:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:34 smithi002 ceph-mon[17167]: pgmap v1522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:34 smithi016 ceph-mon[22010]: pgmap v1522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:34 smithi079 ceph-mon[23158]: pgmap v1522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:36 smithi002 ceph-mon[17167]: pgmap v1523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:36 smithi016 ceph-mon[22010]: pgmap v1523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:36 smithi079 ceph-mon[23158]: pgmap v1523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2219.899035 secs 2022-03-21T14:48:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2219.899035 secs 2022-03-21T14:48:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2219.899035 secs 2022-03-21T14:48:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:38 smithi002 ceph-mon[17167]: pgmap v1524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:39.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:38 smithi016 ceph-mon[22010]: pgmap v1524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:38 smithi079 ceph-mon[23158]: pgmap v1524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:40 smithi002 ceph-mon[17167]: pgmap v1525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:40 smithi016 ceph-mon[22010]: pgmap v1525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:40 smithi079 ceph-mon[23158]: pgmap v1525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:42 smithi002 ceph-mon[17167]: pgmap v1526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2224.899159 secs 2022-03-21T14:48:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:42 smithi016 ceph-mon[22010]: pgmap v1526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2224.899159 secs 2022-03-21T14:48:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:42 smithi079 ceph-mon[23158]: pgmap v1526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2224.899159 secs 2022-03-21T14:48:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:44 smithi002 ceph-mon[17167]: pgmap v1527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:44 smithi016 ceph-mon[22010]: pgmap v1527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:44 smithi079 ceph-mon[23158]: pgmap v1527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:46 smithi002 ceph-mon[17167]: pgmap v1528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:46 smithi016 ceph-mon[22010]: pgmap v1528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:46 smithi079 ceph-mon[23158]: pgmap v1528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2229.899252 secs 2022-03-21T14:48:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2229.899252 secs 2022-03-21T14:48:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2229.899252 secs 2022-03-21T14:48:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:48 smithi002 ceph-mon[17167]: pgmap v1529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:49.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:48 smithi016 ceph-mon[22010]: pgmap v1529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:48 smithi079 ceph-mon[23158]: pgmap v1529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:50 smithi002 ceph-mon[17167]: pgmap v1530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:50 smithi016 ceph-mon[22010]: pgmap v1530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:50 smithi079 ceph-mon[23158]: pgmap v1530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:52 smithi002 ceph-mon[17167]: pgmap v1531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2234.899368 secs 2022-03-21T14:48:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:52 smithi016 ceph-mon[22010]: pgmap v1531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2234.899368 secs 2022-03-21T14:48:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:52 smithi079 ceph-mon[23158]: pgmap v1531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:48:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2234.899368 secs 2022-03-21T14:48:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:54 smithi002 ceph-mon[17167]: pgmap v1532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:54 smithi016 ceph-mon[22010]: pgmap v1532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:54 smithi079 ceph-mon[23158]: pgmap v1532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:48:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:56 smithi002 ceph-mon[17167]: pgmap v1533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:56 smithi016 ceph-mon[22010]: pgmap v1533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:56 smithi079 ceph-mon[23158]: pgmap v1533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:48:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2239.899485 secs 2022-03-21T14:48:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2239.899485 secs 2022-03-21T14:48:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2239.899485 secs 2022-03-21T14:48:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:48:58 smithi002 ceph-mon[17167]: pgmap v1534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:48:58 smithi016 ceph-mon[22010]: pgmap v1534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:48:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:48:58 smithi079 ceph-mon[23158]: pgmap v1534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:00 smithi002 ceph-mon[17167]: pgmap v1535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:00 smithi016 ceph-mon[22010]: pgmap v1535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:00 smithi079 ceph-mon[23158]: pgmap v1535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:02 smithi002 ceph-mon[17167]: pgmap v1536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2244.899641 secs 2022-03-21T14:49:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:02 smithi016 ceph-mon[22010]: pgmap v1536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2244.899641 secs 2022-03-21T14:49:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:02 smithi079 ceph-mon[23158]: pgmap v1536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2244.899641 secs 2022-03-21T14:49:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:04 smithi002 ceph-mon[17167]: pgmap v1537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:04 smithi016 ceph-mon[22010]: pgmap v1537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:04 smithi079 ceph-mon[23158]: pgmap v1537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:06 smithi002 ceph-mon[17167]: pgmap v1538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:06 smithi016 ceph-mon[22010]: pgmap v1538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:06 smithi079 ceph-mon[23158]: pgmap v1538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2249.899765 secs 2022-03-21T14:49:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2249.899765 secs 2022-03-21T14:49:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2249.899765 secs 2022-03-21T14:49:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:08 smithi002 ceph-mon[17167]: pgmap v1539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:08 smithi016 ceph-mon[22010]: pgmap v1539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:08 smithi079 ceph-mon[23158]: pgmap v1539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:10 smithi079 ceph-mon[23158]: pgmap v1540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:49:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:49:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:49:10.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:10 smithi016 ceph-mon[22010]: pgmap v1540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:10.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:49:10.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:49:10.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:49:10.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:10 smithi002 ceph-mon[17167]: pgmap v1540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:49:12.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:49:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:11 smithi002 ceph-mon[17167]: pgmap v1541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:12.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:49:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:11 smithi016 ceph-mon[22010]: pgmap v1541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:12.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:49:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:11 smithi079 ceph-mon[23158]: pgmap v1541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2254.899841 secs 2022-03-21T14:49:13.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2254.899841 secs 2022-03-21T14:49:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2254.899841 secs 2022-03-21T14:49:14.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:14 smithi002 ceph-mon[17167]: pgmap v1542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:14 smithi016 ceph-mon[22010]: pgmap v1542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:14.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:14 smithi079 ceph-mon[23158]: pgmap v1542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:16 smithi016 ceph-mon[22010]: pgmap v1543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:16 smithi079 ceph-mon[23158]: pgmap v1543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:16 smithi002 ceph-mon[17167]: pgmap v1543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:18 smithi016 ceph-mon[22010]: pgmap v1544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2259.899962 secs 2022-03-21T14:49:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:18 smithi079 ceph-mon[23158]: pgmap v1544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2259.899962 secs 2022-03-21T14:49:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:18 smithi002 ceph-mon[17167]: pgmap v1544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2259.899962 secs 2022-03-21T14:49:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:20 smithi016 ceph-mon[22010]: pgmap v1545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:20 smithi079 ceph-mon[23158]: pgmap v1545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:20 smithi002 ceph-mon[17167]: pgmap v1545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:49:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:49:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:49:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:49:22.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:22 smithi016 ceph-mon[22010]: pgmap v1546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:22 smithi079 ceph-mon[23158]: pgmap v1546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:22 smithi002 ceph-mon[17167]: pgmap v1546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2264.900082 secs 2022-03-21T14:49:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2264.900082 secs 2022-03-21T14:49:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2264.900082 secs 2022-03-21T14:49:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:24 smithi079 ceph-mon[23158]: pgmap v1547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:24 smithi002 ceph-mon[17167]: pgmap v1547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:24 smithi016 ceph-mon[22010]: pgmap v1547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:26 smithi079 ceph-mon[23158]: pgmap v1548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:26 smithi002 ceph-mon[17167]: pgmap v1548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:26 smithi016 ceph-mon[22010]: pgmap v1548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2269.900176 secs 2022-03-21T14:49:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:28 smithi079 ceph-mon[23158]: pgmap v1549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2269.900176 secs 2022-03-21T14:49:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:28 smithi002 ceph-mon[17167]: pgmap v1549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2269.900176 secs 2022-03-21T14:49:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:28 smithi016 ceph-mon[22010]: pgmap v1549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:30 smithi079 ceph-mon[23158]: pgmap v1550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:30 smithi002 ceph-mon[17167]: pgmap v1550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:30 smithi016 ceph-mon[22010]: pgmap v1550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:32 smithi079 ceph-mon[23158]: pgmap v1551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:32 smithi002 ceph-mon[17167]: pgmap v1551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:32 smithi016 ceph-mon[22010]: pgmap v1551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2274.900281 secs 2022-03-21T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2274.900281 secs 2022-03-21T14:49:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2274.900281 secs 2022-03-21T14:49:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:34 smithi079 ceph-mon[23158]: pgmap v1552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:34 smithi002 ceph-mon[17167]: pgmap v1552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:34 smithi016 ceph-mon[22010]: pgmap v1552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:36 smithi079 ceph-mon[23158]: pgmap v1553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:36 smithi002 ceph-mon[17167]: pgmap v1553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:36 smithi016 ceph-mon[22010]: pgmap v1553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2279.900397 secs 2022-03-21T14:49:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:38 smithi079 ceph-mon[23158]: pgmap v1554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2279.900397 secs 2022-03-21T14:49:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:38 smithi002 ceph-mon[17167]: pgmap v1554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2279.900397 secs 2022-03-21T14:49:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:38 smithi016 ceph-mon[22010]: pgmap v1554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:40 smithi079 ceph-mon[23158]: pgmap v1555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:40 smithi002 ceph-mon[17167]: pgmap v1555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:40 smithi016 ceph-mon[22010]: pgmap v1555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:42 smithi079 ceph-mon[23158]: pgmap v1556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:42 smithi002 ceph-mon[17167]: pgmap v1556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:42 smithi016 ceph-mon[22010]: pgmap v1556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2284.900519 secs 2022-03-21T14:49:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2284.900519 secs 2022-03-21T14:49:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2284.900519 secs 2022-03-21T14:49:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:44 smithi079 ceph-mon[23158]: pgmap v1557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:44 smithi002 ceph-mon[17167]: pgmap v1557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:44 smithi016 ceph-mon[22010]: pgmap v1557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:46 smithi079 ceph-mon[23158]: pgmap v1558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:46 smithi002 ceph-mon[17167]: pgmap v1558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:46 smithi016 ceph-mon[22010]: pgmap v1558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2289.900638 secs 2022-03-21T14:49:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:48 smithi079 ceph-mon[23158]: pgmap v1559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2289.900638 secs 2022-03-21T14:49:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:48 smithi002 ceph-mon[17167]: pgmap v1559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2289.900638 secs 2022-03-21T14:49:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:48 smithi016 ceph-mon[22010]: pgmap v1559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:50 smithi079 ceph-mon[23158]: pgmap v1560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:50 smithi002 ceph-mon[17167]: pgmap v1560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:50 smithi016 ceph-mon[22010]: pgmap v1560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:52 smithi002 ceph-mon[17167]: pgmap v1561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:52 smithi016 ceph-mon[22010]: pgmap v1561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:52 smithi079 ceph-mon[23158]: pgmap v1561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:49:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2294.900757 secs 2022-03-21T14:49:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2294.900757 secs 2022-03-21T14:49:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2294.900757 secs 2022-03-21T14:49:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:54 smithi002 ceph-mon[17167]: pgmap v1562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:54 smithi016 ceph-mon[22010]: pgmap v1562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:54 smithi079 ceph-mon[23158]: pgmap v1562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:49:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:56 smithi002 ceph-mon[17167]: pgmap v1563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:56 smithi016 ceph-mon[22010]: pgmap v1563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:56 smithi079 ceph-mon[23158]: pgmap v1563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:49:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2299.900830 secs 2022-03-21T14:49:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:49:58 smithi002 ceph-mon[17167]: pgmap v1564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:58.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2299.900830 secs 2022-03-21T14:49:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:49:58 smithi016 ceph-mon[22010]: pgmap v1564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:49:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2299.900830 secs 2022-03-21T14:49:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:49:58 smithi079 ceph-mon[23158]: pgmap v1564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:00 smithi002 ceph-mon[17167]: pgmap v1565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:50:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:00 smithi016 ceph-mon[22010]: pgmap v1565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:50:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:00 smithi079 ceph-mon[23158]: pgmap v1565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T14:50:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:02 smithi002 ceph-mon[17167]: pgmap v1566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:02 smithi016 ceph-mon[22010]: pgmap v1566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:02 smithi079 ceph-mon[23158]: pgmap v1566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2304.900922 secs 2022-03-21T14:50:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2304.900922 secs 2022-03-21T14:50:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2304.900922 secs 2022-03-21T14:50:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:04 smithi002 ceph-mon[17167]: pgmap v1567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:50:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:04 smithi016 ceph-mon[22010]: pgmap v1567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:50:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:04 smithi079 ceph-mon[23158]: pgmap v1567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:50:06.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:06 smithi002 ceph-mon[17167]: pgmap v1568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:06 smithi016 ceph-mon[22010]: pgmap v1568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:06 smithi079 ceph-mon[23158]: pgmap v1568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:50:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2309.901039 secs 2022-03-21T14:50:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:08 smithi002 ceph-mon[17167]: pgmap v1569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2309.901039 secs 2022-03-21T14:50:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:08 smithi016 ceph-mon[22010]: pgmap v1569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2309.901039 secs 2022-03-21T14:50:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:08 smithi079 ceph-mon[23158]: pgmap v1569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:10 smithi002 ceph-mon[17167]: pgmap v1570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:10.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:10 smithi016 ceph-mon[22010]: pgmap v1570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:10 smithi079 ceph-mon[23158]: pgmap v1570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:50:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:50:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:50:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:50:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:50:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:50:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:50:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:50:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:50:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:12 smithi002 ceph-mon[17167]: pgmap v1571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:12 smithi016 ceph-mon[22010]: pgmap v1571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:12 smithi079 ceph-mon[23158]: pgmap v1571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2314.901117 secs 2022-03-21T14:50:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2314.901117 secs 2022-03-21T14:50:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2314.901117 secs 2022-03-21T14:50:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:14 smithi002 ceph-mon[17167]: pgmap v1572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:14 smithi016 ceph-mon[22010]: pgmap v1572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:14 smithi079 ceph-mon[23158]: pgmap v1572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:15 smithi002 ceph-mon[17167]: pgmap v1573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:16.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:15 smithi016 ceph-mon[22010]: pgmap v1573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:15 smithi079 ceph-mon[23158]: pgmap v1573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:50:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2319.901186 secs 2022-03-21T14:50:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:18 smithi002 ceph-mon[17167]: pgmap v1574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2319.901186 secs 2022-03-21T14:50:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:18 smithi016 ceph-mon[22010]: pgmap v1574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2319.901186 secs 2022-03-21T14:50:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:18 smithi079 ceph-mon[23158]: pgmap v1574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:20 smithi016 ceph-mon[22010]: pgmap v1575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:20 smithi079 ceph-mon[23158]: pgmap v1575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:20 smithi002 ceph-mon[17167]: pgmap v1575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:50:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:50:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:50:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:50:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:22 smithi079 ceph-mon[23158]: pgmap v1576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:22 smithi002 ceph-mon[17167]: pgmap v1576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:22 smithi016 ceph-mon[22010]: pgmap v1576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2324.901282 secs 2022-03-21T14:50:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2324.901282 secs 2022-03-21T14:50:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2324.901282 secs 2022-03-21T14:50:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:24 smithi079 ceph-mon[23158]: pgmap v1577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:24 smithi002 ceph-mon[17167]: pgmap v1577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:24 smithi016 ceph-mon[22010]: pgmap v1577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:26 smithi079 ceph-mon[23158]: pgmap v1578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:26 smithi002 ceph-mon[17167]: pgmap v1578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:26 smithi016 ceph-mon[22010]: pgmap v1578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2329.901400 secs 2022-03-21T14:50:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:28 smithi079 ceph-mon[23158]: pgmap v1579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2329.901400 secs 2022-03-21T14:50:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:28 smithi002 ceph-mon[17167]: pgmap v1579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2329.901400 secs 2022-03-21T14:50:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:28 smithi016 ceph-mon[22010]: pgmap v1579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:30 smithi079 ceph-mon[23158]: pgmap v1580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:30 smithi002 ceph-mon[17167]: pgmap v1580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:30 smithi016 ceph-mon[22010]: pgmap v1580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:32 smithi079 ceph-mon[23158]: pgmap v1581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:32 smithi002 ceph-mon[17167]: pgmap v1581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:32 smithi016 ceph-mon[22010]: pgmap v1581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2334.901462 secs 2022-03-21T14:50:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2334.901462 secs 2022-03-21T14:50:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2334.901462 secs 2022-03-21T14:50:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:34 smithi079 ceph-mon[23158]: pgmap v1582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:34 smithi002 ceph-mon[17167]: pgmap v1582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:34 smithi016 ceph-mon[22010]: pgmap v1582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:36 smithi079 ceph-mon[23158]: pgmap v1583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:36 smithi002 ceph-mon[17167]: pgmap v1583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:36 smithi016 ceph-mon[22010]: pgmap v1583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2339.901575 secs 2022-03-21T14:50:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:38 smithi079 ceph-mon[23158]: pgmap v1584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2339.901575 secs 2022-03-21T14:50:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:38 smithi002 ceph-mon[17167]: pgmap v1584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2339.901575 secs 2022-03-21T14:50:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:38 smithi016 ceph-mon[22010]: pgmap v1584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:40 smithi079 ceph-mon[23158]: pgmap v1585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:40 smithi002 ceph-mon[17167]: pgmap v1585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:40 smithi016 ceph-mon[22010]: pgmap v1585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:42 smithi079 ceph-mon[23158]: pgmap v1586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:42 smithi002 ceph-mon[17167]: pgmap v1586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:42 smithi016 ceph-mon[22010]: pgmap v1586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2344.901641 secs 2022-03-21T14:50:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2344.901641 secs 2022-03-21T14:50:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2344.901641 secs 2022-03-21T14:50:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:44 smithi002 ceph-mon[17167]: pgmap v1587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:44 smithi016 ceph-mon[22010]: pgmap v1587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:44 smithi079 ceph-mon[23158]: pgmap v1587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:46 smithi002 ceph-mon[17167]: pgmap v1588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:46 smithi016 ceph-mon[22010]: pgmap v1588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:46 smithi079 ceph-mon[23158]: pgmap v1588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2349.901718 secs 2022-03-21T14:50:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:48 smithi002 ceph-mon[17167]: pgmap v1589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2349.901718 secs 2022-03-21T14:50:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:48 smithi016 ceph-mon[22010]: pgmap v1589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2349.901718 secs 2022-03-21T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:48 smithi079 ceph-mon[23158]: pgmap v1589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:50 smithi002 ceph-mon[17167]: pgmap v1590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:50 smithi016 ceph-mon[22010]: pgmap v1590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:50 smithi079 ceph-mon[23158]: pgmap v1590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:52 smithi002 ceph-mon[17167]: pgmap v1591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:52 smithi016 ceph-mon[22010]: pgmap v1591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:52 smithi079 ceph-mon[23158]: pgmap v1591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:50:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2354.901834 secs 2022-03-21T14:50:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2354.901834 secs 2022-03-21T14:50:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2354.901834 secs 2022-03-21T14:50:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:54 smithi002 ceph-mon[17167]: pgmap v1592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:54 smithi016 ceph-mon[22010]: pgmap v1592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:54 smithi079 ceph-mon[23158]: pgmap v1592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:50:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:56 smithi002 ceph-mon[17167]: pgmap v1593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:56 smithi016 ceph-mon[22010]: pgmap v1593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:56 smithi079 ceph-mon[23158]: pgmap v1593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:50:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2359.901954 secs 2022-03-21T14:50:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:50:58 smithi002 ceph-mon[17167]: pgmap v1594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2359.901954 secs 2022-03-21T14:50:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:50:58 smithi016 ceph-mon[22010]: pgmap v1594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:50:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2359.901954 secs 2022-03-21T14:50:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:50:58 smithi079 ceph-mon[23158]: pgmap v1594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:00 smithi002 ceph-mon[17167]: pgmap v1595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:00.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:00 smithi016 ceph-mon[22010]: pgmap v1595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:00 smithi079 ceph-mon[23158]: pgmap v1595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:02 smithi002 ceph-mon[17167]: pgmap v1596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:02 smithi016 ceph-mon[22010]: pgmap v1596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:02 smithi079 ceph-mon[23158]: pgmap v1596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2364.902068 secs 2022-03-21T14:51:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2364.902068 secs 2022-03-21T14:51:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2364.902068 secs 2022-03-21T14:51:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:04 smithi002 ceph-mon[17167]: pgmap v1597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:04 smithi016 ceph-mon[22010]: pgmap v1597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:04 smithi079 ceph-mon[23158]: pgmap v1597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:06 smithi002 ceph-mon[17167]: pgmap v1598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:06 smithi016 ceph-mon[22010]: pgmap v1598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:06 smithi079 ceph-mon[23158]: pgmap v1598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2369.902184 secs 2022-03-21T14:51:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:08 smithi002 ceph-mon[17167]: pgmap v1599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:08.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2369.902184 secs 2022-03-21T14:51:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:08 smithi016 ceph-mon[22010]: pgmap v1599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2369.902184 secs 2022-03-21T14:51:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:08 smithi079 ceph-mon[23158]: pgmap v1599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:10 smithi002 ceph-mon[17167]: pgmap v1600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:10 smithi016 ceph-mon[22010]: pgmap v1600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:10 smithi079 ceph-mon[23158]: pgmap v1600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:12 smithi002 ceph-mon[17167]: pgmap v1601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:12.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:12 smithi016 ceph-mon[22010]: pgmap v1601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:12 smithi079 ceph-mon[23158]: pgmap v1601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2374.902300 secs 2022-03-21T14:51:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2374.902300 secs 2022-03-21T14:51:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2374.902300 secs 2022-03-21T14:51:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:14 smithi002 ceph-mon[17167]: pgmap v1602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:14 smithi016 ceph-mon[22010]: pgmap v1602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:14 smithi079 ceph-mon[23158]: pgmap v1602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:16 smithi002 ceph-mon[17167]: pgmap v1603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:51:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:51:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:51:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:16 smithi016 ceph-mon[22010]: pgmap v1603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:51:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:51:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:51:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:16 smithi079 ceph-mon[23158]: pgmap v1603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:51:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:51:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:51:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:51:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2379.902423 secs 2022-03-21T14:51:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:51:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2379.902423 secs 2022-03-21T14:51:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:51:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2379.902423 secs 2022-03-21T14:51:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:18 smithi016 ceph-mon[22010]: pgmap v1604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:18 smithi079 ceph-mon[23158]: pgmap v1604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:18 smithi002 ceph-mon[17167]: pgmap v1604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:20 smithi016 ceph-mon[22010]: pgmap v1605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:20 smithi079 ceph-mon[23158]: pgmap v1605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:20 smithi002 ceph-mon[17167]: pgmap v1605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:51:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:51:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:51:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:51:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:51:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:22 smithi016 ceph-mon[22010]: pgmap v1606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2384.902518 secs 2022-03-21T14:51:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:22 smithi079 ceph-mon[23158]: pgmap v1606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2384.902518 secs 2022-03-21T14:51:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:22 smithi002 ceph-mon[17167]: pgmap v1606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2384.902518 secs 2022-03-21T14:51:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:24 smithi016 ceph-mon[22010]: pgmap v1607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:24 smithi079 ceph-mon[23158]: pgmap v1607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:24 smithi002 ceph-mon[17167]: pgmap v1607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:26 smithi016 ceph-mon[22010]: pgmap v1608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:26 smithi079 ceph-mon[23158]: pgmap v1608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:26 smithi002 ceph-mon[17167]: pgmap v1608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2389.902637 secs 2022-03-21T14:51:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2389.902637 secs 2022-03-21T14:51:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2389.902637 secs 2022-03-21T14:51:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:28 smithi016 ceph-mon[22010]: pgmap v1609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:28 smithi079 ceph-mon[23158]: pgmap v1609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:28 smithi002 ceph-mon[17167]: pgmap v1609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:30 smithi016 ceph-mon[22010]: pgmap v1610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:30 smithi079 ceph-mon[23158]: pgmap v1610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:30 smithi002 ceph-mon[17167]: pgmap v1610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:32 smithi016 ceph-mon[22010]: pgmap v1611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2394.902756 secs 2022-03-21T14:51:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:32 smithi079 ceph-mon[23158]: pgmap v1611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2394.902756 secs 2022-03-21T14:51:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:32 smithi002 ceph-mon[17167]: pgmap v1611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2394.902756 secs 2022-03-21T14:51:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:34 smithi016 ceph-mon[22010]: pgmap v1612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:34 smithi079 ceph-mon[23158]: pgmap v1612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:34 smithi002 ceph-mon[17167]: pgmap v1612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:36 smithi079 ceph-mon[23158]: pgmap v1613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:36 smithi002 ceph-mon[17167]: pgmap v1613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:36 smithi016 ceph-mon[22010]: pgmap v1613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:37.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2399.902864 secs 2022-03-21T14:51:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2399.902864 secs 2022-03-21T14:51:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2399.902864 secs 2022-03-21T14:51:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:38 smithi079 ceph-mon[23158]: pgmap v1614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:38 smithi002 ceph-mon[17167]: pgmap v1614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:38 smithi016 ceph-mon[22010]: pgmap v1614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:40 smithi079 ceph-mon[23158]: pgmap v1615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:40 smithi002 ceph-mon[17167]: pgmap v1615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:40 smithi016 ceph-mon[22010]: pgmap v1615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:42 smithi079 ceph-mon[23158]: pgmap v1616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2404.902986 secs 2022-03-21T14:51:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:42 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 2404.788182 seconds ago 2022-03-21T14:51:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:42 smithi002 ceph-mon[17167]: pgmap v1616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2404.902986 secs 2022-03-21T14:51:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:42 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 2404.788182 seconds ago 2022-03-21T14:51:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:42 smithi016 ceph-mon[22010]: pgmap v1616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2404.902986 secs 2022-03-21T14:51:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:42 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 2404.788182 seconds ago 2022-03-21T14:51:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:44 smithi079 ceph-mon[23158]: pgmap v1617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:44 smithi002 ceph-mon[17167]: pgmap v1617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:44 smithi016 ceph-mon[22010]: pgmap v1617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:46 smithi079 ceph-mon[23158]: pgmap v1618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:46 smithi002 ceph-mon[17167]: pgmap v1618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:46 smithi016 ceph-mon[22010]: pgmap v1618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2409.903107 secs 2022-03-21T14:51:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2409.903107 secs 2022-03-21T14:51:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2409.903107 secs 2022-03-21T14:51:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:48 smithi079 ceph-mon[23158]: pgmap v1619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:48 smithi002 ceph-mon[17167]: pgmap v1619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:48 smithi016 ceph-mon[22010]: pgmap v1619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:50 smithi079 ceph-mon[23158]: pgmap v1620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:51.253 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:50 smithi016 ceph-mon[22010]: pgmap v1620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:50 smithi002 ceph-mon[17167]: pgmap v1620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:52 smithi079 ceph-mon[23158]: pgmap v1621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2414.903241 secs 2022-03-21T14:51:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:52 smithi002 ceph-mon[17167]: pgmap v1621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2414.903241 secs 2022-03-21T14:51:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:52 smithi016 ceph-mon[22010]: pgmap v1621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:51:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2414.903241 secs 2022-03-21T14:51:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:54 smithi079 ceph-mon[23158]: pgmap v1622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:54 smithi002 ceph-mon[17167]: pgmap v1622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:54 smithi016 ceph-mon[22010]: pgmap v1622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:51:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:56 smithi079 ceph-mon[23158]: pgmap v1623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:56 smithi002 ceph-mon[17167]: pgmap v1623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:56 smithi016 ceph-mon[22010]: pgmap v1623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:51:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2419.903359 secs 2022-03-21T14:51:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2419.903359 secs 2022-03-21T14:51:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2419.903359 secs 2022-03-21T14:51:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:51:58 smithi079 ceph-mon[23158]: pgmap v1624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:51:58 smithi002 ceph-mon[17167]: pgmap v1624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:51:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:51:58 smithi016 ceph-mon[22010]: pgmap v1624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:00 smithi079 ceph-mon[23158]: pgmap v1625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:00 smithi002 ceph-mon[17167]: pgmap v1625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:00 smithi016 ceph-mon[22010]: pgmap v1625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:02 smithi002 ceph-mon[17167]: pgmap v1626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2424.903427 secs 2022-03-21T14:52:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:02 smithi016 ceph-mon[22010]: pgmap v1626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2424.903427 secs 2022-03-21T14:52:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:02 smithi079 ceph-mon[23158]: pgmap v1626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2424.903427 secs 2022-03-21T14:52:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:04 smithi002 ceph-mon[17167]: pgmap v1627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:04 smithi016 ceph-mon[22010]: pgmap v1627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:04 smithi079 ceph-mon[23158]: pgmap v1627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:06 smithi002 ceph-mon[17167]: pgmap v1628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:06 smithi016 ceph-mon[22010]: pgmap v1628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:06 smithi079 ceph-mon[23158]: pgmap v1628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2429.903585 secs 2022-03-21T14:52:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2429.903585 secs 2022-03-21T14:52:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2429.903585 secs 2022-03-21T14:52:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:08 smithi002 ceph-mon[17167]: pgmap v1629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:08 smithi016 ceph-mon[22010]: pgmap v1629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:08 smithi079 ceph-mon[23158]: pgmap v1629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:10 smithi002 ceph-mon[17167]: pgmap v1630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:10 smithi016 ceph-mon[22010]: pgmap v1630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:10 smithi079 ceph-mon[23158]: pgmap v1630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:12 smithi002 ceph-mon[17167]: pgmap v1631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2434.903741 secs 2022-03-21T14:52:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:12 smithi016 ceph-mon[22010]: pgmap v1631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2434.903741 secs 2022-03-21T14:52:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:12 smithi079 ceph-mon[23158]: pgmap v1631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2434.903741 secs 2022-03-21T14:52:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:14 smithi002 ceph-mon[17167]: pgmap v1632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:14 smithi016 ceph-mon[22010]: pgmap v1632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:14 smithi079 ceph-mon[23158]: pgmap v1632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:16 smithi016 ceph-mon[22010]: pgmap v1633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:52:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:52:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:52:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:16 smithi079 ceph-mon[23158]: pgmap v1633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:52:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:52:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:52:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:16 smithi002 ceph-mon[17167]: pgmap v1633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:52:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:52:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:52:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2439.903862 secs 2022-03-21T14:52:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:18 smithi002 ceph-mon[17167]: pgmap v1634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2439.903862 secs 2022-03-21T14:52:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:18 smithi016 ceph-mon[22010]: pgmap v1634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2439.903862 secs 2022-03-21T14:52:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:18 smithi079 ceph-mon[23158]: pgmap v1634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:52:20.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:20 smithi079 ceph-mon[23158]: pgmap v1635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:20 smithi002 ceph-mon[17167]: pgmap v1635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:20 smithi016 ceph-mon[22010]: pgmap v1635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:52:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:52:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:52:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:52:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:52:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:52:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:22 smithi079 ceph-mon[23158]: pgmap v1636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:22 smithi002 ceph-mon[17167]: pgmap v1636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:22 smithi016 ceph-mon[22010]: pgmap v1636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2444.904006 secs 2022-03-21T14:52:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2444.904006 secs 2022-03-21T14:52:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2444.904006 secs 2022-03-21T14:52:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:24 smithi002 ceph-mon[17167]: pgmap v1637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:24 smithi016 ceph-mon[22010]: pgmap v1637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:24 smithi079 ceph-mon[23158]: pgmap v1637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:26 smithi002 ceph-mon[17167]: pgmap v1638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:26 smithi016 ceph-mon[22010]: pgmap v1638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:26 smithi079 ceph-mon[23158]: pgmap v1638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2449.904130 secs 2022-03-21T14:52:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:28 smithi002 ceph-mon[17167]: pgmap v1639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2449.904130 secs 2022-03-21T14:52:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:28 smithi016 ceph-mon[22010]: pgmap v1639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2449.904130 secs 2022-03-21T14:52:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:28 smithi079 ceph-mon[23158]: pgmap v1639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:30 smithi002 ceph-mon[17167]: pgmap v1640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:30 smithi016 ceph-mon[22010]: pgmap v1640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:30 smithi079 ceph-mon[23158]: pgmap v1640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:32 smithi002 ceph-mon[17167]: pgmap v1641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:32 smithi016 ceph-mon[22010]: pgmap v1641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:32 smithi079 ceph-mon[23158]: pgmap v1641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2454.904256 secs 2022-03-21T14:52:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2454.904256 secs 2022-03-21T14:52:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2454.904256 secs 2022-03-21T14:52:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:34 smithi002 ceph-mon[17167]: pgmap v1642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:34 smithi016 ceph-mon[22010]: pgmap v1642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:34 smithi079 ceph-mon[23158]: pgmap v1642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:36 smithi002 ceph-mon[17167]: pgmap v1643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:36 smithi016 ceph-mon[22010]: pgmap v1643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:36 smithi079 ceph-mon[23158]: pgmap v1643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2459.904368 secs 2022-03-21T14:52:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:38 smithi002 ceph-mon[17167]: pgmap v1644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2459.904368 secs 2022-03-21T14:52:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:38 smithi016 ceph-mon[22010]: pgmap v1644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2459.904368 secs 2022-03-21T14:52:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:38 smithi079 ceph-mon[23158]: pgmap v1644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:40 smithi002 ceph-mon[17167]: pgmap v1645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:40 smithi016 ceph-mon[22010]: pgmap v1645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:40 smithi079 ceph-mon[23158]: pgmap v1645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:42 smithi002 ceph-mon[17167]: pgmap v1646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:42 smithi016 ceph-mon[22010]: pgmap v1646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:42 smithi079 ceph-mon[23158]: pgmap v1646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2464.904511 secs 2022-03-21T14:52:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2464.904511 secs 2022-03-21T14:52:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2464.904511 secs 2022-03-21T14:52:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:44 smithi002 ceph-mon[17167]: pgmap v1647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:44 smithi016 ceph-mon[22010]: pgmap v1647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:44 smithi079 ceph-mon[23158]: pgmap v1647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:46 smithi002 ceph-mon[17167]: pgmap v1648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:46 smithi016 ceph-mon[22010]: pgmap v1648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:46 smithi079 ceph-mon[23158]: pgmap v1648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2469.904662 secs 2022-03-21T14:52:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:48 smithi002 ceph-mon[17167]: pgmap v1649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2469.904662 secs 2022-03-21T14:52:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:48 smithi016 ceph-mon[22010]: pgmap v1649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2469.904662 secs 2022-03-21T14:52:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:48 smithi079 ceph-mon[23158]: pgmap v1649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:50 smithi002 ceph-mon[17167]: pgmap v1650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:50 smithi016 ceph-mon[22010]: pgmap v1650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:50 smithi079 ceph-mon[23158]: pgmap v1650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:52 smithi002 ceph-mon[17167]: pgmap v1651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:52 smithi016 ceph-mon[22010]: pgmap v1651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:52 smithi079 ceph-mon[23158]: pgmap v1651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:52:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2474.904787 secs 2022-03-21T14:52:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2474.904787 secs 2022-03-21T14:52:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2474.904787 secs 2022-03-21T14:52:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:54 smithi002 ceph-mon[17167]: pgmap v1652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:54 smithi016 ceph-mon[22010]: pgmap v1652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:54 smithi079 ceph-mon[23158]: pgmap v1652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:52:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:56 smithi002 ceph-mon[17167]: pgmap v1653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:56 smithi016 ceph-mon[22010]: pgmap v1653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:56 smithi079 ceph-mon[23158]: pgmap v1653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:52:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2479.904907 secs 2022-03-21T14:52:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:52:58 smithi002 ceph-mon[17167]: pgmap v1654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2479.904907 secs 2022-03-21T14:52:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:52:58 smithi016 ceph-mon[22010]: pgmap v1654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:52:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2479.904907 secs 2022-03-21T14:52:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:52:58 smithi079 ceph-mon[23158]: pgmap v1654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:00 smithi002 ceph-mon[17167]: pgmap v1655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:00 smithi016 ceph-mon[22010]: pgmap v1655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:00 smithi079 ceph-mon[23158]: pgmap v1655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:02 smithi002 ceph-mon[17167]: pgmap v1656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:02 smithi016 ceph-mon[22010]: pgmap v1656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:02 smithi079 ceph-mon[23158]: pgmap v1656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2484.905032 secs 2022-03-21T14:53:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2484.905032 secs 2022-03-21T14:53:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2484.905032 secs 2022-03-21T14:53:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:04 smithi002 ceph-mon[17167]: pgmap v1657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:04.933 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:04 smithi016 ceph-mon[22010]: pgmap v1657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:04 smithi079 ceph-mon[23158]: pgmap v1657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:06 smithi002 ceph-mon[17167]: pgmap v1658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:06 smithi016 ceph-mon[22010]: pgmap v1658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:06 smithi079 ceph-mon[23158]: pgmap v1658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2489.905168 secs 2022-03-21T14:53:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:08 smithi002 ceph-mon[17167]: pgmap v1659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2489.905168 secs 2022-03-21T14:53:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:08 smithi016 ceph-mon[22010]: pgmap v1659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2489.905168 secs 2022-03-21T14:53:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:08 smithi079 ceph-mon[23158]: pgmap v1659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:10 smithi002 ceph-mon[17167]: pgmap v1660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:10 smithi016 ceph-mon[22010]: pgmap v1660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:10 smithi079 ceph-mon[23158]: pgmap v1660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:12 smithi002 ceph-mon[17167]: pgmap v1661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2494.905292 secs 2022-03-21T14:53:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:12 smithi016 ceph-mon[22010]: pgmap v1661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2494.905292 secs 2022-03-21T14:53:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:12 smithi079 ceph-mon[23158]: pgmap v1661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2494.905292 secs 2022-03-21T14:53:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:14 smithi002 ceph-mon[17167]: pgmap v1662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:14 smithi016 ceph-mon[22010]: pgmap v1662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:14 smithi079 ceph-mon[23158]: pgmap v1662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:16 smithi002 ceph-mon[17167]: pgmap v1663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:16 smithi016 ceph-mon[22010]: pgmap v1663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:16 smithi079 ceph-mon[23158]: pgmap v1663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2499.905417 secs 2022-03-21T14:53:17.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2499.905417 secs 2022-03-21T14:53:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2499.905417 secs 2022-03-21T14:53:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:18 smithi002 ceph-mon[17167]: pgmap v1664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:53:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:53:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:53:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:18 smithi016 ceph-mon[22010]: pgmap v1664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:53:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:53:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:53:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:18 smithi079 ceph-mon[23158]: pgmap v1664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:53:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:53:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:53:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:53:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:19 smithi002 ceph-mon[17167]: pgmap v1665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:53:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:19 smithi016 ceph-mon[22010]: pgmap v1665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:53:20.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:19 smithi079 ceph-mon[23158]: pgmap v1665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:53:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:53:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:53:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:53:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:53:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:53:22.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:21 smithi002 ceph-mon[17167]: pgmap v1666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:21 smithi016 ceph-mon[22010]: pgmap v1666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:22.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:21 smithi079 ceph-mon[23158]: pgmap v1666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2504.905584 secs 2022-03-21T14:53:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2504.905584 secs 2022-03-21T14:53:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2504.905584 secs 2022-03-21T14:53:24.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:23 smithi002 ceph-mon[17167]: pgmap v1667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:23 smithi016 ceph-mon[22010]: pgmap v1667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:23 smithi079 ceph-mon[23158]: pgmap v1667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:26 smithi002 ceph-mon[17167]: pgmap v1668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:26 smithi016 ceph-mon[22010]: pgmap v1668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:26 smithi079 ceph-mon[23158]: pgmap v1668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2509.905701 secs 2022-03-21T14:53:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:28 smithi002 ceph-mon[17167]: pgmap v1669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2509.905701 secs 2022-03-21T14:53:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:28 smithi016 ceph-mon[22010]: pgmap v1669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2509.905701 secs 2022-03-21T14:53:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:28 smithi079 ceph-mon[23158]: pgmap v1669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:30 smithi002 ceph-mon[17167]: pgmap v1670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:30.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:30 smithi016 ceph-mon[22010]: pgmap v1670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:30 smithi079 ceph-mon[23158]: pgmap v1670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:32 smithi002 ceph-mon[17167]: pgmap v1671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:32 smithi016 ceph-mon[22010]: pgmap v1671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:32 smithi079 ceph-mon[23158]: pgmap v1671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2514.905826 secs 2022-03-21T14:53:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2514.905826 secs 2022-03-21T14:53:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2514.905826 secs 2022-03-21T14:53:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:34 smithi002 ceph-mon[17167]: pgmap v1672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:34 smithi016 ceph-mon[22010]: pgmap v1672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:34 smithi079 ceph-mon[23158]: pgmap v1672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:36 smithi002 ceph-mon[17167]: pgmap v1673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:36 smithi016 ceph-mon[22010]: pgmap v1673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:36 smithi079 ceph-mon[23158]: pgmap v1673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:38.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2519.905943 secs 2022-03-21T14:53:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:38 smithi002 ceph-mon[17167]: pgmap v1674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2519.905943 secs 2022-03-21T14:53:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:38 smithi016 ceph-mon[22010]: pgmap v1674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2519.905943 secs 2022-03-21T14:53:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:38 smithi079 ceph-mon[23158]: pgmap v1674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:40 smithi002 ceph-mon[17167]: pgmap v1675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:40 smithi016 ceph-mon[22010]: pgmap v1675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:40 smithi079 ceph-mon[23158]: pgmap v1675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:42 smithi002 ceph-mon[17167]: pgmap v1676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:42 smithi016 ceph-mon[22010]: pgmap v1676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:42 smithi079 ceph-mon[23158]: pgmap v1676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2524.906076 secs 2022-03-21T14:53:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2524.906076 secs 2022-03-21T14:53:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2524.906076 secs 2022-03-21T14:53:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:44 smithi002 ceph-mon[17167]: pgmap v1677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:44 smithi016 ceph-mon[22010]: pgmap v1677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:44 smithi079 ceph-mon[23158]: pgmap v1677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:46 smithi002 ceph-mon[17167]: pgmap v1678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:46 smithi016 ceph-mon[22010]: pgmap v1678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:46 smithi079 ceph-mon[23158]: pgmap v1678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:48.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2529.906175 secs 2022-03-21T14:53:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:48 smithi002 ceph-mon[17167]: pgmap v1679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:48.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2529.906175 secs 2022-03-21T14:53:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:48 smithi016 ceph-mon[22010]: pgmap v1679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2529.906175 secs 2022-03-21T14:53:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:48 smithi079 ceph-mon[23158]: pgmap v1679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:50 smithi002 ceph-mon[17167]: pgmap v1680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:50.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:50 smithi016 ceph-mon[22010]: pgmap v1680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:50 smithi079 ceph-mon[23158]: pgmap v1680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:52.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:52 smithi002 ceph-mon[17167]: pgmap v1681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:52 smithi016 ceph-mon[22010]: pgmap v1681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:52 smithi079 ceph-mon[23158]: pgmap v1681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:53:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2534.906317 secs 2022-03-21T14:53:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2534.906317 secs 2022-03-21T14:53:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2534.906317 secs 2022-03-21T14:53:54.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:54 smithi002 ceph-mon[17167]: pgmap v1682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:54 smithi016 ceph-mon[22010]: pgmap v1682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:54 smithi079 ceph-mon[23158]: pgmap v1682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:53:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:56 smithi002 ceph-mon[17167]: pgmap v1683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:56 smithi016 ceph-mon[22010]: pgmap v1683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:56 smithi079 ceph-mon[23158]: pgmap v1683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:53:58.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2539.906436 secs 2022-03-21T14:53:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:53:58 smithi002 ceph-mon[17167]: pgmap v1684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2539.906436 secs 2022-03-21T14:53:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:53:58 smithi016 ceph-mon[22010]: pgmap v1684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:53:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2539.906436 secs 2022-03-21T14:53:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:53:58 smithi079 ceph-mon[23158]: pgmap v1684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:00 smithi002 ceph-mon[17167]: pgmap v1685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:00 smithi016 ceph-mon[22010]: pgmap v1685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:00 smithi079 ceph-mon[23158]: pgmap v1685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:02 smithi002 ceph-mon[17167]: pgmap v1686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:02 smithi016 ceph-mon[22010]: pgmap v1686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:02 smithi079 ceph-mon[23158]: pgmap v1686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2544.906576 secs 2022-03-21T14:54:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2544.906576 secs 2022-03-21T14:54:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2544.906576 secs 2022-03-21T14:54:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:04 smithi002 ceph-mon[17167]: pgmap v1687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:04 smithi016 ceph-mon[22010]: pgmap v1687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:04 smithi079 ceph-mon[23158]: pgmap v1687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:06 smithi002 ceph-mon[17167]: pgmap v1688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:06 smithi016 ceph-mon[22010]: pgmap v1688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:06 smithi079 ceph-mon[23158]: pgmap v1688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:08.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2549.906696 secs 2022-03-21T14:54:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:08 smithi002 ceph-mon[17167]: pgmap v1689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2549.906696 secs 2022-03-21T14:54:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:08 smithi016 ceph-mon[22010]: pgmap v1689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2549.906696 secs 2022-03-21T14:54:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:08 smithi079 ceph-mon[23158]: pgmap v1689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:10 smithi002 ceph-mon[17167]: pgmap v1690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:10 smithi016 ceph-mon[22010]: pgmap v1690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:10 smithi079 ceph-mon[23158]: pgmap v1690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:12 smithi002 ceph-mon[17167]: pgmap v1691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2554.906755 secs 2022-03-21T14:54:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:12 smithi016 ceph-mon[22010]: pgmap v1691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2554.906755 secs 2022-03-21T14:54:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:12 smithi079 ceph-mon[23158]: pgmap v1691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2554.906755 secs 2022-03-21T14:54:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:14 smithi002 ceph-mon[17167]: pgmap v1692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:14 smithi016 ceph-mon[22010]: pgmap v1692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:14 smithi079 ceph-mon[23158]: pgmap v1692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:16 smithi002 ceph-mon[17167]: pgmap v1693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:16 smithi016 ceph-mon[22010]: pgmap v1693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:16 smithi079 ceph-mon[23158]: pgmap v1693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2559.906834 secs 2022-03-21T14:54:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2559.906834 secs 2022-03-21T14:54:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2559.906834 secs 2022-03-21T14:54:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:18 smithi002 ceph-mon[17167]: pgmap v1694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:18 smithi016 ceph-mon[22010]: pgmap v1694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:18 smithi079 ceph-mon[23158]: pgmap v1694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:54:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:54:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:54:19.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:54:19.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:54:19.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:54:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:54:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:54:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:54:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:20 smithi002 ceph-mon[17167]: pgmap v1695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:54:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:54:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:54:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:20 smithi016 ceph-mon[22010]: pgmap v1695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:54:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:54:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:54:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:20 smithi079 ceph-mon[23158]: pgmap v1695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:54:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:54:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:54:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:22 smithi002 ceph-mon[17167]: pgmap v1696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2564.906910 secs 2022-03-21T14:54:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:22 smithi016 ceph-mon[22010]: pgmap v1696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2564.906910 secs 2022-03-21T14:54:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:22 smithi079 ceph-mon[23158]: pgmap v1696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2564.906910 secs 2022-03-21T14:54:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:24 smithi002 ceph-mon[17167]: pgmap v1697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:24 smithi016 ceph-mon[22010]: pgmap v1697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:24 smithi079 ceph-mon[23158]: pgmap v1697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:26 smithi002 ceph-mon[17167]: pgmap v1698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:26 smithi016 ceph-mon[22010]: pgmap v1698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:26 smithi079 ceph-mon[23158]: pgmap v1698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2569.907015 secs 2022-03-21T14:54:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2569.907015 secs 2022-03-21T14:54:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2569.907015 secs 2022-03-21T14:54:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:28 smithi002 ceph-mon[17167]: pgmap v1699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:29.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:28 smithi016 ceph-mon[22010]: pgmap v1699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:28 smithi079 ceph-mon[23158]: pgmap v1699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:30 smithi016 ceph-mon[22010]: pgmap v1700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:30 smithi079 ceph-mon[23158]: pgmap v1700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:30 smithi002 ceph-mon[17167]: pgmap v1700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:32 smithi016 ceph-mon[22010]: pgmap v1701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2574.907137 secs 2022-03-21T14:54:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:32 smithi079 ceph-mon[23158]: pgmap v1701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2574.907137 secs 2022-03-21T14:54:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:32 smithi002 ceph-mon[17167]: pgmap v1701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2574.907137 secs 2022-03-21T14:54:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:34 smithi016 ceph-mon[22010]: pgmap v1702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:34 smithi079 ceph-mon[23158]: pgmap v1702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:34 smithi002 ceph-mon[17167]: pgmap v1702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:36 smithi016 ceph-mon[22010]: pgmap v1703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:36 smithi079 ceph-mon[23158]: pgmap v1703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:36 smithi002 ceph-mon[17167]: pgmap v1703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2579.907234 secs 2022-03-21T14:54:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2579.907234 secs 2022-03-21T14:54:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2579.907234 secs 2022-03-21T14:54:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:38 smithi016 ceph-mon[22010]: pgmap v1704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:38 smithi079 ceph-mon[23158]: pgmap v1704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:38 smithi002 ceph-mon[17167]: pgmap v1704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:40 smithi016 ceph-mon[22010]: pgmap v1705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:40 smithi079 ceph-mon[23158]: pgmap v1705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:40 smithi002 ceph-mon[17167]: pgmap v1705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:42 smithi016 ceph-mon[22010]: pgmap v1706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2584.907337 secs 2022-03-21T14:54:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:42 smithi079 ceph-mon[23158]: pgmap v1706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2584.907337 secs 2022-03-21T14:54:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:42 smithi002 ceph-mon[17167]: pgmap v1706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2584.907337 secs 2022-03-21T14:54:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:44 smithi016 ceph-mon[22010]: pgmap v1707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:44 smithi079 ceph-mon[23158]: pgmap v1707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:44 smithi002 ceph-mon[17167]: pgmap v1707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:46 smithi016 ceph-mon[22010]: pgmap v1708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:46 smithi079 ceph-mon[23158]: pgmap v1708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:46 smithi002 ceph-mon[17167]: pgmap v1708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2589.907401 secs 2022-03-21T14:54:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2589.907401 secs 2022-03-21T14:54:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2589.907401 secs 2022-03-21T14:54:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:48 smithi016 ceph-mon[22010]: pgmap v1709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:48 smithi079 ceph-mon[23158]: pgmap v1709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:48 smithi002 ceph-mon[17167]: pgmap v1709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:50 smithi016 ceph-mon[22010]: pgmap v1710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:50 smithi079 ceph-mon[23158]: pgmap v1710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:50 smithi002 ceph-mon[17167]: pgmap v1710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:52 smithi079 ceph-mon[23158]: pgmap v1711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2594.907516 secs 2022-03-21T14:54:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:52 smithi002 ceph-mon[17167]: pgmap v1711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2594.907516 secs 2022-03-21T14:54:53.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:52 smithi016 ceph-mon[22010]: pgmap v1711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:54:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2594.907516 secs 2022-03-21T14:54:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:54 smithi079 ceph-mon[23158]: pgmap v1712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:54 smithi002 ceph-mon[17167]: pgmap v1712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:54 smithi016 ceph-mon[22010]: pgmap v1712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:54:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:55 smithi079 ceph-mon[23158]: pgmap v1713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:56.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:55 smithi002 ceph-mon[17167]: pgmap v1713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:55 smithi016 ceph-mon[22010]: pgmap v1713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:54:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2599.907635 secs 2022-03-21T14:54:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:54:58 smithi016 ceph-mon[22010]: pgmap v1714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2599.907635 secs 2022-03-21T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:54:58 smithi079 ceph-mon[23158]: pgmap v1714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:54:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2599.907635 secs 2022-03-21T14:54:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:54:58 smithi002 ceph-mon[17167]: pgmap v1714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:00.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:00 smithi002 ceph-mon[17167]: pgmap v1715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:00 smithi016 ceph-mon[22010]: pgmap v1715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:00 smithi079 ceph-mon[23158]: pgmap v1715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:02.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:02 smithi002 ceph-mon[17167]: pgmap v1716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:02 smithi016 ceph-mon[22010]: pgmap v1716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:02 smithi079 ceph-mon[23158]: pgmap v1716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2604.907752 secs 2022-03-21T14:55:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2604.907752 secs 2022-03-21T14:55:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2604.907752 secs 2022-03-21T14:55:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:04 smithi002 ceph-mon[17167]: pgmap v1717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:04 smithi016 ceph-mon[22010]: pgmap v1717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:04 smithi079 ceph-mon[23158]: pgmap v1717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:06 smithi002 ceph-mon[17167]: pgmap v1718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:06 smithi016 ceph-mon[22010]: pgmap v1718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:06 smithi079 ceph-mon[23158]: pgmap v1718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2609.907866 secs 2022-03-21T14:55:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:08 smithi002 ceph-mon[17167]: pgmap v1719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2609.907866 secs 2022-03-21T14:55:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:08 smithi016 ceph-mon[22010]: pgmap v1719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2609.907866 secs 2022-03-21T14:55:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:08 smithi079 ceph-mon[23158]: pgmap v1719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:10.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:10 smithi002 ceph-mon[17167]: pgmap v1720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:10 smithi016 ceph-mon[22010]: pgmap v1720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:10 smithi079 ceph-mon[23158]: pgmap v1720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:12.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:12 smithi002 ceph-mon[17167]: pgmap v1721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:12 smithi016 ceph-mon[22010]: pgmap v1721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:12 smithi079 ceph-mon[23158]: pgmap v1721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2614.907962 secs 2022-03-21T14:55:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2614.907962 secs 2022-03-21T14:55:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2614.907962 secs 2022-03-21T14:55:14.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:14 smithi002 ceph-mon[17167]: pgmap v1722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:14 smithi016 ceph-mon[22010]: pgmap v1722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:14 smithi079 ceph-mon[23158]: pgmap v1722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:16 smithi002 ceph-mon[17167]: pgmap v1723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:16 smithi016 ceph-mon[22010]: pgmap v1723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:16 smithi079 ceph-mon[23158]: pgmap v1723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:18.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2619.908060 secs 2022-03-21T14:55:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:18 smithi002 ceph-mon[17167]: pgmap v1724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2619.908060 secs 2022-03-21T14:55:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:18 smithi016 ceph-mon[22010]: pgmap v1724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2619.908060 secs 2022-03-21T14:55:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:18 smithi079 ceph-mon[23158]: pgmap v1724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:20 smithi002 ceph-mon[17167]: pgmap v1725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:55:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:55:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:55:20.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:20 smithi016 ceph-mon[22010]: pgmap v1725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:55:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:55:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:20 smithi079 ceph-mon[23158]: pgmap v1725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:55:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:55:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:55:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:55:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:55:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:55:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:55:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:22 smithi079 ceph-mon[23158]: pgmap v1726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2624.908193 secs 2022-03-21T14:55:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:22 smithi002 ceph-mon[17167]: pgmap v1726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2624.908193 secs 2022-03-21T14:55:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:22 smithi016 ceph-mon[22010]: pgmap v1726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2624.908193 secs 2022-03-21T14:55:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:24 smithi079 ceph-mon[23158]: pgmap v1727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:24 smithi002 ceph-mon[17167]: pgmap v1727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:24 smithi016 ceph-mon[22010]: pgmap v1727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:26 smithi079 ceph-mon[23158]: pgmap v1728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:26 smithi002 ceph-mon[17167]: pgmap v1728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:26 smithi016 ceph-mon[22010]: pgmap v1728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2629.908302 secs 2022-03-21T14:55:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2629.908302 secs 2022-03-21T14:55:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2629.908302 secs 2022-03-21T14:55:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:28 smithi002 ceph-mon[17167]: pgmap v1729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:29.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:28 smithi016 ceph-mon[22010]: pgmap v1729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:28 smithi079 ceph-mon[23158]: pgmap v1729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:30 smithi002 ceph-mon[17167]: pgmap v1730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:30 smithi016 ceph-mon[22010]: pgmap v1730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:30 smithi079 ceph-mon[23158]: pgmap v1730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:32 smithi002 ceph-mon[17167]: pgmap v1731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2634.908436 secs 2022-03-21T14:55:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:32 smithi016 ceph-mon[22010]: pgmap v1731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2634.908436 secs 2022-03-21T14:55:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:32 smithi079 ceph-mon[23158]: pgmap v1731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2634.908436 secs 2022-03-21T14:55:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:34 smithi002 ceph-mon[17167]: pgmap v1732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:34 smithi016 ceph-mon[22010]: pgmap v1732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:34 smithi079 ceph-mon[23158]: pgmap v1732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:36 smithi002 ceph-mon[17167]: pgmap v1733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:36 smithi016 ceph-mon[22010]: pgmap v1733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:36 smithi079 ceph-mon[23158]: pgmap v1733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2639.908578 secs 2022-03-21T14:55:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2639.908578 secs 2022-03-21T14:55:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2639.908578 secs 2022-03-21T14:55:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:38 smithi002 ceph-mon[17167]: pgmap v1734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:39.306 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:38 smithi079 ceph-mon[23158]: pgmap v1734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:39.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:38 smithi016 ceph-mon[22010]: pgmap v1734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:40 smithi002 ceph-mon[17167]: pgmap v1735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:40 smithi016 ceph-mon[22010]: pgmap v1735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:40 smithi079 ceph-mon[23158]: pgmap v1735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:42 smithi002 ceph-mon[17167]: pgmap v1736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2644.908717 secs 2022-03-21T14:55:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:42 smithi016 ceph-mon[22010]: pgmap v1736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2644.908717 secs 2022-03-21T14:55:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:42 smithi079 ceph-mon[23158]: pgmap v1736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2644.908717 secs 2022-03-21T14:55:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:44 smithi002 ceph-mon[17167]: pgmap v1737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:44 smithi016 ceph-mon[22010]: pgmap v1737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:44 smithi079 ceph-mon[23158]: pgmap v1737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:46 smithi002 ceph-mon[17167]: pgmap v1738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:46 smithi016 ceph-mon[22010]: pgmap v1738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:46 smithi079 ceph-mon[23158]: pgmap v1738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2649.908782 secs 2022-03-21T14:55:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2649.908782 secs 2022-03-21T14:55:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2649.908782 secs 2022-03-21T14:55:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:48 smithi002 ceph-mon[17167]: pgmap v1739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:48 smithi016 ceph-mon[22010]: pgmap v1739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:48 smithi079 ceph-mon[23158]: pgmap v1739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:50 smithi002 ceph-mon[17167]: pgmap v1740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:50 smithi016 ceph-mon[22010]: pgmap v1740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:50 smithi079 ceph-mon[23158]: pgmap v1740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:52 smithi002 ceph-mon[17167]: pgmap v1741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2654.908901 secs 2022-03-21T14:55:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:52 smithi016 ceph-mon[22010]: pgmap v1741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2654.908901 secs 2022-03-21T14:55:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:52 smithi079 ceph-mon[23158]: pgmap v1741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:55:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2654.908901 secs 2022-03-21T14:55:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:54 smithi002 ceph-mon[17167]: pgmap v1742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:54 smithi016 ceph-mon[22010]: pgmap v1742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:54 smithi079 ceph-mon[23158]: pgmap v1742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:55:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:56 smithi002 ceph-mon[17167]: pgmap v1743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:56 smithi016 ceph-mon[22010]: pgmap v1743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:56 smithi079 ceph-mon[23158]: pgmap v1743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:55:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2659.909039 secs 2022-03-21T14:55:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2659.909039 secs 2022-03-21T14:55:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2659.909039 secs 2022-03-21T14:55:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:55:58 smithi002 ceph-mon[17167]: pgmap v1744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:55:58 smithi016 ceph-mon[22010]: pgmap v1744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:55:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:55:58 smithi079 ceph-mon[23158]: pgmap v1744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:00 smithi002 ceph-mon[17167]: pgmap v1745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:00 smithi016 ceph-mon[22010]: pgmap v1745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:00 smithi079 ceph-mon[23158]: pgmap v1745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:02 smithi002 ceph-mon[17167]: pgmap v1746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2664.909178 secs 2022-03-21T14:56:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:02 smithi016 ceph-mon[22010]: pgmap v1746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2664.909178 secs 2022-03-21T14:56:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:02 smithi079 ceph-mon[23158]: pgmap v1746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2664.909178 secs 2022-03-21T14:56:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:04 smithi002 ceph-mon[17167]: pgmap v1747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:04 smithi016 ceph-mon[22010]: pgmap v1747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:04 smithi079 ceph-mon[23158]: pgmap v1747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:06 smithi002 ceph-mon[17167]: pgmap v1748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:06 smithi016 ceph-mon[22010]: pgmap v1748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:06 smithi079 ceph-mon[23158]: pgmap v1748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2669.909318 secs 2022-03-21T14:56:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2669.909318 secs 2022-03-21T14:56:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2669.909318 secs 2022-03-21T14:56:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:08 smithi002 ceph-mon[17167]: pgmap v1749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:08 smithi016 ceph-mon[22010]: pgmap v1749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:08 smithi079 ceph-mon[23158]: pgmap v1749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:10 smithi002 ceph-mon[17167]: pgmap v1750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:10 smithi016 ceph-mon[22010]: pgmap v1750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:10 smithi079 ceph-mon[23158]: pgmap v1750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:12 smithi002 ceph-mon[17167]: pgmap v1751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2674.909443 secs 2022-03-21T14:56:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:12 smithi016 ceph-mon[22010]: pgmap v1751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2674.909443 secs 2022-03-21T14:56:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:12 smithi079 ceph-mon[23158]: pgmap v1751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2674.909443 secs 2022-03-21T14:56:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:14 smithi002 ceph-mon[17167]: pgmap v1752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:14 smithi016 ceph-mon[22010]: pgmap v1752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:14 smithi079 ceph-mon[23158]: pgmap v1752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:16 smithi002 ceph-mon[17167]: pgmap v1753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:16 smithi016 ceph-mon[22010]: pgmap v1753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:16 smithi079 ceph-mon[23158]: pgmap v1753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2679.909583 secs 2022-03-21T14:56:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2679.909583 secs 2022-03-21T14:56:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2679.909583 secs 2022-03-21T14:56:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:18 smithi002 ceph-mon[17167]: pgmap v1754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:18 smithi016 ceph-mon[22010]: pgmap v1754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:18 smithi079 ceph-mon[23158]: pgmap v1754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: pgmap v1755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:56:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:56:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:56:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:56:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:56:21.142 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: pgmap v1755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:21.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:56:21.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:56:21.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:56:21.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:56:21.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:56:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: pgmap v1755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:56:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:56:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:56:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:56:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:56:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:22 smithi002 ceph-mon[17167]: pgmap v1756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:56:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:22 smithi016 ceph-mon[22010]: pgmap v1756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:56:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:22 smithi079 ceph-mon[23158]: pgmap v1756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:56:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2684.909692 secs 2022-03-21T14:56:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2684.909692 secs 2022-03-21T14:56:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2684.909692 secs 2022-03-21T14:56:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:24 smithi002 ceph-mon[17167]: pgmap v1757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:24 smithi016 ceph-mon[22010]: pgmap v1757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:24.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:24 smithi079 ceph-mon[23158]: pgmap v1757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:26 smithi002 ceph-mon[17167]: pgmap v1758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:26 smithi016 ceph-mon[22010]: pgmap v1758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:26 smithi079 ceph-mon[23158]: pgmap v1758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2689.909759 secs 2022-03-21T14:56:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:28 smithi002 ceph-mon[17167]: pgmap v1759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2689.909759 secs 2022-03-21T14:56:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:28 smithi016 ceph-mon[22010]: pgmap v1759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2689.909759 secs 2022-03-21T14:56:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:28 smithi079 ceph-mon[23158]: pgmap v1759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:30 smithi002 ceph-mon[17167]: pgmap v1760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:30 smithi016 ceph-mon[22010]: pgmap v1760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:30 smithi079 ceph-mon[23158]: pgmap v1760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:32 smithi002 ceph-mon[17167]: pgmap v1761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:32 smithi016 ceph-mon[22010]: pgmap v1761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:32 smithi079 ceph-mon[23158]: pgmap v1761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2694.909885 secs 2022-03-21T14:56:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2694.909885 secs 2022-03-21T14:56:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2694.909885 secs 2022-03-21T14:56:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:34 smithi002 ceph-mon[17167]: pgmap v1762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:34 smithi016 ceph-mon[22010]: pgmap v1762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:35.088 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:34 smithi079 ceph-mon[23158]: pgmap v1762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:36 smithi002 ceph-mon[17167]: pgmap v1763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:36 smithi016 ceph-mon[22010]: pgmap v1763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:36 smithi079 ceph-mon[23158]: pgmap v1763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2699.909997 secs 2022-03-21T14:56:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2699.909997 secs 2022-03-21T14:56:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2699.909997 secs 2022-03-21T14:56:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:38 smithi016 ceph-mon[22010]: pgmap v1764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:38 smithi079 ceph-mon[23158]: pgmap v1764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:38 smithi002 ceph-mon[17167]: pgmap v1764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:40 smithi016 ceph-mon[22010]: pgmap v1765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:40 smithi079 ceph-mon[23158]: pgmap v1765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:40 smithi002 ceph-mon[17167]: pgmap v1765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:42 smithi079 ceph-mon[23158]: pgmap v1766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2704.910113 secs 2022-03-21T14:56:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:42 smithi002 ceph-mon[17167]: pgmap v1766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2704.910113 secs 2022-03-21T14:56:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:42 smithi016 ceph-mon[22010]: pgmap v1766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2704.910113 secs 2022-03-21T14:56:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:44 smithi079 ceph-mon[23158]: pgmap v1767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:44 smithi002 ceph-mon[17167]: pgmap v1767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:44 smithi016 ceph-mon[22010]: pgmap v1767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:46 smithi079 ceph-mon[23158]: pgmap v1768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:46 smithi002 ceph-mon[17167]: pgmap v1768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:46 smithi016 ceph-mon[22010]: pgmap v1768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2709.910229 secs 2022-03-21T14:56:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2709.910229 secs 2022-03-21T14:56:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2709.910229 secs 2022-03-21T14:56:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:48 smithi079 ceph-mon[23158]: pgmap v1769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:48 smithi002 ceph-mon[17167]: pgmap v1769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:48 smithi016 ceph-mon[22010]: pgmap v1769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:50 smithi079 ceph-mon[23158]: pgmap v1770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:50 smithi002 ceph-mon[17167]: pgmap v1770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:50 smithi016 ceph-mon[22010]: pgmap v1770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:52 smithi079 ceph-mon[23158]: pgmap v1771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2714.910346 secs 2022-03-21T14:56:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:52 smithi002 ceph-mon[17167]: pgmap v1771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2714.910346 secs 2022-03-21T14:56:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:52 smithi016 ceph-mon[22010]: pgmap v1771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:56:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2714.910346 secs 2022-03-21T14:56:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:54 smithi079 ceph-mon[23158]: pgmap v1772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:54 smithi002 ceph-mon[17167]: pgmap v1772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:54 smithi016 ceph-mon[22010]: pgmap v1772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:56:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:56 smithi079 ceph-mon[23158]: pgmap v1773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:56 smithi002 ceph-mon[17167]: pgmap v1773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:56 smithi016 ceph-mon[22010]: pgmap v1773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:56:57.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2719.910496 secs 2022-03-21T14:56:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2719.910496 secs 2022-03-21T14:56:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2719.910496 secs 2022-03-21T14:56:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:56:58 smithi079 ceph-mon[23158]: pgmap v1774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:56:58 smithi002 ceph-mon[17167]: pgmap v1774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:56:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:56:58 smithi016 ceph-mon[22010]: pgmap v1774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:00 smithi079 ceph-mon[23158]: pgmap v1775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:00 smithi002 ceph-mon[17167]: pgmap v1775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:00 smithi016 ceph-mon[22010]: pgmap v1775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:02 smithi079 ceph-mon[23158]: pgmap v1776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2724.910649 secs 2022-03-21T14:57:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:02 smithi002 ceph-mon[17167]: pgmap v1776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2724.910649 secs 2022-03-21T14:57:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:02 smithi016 ceph-mon[22010]: pgmap v1776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2724.910649 secs 2022-03-21T14:57:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:04 smithi079 ceph-mon[23158]: pgmap v1777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:04 smithi002 ceph-mon[17167]: pgmap v1777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:04 smithi016 ceph-mon[22010]: pgmap v1777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:06 smithi079 ceph-mon[23158]: pgmap v1778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:06 smithi002 ceph-mon[17167]: pgmap v1778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:06 smithi016 ceph-mon[22010]: pgmap v1778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2729.910764 secs 2022-03-21T14:57:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2729.910764 secs 2022-03-21T14:57:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2729.910764 secs 2022-03-21T14:57:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:08 smithi002 ceph-mon[17167]: pgmap v1779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:08 smithi016 ceph-mon[22010]: pgmap v1779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:08 smithi079 ceph-mon[23158]: pgmap v1779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:10 smithi002 ceph-mon[17167]: pgmap v1780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:10 smithi016 ceph-mon[22010]: pgmap v1780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:10 smithi079 ceph-mon[23158]: pgmap v1780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:12 smithi002 ceph-mon[17167]: pgmap v1781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2734.910882 secs 2022-03-21T14:57:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:12 smithi016 ceph-mon[22010]: pgmap v1781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2734.910882 secs 2022-03-21T14:57:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:12 smithi079 ceph-mon[23158]: pgmap v1781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2734.910882 secs 2022-03-21T14:57:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:14 smithi002 ceph-mon[17167]: pgmap v1782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:14 smithi016 ceph-mon[22010]: pgmap v1782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:14 smithi079 ceph-mon[23158]: pgmap v1782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:16 smithi002 ceph-mon[17167]: pgmap v1783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:16 smithi016 ceph-mon[22010]: pgmap v1783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:16 smithi079 ceph-mon[23158]: pgmap v1783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2739.910982 secs 2022-03-21T14:57:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2739.910982 secs 2022-03-21T14:57:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2739.910982 secs 2022-03-21T14:57:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:18 smithi002 ceph-mon[17167]: pgmap v1784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:18 smithi016 ceph-mon[22010]: pgmap v1784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:18 smithi079 ceph-mon[23158]: pgmap v1784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:20 smithi002 ceph-mon[17167]: pgmap v1785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:57:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:57:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:20 smithi016 ceph-mon[22010]: pgmap v1785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:57:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:57:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:20 smithi079 ceph-mon[23158]: pgmap v1785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:57:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:57:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:57:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:57:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:57:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:57:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:57:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:57:22.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:57:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:57:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:57:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: pgmap v1786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2744.911094 secs 2022-03-21T14:57:23.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:57:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: pgmap v1786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2744.911094 secs 2022-03-21T14:57:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:57:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: pgmap v1786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2744.911094 secs 2022-03-21T14:57:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T14:57:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:57:24.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:57:24.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:23 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:57:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:57:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:57:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:57:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:57:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:57:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:57:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T14:57:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T14:57:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T14:57:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T14:57:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T14:57:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T14:57:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:24 smithi002 ceph-mon[17167]: pgmap v1787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:24 smithi016 ceph-mon[22010]: pgmap v1787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:24 smithi079 ceph-mon[23158]: pgmap v1787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:26 smithi002 ceph-mon[17167]: pgmap v1788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:26 smithi016 ceph-mon[22010]: pgmap v1788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:26 smithi079 ceph-mon[23158]: pgmap v1788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2749.911214 secs 2022-03-21T14:57:28.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2749.911214 secs 2022-03-21T14:57:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2749.911214 secs 2022-03-21T14:57:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:28 smithi002 ceph-mon[17167]: pgmap v1789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:28 smithi016 ceph-mon[22010]: pgmap v1789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:28 smithi079 ceph-mon[23158]: pgmap v1789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:30 smithi002 ceph-mon[17167]: pgmap v1790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:30 smithi016 ceph-mon[22010]: pgmap v1790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:30 smithi079 ceph-mon[23158]: pgmap v1790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:32 smithi002 ceph-mon[17167]: pgmap v1791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2754.911331 secs 2022-03-21T14:57:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:32 smithi016 ceph-mon[22010]: pgmap v1791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2754.911331 secs 2022-03-21T14:57:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:32 smithi079 ceph-mon[23158]: pgmap v1791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2754.911331 secs 2022-03-21T14:57:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:34 smithi002 ceph-mon[17167]: pgmap v1792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:34 smithi016 ceph-mon[22010]: pgmap v1792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:34 smithi079 ceph-mon[23158]: pgmap v1792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:36 smithi002 ceph-mon[17167]: pgmap v1793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:36 smithi016 ceph-mon[22010]: pgmap v1793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:36 smithi079 ceph-mon[23158]: pgmap v1793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2759.911467 secs 2022-03-21T14:57:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2759.911467 secs 2022-03-21T14:57:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2759.911467 secs 2022-03-21T14:57:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:38 smithi002 ceph-mon[17167]: pgmap v1794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:38 smithi016 ceph-mon[22010]: pgmap v1794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:38 smithi079 ceph-mon[23158]: pgmap v1794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:40 smithi002 ceph-mon[17167]: pgmap v1795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:40 smithi016 ceph-mon[22010]: pgmap v1795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:40 smithi079 ceph-mon[23158]: pgmap v1795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:42 smithi002 ceph-mon[17167]: pgmap v1796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2764.911539 secs 2022-03-21T14:57:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:42 smithi016 ceph-mon[22010]: pgmap v1796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2764.911539 secs 2022-03-21T14:57:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:42 smithi079 ceph-mon[23158]: pgmap v1796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2764.911539 secs 2022-03-21T14:57:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:44 smithi002 ceph-mon[17167]: pgmap v1797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:57:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:44 smithi016 ceph-mon[22010]: pgmap v1797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:57:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:44 smithi079 ceph-mon[23158]: pgmap v1797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T14:57:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:46 smithi002 ceph-mon[17167]: pgmap v1798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:46 smithi016 ceph-mon[22010]: pgmap v1798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:46 smithi079 ceph-mon[23158]: pgmap v1798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T14:57:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2769.911616 secs 2022-03-21T14:57:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2769.911616 secs 2022-03-21T14:57:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2769.911616 secs 2022-03-21T14:57:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:48 smithi002 ceph-mon[17167]: pgmap v1799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:48 smithi016 ceph-mon[22010]: pgmap v1799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:48 smithi079 ceph-mon[23158]: pgmap v1799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:50 smithi002 ceph-mon[17167]: pgmap v1800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:50 smithi016 ceph-mon[22010]: pgmap v1800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:50 smithi079 ceph-mon[23158]: pgmap v1800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:52 smithi002 ceph-mon[17167]: pgmap v1801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2774.911681 secs 2022-03-21T14:57:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:52 smithi016 ceph-mon[22010]: pgmap v1801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2774.911681 secs 2022-03-21T14:57:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:52 smithi079 ceph-mon[23158]: pgmap v1801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:57:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2774.911681 secs 2022-03-21T14:57:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:54 smithi002 ceph-mon[17167]: pgmap v1802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:54 smithi016 ceph-mon[22010]: pgmap v1802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:54 smithi079 ceph-mon[23158]: pgmap v1802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:57:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:56 smithi002 ceph-mon[17167]: pgmap v1803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:56 smithi016 ceph-mon[22010]: pgmap v1803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:56 smithi079 ceph-mon[23158]: pgmap v1803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:57:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2779.911758 secs 2022-03-21T14:57:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2779.911758 secs 2022-03-21T14:57:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2779.911758 secs 2022-03-21T14:57:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:57:58 smithi002 ceph-mon[17167]: pgmap v1804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:57:58 smithi016 ceph-mon[22010]: pgmap v1804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:57:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:57:58 smithi079 ceph-mon[23158]: pgmap v1804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:00 smithi002 ceph-mon[17167]: pgmap v1805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:00 smithi016 ceph-mon[22010]: pgmap v1805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:00 smithi079 ceph-mon[23158]: pgmap v1805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:02 smithi002 ceph-mon[17167]: pgmap v1806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2784.911855 secs 2022-03-21T14:58:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:02 smithi016 ceph-mon[22010]: pgmap v1806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2784.911855 secs 2022-03-21T14:58:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:02 smithi079 ceph-mon[23158]: pgmap v1806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2784.911855 secs 2022-03-21T14:58:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:04 smithi002 ceph-mon[17167]: pgmap v1807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:04 smithi016 ceph-mon[22010]: pgmap v1807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:04 smithi079 ceph-mon[23158]: pgmap v1807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:06 smithi002 ceph-mon[17167]: pgmap v1808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:06 smithi016 ceph-mon[22010]: pgmap v1808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:06 smithi079 ceph-mon[23158]: pgmap v1808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2789.911931 secs 2022-03-21T14:58:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2789.911931 secs 2022-03-21T14:58:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2789.911931 secs 2022-03-21T14:58:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:08 smithi016 ceph-mon[22010]: pgmap v1809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:08 smithi079 ceph-mon[23158]: pgmap v1809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:08 smithi002 ceph-mon[17167]: pgmap v1809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:10 smithi016 ceph-mon[22010]: pgmap v1810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:10 smithi079 ceph-mon[23158]: pgmap v1810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:10 smithi002 ceph-mon[17167]: pgmap v1810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:13.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:12 smithi016 ceph-mon[22010]: pgmap v1811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2794.912048 secs 2022-03-21T14:58:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:12 smithi079 ceph-mon[23158]: pgmap v1811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2794.912048 secs 2022-03-21T14:58:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:12 smithi002 ceph-mon[17167]: pgmap v1811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2794.912048 secs 2022-03-21T14:58:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:14 smithi016 ceph-mon[22010]: pgmap v1812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:14 smithi079 ceph-mon[23158]: pgmap v1812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:14 smithi002 ceph-mon[17167]: pgmap v1812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:16 smithi016 ceph-mon[22010]: pgmap v1813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:16 smithi079 ceph-mon[23158]: pgmap v1813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:16 smithi002 ceph-mon[17167]: pgmap v1813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2799.912165 secs 2022-03-21T14:58:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2799.912165 secs 2022-03-21T14:58:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2799.912165 secs 2022-03-21T14:58:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:18 smithi016 ceph-mon[22010]: pgmap v1814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:18 smithi079 ceph-mon[23158]: pgmap v1814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:18 smithi002 ceph-mon[17167]: pgmap v1814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:20 smithi016 ceph-mon[22010]: pgmap v1815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:58:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:58:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:20 smithi079 ceph-mon[23158]: pgmap v1815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:58:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:58:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:20 smithi002 ceph-mon[17167]: pgmap v1815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:58:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:58:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:22 smithi016 ceph-mon[22010]: pgmap v1816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2804.912293 secs 2022-03-21T14:58:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:58:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:58:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:58:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:22 smithi079 ceph-mon[23158]: pgmap v1816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2804.912293 secs 2022-03-21T14:58:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:58:23.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:22 smithi002 ceph-mon[17167]: pgmap v1816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:23.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2804.912293 secs 2022-03-21T14:58:23.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:58:23.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:58:23.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:58:24.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:58:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:24 smithi002 ceph-mon[17167]: pgmap v1817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:24.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:58:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:24 smithi016 ceph-mon[22010]: pgmap v1817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:58:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:24 smithi079 ceph-mon[23158]: pgmap v1817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:26 smithi002 ceph-mon[17167]: pgmap v1818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:26 smithi016 ceph-mon[22010]: pgmap v1818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:26 smithi079 ceph-mon[23158]: pgmap v1818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:28.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2809.912392 secs 2022-03-21T14:58:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:28 smithi002 ceph-mon[17167]: pgmap v1819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:28.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2809.912392 secs 2022-03-21T14:58:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:28 smithi016 ceph-mon[22010]: pgmap v1819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2809.912392 secs 2022-03-21T14:58:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:28 smithi079 ceph-mon[23158]: pgmap v1819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:30.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:30 smithi002 ceph-mon[17167]: pgmap v1820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:30 smithi016 ceph-mon[22010]: pgmap v1820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:30 smithi079 ceph-mon[23158]: pgmap v1820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:32 smithi002 ceph-mon[17167]: pgmap v1821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:32.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:32 smithi016 ceph-mon[22010]: pgmap v1821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:32 smithi079 ceph-mon[23158]: pgmap v1821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2814.912528 secs 2022-03-21T14:58:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2814.912528 secs 2022-03-21T14:58:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2814.912528 secs 2022-03-21T14:58:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:34 smithi002 ceph-mon[17167]: pgmap v1822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:34 smithi016 ceph-mon[22010]: pgmap v1822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:34 smithi079 ceph-mon[23158]: pgmap v1822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:36 smithi002 ceph-mon[17167]: pgmap v1823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:36 smithi016 ceph-mon[22010]: pgmap v1823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:36 smithi079 ceph-mon[23158]: pgmap v1823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2819.912661 secs 2022-03-21T14:58:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:38 smithi016 ceph-mon[22010]: pgmap v1824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2819.912661 secs 2022-03-21T14:58:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:38 smithi079 ceph-mon[23158]: pgmap v1824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2819.912661 secs 2022-03-21T14:58:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:38 smithi002 ceph-mon[17167]: pgmap v1824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:40 smithi016 ceph-mon[22010]: pgmap v1825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:40 smithi079 ceph-mon[23158]: pgmap v1825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:40 smithi002 ceph-mon[17167]: pgmap v1825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:42 smithi016 ceph-mon[22010]: pgmap v1826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:42 smithi079 ceph-mon[23158]: pgmap v1826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:42 smithi002 ceph-mon[17167]: pgmap v1826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2824.912724 secs 2022-03-21T14:58:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2824.912724 secs 2022-03-21T14:58:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2824.912724 secs 2022-03-21T14:58:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:44 smithi016 ceph-mon[22010]: pgmap v1827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:44 smithi079 ceph-mon[23158]: pgmap v1827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:44 smithi002 ceph-mon[17167]: pgmap v1827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:46 smithi016 ceph-mon[22010]: pgmap v1828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:46 smithi079 ceph-mon[23158]: pgmap v1828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:46 smithi002 ceph-mon[17167]: pgmap v1828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2829.912814 secs 2022-03-21T14:58:48.014 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2829.912814 secs 2022-03-21T14:58:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2829.912814 secs 2022-03-21T14:58:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:48 smithi016 ceph-mon[22010]: pgmap v1829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:48 smithi079 ceph-mon[23158]: pgmap v1829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:48 smithi002 ceph-mon[17167]: pgmap v1829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:50 smithi016 ceph-mon[22010]: pgmap v1830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:50 smithi079 ceph-mon[23158]: pgmap v1830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:50 smithi002 ceph-mon[17167]: pgmap v1830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:52 smithi016 ceph-mon[22010]: pgmap v1831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2834.912928 secs 2022-03-21T14:58:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:52 smithi079 ceph-mon[23158]: pgmap v1831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2834.912928 secs 2022-03-21T14:58:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:52 smithi002 ceph-mon[17167]: pgmap v1831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:58:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2834.912928 secs 2022-03-21T14:58:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:54 smithi016 ceph-mon[22010]: pgmap v1832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:54 smithi079 ceph-mon[23158]: pgmap v1832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:54 smithi002 ceph-mon[17167]: pgmap v1832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:58:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:56 smithi016 ceph-mon[22010]: pgmap v1833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:56 smithi079 ceph-mon[23158]: pgmap v1833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:56 smithi002 ceph-mon[17167]: pgmap v1833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:58:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2839.913045 secs 2022-03-21T14:58:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2839.913045 secs 2022-03-21T14:58:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2839.913045 secs 2022-03-21T14:58:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:58:58 smithi079 ceph-mon[23158]: pgmap v1834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:58:58 smithi002 ceph-mon[17167]: pgmap v1834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:58:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:58:58 smithi016 ceph-mon[22010]: pgmap v1834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:00 smithi079 ceph-mon[23158]: pgmap v1835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:00 smithi002 ceph-mon[17167]: pgmap v1835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:00 smithi016 ceph-mon[22010]: pgmap v1835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:02 smithi079 ceph-mon[23158]: pgmap v1836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2844.913167 secs 2022-03-21T14:59:03.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:02 smithi002 ceph-mon[17167]: pgmap v1836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:03.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2844.913167 secs 2022-03-21T14:59:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:02 smithi016 ceph-mon[22010]: pgmap v1836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2844.913167 secs 2022-03-21T14:59:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:04 smithi079 ceph-mon[23158]: pgmap v1837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:04 smithi002 ceph-mon[17167]: pgmap v1837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:04 smithi016 ceph-mon[22010]: pgmap v1837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:06 smithi079 ceph-mon[23158]: pgmap v1838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:06 smithi002 ceph-mon[17167]: pgmap v1838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:06 smithi016 ceph-mon[22010]: pgmap v1838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2849.913238 secs 2022-03-21T14:59:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2849.913238 secs 2022-03-21T14:59:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2849.913238 secs 2022-03-21T14:59:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:08 smithi079 ceph-mon[23158]: pgmap v1839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:08 smithi002 ceph-mon[17167]: pgmap v1839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:08 smithi016 ceph-mon[22010]: pgmap v1839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:10 smithi079 ceph-mon[23158]: pgmap v1840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:10 smithi002 ceph-mon[17167]: pgmap v1840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:10 smithi016 ceph-mon[22010]: pgmap v1840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:12 smithi079 ceph-mon[23158]: pgmap v1841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2854.913350 secs 2022-03-21T14:59:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:12 smithi002 ceph-mon[17167]: pgmap v1841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2854.913350 secs 2022-03-21T14:59:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:12 smithi016 ceph-mon[22010]: pgmap v1841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2854.913350 secs 2022-03-21T14:59:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:14 smithi079 ceph-mon[23158]: pgmap v1842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:14 smithi002 ceph-mon[17167]: pgmap v1842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:14 smithi016 ceph-mon[22010]: pgmap v1842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:16 smithi079 ceph-mon[23158]: pgmap v1843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:16 smithi002 ceph-mon[17167]: pgmap v1843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:16 smithi016 ceph-mon[22010]: pgmap v1843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2859.913468 secs 2022-03-21T14:59:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2859.913468 secs 2022-03-21T14:59:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2859.913468 secs 2022-03-21T14:59:18.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:18 smithi079 ceph-mon[23158]: pgmap v1844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:18 smithi002 ceph-mon[17167]: pgmap v1844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:18 smithi016 ceph-mon[22010]: pgmap v1844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:20 smithi079 ceph-mon[23158]: pgmap v1845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:59:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:59:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:20 smithi002 ceph-mon[17167]: pgmap v1845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:59:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:59:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:20 smithi016 ceph-mon[22010]: pgmap v1845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T14:59:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T14:59:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:22 smithi079 ceph-mon[23158]: pgmap v1846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2864.913586 secs 2022-03-21T14:59:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:22 smithi002 ceph-mon[17167]: pgmap v1846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2864.913586 secs 2022-03-21T14:59:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:22 smithi016 ceph-mon[22010]: pgmap v1846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2864.913586 secs 2022-03-21T14:59:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:59:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:59:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:59:24.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:59:24.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:59:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:59:24.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T14:59:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T14:59:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T14:59:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:24 smithi002 ceph-mon[17167]: pgmap v1847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:59:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:24 smithi016 ceph-mon[22010]: pgmap v1847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:59:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:24 smithi079 ceph-mon[23158]: pgmap v1847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T14:59:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:26 smithi002 ceph-mon[17167]: pgmap v1848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:26 smithi016 ceph-mon[22010]: pgmap v1848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:26 smithi079 ceph-mon[23158]: pgmap v1848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2869.913663 secs 2022-03-21T14:59:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2869.913663 secs 2022-03-21T14:59:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2869.913663 secs 2022-03-21T14:59:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:28 smithi002 ceph-mon[17167]: pgmap v1849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:29.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:28 smithi016 ceph-mon[22010]: pgmap v1849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:28 smithi079 ceph-mon[23158]: pgmap v1849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:30 smithi002 ceph-mon[17167]: pgmap v1850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:30 smithi016 ceph-mon[22010]: pgmap v1850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:30 smithi079 ceph-mon[23158]: pgmap v1850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:32 smithi002 ceph-mon[17167]: pgmap v1851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2874.913760 secs 2022-03-21T14:59:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:32 smithi016 ceph-mon[22010]: pgmap v1851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2874.913760 secs 2022-03-21T14:59:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:32 smithi079 ceph-mon[23158]: pgmap v1851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2874.913760 secs 2022-03-21T14:59:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:34 smithi002 ceph-mon[17167]: pgmap v1852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:34 smithi016 ceph-mon[22010]: pgmap v1852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:34 smithi079 ceph-mon[23158]: pgmap v1852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:36 smithi002 ceph-mon[17167]: pgmap v1853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:36 smithi016 ceph-mon[22010]: pgmap v1853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:36 smithi079 ceph-mon[23158]: pgmap v1853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2879.913825 secs 2022-03-21T14:59:38.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2879.913825 secs 2022-03-21T14:59:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2879.913825 secs 2022-03-21T14:59:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:38 smithi002 ceph-mon[17167]: pgmap v1854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:39.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:38 smithi016 ceph-mon[22010]: pgmap v1854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:38 smithi079 ceph-mon[23158]: pgmap v1854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:40 smithi002 ceph-mon[17167]: pgmap v1855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:40 smithi016 ceph-mon[22010]: pgmap v1855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:40 smithi079 ceph-mon[23158]: pgmap v1855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: pgmap v1856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2884.913968 secs 2022-03-21T14:59:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: slow request 2884.809197 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: slow request 2884.598013 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: slow request 2884.910607 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: slow request 2884.861469 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:42 smithi002 ceph-mon[17167]: slow request 2884.836209 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: pgmap v1856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2884.913968 secs 2022-03-21T14:59:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: slow request 2884.809197 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: slow request 2884.598013 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: slow request 2884.910607 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: slow request 2884.861469 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:42 smithi016 ceph-mon[22010]: slow request 2884.836209 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: pgmap v1856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2884.913968 secs 2022-03-21T14:59:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: slow request 2884.809197 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: slow request 2884.598013 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: slow request 2884.910607 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: slow request 2884.861469 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:42 smithi079 ceph-mon[23158]: slow request 2884.836209 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:44 smithi002 ceph-mon[17167]: pgmap v1857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:44 smithi016 ceph-mon[22010]: pgmap v1857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:44 smithi079 ceph-mon[23158]: pgmap v1857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:46 smithi002 ceph-mon[17167]: pgmap v1858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:46 smithi016 ceph-mon[22010]: pgmap v1858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:46 smithi079 ceph-mon[23158]: pgmap v1858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2889.914103 secs 2022-03-21T14:59:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: slow request 2889.806065 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: slow request 2889.885397 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: slow request 2889.803106 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: slow request 2889.907302 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:47 smithi002 ceph-mon[17167]: slow request 2889.858327 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2889.914103 secs 2022-03-21T14:59:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: slow request 2889.806065 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: slow request 2889.885397 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: slow request 2889.803106 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: slow request 2889.907302 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:47 smithi016 ceph-mon[22010]: slow request 2889.858327 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2889.914103 secs 2022-03-21T14:59:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: slow request 2889.806065 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: slow request 2889.885397 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: slow request 2889.803106 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: slow request 2889.907302 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:47 smithi079 ceph-mon[23158]: slow request 2889.858327 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:48 smithi002 ceph-mon[17167]: pgmap v1859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:49.303 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:48 smithi079 ceph-mon[23158]: pgmap v1859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:48 smithi016 ceph-mon[22010]: pgmap v1859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:50 smithi002 ceph-mon[17167]: pgmap v1860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:50 smithi016 ceph-mon[22010]: pgmap v1860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:50 smithi079 ceph-mon[23158]: pgmap v1860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: pgmap v1861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2894.914225 secs 2022-03-21T14:59:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: slow request 2894.800210 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: slow request 2894.882794 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: slow request 2894.832368 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: slow request 2894.797463 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:52 smithi002 ceph-mon[17167]: slow request 2894.904246 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: pgmap v1861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2894.914225 secs 2022-03-21T14:59:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: slow request 2894.800210 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: slow request 2894.882794 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: slow request 2894.832368 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: slow request 2894.797463 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:52 smithi016 ceph-mon[22010]: slow request 2894.904246 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: pgmap v1861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T14:59:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2894.914225 secs 2022-03-21T14:59:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: slow request 2894.800210 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: slow request 2894.882794 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: slow request 2894.832368 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: slow request 2894.797463 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:52 smithi079 ceph-mon[23158]: slow request 2894.904246 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:54 smithi002 ceph-mon[17167]: pgmap v1862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:54 smithi016 ceph-mon[22010]: pgmap v1862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:54 smithi079 ceph-mon[23158]: pgmap v1862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T14:59:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:56 smithi002 ceph-mon[17167]: pgmap v1863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:56 smithi016 ceph-mon[22010]: pgmap v1863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:56 smithi079 ceph-mon[23158]: pgmap v1863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T14:59:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2899.914374 secs 2022-03-21T14:59:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: slow request 2899.880131 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: slow request 2899.855278 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: slow request 2899.828366 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: slow request 2899.901478 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:57 smithi002 ceph-mon[17167]: slow request 2899.877064 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2899.914374 secs 2022-03-21T14:59:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: slow request 2899.880131 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: slow request 2899.855278 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: slow request 2899.828366 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: slow request 2899.901478 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:57 smithi016 ceph-mon[22010]: slow request 2899.877064 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2899.914374 secs 2022-03-21T14:59:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: slow request 2899.880131 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: slow request 2899.855278 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: slow request 2899.828366 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: slow request 2899.901478 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:57 smithi079 ceph-mon[23158]: slow request 2899.877064 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T14:59:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 14:59:58 smithi002 ceph-mon[17167]: pgmap v1864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 14:59:58 smithi016 ceph-mon[22010]: pgmap v1864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T14:59:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 14:59:58 smithi079 ceph-mon[23158]: pgmap v1864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:00 smithi002 ceph-mon[17167]: pgmap v1865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:00:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:00 smithi016 ceph-mon[22010]: pgmap v1865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:00:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:00 smithi079 ceph-mon[23158]: pgmap v1865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:00:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: pgmap v1866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2904.914514 secs 2022-03-21T15:00:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: slow request 2904.852615 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: slow request 2904.825770 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: slow request 2904.849652 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: slow request 2904.898233 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:02 smithi002 ceph-mon[17167]: slow request 2904.874530 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: pgmap v1866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2904.914514 secs 2022-03-21T15:00:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: slow request 2904.852615 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: slow request 2904.825770 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: slow request 2904.849652 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: slow request 2904.898233 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:02 smithi016 ceph-mon[22010]: slow request 2904.874530 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: pgmap v1866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2904.914514 secs 2022-03-21T15:00:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: slow request 2904.852615 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: slow request 2904.825770 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: slow request 2904.849652 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: slow request 2904.898233 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:02 smithi079 ceph-mon[23158]: slow request 2904.874530 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:04 smithi002 ceph-mon[17167]: pgmap v1867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:04 smithi016 ceph-mon[22010]: pgmap v1867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:04 smithi079 ceph-mon[23158]: pgmap v1867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:06 smithi002 ceph-mon[17167]: pgmap v1868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:06 smithi016 ceph-mon[22010]: pgmap v1868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:06 smithi079 ceph-mon[23158]: pgmap v1868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2909.914595 secs 2022-03-21T15:00:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: slow request 2909.822657 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: slow request 2909.871762 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: slow request 2909.846658 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: slow request 2909.895253 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:07 smithi002 ceph-mon[17167]: slow request 2909.818896 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2909.914595 secs 2022-03-21T15:00:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: slow request 2909.822657 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: slow request 2909.871762 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: slow request 2909.846658 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: slow request 2909.895253 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:07 smithi016 ceph-mon[22010]: slow request 2909.818896 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2909.914595 secs 2022-03-21T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: slow request 2909.822657 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: slow request 2909.871762 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: slow request 2909.846658 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: slow request 2909.895253 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:07 smithi079 ceph-mon[23158]: slow request 2909.818896 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:08 smithi002 ceph-mon[17167]: pgmap v1869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:08 smithi016 ceph-mon[22010]: pgmap v1869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:08 smithi079 ceph-mon[23158]: pgmap v1869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:10 smithi002 ceph-mon[17167]: pgmap v1870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:10 smithi016 ceph-mon[22010]: pgmap v1870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:10 smithi079 ceph-mon[23158]: pgmap v1870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: pgmap v1871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2914.914723 secs 2022-03-21T15:00:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: slow request 2914.798617 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:00:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: slow request 2914.816021 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: slow request 2914.891922 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: slow request 2914.868521 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:12 smithi002 ceph-mon[17167]: slow request 2914.843463 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: pgmap v1871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2914.914723 secs 2022-03-21T15:00:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: slow request 2914.798617 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:00:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: slow request 2914.816021 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: slow request 2914.891922 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: slow request 2914.868521 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:12 smithi016 ceph-mon[22010]: slow request 2914.843463 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: pgmap v1871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2914.914723 secs 2022-03-21T15:00:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: slow request 2914.798617 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:00:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: slow request 2914.816021 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: slow request 2914.891922 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: slow request 2914.868521 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:12 smithi079 ceph-mon[23158]: slow request 2914.843463 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:14 smithi002 ceph-mon[17167]: pgmap v1872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:14 smithi016 ceph-mon[22010]: pgmap v1872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:14 smithi079 ceph-mon[23158]: pgmap v1872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:16 smithi002 ceph-mon[17167]: pgmap v1873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:16 smithi016 ceph-mon[22010]: pgmap v1873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:16 smithi079 ceph-mon[23158]: pgmap v1873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 2919.914802 secs 2022-03-21T15:00:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: slow request 2919.812778 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: slow request 2919.914801 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: slow request 2919.865578 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: slow request 2919.840628 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:17 smithi002 ceph-mon[17167]: slow request 2919.889075 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 2919.914802 secs 2022-03-21T15:00:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: slow request 2919.812778 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: slow request 2919.914801 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: slow request 2919.865578 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: slow request 2919.840628 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:17 smithi016 ceph-mon[22010]: slow request 2919.889075 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 2919.914802 secs 2022-03-21T15:00:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: slow request 2919.812778 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: slow request 2919.914801 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: slow request 2919.865578 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: slow request 2919.840628 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:17 smithi079 ceph-mon[23158]: slow request 2919.889075 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:00:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:18 smithi002 ceph-mon[17167]: pgmap v1874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:18 smithi016 ceph-mon[22010]: pgmap v1874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:18 smithi079 ceph-mon[23158]: pgmap v1874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:20 smithi002 ceph-mon[17167]: pgmap v1875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:00:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:00:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:20 smithi016 ceph-mon[22010]: pgmap v1875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:00:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:00:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:20 smithi079 ceph-mon[23158]: pgmap v1875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:00:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:00:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:22 smithi002 ceph-mon[17167]: pgmap v1876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2924.914918 secs 2022-03-21T15:00:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:22 smithi016 ceph-mon[22010]: pgmap v1876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2924.914918 secs 2022-03-21T15:00:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:22 smithi079 ceph-mon[23158]: pgmap v1876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2924.914918 secs 2022-03-21T15:00:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:24 smithi002 ceph-mon[17167]: pgmap v1877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:00:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:00:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:00:25.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:24 smithi016 ceph-mon[22010]: pgmap v1877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:00:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:00:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:00:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:24 smithi079 ceph-mon[23158]: pgmap v1877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:00:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:00:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:00:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:26 smithi079 ceph-mon[23158]: pgmap v1878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:26 smithi002 ceph-mon[17167]: pgmap v1878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:26 smithi016 ceph-mon[22010]: pgmap v1878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2929.915038 secs 2022-03-21T15:00:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2929.915038 secs 2022-03-21T15:00:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2929.915038 secs 2022-03-21T15:00:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:28 smithi002 ceph-mon[17167]: pgmap v1879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:28 smithi079 ceph-mon[23158]: pgmap v1879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:28 smithi016 ceph-mon[22010]: pgmap v1879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:00:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:30 smithi002 ceph-mon[17167]: pgmap v1880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:30 smithi016 ceph-mon[22010]: pgmap v1880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:30 smithi079 ceph-mon[23158]: pgmap v1880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:32 smithi002 ceph-mon[17167]: pgmap v1881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2934.915155 secs 2022-03-21T15:00:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:32 smithi016 ceph-mon[22010]: pgmap v1881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2934.915155 secs 2022-03-21T15:00:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:32 smithi079 ceph-mon[23158]: pgmap v1881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2934.915155 secs 2022-03-21T15:00:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:34 smithi002 ceph-mon[17167]: pgmap v1882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:34 smithi016 ceph-mon[22010]: pgmap v1882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:34 smithi079 ceph-mon[23158]: pgmap v1882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:36 smithi002 ceph-mon[17167]: pgmap v1883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:36 smithi016 ceph-mon[22010]: pgmap v1883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:36 smithi079 ceph-mon[23158]: pgmap v1883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2939.915274 secs 2022-03-21T15:00:38.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2939.915274 secs 2022-03-21T15:00:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2939.915274 secs 2022-03-21T15:00:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:38 smithi002 ceph-mon[17167]: pgmap v1884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:38 smithi016 ceph-mon[22010]: pgmap v1884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:38 smithi079 ceph-mon[23158]: pgmap v1884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:40 smithi016 ceph-mon[22010]: pgmap v1885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:40 smithi079 ceph-mon[23158]: pgmap v1885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:40 smithi002 ceph-mon[17167]: pgmap v1885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:42 smithi016 ceph-mon[22010]: pgmap v1886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2944.915371 secs 2022-03-21T15:00:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:42 smithi079 ceph-mon[23158]: pgmap v1886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2944.915371 secs 2022-03-21T15:00:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:42 smithi002 ceph-mon[17167]: pgmap v1886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2944.915371 secs 2022-03-21T15:00:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:44 smithi016 ceph-mon[22010]: pgmap v1887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:44 smithi079 ceph-mon[23158]: pgmap v1887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:44 smithi002 ceph-mon[17167]: pgmap v1887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:46 smithi016 ceph-mon[22010]: pgmap v1888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:46 smithi079 ceph-mon[23158]: pgmap v1888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:46 smithi002 ceph-mon[17167]: pgmap v1888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:47.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2949.915488 secs 2022-03-21T15:00:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2949.915488 secs 2022-03-21T15:00:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2949.915488 secs 2022-03-21T15:00:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:48 smithi016 ceph-mon[22010]: pgmap v1889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:48 smithi079 ceph-mon[23158]: pgmap v1889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:48 smithi002 ceph-mon[17167]: pgmap v1889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:50 smithi079 ceph-mon[23158]: pgmap v1890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:50 smithi002 ceph-mon[17167]: pgmap v1890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:50 smithi016 ceph-mon[22010]: pgmap v1890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:52 smithi079 ceph-mon[23158]: pgmap v1891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2954.915619 secs 2022-03-21T15:00:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:52 smithi002 ceph-mon[17167]: pgmap v1891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2954.915619 secs 2022-03-21T15:00:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:52 smithi016 ceph-mon[22010]: pgmap v1891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:00:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2954.915619 secs 2022-03-21T15:00:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:54 smithi079 ceph-mon[23158]: pgmap v1892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:54 smithi002 ceph-mon[17167]: pgmap v1892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:54 smithi016 ceph-mon[22010]: pgmap v1892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:00:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:56 smithi079 ceph-mon[23158]: pgmap v1893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:56 smithi002 ceph-mon[17167]: pgmap v1893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:56 smithi016 ceph-mon[22010]: pgmap v1893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:00:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2959.915763 secs 2022-03-21T15:00:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2959.915763 secs 2022-03-21T15:00:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2959.915763 secs 2022-03-21T15:00:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:00:58 smithi079 ceph-mon[23158]: pgmap v1894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:00:58 smithi002 ceph-mon[17167]: pgmap v1894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:00:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:00:58 smithi016 ceph-mon[22010]: pgmap v1894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:00 smithi079 ceph-mon[23158]: pgmap v1895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:00 smithi002 ceph-mon[17167]: pgmap v1895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:00 smithi016 ceph-mon[22010]: pgmap v1895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:02 smithi079 ceph-mon[23158]: pgmap v1896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2964.915843 secs 2022-03-21T15:01:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:02 smithi002 ceph-mon[17167]: pgmap v1896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2964.915843 secs 2022-03-21T15:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:02 smithi016 ceph-mon[22010]: pgmap v1896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2964.915843 secs 2022-03-21T15:01:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:04 smithi079 ceph-mon[23158]: pgmap v1897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:04 smithi002 ceph-mon[17167]: pgmap v1897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:04 smithi016 ceph-mon[22010]: pgmap v1897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:06 smithi079 ceph-mon[23158]: pgmap v1898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:06 smithi002 ceph-mon[17167]: pgmap v1898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:06 smithi016 ceph-mon[22010]: pgmap v1898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2969.915918 secs 2022-03-21T15:01:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2969.915918 secs 2022-03-21T15:01:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2969.915918 secs 2022-03-21T15:01:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:08 smithi079 ceph-mon[23158]: pgmap v1899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:08 smithi002 ceph-mon[17167]: pgmap v1899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:08 smithi016 ceph-mon[22010]: pgmap v1899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:10 smithi079 ceph-mon[23158]: pgmap v1900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:10 smithi002 ceph-mon[17167]: pgmap v1900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:10 smithi016 ceph-mon[22010]: pgmap v1900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:12 smithi079 ceph-mon[23158]: pgmap v1901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2974.916040 secs 2022-03-21T15:01:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:12 smithi002 ceph-mon[17167]: pgmap v1901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2974.916040 secs 2022-03-21T15:01:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:12 smithi016 ceph-mon[22010]: pgmap v1901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2974.916040 secs 2022-03-21T15:01:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:14 smithi079 ceph-mon[23158]: pgmap v1902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:14 smithi002 ceph-mon[17167]: pgmap v1902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:14 smithi016 ceph-mon[22010]: pgmap v1902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:16 smithi079 ceph-mon[23158]: pgmap v1903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:16 smithi002 ceph-mon[17167]: pgmap v1903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:16 smithi016 ceph-mon[22010]: pgmap v1903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2979.916154 secs 2022-03-21T15:01:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2979.916154 secs 2022-03-21T15:01:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2979.916154 secs 2022-03-21T15:01:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:18 smithi002 ceph-mon[17167]: pgmap v1904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:18 smithi016 ceph-mon[22010]: pgmap v1904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:18 smithi079 ceph-mon[23158]: pgmap v1904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:20 smithi002 ceph-mon[17167]: pgmap v1905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:01:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:01:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:20 smithi016 ceph-mon[22010]: pgmap v1905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:01:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:01:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:20 smithi079 ceph-mon[23158]: pgmap v1905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:01:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:01:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:22 smithi002 ceph-mon[17167]: pgmap v1906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2984.916229 secs 2022-03-21T15:01:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:22 smithi016 ceph-mon[22010]: pgmap v1906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2984.916229 secs 2022-03-21T15:01:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:22 smithi079 ceph-mon[23158]: pgmap v1906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2984.916229 secs 2022-03-21T15:01:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:24 smithi002 ceph-mon[17167]: pgmap v1907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:25.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:24 smithi016 ceph-mon[22010]: pgmap v1907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:24 smithi079 ceph-mon[23158]: pgmap v1907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:26 smithi002 ceph-mon[17167]: pgmap v1908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:26 smithi016 ceph-mon[22010]: pgmap v1908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:26 smithi079 ceph-mon[23158]: pgmap v1908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2989.916337 secs 2022-03-21T15:01:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2989.916337 secs 2022-03-21T15:01:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2989.916337 secs 2022-03-21T15:01:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:28 smithi002 ceph-mon[17167]: pgmap v1909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:29.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:28 smithi016 ceph-mon[22010]: pgmap v1909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:28 smithi079 ceph-mon[23158]: pgmap v1909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:01:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:01:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:01:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:30.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:01:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:01:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:01:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:01:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:01:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:01:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:30 smithi002 ceph-mon[17167]: pgmap v1910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:30 smithi016 ceph-mon[22010]: pgmap v1910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:30 smithi079 ceph-mon[23158]: pgmap v1910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:01:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:32 smithi002 ceph-mon[17167]: pgmap v1911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2994.916475 secs 2022-03-21T15:01:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:32 smithi016 ceph-mon[22010]: pgmap v1911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2994.916475 secs 2022-03-21T15:01:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:32 smithi079 ceph-mon[23158]: pgmap v1911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2994.916475 secs 2022-03-21T15:01:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:34 smithi002 ceph-mon[17167]: pgmap v1912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:34 smithi016 ceph-mon[22010]: pgmap v1912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:34 smithi079 ceph-mon[23158]: pgmap v1912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:36 smithi002 ceph-mon[17167]: pgmap v1913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:36 smithi016 ceph-mon[22010]: pgmap v1913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:36 smithi079 ceph-mon[23158]: pgmap v1913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 2999.916593 secs 2022-03-21T15:01:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 2999.916593 secs 2022-03-21T15:01:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 2999.916593 secs 2022-03-21T15:01:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:38 smithi002 ceph-mon[17167]: pgmap v1914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:38 smithi016 ceph-mon[22010]: pgmap v1914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:38 smithi079 ceph-mon[23158]: pgmap v1914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:40 smithi002 ceph-mon[17167]: pgmap v1915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:40 smithi016 ceph-mon[22010]: pgmap v1915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:40 smithi079 ceph-mon[23158]: pgmap v1915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:42 smithi002 ceph-mon[17167]: pgmap v1916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3004.916685 secs 2022-03-21T15:01:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:42 smithi016 ceph-mon[22010]: pgmap v1916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3004.916685 secs 2022-03-21T15:01:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:42 smithi079 ceph-mon[23158]: pgmap v1916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3004.916685 secs 2022-03-21T15:01:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:44 smithi002 ceph-mon[17167]: pgmap v1917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:44 smithi016 ceph-mon[22010]: pgmap v1917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:44 smithi079 ceph-mon[23158]: pgmap v1917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:46 smithi002 ceph-mon[17167]: pgmap v1918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:46 smithi016 ceph-mon[22010]: pgmap v1918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:46 smithi079 ceph-mon[23158]: pgmap v1918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3009.916765 secs 2022-03-21T15:01:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3009.916765 secs 2022-03-21T15:01:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3009.916765 secs 2022-03-21T15:01:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:48 smithi002 ceph-mon[17167]: pgmap v1919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:48 smithi016 ceph-mon[22010]: pgmap v1919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:48 smithi079 ceph-mon[23158]: pgmap v1919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:50 smithi002 ceph-mon[17167]: pgmap v1920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:50 smithi016 ceph-mon[22010]: pgmap v1920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:50 smithi079 ceph-mon[23158]: pgmap v1920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:52 smithi002 ceph-mon[17167]: pgmap v1921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3014.916839 secs 2022-03-21T15:01:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:52 smithi016 ceph-mon[22010]: pgmap v1921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3014.916839 secs 2022-03-21T15:01:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:52 smithi079 ceph-mon[23158]: pgmap v1921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:01:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3014.916839 secs 2022-03-21T15:01:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:54 smithi002 ceph-mon[17167]: pgmap v1922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:54 smithi016 ceph-mon[22010]: pgmap v1922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:54 smithi079 ceph-mon[23158]: pgmap v1922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:01:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:56 smithi002 ceph-mon[17167]: pgmap v1923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:56 smithi016 ceph-mon[22010]: pgmap v1923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:56 smithi079 ceph-mon[23158]: pgmap v1923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:01:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3019.916972 secs 2022-03-21T15:01:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3019.916972 secs 2022-03-21T15:01:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3019.916972 secs 2022-03-21T15:01:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:01:58 smithi002 ceph-mon[17167]: pgmap v1924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:01:58 smithi016 ceph-mon[22010]: pgmap v1924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:01:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:01:58 smithi079 ceph-mon[23158]: pgmap v1924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:00 smithi002 ceph-mon[17167]: pgmap v1925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:00 smithi016 ceph-mon[22010]: pgmap v1925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:00 smithi079 ceph-mon[23158]: pgmap v1925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:02 smithi002 ceph-mon[17167]: pgmap v1926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3024.917089 secs 2022-03-21T15:02:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:02 smithi016 ceph-mon[22010]: pgmap v1926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3024.917089 secs 2022-03-21T15:02:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:02 smithi079 ceph-mon[23158]: pgmap v1926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3024.917089 secs 2022-03-21T15:02:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:04 smithi002 ceph-mon[17167]: pgmap v1927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:04 smithi016 ceph-mon[22010]: pgmap v1927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:04 smithi079 ceph-mon[23158]: pgmap v1927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:06 smithi002 ceph-mon[17167]: pgmap v1928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:06 smithi016 ceph-mon[22010]: pgmap v1928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:06 smithi079 ceph-mon[23158]: pgmap v1928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3029.917227 secs 2022-03-21T15:02:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3029.917227 secs 2022-03-21T15:02:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3029.917227 secs 2022-03-21T15:02:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:08 smithi002 ceph-mon[17167]: pgmap v1929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:08 smithi016 ceph-mon[22010]: pgmap v1929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:08 smithi079 ceph-mon[23158]: pgmap v1929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:10 smithi002 ceph-mon[17167]: pgmap v1930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:10 smithi016 ceph-mon[22010]: pgmap v1930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:10 smithi079 ceph-mon[23158]: pgmap v1930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:12 smithi002 ceph-mon[17167]: pgmap v1931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3034.917346 secs 2022-03-21T15:02:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:12 smithi016 ceph-mon[22010]: pgmap v1931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3034.917346 secs 2022-03-21T15:02:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:12 smithi079 ceph-mon[23158]: pgmap v1931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3034.917346 secs 2022-03-21T15:02:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:14 smithi002 ceph-mon[17167]: pgmap v1932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:14 smithi016 ceph-mon[22010]: pgmap v1932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:14 smithi079 ceph-mon[23158]: pgmap v1932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:16 smithi002 ceph-mon[17167]: pgmap v1933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:16 smithi016 ceph-mon[22010]: pgmap v1933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:16 smithi079 ceph-mon[23158]: pgmap v1933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3039.917476 secs 2022-03-21T15:02:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3039.917476 secs 2022-03-21T15:02:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3039.917476 secs 2022-03-21T15:02:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:18 smithi016 ceph-mon[22010]: pgmap v1934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:18 smithi079 ceph-mon[23158]: pgmap v1934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:18 smithi002 ceph-mon[17167]: pgmap v1934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:20 smithi016 ceph-mon[22010]: pgmap v1935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:02:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:02:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:20 smithi079 ceph-mon[23158]: pgmap v1935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:02:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:02:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:20 smithi002 ceph-mon[17167]: pgmap v1935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:02:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:02:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:22 smithi016 ceph-mon[22010]: pgmap v1936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3044.917594 secs 2022-03-21T15:02:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:22 smithi079 ceph-mon[23158]: pgmap v1936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3044.917594 secs 2022-03-21T15:02:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:22 smithi002 ceph-mon[17167]: pgmap v1936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3044.917594 secs 2022-03-21T15:02:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:24 smithi016 ceph-mon[22010]: pgmap v1937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:24 smithi079 ceph-mon[23158]: pgmap v1937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:24 smithi002 ceph-mon[17167]: pgmap v1937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:26 smithi016 ceph-mon[22010]: pgmap v1938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:26 smithi079 ceph-mon[23158]: pgmap v1938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:26 smithi002 ceph-mon[17167]: pgmap v1938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3049.917632 secs 2022-03-21T15:02:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3049.917632 secs 2022-03-21T15:02:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3049.917632 secs 2022-03-21T15:02:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:28 smithi016 ceph-mon[22010]: pgmap v1939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:28 smithi079 ceph-mon[23158]: pgmap v1939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:28 smithi002 ceph-mon[17167]: pgmap v1939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: pgmap v1940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:02:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:02:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:02:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:31.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: pgmap v1940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:31.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:02:31.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:02:31.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:02:31.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:31.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: pgmap v1940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:02:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:02:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:02:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:02:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:32 smithi079 ceph-mon[23158]: pgmap v1941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3054.917701 secs 2022-03-21T15:02:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:32 smithi002 ceph-mon[17167]: pgmap v1941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3054.917701 secs 2022-03-21T15:02:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:32 smithi016 ceph-mon[22010]: pgmap v1941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3054.917701 secs 2022-03-21T15:02:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:34 smithi079 ceph-mon[23158]: pgmap v1942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:34 smithi002 ceph-mon[17167]: pgmap v1942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:34 smithi016 ceph-mon[22010]: pgmap v1942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:36 smithi079 ceph-mon[23158]: pgmap v1943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:36 smithi002 ceph-mon[17167]: pgmap v1943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:36 smithi016 ceph-mon[22010]: pgmap v1943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3059.917767 secs 2022-03-21T15:02:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3059.917767 secs 2022-03-21T15:02:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3059.917767 secs 2022-03-21T15:02:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:38 smithi079 ceph-mon[23158]: pgmap v1944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:38 smithi002 ceph-mon[17167]: pgmap v1944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:38 smithi016 ceph-mon[22010]: pgmap v1944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:40 smithi079 ceph-mon[23158]: pgmap v1945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:40 smithi002 ceph-mon[17167]: pgmap v1945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:40 smithi016 ceph-mon[22010]: pgmap v1945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:42 smithi079 ceph-mon[23158]: pgmap v1946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3064.917876 secs 2022-03-21T15:02:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:42 smithi002 ceph-mon[17167]: pgmap v1946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3064.917876 secs 2022-03-21T15:02:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:42 smithi016 ceph-mon[22010]: pgmap v1946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3064.917876 secs 2022-03-21T15:02:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:44 smithi079 ceph-mon[23158]: pgmap v1947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:44 smithi002 ceph-mon[17167]: pgmap v1947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:44 smithi016 ceph-mon[22010]: pgmap v1947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:46 smithi079 ceph-mon[23158]: pgmap v1948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:46 smithi002 ceph-mon[17167]: pgmap v1948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:46 smithi016 ceph-mon[22010]: pgmap v1948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3069.918012 secs 2022-03-21T15:02:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3069.918012 secs 2022-03-21T15:02:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3069.918012 secs 2022-03-21T15:02:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:48 smithi079 ceph-mon[23158]: pgmap v1949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:48 smithi002 ceph-mon[17167]: pgmap v1949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:48 smithi016 ceph-mon[22010]: pgmap v1949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:50 smithi079 ceph-mon[23158]: pgmap v1950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:50 smithi002 ceph-mon[17167]: pgmap v1950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:50 smithi016 ceph-mon[22010]: pgmap v1950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:52 smithi079 ceph-mon[23158]: pgmap v1951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3074.918144 secs 2022-03-21T15:02:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:52 smithi002 ceph-mon[17167]: pgmap v1951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3074.918144 secs 2022-03-21T15:02:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:52 smithi016 ceph-mon[22010]: pgmap v1951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:02:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3074.918144 secs 2022-03-21T15:02:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:54 smithi002 ceph-mon[17167]: pgmap v1952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:54 smithi016 ceph-mon[22010]: pgmap v1952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:54 smithi079 ceph-mon[23158]: pgmap v1952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:02:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:56 smithi002 ceph-mon[17167]: pgmap v1953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:56 smithi016 ceph-mon[22010]: pgmap v1953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:56 smithi079 ceph-mon[23158]: pgmap v1953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:02:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3079.918255 secs 2022-03-21T15:02:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3079.918255 secs 2022-03-21T15:02:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3079.918255 secs 2022-03-21T15:02:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:02:58 smithi002 ceph-mon[17167]: pgmap v1954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:02:58 smithi016 ceph-mon[22010]: pgmap v1954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:02:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:02:58 smithi079 ceph-mon[23158]: pgmap v1954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:00 smithi002 ceph-mon[17167]: pgmap v1955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:01.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:00 smithi016 ceph-mon[22010]: pgmap v1955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:00 smithi079 ceph-mon[23158]: pgmap v1955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:02 smithi002 ceph-mon[17167]: pgmap v1956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3084.918355 secs 2022-03-21T15:03:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:02 smithi016 ceph-mon[22010]: pgmap v1956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3084.918355 secs 2022-03-21T15:03:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:02 smithi079 ceph-mon[23158]: pgmap v1956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3084.918355 secs 2022-03-21T15:03:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:04 smithi002 ceph-mon[17167]: pgmap v1957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:04 smithi016 ceph-mon[22010]: pgmap v1957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:04 smithi079 ceph-mon[23158]: pgmap v1957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:06 smithi002 ceph-mon[17167]: pgmap v1958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:06 smithi016 ceph-mon[22010]: pgmap v1958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:07.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:06 smithi079 ceph-mon[23158]: pgmap v1958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3089.918472 secs 2022-03-21T15:03:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3089.918472 secs 2022-03-21T15:03:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3089.918472 secs 2022-03-21T15:03:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:09 smithi002 ceph-mon[17167]: pgmap v1959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:09 smithi016 ceph-mon[22010]: pgmap v1959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:09 smithi079 ceph-mon[23158]: pgmap v1959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:11 smithi002 ceph-mon[17167]: pgmap v1960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:11 smithi016 ceph-mon[22010]: pgmap v1960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:11 smithi079 ceph-mon[23158]: pgmap v1960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:13 smithi002 ceph-mon[17167]: pgmap v1961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3094.918602 secs 2022-03-21T15:03:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:13 smithi016 ceph-mon[22010]: pgmap v1961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3094.918602 secs 2022-03-21T15:03:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:13 smithi079 ceph-mon[23158]: pgmap v1961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3094.918602 secs 2022-03-21T15:03:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:15 smithi002 ceph-mon[17167]: pgmap v1962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:15 smithi016 ceph-mon[22010]: pgmap v1962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:15 smithi079 ceph-mon[23158]: pgmap v1962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:17 smithi002 ceph-mon[17167]: pgmap v1963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:17 smithi016 ceph-mon[22010]: pgmap v1963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:17 smithi079 ceph-mon[23158]: pgmap v1963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3099.918728 secs 2022-03-21T15:03:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:18 smithi002 ceph-mon[17167]: pgmap v1964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3099.918728 secs 2022-03-21T15:03:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:18 smithi016 ceph-mon[22010]: pgmap v1964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3099.918728 secs 2022-03-21T15:03:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:18 smithi079 ceph-mon[23158]: pgmap v1964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:20 smithi079 ceph-mon[23158]: pgmap v1965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:03:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:03:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:20 smithi002 ceph-mon[17167]: pgmap v1965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:03:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:03:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:20 smithi016 ceph-mon[22010]: pgmap v1965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:03:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:03:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:22 smithi079 ceph-mon[23158]: pgmap v1966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3104.918852 secs 2022-03-21T15:03:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:22 smithi002 ceph-mon[17167]: pgmap v1966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3104.918852 secs 2022-03-21T15:03:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:22 smithi016 ceph-mon[22010]: pgmap v1966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3104.918852 secs 2022-03-21T15:03:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:24 smithi079 ceph-mon[23158]: pgmap v1967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:24 smithi002 ceph-mon[17167]: pgmap v1967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:24 smithi016 ceph-mon[22010]: pgmap v1967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:26 smithi079 ceph-mon[23158]: pgmap v1968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:26 smithi002 ceph-mon[17167]: pgmap v1968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:26 smithi016 ceph-mon[22010]: pgmap v1968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3109.918969 secs 2022-03-21T15:03:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3109.918969 secs 2022-03-21T15:03:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3109.918969 secs 2022-03-21T15:03:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:28 smithi002 ceph-mon[17167]: pgmap v1969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:28 smithi016 ceph-mon[22010]: pgmap v1969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:28 smithi079 ceph-mon[23158]: pgmap v1969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:30 smithi002 ceph-mon[17167]: pgmap v1970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:30 smithi016 ceph-mon[22010]: pgmap v1970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:30 smithi079 ceph-mon[23158]: pgmap v1970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:32.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: pgmap v1971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:03:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:03:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:03:32.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3114.919102 secs 2022-03-21T15:03:32.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:03:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: pgmap v1971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:03:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:03:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:03:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3114.919102 secs 2022-03-21T15:03:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: pgmap v1971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:03:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3114.919102 secs 2022-03-21T15:03:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:03:35.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:34 smithi002 ceph-mon[17167]: pgmap v1972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:34 smithi016 ceph-mon[22010]: pgmap v1972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:34 smithi079 ceph-mon[23158]: pgmap v1972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:36 smithi002 ceph-mon[17167]: pgmap v1973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:36 smithi016 ceph-mon[22010]: pgmap v1973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:36 smithi079 ceph-mon[23158]: pgmap v1973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3119.919236 secs 2022-03-21T15:03:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3119.919236 secs 2022-03-21T15:03:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3119.919236 secs 2022-03-21T15:03:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:38 smithi002 ceph-mon[17167]: pgmap v1974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:38 smithi016 ceph-mon[22010]: pgmap v1974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:38 smithi079 ceph-mon[23158]: pgmap v1974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:40 smithi002 ceph-mon[17167]: pgmap v1975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:40 smithi016 ceph-mon[22010]: pgmap v1975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:40 smithi079 ceph-mon[23158]: pgmap v1975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:42 smithi002 ceph-mon[17167]: pgmap v1976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3124.919352 secs 2022-03-21T15:03:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:42 smithi016 ceph-mon[22010]: pgmap v1976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3124.919352 secs 2022-03-21T15:03:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:42 smithi079 ceph-mon[23158]: pgmap v1976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3124.919352 secs 2022-03-21T15:03:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:44 smithi002 ceph-mon[17167]: pgmap v1977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:44 smithi016 ceph-mon[22010]: pgmap v1977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:44 smithi079 ceph-mon[23158]: pgmap v1977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:46 smithi002 ceph-mon[17167]: pgmap v1978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:46 smithi016 ceph-mon[22010]: pgmap v1978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:46 smithi079 ceph-mon[23158]: pgmap v1978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3129.919488 secs 2022-03-21T15:03:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3129.919488 secs 2022-03-21T15:03:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3129.919488 secs 2022-03-21T15:03:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:48 smithi002 ceph-mon[17167]: pgmap v1979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:48 smithi016 ceph-mon[22010]: pgmap v1979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:48 smithi079 ceph-mon[23158]: pgmap v1979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:50 smithi002 ceph-mon[17167]: pgmap v1980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:50 smithi016 ceph-mon[22010]: pgmap v1980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:50 smithi079 ceph-mon[23158]: pgmap v1980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:52 smithi002 ceph-mon[17167]: pgmap v1981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3134.919607 secs 2022-03-21T15:03:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:52 smithi016 ceph-mon[22010]: pgmap v1981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3134.919607 secs 2022-03-21T15:03:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:52 smithi079 ceph-mon[23158]: pgmap v1981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:03:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3134.919607 secs 2022-03-21T15:03:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:54 smithi002 ceph-mon[17167]: pgmap v1982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:54 smithi016 ceph-mon[22010]: pgmap v1982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:54 smithi079 ceph-mon[23158]: pgmap v1982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:03:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:56 smithi002 ceph-mon[17167]: pgmap v1983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:56 smithi016 ceph-mon[22010]: pgmap v1983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:56 smithi079 ceph-mon[23158]: pgmap v1983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:03:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3139.919697 secs 2022-03-21T15:03:58.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3139.919697 secs 2022-03-21T15:03:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3139.919697 secs 2022-03-21T15:03:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:03:58 smithi002 ceph-mon[17167]: pgmap v1984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:03:58 smithi016 ceph-mon[22010]: pgmap v1984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:03:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:03:58 smithi079 ceph-mon[23158]: pgmap v1984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:00 smithi002 ceph-mon[17167]: pgmap v1985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:00 smithi016 ceph-mon[22010]: pgmap v1985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:00 smithi079 ceph-mon[23158]: pgmap v1985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:02 smithi002 ceph-mon[17167]: pgmap v1986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3144.919816 secs 2022-03-21T15:04:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:02 smithi016 ceph-mon[22010]: pgmap v1986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3144.919816 secs 2022-03-21T15:04:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:02 smithi079 ceph-mon[23158]: pgmap v1986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3144.919816 secs 2022-03-21T15:04:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:04 smithi002 ceph-mon[17167]: pgmap v1987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:04 smithi016 ceph-mon[22010]: pgmap v1987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:04 smithi079 ceph-mon[23158]: pgmap v1987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:06 smithi002 ceph-mon[17167]: pgmap v1988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:06 smithi016 ceph-mon[22010]: pgmap v1988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:06 smithi079 ceph-mon[23158]: pgmap v1988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3149.919925 secs 2022-03-21T15:04:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3149.919925 secs 2022-03-21T15:04:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3149.919925 secs 2022-03-21T15:04:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:08 smithi002 ceph-mon[17167]: pgmap v1989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:09.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:08 smithi016 ceph-mon[22010]: pgmap v1989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:08 smithi079 ceph-mon[23158]: pgmap v1989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:10 smithi002 ceph-mon[17167]: pgmap v1990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:10 smithi016 ceph-mon[22010]: pgmap v1990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:10 smithi079 ceph-mon[23158]: pgmap v1990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:12 smithi002 ceph-mon[17167]: pgmap v1991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3154.920049 secs 2022-03-21T15:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:12 smithi016 ceph-mon[22010]: pgmap v1991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3154.920049 secs 2022-03-21T15:04:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:12 smithi079 ceph-mon[23158]: pgmap v1991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3154.920049 secs 2022-03-21T15:04:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:14 smithi002 ceph-mon[17167]: pgmap v1992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:14 smithi016 ceph-mon[22010]: pgmap v1992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:14 smithi079 ceph-mon[23158]: pgmap v1992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:16 smithi002 ceph-mon[17167]: pgmap v1993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:16 smithi016 ceph-mon[22010]: pgmap v1993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:16 smithi079 ceph-mon[23158]: pgmap v1993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3159.920137 secs 2022-03-21T15:04:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3159.920137 secs 2022-03-21T15:04:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3159.920137 secs 2022-03-21T15:04:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:18 smithi002 ceph-mon[17167]: pgmap v1994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:18 smithi016 ceph-mon[22010]: pgmap v1994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:18 smithi079 ceph-mon[23158]: pgmap v1994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:20 smithi002 ceph-mon[17167]: pgmap v1995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:04:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:20 smithi016 ceph-mon[22010]: pgmap v1995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:04:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:20 smithi079 ceph-mon[23158]: pgmap v1995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:04:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:04:23.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:22 smithi002 ceph-mon[17167]: pgmap v1996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3164.920258 secs 2022-03-21T15:04:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:22 smithi016 ceph-mon[22010]: pgmap v1996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3164.920258 secs 2022-03-21T15:04:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:22 smithi079 ceph-mon[23158]: pgmap v1996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3164.920258 secs 2022-03-21T15:04:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:24 smithi002 ceph-mon[17167]: pgmap v1997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:24 smithi016 ceph-mon[22010]: pgmap v1997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:24 smithi079 ceph-mon[23158]: pgmap v1997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:26 smithi002 ceph-mon[17167]: pgmap v1998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:26 smithi016 ceph-mon[22010]: pgmap v1998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:26 smithi079 ceph-mon[23158]: pgmap v1998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3169.920372 secs 2022-03-21T15:04:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3169.920372 secs 2022-03-21T15:04:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3169.920372 secs 2022-03-21T15:04:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:28 smithi002 ceph-mon[17167]: pgmap v1999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:28 smithi016 ceph-mon[22010]: pgmap v1999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:28 smithi079 ceph-mon[23158]: pgmap v1999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:30 smithi002 ceph-mon[17167]: pgmap v2000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:30 smithi016 ceph-mon[22010]: pgmap v2000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:30 smithi079 ceph-mon[23158]: pgmap v2000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:32 smithi079 ceph-mon[23158]: pgmap v2001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3174.920503 secs 2022-03-21T15:04:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:04:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:04:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:04:33.084 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:32 smithi016 ceph-mon[22010]: pgmap v2001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:33.085 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3174.920503 secs 2022-03-21T15:04:33.085 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:04:33.085 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:04:33.085 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:04:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:32 smithi002 ceph-mon[17167]: pgmap v2001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3174.920503 secs 2022-03-21T15:04:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:04:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:04:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:04:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:04:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:34 smithi002 ceph-mon[17167]: pgmap v2002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:04:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:34 smithi016 ceph-mon[22010]: pgmap v2002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:04:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:34 smithi079 ceph-mon[23158]: pgmap v2002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:36 smithi002 ceph-mon[17167]: pgmap v2003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:36 smithi016 ceph-mon[22010]: pgmap v2003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:36 smithi079 ceph-mon[23158]: pgmap v2003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3179.920636 secs 2022-03-21T15:04:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3179.920636 secs 2022-03-21T15:04:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3179.920636 secs 2022-03-21T15:04:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:38 smithi002 ceph-mon[17167]: pgmap v2004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:38 smithi016 ceph-mon[22010]: pgmap v2004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:38 smithi079 ceph-mon[23158]: pgmap v2004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:41.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:40 smithi002 ceph-mon[17167]: pgmap v2005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:40 smithi016 ceph-mon[22010]: pgmap v2005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:40 smithi079 ceph-mon[23158]: pgmap v2005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:43.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:42 smithi002 ceph-mon[17167]: pgmap v2006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3184.920735 secs 2022-03-21T15:04:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:42 smithi016 ceph-mon[22010]: pgmap v2006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3184.920735 secs 2022-03-21T15:04:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:42 smithi079 ceph-mon[23158]: pgmap v2006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:04:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3184.920735 secs 2022-03-21T15:04:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:44 smithi002 ceph-mon[17167]: pgmap v2007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:44 smithi016 ceph-mon[22010]: pgmap v2007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:44 smithi079 ceph-mon[23158]: pgmap v2007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:46 smithi002 ceph-mon[17167]: pgmap v2008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:46 smithi016 ceph-mon[22010]: pgmap v2008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:46 smithi079 ceph-mon[23158]: pgmap v2008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3189.920845 secs 2022-03-21T15:04:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3189.920845 secs 2022-03-21T15:04:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3189.920845 secs 2022-03-21T15:04:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:48 smithi002 ceph-mon[17167]: pgmap v2009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:48 smithi016 ceph-mon[22010]: pgmap v2009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:48 smithi079 ceph-mon[23158]: pgmap v2009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:50 smithi002 ceph-mon[17167]: pgmap v2010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:50 smithi016 ceph-mon[22010]: pgmap v2010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:50 smithi079 ceph-mon[23158]: pgmap v2010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:52 smithi002 ceph-mon[17167]: pgmap v2011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3194.920968 secs 2022-03-21T15:04:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:52 smithi016 ceph-mon[22010]: pgmap v2011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3194.920968 secs 2022-03-21T15:04:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:52 smithi079 ceph-mon[23158]: pgmap v2011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3194.920968 secs 2022-03-21T15:04:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:54 smithi002 ceph-mon[17167]: pgmap v2012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:54 smithi016 ceph-mon[22010]: pgmap v2012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:54 smithi079 ceph-mon[23158]: pgmap v2012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:04:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:56 smithi002 ceph-mon[17167]: pgmap v2013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:56 smithi016 ceph-mon[22010]: pgmap v2013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:56 smithi079 ceph-mon[23158]: pgmap v2013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:04:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3199.921074 secs 2022-03-21T15:04:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3199.921074 secs 2022-03-21T15:04:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3199.921074 secs 2022-03-21T15:04:59.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:04:58 smithi002 ceph-mon[17167]: pgmap v2014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:04:58 smithi016 ceph-mon[22010]: pgmap v2014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:04:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:04:58 smithi079 ceph-mon[23158]: pgmap v2014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:00 smithi002 ceph-mon[17167]: pgmap v2015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:00 smithi016 ceph-mon[22010]: pgmap v2015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:00 smithi079 ceph-mon[23158]: pgmap v2015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:02 smithi002 ceph-mon[17167]: pgmap v2016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3204.921212 secs 2022-03-21T15:05:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:02 smithi016 ceph-mon[22010]: pgmap v2016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3204.921212 secs 2022-03-21T15:05:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:02 smithi079 ceph-mon[23158]: pgmap v2016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3204.921212 secs 2022-03-21T15:05:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:04 smithi002 ceph-mon[17167]: pgmap v2017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:04 smithi016 ceph-mon[22010]: pgmap v2017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:04 smithi079 ceph-mon[23158]: pgmap v2017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:06 smithi002 ceph-mon[17167]: pgmap v2018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:06 smithi016 ceph-mon[22010]: pgmap v2018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:06 smithi079 ceph-mon[23158]: pgmap v2018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3209.921352 secs 2022-03-21T15:05:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3209.921352 secs 2022-03-21T15:05:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3209.921352 secs 2022-03-21T15:05:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:08 smithi002 ceph-mon[17167]: pgmap v2019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:08 smithi016 ceph-mon[22010]: pgmap v2019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:08 smithi079 ceph-mon[23158]: pgmap v2019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:10 smithi002 ceph-mon[17167]: pgmap v2020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:10 smithi016 ceph-mon[22010]: pgmap v2020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:10 smithi079 ceph-mon[23158]: pgmap v2020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:12 smithi002 ceph-mon[17167]: pgmap v2021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3214.921471 secs 2022-03-21T15:05:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:12 smithi016 ceph-mon[22010]: pgmap v2021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3214.921471 secs 2022-03-21T15:05:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:12 smithi079 ceph-mon[23158]: pgmap v2021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3214.921471 secs 2022-03-21T15:05:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:14 smithi002 ceph-mon[17167]: pgmap v2022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:14 smithi016 ceph-mon[22010]: pgmap v2022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:14 smithi079 ceph-mon[23158]: pgmap v2022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:16 smithi002 ceph-mon[17167]: pgmap v2023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:16 smithi016 ceph-mon[22010]: pgmap v2023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:16 smithi079 ceph-mon[23158]: pgmap v2023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3219.921575 secs 2022-03-21T15:05:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3219.921575 secs 2022-03-21T15:05:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3219.921575 secs 2022-03-21T15:05:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:18 smithi002 ceph-mon[17167]: pgmap v2024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:18 smithi016 ceph-mon[22010]: pgmap v2024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:18 smithi079 ceph-mon[23158]: pgmap v2024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:20 smithi002 ceph-mon[17167]: pgmap v2025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:05:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:05:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:20 smithi016 ceph-mon[22010]: pgmap v2025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:05:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:05:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:20 smithi079 ceph-mon[23158]: pgmap v2025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:05:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:05:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:22 smithi002 ceph-mon[17167]: pgmap v2026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3224.921691 secs 2022-03-21T15:05:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:22 smithi016 ceph-mon[22010]: pgmap v2026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3224.921691 secs 2022-03-21T15:05:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:22 smithi079 ceph-mon[23158]: pgmap v2026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3224.921691 secs 2022-03-21T15:05:25.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:24 smithi002 ceph-mon[17167]: pgmap v2027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:25.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:24 smithi016 ceph-mon[22010]: pgmap v2027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:24 smithi079 ceph-mon[23158]: pgmap v2027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:26 smithi002 ceph-mon[17167]: pgmap v2028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:26 smithi016 ceph-mon[22010]: pgmap v2028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:26 smithi079 ceph-mon[23158]: pgmap v2028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:28.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3229.921784 secs 2022-03-21T15:05:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3229.921784 secs 2022-03-21T15:05:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3229.921784 secs 2022-03-21T15:05:29.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:28 smithi002 ceph-mon[17167]: pgmap v2029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:28 smithi016 ceph-mon[22010]: pgmap v2029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:28 smithi079 ceph-mon[23158]: pgmap v2029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:31.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:30 smithi002 ceph-mon[17167]: pgmap v2030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:30 smithi016 ceph-mon[22010]: pgmap v2030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:30 smithi079 ceph-mon[23158]: pgmap v2030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:33.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:32 smithi002 ceph-mon[17167]: pgmap v2031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3234.921906 secs 2022-03-21T15:05:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:32 smithi016 ceph-mon[22010]: pgmap v2031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3234.921906 secs 2022-03-21T15:05:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:32 smithi079 ceph-mon[23158]: pgmap v2031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3234.921906 secs 2022-03-21T15:05:34.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:05:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:05:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:05:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:05:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:05:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:05:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:05:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:05:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:05:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:34 smithi002 ceph-mon[17167]: pgmap v2032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:05:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:34 smithi016 ceph-mon[22010]: pgmap v2032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:05:35.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:34 smithi079 ceph-mon[23158]: pgmap v2032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:05:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:36 smithi016 ceph-mon[22010]: pgmap v2033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:36 smithi079 ceph-mon[23158]: pgmap v2033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:36 smithi002 ceph-mon[17167]: pgmap v2033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3239.922032 secs 2022-03-21T15:05:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3239.922032 secs 2022-03-21T15:05:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3239.922032 secs 2022-03-21T15:05:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:38 smithi016 ceph-mon[22010]: pgmap v2034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:38 smithi079 ceph-mon[23158]: pgmap v2034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:38 smithi002 ceph-mon[17167]: pgmap v2034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:40 smithi016 ceph-mon[22010]: pgmap v2035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:40 smithi079 ceph-mon[23158]: pgmap v2035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:40 smithi002 ceph-mon[17167]: pgmap v2035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:42 smithi016 ceph-mon[22010]: pgmap v2036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3244.922151 secs 2022-03-21T15:05:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:42 smithi079 ceph-mon[23158]: pgmap v2036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3244.922151 secs 2022-03-21T15:05:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:42 smithi002 ceph-mon[17167]: pgmap v2036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3244.922151 secs 2022-03-21T15:05:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:44 smithi016 ceph-mon[22010]: pgmap v2037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:44 smithi079 ceph-mon[23158]: pgmap v2037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:44 smithi002 ceph-mon[17167]: pgmap v2037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:46 smithi016 ceph-mon[22010]: pgmap v2038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:46 smithi079 ceph-mon[23158]: pgmap v2038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:46 smithi002 ceph-mon[17167]: pgmap v2038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3249.922269 secs 2022-03-21T15:05:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3249.922269 secs 2022-03-21T15:05:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3249.922269 secs 2022-03-21T15:05:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:48 smithi016 ceph-mon[22010]: pgmap v2039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:48 smithi079 ceph-mon[23158]: pgmap v2039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:48 smithi002 ceph-mon[17167]: pgmap v2039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:50 smithi016 ceph-mon[22010]: pgmap v2040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:50 smithi079 ceph-mon[23158]: pgmap v2040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:50 smithi002 ceph-mon[17167]: pgmap v2040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:05:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:52 smithi016 ceph-mon[22010]: pgmap v2041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3254.922372 secs 2022-03-21T15:05:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:52 smithi079 ceph-mon[23158]: pgmap v2041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3254.922372 secs 2022-03-21T15:05:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:52 smithi002 ceph-mon[17167]: pgmap v2041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3254.922372 secs 2022-03-21T15:05:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:54 smithi016 ceph-mon[22010]: pgmap v2042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:54 smithi079 ceph-mon[23158]: pgmap v2042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:54 smithi002 ceph-mon[17167]: pgmap v2042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:56 smithi016 ceph-mon[22010]: pgmap v2043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:56 smithi079 ceph-mon[23158]: pgmap v2043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:56 smithi002 ceph-mon[17167]: pgmap v2043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:05:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3259.922507 secs 2022-03-21T15:05:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3259.922507 secs 2022-03-21T15:05:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3259.922507 secs 2022-03-21T15:05:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:05:58 smithi079 ceph-mon[23158]: pgmap v2044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:05:58 smithi002 ceph-mon[17167]: pgmap v2044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:05:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:05:58 smithi016 ceph-mon[22010]: pgmap v2044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:00 smithi079 ceph-mon[23158]: pgmap v2045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:00 smithi002 ceph-mon[17167]: pgmap v2045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:00 smithi016 ceph-mon[22010]: pgmap v2045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:02 smithi079 ceph-mon[23158]: pgmap v2046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3264.922619 secs 2022-03-21T15:06:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:02 smithi002 ceph-mon[17167]: pgmap v2046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3264.922619 secs 2022-03-21T15:06:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:02 smithi016 ceph-mon[22010]: pgmap v2046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3264.922619 secs 2022-03-21T15:06:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:04 smithi079 ceph-mon[23158]: pgmap v2047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:04 smithi002 ceph-mon[17167]: pgmap v2047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:04 smithi016 ceph-mon[22010]: pgmap v2047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:06 smithi079 ceph-mon[23158]: pgmap v2048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:06 smithi002 ceph-mon[17167]: pgmap v2048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:06 smithi016 ceph-mon[22010]: pgmap v2048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3269.922729 secs 2022-03-21T15:06:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3269.922729 secs 2022-03-21T15:06:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3269.922729 secs 2022-03-21T15:06:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:08 smithi079 ceph-mon[23158]: pgmap v2049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:08 smithi002 ceph-mon[17167]: pgmap v2049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:08 smithi016 ceph-mon[22010]: pgmap v2049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:10 smithi079 ceph-mon[23158]: pgmap v2050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:10 smithi002 ceph-mon[17167]: pgmap v2050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:10 smithi016 ceph-mon[22010]: pgmap v2050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:12 smithi079 ceph-mon[23158]: pgmap v2051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3274.922864 secs 2022-03-21T15:06:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:12 smithi002 ceph-mon[17167]: pgmap v2051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3274.922864 secs 2022-03-21T15:06:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:12 smithi016 ceph-mon[22010]: pgmap v2051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3274.922864 secs 2022-03-21T15:06:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:14 smithi079 ceph-mon[23158]: pgmap v2052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:14 smithi002 ceph-mon[17167]: pgmap v2052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:14 smithi016 ceph-mon[22010]: pgmap v2052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:16 smithi079 ceph-mon[23158]: pgmap v2053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:16 smithi002 ceph-mon[17167]: pgmap v2053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:16 smithi016 ceph-mon[22010]: pgmap v2053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:18.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3279.923004 secs 2022-03-21T15:06:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3279.923004 secs 2022-03-21T15:06:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3279.923004 secs 2022-03-21T15:06:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:18 smithi079 ceph-mon[23158]: pgmap v2054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:18 smithi002 ceph-mon[17167]: pgmap v2054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:18 smithi016 ceph-mon[22010]: pgmap v2054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:20 smithi079 ceph-mon[23158]: pgmap v2055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:06:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:20 smithi002 ceph-mon[17167]: pgmap v2055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:06:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:20 smithi016 ceph-mon[22010]: pgmap v2055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:06:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:06:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:22 smithi002 ceph-mon[17167]: pgmap v2056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3284.923129 secs 2022-03-21T15:06:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:22 smithi016 ceph-mon[22010]: pgmap v2056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3284.923129 secs 2022-03-21T15:06:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:22 smithi079 ceph-mon[23158]: pgmap v2056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3284.923129 secs 2022-03-21T15:06:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:24 smithi002 ceph-mon[17167]: pgmap v2057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:24 smithi016 ceph-mon[22010]: pgmap v2057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:24 smithi079 ceph-mon[23158]: pgmap v2057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:26 smithi002 ceph-mon[17167]: pgmap v2058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:26 smithi016 ceph-mon[22010]: pgmap v2058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:26 smithi079 ceph-mon[23158]: pgmap v2058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3289.923263 secs 2022-03-21T15:06:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3289.923263 secs 2022-03-21T15:06:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3289.923263 secs 2022-03-21T15:06:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:28 smithi002 ceph-mon[17167]: pgmap v2059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:29.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:28 smithi016 ceph-mon[22010]: pgmap v2059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:28 smithi079 ceph-mon[23158]: pgmap v2059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:30 smithi002 ceph-mon[17167]: pgmap v2060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:30 smithi016 ceph-mon[22010]: pgmap v2060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:30 smithi079 ceph-mon[23158]: pgmap v2060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:32 smithi002 ceph-mon[17167]: pgmap v2061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3294.923401 secs 2022-03-21T15:06:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:32 smithi016 ceph-mon[22010]: pgmap v2061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3294.923401 secs 2022-03-21T15:06:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:32 smithi079 ceph-mon[23158]: pgmap v2061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3294.923401 secs 2022-03-21T15:06:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:34 smithi016 ceph-mon[22010]: pgmap v2062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:06:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:06:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:06:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:34 smithi079 ceph-mon[23158]: pgmap v2062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:06:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:06:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:06:35.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:34 smithi002 ceph-mon[17167]: pgmap v2062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:35.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:06:35.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:06:35.259 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:06:37.260 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:37 smithi002 ceph-mon[17167]: pgmap v2063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:37.269 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:37 smithi016 ceph-mon[22010]: pgmap v2063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:37 smithi079 ceph-mon[23158]: pgmap v2063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:38.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3299.923462 secs 2022-03-21T15:06:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:38 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T15:06:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:38.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:38 smithi002 ceph-mon[17167]: pgmap v2064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3299.923462 secs 2022-03-21T15:06:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:38 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T15:06:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:38 smithi016 ceph-mon[22010]: pgmap v2064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3299.923462 secs 2022-03-21T15:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:38 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T15:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:38 smithi079 ceph-mon[23158]: pgmap v2064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:40 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T15:06:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:40 smithi002 ceph-mon[17167]: pgmap v2065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:40 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T15:06:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:40 smithi016 ceph-mon[22010]: pgmap v2065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:40 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T15:06:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:06:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:40 smithi079 ceph-mon[23158]: pgmap v2065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:42 smithi002 ceph-mon[17167]: pgmap v2066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:42 smithi016 ceph-mon[22010]: pgmap v2066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:42 smithi079 ceph-mon[23158]: pgmap v2066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3304.923586 secs 2022-03-21T15:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3304.923586 secs 2022-03-21T15:06:44.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3304.923586 secs 2022-03-21T15:06:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:44 smithi016 ceph-mon[22010]: pgmap v2067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:44 smithi079 ceph-mon[23158]: pgmap v2067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:45.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:44 smithi002 ceph-mon[17167]: pgmap v2067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:46 smithi002 ceph-mon[17167]: pgmap v2068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:46 smithi016 ceph-mon[22010]: pgmap v2068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:46 smithi079 ceph-mon[23158]: pgmap v2068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3309.923700 secs 2022-03-21T15:06:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3309.923700 secs 2022-03-21T15:06:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3309.923700 secs 2022-03-21T15:06:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:48 smithi002 ceph-mon[17167]: pgmap v2069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:48 smithi016 ceph-mon[22010]: pgmap v2069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:48 smithi079 ceph-mon[23158]: pgmap v2069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:51.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:50 smithi002 ceph-mon[17167]: pgmap v2070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:50 smithi016 ceph-mon[22010]: pgmap v2070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:51.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:50 smithi079 ceph-mon[23158]: pgmap v2070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:52 smithi002 ceph-mon[17167]: pgmap v2071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3314.923820 secs 2022-03-21T15:06:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:52 smithi016 ceph-mon[22010]: pgmap v2071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3314.923820 secs 2022-03-21T15:06:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:52 smithi079 ceph-mon[23158]: pgmap v2071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3314.923820 secs 2022-03-21T15:06:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:54 smithi002 ceph-mon[17167]: pgmap v2072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:54 smithi016 ceph-mon[22010]: pgmap v2072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:54 smithi079 ceph-mon[23158]: pgmap v2072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:56 smithi002 ceph-mon[17167]: pgmap v2073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:56 smithi016 ceph-mon[22010]: pgmap v2073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:56 smithi079 ceph-mon[23158]: pgmap v2073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:06:58.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3319.923880 secs 2022-03-21T15:06:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3319.923880 secs 2022-03-21T15:06:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3319.923880 secs 2022-03-21T15:06:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:06:58 smithi002 ceph-mon[17167]: pgmap v2074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:06:58 smithi016 ceph-mon[22010]: pgmap v2074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:06:59.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:06:58 smithi079 ceph-mon[23158]: pgmap v2074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:01.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:00 smithi002 ceph-mon[17167]: pgmap v2075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:00 smithi016 ceph-mon[22010]: pgmap v2075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:00 smithi079 ceph-mon[23158]: pgmap v2075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:02 smithi002 ceph-mon[17167]: pgmap v2076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3324.924020 secs 2022-03-21T15:07:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:02 smithi016 ceph-mon[22010]: pgmap v2076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3324.924020 secs 2022-03-21T15:07:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:02 smithi079 ceph-mon[23158]: pgmap v2076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3324.924020 secs 2022-03-21T15:07:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:04 smithi002 ceph-mon[17167]: pgmap v2077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:04 smithi016 ceph-mon[22010]: pgmap v2077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:05.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:04 smithi079 ceph-mon[23158]: pgmap v2077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:06 smithi002 ceph-mon[17167]: pgmap v2078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:06 smithi016 ceph-mon[22010]: pgmap v2078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:06 smithi079 ceph-mon[23158]: pgmap v2078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3329.924157 secs 2022-03-21T15:07:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3329.924157 secs 2022-03-21T15:07:08.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3329.924157 secs 2022-03-21T15:07:09.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:08 smithi002 ceph-mon[17167]: pgmap v2079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:08 smithi016 ceph-mon[22010]: pgmap v2079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:08 smithi079 ceph-mon[23158]: pgmap v2079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:11.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:10 smithi002 ceph-mon[17167]: pgmap v2080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:10 smithi016 ceph-mon[22010]: pgmap v2080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:11.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:10 smithi079 ceph-mon[23158]: pgmap v2080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:12 smithi002 ceph-mon[17167]: pgmap v2081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3334.924275 secs 2022-03-21T15:07:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:12 smithi016 ceph-mon[22010]: pgmap v2081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3334.924275 secs 2022-03-21T15:07:13.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:12 smithi079 ceph-mon[23158]: pgmap v2081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3334.924275 secs 2022-03-21T15:07:15.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:14 smithi002 ceph-mon[17167]: pgmap v2082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:14 smithi016 ceph-mon[22010]: pgmap v2082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:15.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:14 smithi079 ceph-mon[23158]: pgmap v2082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:16 smithi002 ceph-mon[17167]: pgmap v2083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:16 smithi016 ceph-mon[22010]: pgmap v2083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:16 smithi079 ceph-mon[23158]: pgmap v2083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3339.924393 secs 2022-03-21T15:07:18.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3339.924393 secs 2022-03-21T15:07:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3339.924393 secs 2022-03-21T15:07:19.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:18 smithi002 ceph-mon[17167]: pgmap v2084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:18 smithi016 ceph-mon[22010]: pgmap v2084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:18 smithi079 ceph-mon[23158]: pgmap v2084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:21.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:20 smithi002 ceph-mon[17167]: pgmap v2085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:07:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:20 smithi016 ceph-mon[22010]: pgmap v2085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:07:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:07:21.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:20 smithi079 ceph-mon[23158]: pgmap v2085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:07:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:07:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:22 smithi002 ceph-mon[17167]: pgmap v2086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3344.924512 secs 2022-03-21T15:07:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:22 smithi016 ceph-mon[22010]: pgmap v2086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3344.924512 secs 2022-03-21T15:07:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:22 smithi079 ceph-mon[23158]: pgmap v2086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3344.924512 secs 2022-03-21T15:07:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:24 smithi016 ceph-mon[22010]: pgmap v2087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:24 smithi079 ceph-mon[23158]: pgmap v2087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:24 smithi002 ceph-mon[17167]: pgmap v2087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:26 smithi016 ceph-mon[22010]: pgmap v2088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:26 smithi079 ceph-mon[23158]: pgmap v2088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:26 smithi002 ceph-mon[17167]: pgmap v2088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3349.924644 secs 2022-03-21T15:07:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3349.924644 secs 2022-03-21T15:07:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3349.924644 secs 2022-03-21T15:07:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:28 smithi016 ceph-mon[22010]: pgmap v2089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:28 smithi079 ceph-mon[23158]: pgmap v2089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:28 smithi002 ceph-mon[17167]: pgmap v2089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:30 smithi016 ceph-mon[22010]: pgmap v2090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:30 smithi079 ceph-mon[23158]: pgmap v2090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:30 smithi002 ceph-mon[17167]: pgmap v2090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:32 smithi016 ceph-mon[22010]: pgmap v2091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:33.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3354.924709 secs 2022-03-21T15:07:33.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:32 smithi079 ceph-mon[23158]: pgmap v2091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3354.924709 secs 2022-03-21T15:07:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:32 smithi002 ceph-mon[17167]: pgmap v2091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3354.924709 secs 2022-03-21T15:07:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:34 smithi016 ceph-mon[22010]: pgmap v2092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:34 smithi079 ceph-mon[23158]: pgmap v2092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:34 smithi002 ceph-mon[17167]: pgmap v2092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:36 smithi016 ceph-mon[22010]: pgmap v2093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:36 smithi079 ceph-mon[23158]: pgmap v2093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:36 smithi002 ceph-mon[17167]: pgmap v2093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3359.924775 secs 2022-03-21T15:07:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3359.924775 secs 2022-03-21T15:07:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3359.924775 secs 2022-03-21T15:07:39.153 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:38 smithi002 ceph-mon[17167]: pgmap v2094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:38 smithi016 ceph-mon[22010]: pgmap v2094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:38 smithi079 ceph-mon[23158]: pgmap v2094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:07:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:07:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:07:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:07:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:07:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:07:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:07:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:07:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:07:40.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:40.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:07:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:07:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: pgmap v2095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:07:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:07:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:07:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:07:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: pgmap v2095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:07:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:07:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:07:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:07:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: pgmap v2095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:07:41.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:40 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:07:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: pgmap v2096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3364.924841 secs 2022-03-21T15:07:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: pgmap v2096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3364.924841 secs 2022-03-21T15:07:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: pgmap v2096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:07:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:07:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3364.924841 secs 2022-03-21T15:07:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:43 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T15:07:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:43 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:07:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:43 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:07:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:43 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T15:07:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:43 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:07:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:43 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:07:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:43 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T15:07:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:43 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:07:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:43 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:07:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:44 smithi079 ceph-mon[23158]: pgmap v2097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:44 smithi002 ceph-mon[17167]: pgmap v2097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:44 smithi016 ceph-mon[22010]: pgmap v2097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:46 smithi079 ceph-mon[23158]: pgmap v2098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:46 smithi002 ceph-mon[17167]: pgmap v2098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:46 smithi016 ceph-mon[22010]: pgmap v2098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3369.924958 secs 2022-03-21T15:07:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3369.924958 secs 2022-03-21T15:07:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3369.924958 secs 2022-03-21T15:07:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:48 smithi079 ceph-mon[23158]: pgmap v2099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:48 smithi002 ceph-mon[17167]: pgmap v2099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:48 smithi016 ceph-mon[22010]: pgmap v2099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:50 smithi079 ceph-mon[23158]: pgmap v2100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:50 smithi002 ceph-mon[17167]: pgmap v2100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:50 smithi016 ceph-mon[22010]: pgmap v2100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:52 smithi079 ceph-mon[23158]: pgmap v2101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3374.925073 secs 2022-03-21T15:07:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:52 smithi002 ceph-mon[17167]: pgmap v2101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3374.925073 secs 2022-03-21T15:07:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:52 smithi016 ceph-mon[22010]: pgmap v2101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3374.925073 secs 2022-03-21T15:07:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:54 smithi079 ceph-mon[23158]: pgmap v2102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:54 smithi002 ceph-mon[17167]: pgmap v2102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:54 smithi016 ceph-mon[22010]: pgmap v2102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:56 smithi079 ceph-mon[23158]: pgmap v2103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:56 smithi002 ceph-mon[17167]: pgmap v2103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:56 smithi016 ceph-mon[22010]: pgmap v2103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:07:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3379.925188 secs 2022-03-21T15:07:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3379.925188 secs 2022-03-21T15:07:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3379.925188 secs 2022-03-21T15:07:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:07:58 smithi079 ceph-mon[23158]: pgmap v2104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:07:58 smithi002 ceph-mon[17167]: pgmap v2104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:07:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:07:58 smithi016 ceph-mon[22010]: pgmap v2104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:00 smithi079 ceph-mon[23158]: pgmap v2105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:00 smithi002 ceph-mon[17167]: pgmap v2105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:00 smithi016 ceph-mon[22010]: pgmap v2105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:02 smithi079 ceph-mon[23158]: pgmap v2106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3384.925318 secs 2022-03-21T15:08:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:02 smithi002 ceph-mon[17167]: pgmap v2106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3384.925318 secs 2022-03-21T15:08:03.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:02 smithi016 ceph-mon[22010]: pgmap v2106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3384.925318 secs 2022-03-21T15:08:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:04 smithi079 ceph-mon[23158]: pgmap v2107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:04 smithi002 ceph-mon[17167]: pgmap v2107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:04 smithi016 ceph-mon[22010]: pgmap v2107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:06 smithi079 ceph-mon[23158]: pgmap v2108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:06 smithi002 ceph-mon[17167]: pgmap v2108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:06 smithi016 ceph-mon[22010]: pgmap v2108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3389.925381 secs 2022-03-21T15:08:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3389.925381 secs 2022-03-21T15:08:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3389.925381 secs 2022-03-21T15:08:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:08 smithi002 ceph-mon[17167]: pgmap v2109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:08 smithi016 ceph-mon[22010]: pgmap v2109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:08 smithi079 ceph-mon[23158]: pgmap v2109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:10 smithi002 ceph-mon[17167]: pgmap v2110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:11.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:10 smithi016 ceph-mon[22010]: pgmap v2110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:10 smithi079 ceph-mon[23158]: pgmap v2110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:12 smithi002 ceph-mon[17167]: pgmap v2111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3394.925523 secs 2022-03-21T15:08:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:12 smithi016 ceph-mon[22010]: pgmap v2111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3394.925523 secs 2022-03-21T15:08:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:12 smithi079 ceph-mon[23158]: pgmap v2111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3394.925523 secs 2022-03-21T15:08:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:14 smithi002 ceph-mon[17167]: pgmap v2112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:14 smithi016 ceph-mon[22010]: pgmap v2112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:14 smithi079 ceph-mon[23158]: pgmap v2112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:16 smithi002 ceph-mon[17167]: pgmap v2113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:16 smithi016 ceph-mon[22010]: pgmap v2113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:16 smithi079 ceph-mon[23158]: pgmap v2113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3399.925629 secs 2022-03-21T15:08:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3399.925629 secs 2022-03-21T15:08:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3399.925629 secs 2022-03-21T15:08:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:19 smithi002 ceph-mon[17167]: pgmap v2114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:19 smithi016 ceph-mon[22010]: pgmap v2114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:19 smithi079 ceph-mon[23158]: pgmap v2114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:21 smithi002 ceph-mon[17167]: pgmap v2115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:08:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:21 smithi016 ceph-mon[22010]: pgmap v2115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:08:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:21 smithi079 ceph-mon[23158]: pgmap v2115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:08:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:23 smithi002 ceph-mon[17167]: pgmap v2116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3404.925730 secs 2022-03-21T15:08:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:23 smithi016 ceph-mon[22010]: pgmap v2116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3404.925730 secs 2022-03-21T15:08:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:23 smithi079 ceph-mon[23158]: pgmap v2116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3404.925730 secs 2022-03-21T15:08:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:25 smithi002 ceph-mon[17167]: pgmap v2117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:25 smithi016 ceph-mon[22010]: pgmap v2117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:25 smithi079 ceph-mon[23158]: pgmap v2117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:27 smithi002 ceph-mon[17167]: pgmap v2118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:27 smithi016 ceph-mon[22010]: pgmap v2118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:27 smithi079 ceph-mon[23158]: pgmap v2118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3409.925871 secs 2022-03-21T15:08:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3409.925871 secs 2022-03-21T15:08:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3409.925871 secs 2022-03-21T15:08:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:29 smithi002 ceph-mon[17167]: pgmap v2119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:29 smithi016 ceph-mon[22010]: pgmap v2119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:29 smithi079 ceph-mon[23158]: pgmap v2119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:31 smithi002 ceph-mon[17167]: pgmap v2120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:31 smithi016 ceph-mon[22010]: pgmap v2120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:31 smithi079 ceph-mon[23158]: pgmap v2120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:33 smithi002 ceph-mon[17167]: pgmap v2121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3414.925995 secs 2022-03-21T15:08:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:33 smithi016 ceph-mon[22010]: pgmap v2121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3414.925995 secs 2022-03-21T15:08:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:33 smithi079 ceph-mon[23158]: pgmap v2121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3414.925995 secs 2022-03-21T15:08:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:35 smithi002 ceph-mon[17167]: pgmap v2122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:35 smithi016 ceph-mon[22010]: pgmap v2122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:35 smithi079 ceph-mon[23158]: pgmap v2122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:37 smithi002 ceph-mon[17167]: pgmap v2123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:37 smithi016 ceph-mon[22010]: pgmap v2123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:37 smithi079 ceph-mon[23158]: pgmap v2123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3419.926140 secs 2022-03-21T15:08:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3419.926140 secs 2022-03-21T15:08:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3419.926140 secs 2022-03-21T15:08:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:39 smithi002 ceph-mon[17167]: pgmap v2124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:39 smithi016 ceph-mon[22010]: pgmap v2124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:39 smithi079 ceph-mon[23158]: pgmap v2124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:41 smithi002 ceph-mon[17167]: pgmap v2125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:41 smithi016 ceph-mon[22010]: pgmap v2125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:41 smithi079 ceph-mon[23158]: pgmap v2125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:43.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:43 smithi002 ceph-mon[17167]: pgmap v2126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:08:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:08:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:08:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3424.926217 secs 2022-03-21T15:08:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:43 smithi016 ceph-mon[22010]: pgmap v2126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:08:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:08:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:08:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3424.926217 secs 2022-03-21T15:08:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:43 smithi079 ceph-mon[23158]: pgmap v2126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:08:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:08:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:08:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3424.926217 secs 2022-03-21T15:08:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:08:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:08:44.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:08:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:45 smithi002 ceph-mon[17167]: pgmap v2127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:45 smithi016 ceph-mon[22010]: pgmap v2127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:45 smithi079 ceph-mon[23158]: pgmap v2127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:47 smithi002 ceph-mon[17167]: pgmap v2128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:47 smithi016 ceph-mon[22010]: pgmap v2128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:47.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:47 smithi079 ceph-mon[23158]: pgmap v2128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3429.926324 secs 2022-03-21T15:08:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3429.926324 secs 2022-03-21T15:08:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3429.926324 secs 2022-03-21T15:08:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:49 smithi002 ceph-mon[17167]: pgmap v2129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:49 smithi016 ceph-mon[22010]: pgmap v2129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:49 smithi079 ceph-mon[23158]: pgmap v2129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:51 smithi002 ceph-mon[17167]: pgmap v2130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:51 smithi016 ceph-mon[22010]: pgmap v2130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:51 smithi079 ceph-mon[23158]: pgmap v2130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:53 smithi002 ceph-mon[17167]: pgmap v2131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3434.926462 secs 2022-03-21T15:08:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:53 smithi016 ceph-mon[22010]: pgmap v2131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3434.926462 secs 2022-03-21T15:08:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:53 smithi079 ceph-mon[23158]: pgmap v2131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3434.926462 secs 2022-03-21T15:08:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:55 smithi002 ceph-mon[17167]: pgmap v2132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:55 smithi016 ceph-mon[22010]: pgmap v2132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:55 smithi079 ceph-mon[23158]: pgmap v2132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:56 smithi016 ceph-mon[22010]: pgmap v2133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:56 smithi079 ceph-mon[23158]: pgmap v2133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:56 smithi002 ceph-mon[17167]: pgmap v2133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:08:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3439.926604 secs 2022-03-21T15:08:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:08:58 smithi016 ceph-mon[22010]: pgmap v2134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3439.926604 secs 2022-03-21T15:08:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:08:58 smithi079 ceph-mon[23158]: pgmap v2134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:08:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3439.926604 secs 2022-03-21T15:08:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:08:58 smithi002 ceph-mon[17167]: pgmap v2134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:00 smithi002 ceph-mon[17167]: pgmap v2135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:00 smithi016 ceph-mon[22010]: pgmap v2135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:00 smithi079 ceph-mon[23158]: pgmap v2135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:03.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:02 smithi002 ceph-mon[17167]: pgmap v2136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3444.926694 secs 2022-03-21T15:09:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:02 smithi016 ceph-mon[22010]: pgmap v2136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3444.926694 secs 2022-03-21T15:09:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:02 smithi079 ceph-mon[23158]: pgmap v2136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3444.926694 secs 2022-03-21T15:09:05.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:04 smithi002 ceph-mon[17167]: pgmap v2137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:04 smithi016 ceph-mon[22010]: pgmap v2137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:04 smithi079 ceph-mon[23158]: pgmap v2137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:06 smithi002 ceph-mon[17167]: pgmap v2138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:06 smithi016 ceph-mon[22010]: pgmap v2138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:06 smithi079 ceph-mon[23158]: pgmap v2138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:08.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3449.926809 secs 2022-03-21T15:09:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3449.926809 secs 2022-03-21T15:09:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3449.926809 secs 2022-03-21T15:09:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:08 smithi002 ceph-mon[17167]: pgmap v2139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:08 smithi016 ceph-mon[22010]: pgmap v2139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:08 smithi079 ceph-mon[23158]: pgmap v2139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:10 smithi002 ceph-mon[17167]: pgmap v2140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:10 smithi016 ceph-mon[22010]: pgmap v2140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:10 smithi079 ceph-mon[23158]: pgmap v2140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:13.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:12 smithi002 ceph-mon[17167]: pgmap v2141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3454.926936 secs 2022-03-21T15:09:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:12 smithi016 ceph-mon[22010]: pgmap v2141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3454.926936 secs 2022-03-21T15:09:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:12 smithi079 ceph-mon[23158]: pgmap v2141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3454.926936 secs 2022-03-21T15:09:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:14 smithi002 ceph-mon[17167]: pgmap v2142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:14 smithi016 ceph-mon[22010]: pgmap v2142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:14 smithi079 ceph-mon[23158]: pgmap v2142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:16 smithi002 ceph-mon[17167]: pgmap v2143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:16 smithi016 ceph-mon[22010]: pgmap v2143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:16 smithi079 ceph-mon[23158]: pgmap v2143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3459.927049 secs 2022-03-21T15:09:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3459.927049 secs 2022-03-21T15:09:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3459.927049 secs 2022-03-21T15:09:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:18 smithi016 ceph-mon[22010]: pgmap v2144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:18 smithi079 ceph-mon[23158]: pgmap v2144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:18 smithi002 ceph-mon[17167]: pgmap v2144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:20 smithi016 ceph-mon[22010]: pgmap v2145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:09:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:09:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:20 smithi079 ceph-mon[23158]: pgmap v2145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:09:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:09:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:20 smithi002 ceph-mon[17167]: pgmap v2145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:09:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:09:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:22 smithi016 ceph-mon[22010]: pgmap v2146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3464.927187 secs 2022-03-21T15:09:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:22 smithi079 ceph-mon[23158]: pgmap v2146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3464.927187 secs 2022-03-21T15:09:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:22 smithi002 ceph-mon[17167]: pgmap v2146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3464.927187 secs 2022-03-21T15:09:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:24 smithi016 ceph-mon[22010]: pgmap v2147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:24 smithi079 ceph-mon[23158]: pgmap v2147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:24 smithi002 ceph-mon[17167]: pgmap v2147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:26 smithi016 ceph-mon[22010]: pgmap v2148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:26 smithi079 ceph-mon[23158]: pgmap v2148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:26 smithi002 ceph-mon[17167]: pgmap v2148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3469.927337 secs 2022-03-21T15:09:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3469.927337 secs 2022-03-21T15:09:28.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3469.927337 secs 2022-03-21T15:09:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:28 smithi016 ceph-mon[22010]: pgmap v2149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:29.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:28 smithi079 ceph-mon[23158]: pgmap v2149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:28 smithi002 ceph-mon[17167]: pgmap v2149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:30 smithi016 ceph-mon[22010]: pgmap v2150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:31.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:30 smithi079 ceph-mon[23158]: pgmap v2150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:30 smithi002 ceph-mon[17167]: pgmap v2150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:32 smithi016 ceph-mon[22010]: pgmap v2151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3474.927457 secs 2022-03-21T15:09:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:32 smithi079 ceph-mon[23158]: pgmap v2151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3474.927457 secs 2022-03-21T15:09:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:32 smithi002 ceph-mon[17167]: pgmap v2151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:33.372 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3474.927457 secs 2022-03-21T15:09:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:34 smithi016 ceph-mon[22010]: pgmap v2152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:34 smithi079 ceph-mon[23158]: pgmap v2152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:34 smithi002 ceph-mon[17167]: pgmap v2152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:36 smithi016 ceph-mon[22010]: pgmap v2153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:36 smithi079 ceph-mon[23158]: pgmap v2153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:36 smithi002 ceph-mon[17167]: pgmap v2153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3479.927592 secs 2022-03-21T15:09:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3479.927592 secs 2022-03-21T15:09:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3479.927592 secs 2022-03-21T15:09:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:38 smithi079 ceph-mon[23158]: pgmap v2154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:38 smithi002 ceph-mon[17167]: pgmap v2154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:38 smithi016 ceph-mon[22010]: pgmap v2154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:40 smithi079 ceph-mon[23158]: pgmap v2155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:41.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:40 smithi002 ceph-mon[17167]: pgmap v2155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:40 smithi016 ceph-mon[22010]: pgmap v2155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:43.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:42 smithi002 ceph-mon[17167]: pgmap v2156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:43.205 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3484.927703 secs 2022-03-21T15:09:43.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:42 smithi079 ceph-mon[23158]: pgmap v2156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3484.927703 secs 2022-03-21T15:09:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:42 smithi016 ceph-mon[22010]: pgmap v2156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3484.927703 secs 2022-03-21T15:09:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:09:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:09:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:09:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:09:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:09:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:09:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:09:44.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:09:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:09:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:09:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:09:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:09:45.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:44 smithi079 ceph-mon[23158]: pgmap v2157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:44 smithi002 ceph-mon[17167]: pgmap v2157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:44 smithi016 ceph-mon[22010]: pgmap v2157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:46 smithi079 ceph-mon[23158]: pgmap v2158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:46 smithi002 ceph-mon[17167]: pgmap v2158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:46 smithi016 ceph-mon[22010]: pgmap v2158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3489.927774 secs 2022-03-21T15:09:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3489.927774 secs 2022-03-21T15:09:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3489.927774 secs 2022-03-21T15:09:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:48 smithi079 ceph-mon[23158]: pgmap v2159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:48 smithi002 ceph-mon[17167]: pgmap v2159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:48 smithi016 ceph-mon[22010]: pgmap v2159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:50 smithi079 ceph-mon[23158]: pgmap v2160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:50 smithi002 ceph-mon[17167]: pgmap v2160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:50 smithi016 ceph-mon[22010]: pgmap v2160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:52 smithi079 ceph-mon[23158]: pgmap v2161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3494.927886 secs 2022-03-21T15:09:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:52 smithi002 ceph-mon[17167]: pgmap v2161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3494.927886 secs 2022-03-21T15:09:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:52 smithi016 ceph-mon[22010]: pgmap v2161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3494.927886 secs 2022-03-21T15:09:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:54 smithi079 ceph-mon[23158]: pgmap v2162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:54 smithi002 ceph-mon[17167]: pgmap v2162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:54 smithi016 ceph-mon[22010]: pgmap v2162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:56 smithi079 ceph-mon[23158]: pgmap v2163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:57.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:56 smithi002 ceph-mon[17167]: pgmap v2163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:56 smithi016 ceph-mon[22010]: pgmap v2163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:09:58.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3499.928001 secs 2022-03-21T15:09:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3499.928001 secs 2022-03-21T15:09:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3499.928001 secs 2022-03-21T15:09:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:09:58 smithi079 ceph-mon[23158]: pgmap v2164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:09:58 smithi002 ceph-mon[17167]: pgmap v2164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:09:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:09:58 smithi016 ceph-mon[22010]: pgmap v2164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:00 smithi079 ceph-mon[23158]: pgmap v2165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:10:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:00 smithi002 ceph-mon[17167]: pgmap v2165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:10:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:00 smithi016 ceph-mon[22010]: pgmap v2165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:10:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:02 smithi079 ceph-mon[23158]: pgmap v2166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3504.928122 secs 2022-03-21T15:10:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:02 smithi002 ceph-mon[17167]: pgmap v2166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3504.928122 secs 2022-03-21T15:10:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:02 smithi016 ceph-mon[22010]: pgmap v2166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3504.928122 secs 2022-03-21T15:10:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:04 smithi002 ceph-mon[17167]: pgmap v2167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:04 smithi016 ceph-mon[22010]: pgmap v2167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:04 smithi079 ceph-mon[23158]: pgmap v2167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:06 smithi002 ceph-mon[17167]: pgmap v2168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:06 smithi016 ceph-mon[22010]: pgmap v2168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:07.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:06 smithi079 ceph-mon[23158]: pgmap v2168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3509.928234 secs 2022-03-21T15:10:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3509.928234 secs 2022-03-21T15:10:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3509.928234 secs 2022-03-21T15:10:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:08 smithi002 ceph-mon[17167]: pgmap v2169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:08 smithi016 ceph-mon[22010]: pgmap v2169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:08 smithi079 ceph-mon[23158]: pgmap v2169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:10 smithi002 ceph-mon[17167]: pgmap v2170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:10 smithi016 ceph-mon[22010]: pgmap v2170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:10 smithi079 ceph-mon[23158]: pgmap v2170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:12 smithi002 ceph-mon[17167]: pgmap v2171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3514.928355 secs 2022-03-21T15:10:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:12 smithi016 ceph-mon[22010]: pgmap v2171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3514.928355 secs 2022-03-21T15:10:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:12 smithi079 ceph-mon[23158]: pgmap v2171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3514.928355 secs 2022-03-21T15:10:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:14 smithi002 ceph-mon[17167]: pgmap v2172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:14 smithi016 ceph-mon[22010]: pgmap v2172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:14 smithi079 ceph-mon[23158]: pgmap v2172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:16 smithi002 ceph-mon[17167]: pgmap v2173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:17 smithi016 ceph-mon[22010]: pgmap v2173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:17 smithi079 ceph-mon[23158]: pgmap v2173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3519.928462 secs 2022-03-21T15:10:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3519.928462 secs 2022-03-21T15:10:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3519.928462 secs 2022-03-21T15:10:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:19 smithi002 ceph-mon[17167]: pgmap v2174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:19 smithi016 ceph-mon[22010]: pgmap v2174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:19 smithi079 ceph-mon[23158]: pgmap v2174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:21 smithi002 ceph-mon[17167]: pgmap v2175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:10:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:10:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:21 smithi016 ceph-mon[22010]: pgmap v2175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:10:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:10:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:21 smithi079 ceph-mon[23158]: pgmap v2175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:10:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:10:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:23 smithi002 ceph-mon[17167]: pgmap v2176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3524.928603 secs 2022-03-21T15:10:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:23 smithi016 ceph-mon[22010]: pgmap v2176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3524.928603 secs 2022-03-21T15:10:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:23 smithi079 ceph-mon[23158]: pgmap v2176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3524.928603 secs 2022-03-21T15:10:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:25 smithi002 ceph-mon[17167]: pgmap v2177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:25 smithi016 ceph-mon[22010]: pgmap v2177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:25 smithi079 ceph-mon[23158]: pgmap v2177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:27 smithi002 ceph-mon[17167]: pgmap v2178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:27 smithi016 ceph-mon[22010]: pgmap v2178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:27 smithi079 ceph-mon[23158]: pgmap v2178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3529.928712 secs 2022-03-21T15:10:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3529.928712 secs 2022-03-21T15:10:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3529.928712 secs 2022-03-21T15:10:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:29 smithi002 ceph-mon[17167]: pgmap v2179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:29 smithi016 ceph-mon[22010]: pgmap v2179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:29 smithi079 ceph-mon[23158]: pgmap v2179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:31 smithi002 ceph-mon[17167]: pgmap v2180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:31 smithi016 ceph-mon[22010]: pgmap v2180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:31 smithi079 ceph-mon[23158]: pgmap v2180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:33 smithi002 ceph-mon[17167]: pgmap v2181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3534.928830 secs 2022-03-21T15:10:33.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:33 smithi016 ceph-mon[22010]: pgmap v2181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:33.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3534.928830 secs 2022-03-21T15:10:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:33 smithi079 ceph-mon[23158]: pgmap v2181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3534.928830 secs 2022-03-21T15:10:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:35 smithi002 ceph-mon[17167]: pgmap v2182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:35 smithi016 ceph-mon[22010]: pgmap v2182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:35 smithi079 ceph-mon[23158]: pgmap v2182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:37 smithi002 ceph-mon[17167]: pgmap v2183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:37 smithi016 ceph-mon[22010]: pgmap v2183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:37 smithi079 ceph-mon[23158]: pgmap v2183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3539.928937 secs 2022-03-21T15:10:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3539.928937 secs 2022-03-21T15:10:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3539.928937 secs 2022-03-21T15:10:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:39 smithi002 ceph-mon[17167]: pgmap v2184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:39 smithi016 ceph-mon[22010]: pgmap v2184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:39 smithi079 ceph-mon[23158]: pgmap v2184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:41 smithi002 ceph-mon[17167]: pgmap v2185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:41 smithi016 ceph-mon[22010]: pgmap v2185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:41 smithi079 ceph-mon[23158]: pgmap v2185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:43 smithi002 ceph-mon[17167]: pgmap v2186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3544.929060 secs 2022-03-21T15:10:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:43 smithi016 ceph-mon[22010]: pgmap v2186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3544.929060 secs 2022-03-21T15:10:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:43 smithi079 ceph-mon[23158]: pgmap v2186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3544.929060 secs 2022-03-21T15:10:44.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:10:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:10:44.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:10:44.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:10:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:10:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:10:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:10:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:10:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:10:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:45 smithi002 ceph-mon[17167]: pgmap v2187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:45 smithi016 ceph-mon[22010]: pgmap v2187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:45 smithi079 ceph-mon[23158]: pgmap v2187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:47 smithi079 ceph-mon[23158]: pgmap v2188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:47 smithi002 ceph-mon[17167]: pgmap v2188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:47 smithi016 ceph-mon[22010]: pgmap v2188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3549.929181 secs 2022-03-21T15:10:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:48 smithi002 ceph-mon[17167]: pgmap v2189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3549.929181 secs 2022-03-21T15:10:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:48 smithi016 ceph-mon[22010]: pgmap v2189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3549.929181 secs 2022-03-21T15:10:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:48 smithi079 ceph-mon[23158]: pgmap v2189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-03-21T15:10:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:10:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:50 smithi016 ceph-mon[22010]: pgmap v2190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:50 smithi079 ceph-mon[23158]: pgmap v2190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:50 smithi002 ceph-mon[17167]: pgmap v2190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 255 B/s wr, 3 op/s 2022-03-21T15:10:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:52 smithi016 ceph-mon[22010]: pgmap v2191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3554.929299 secs 2022-03-21T15:10:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:52 smithi079 ceph-mon[23158]: pgmap v2191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3554.929299 secs 2022-03-21T15:10:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:52 smithi002 ceph-mon[17167]: pgmap v2191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3554.929299 secs 2022-03-21T15:10:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:54 smithi016 ceph-mon[22010]: pgmap v2192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:54 smithi079 ceph-mon[23158]: pgmap v2192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:55.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:54 smithi002 ceph-mon[17167]: pgmap v2192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:56 smithi016 ceph-mon[22010]: pgmap v2193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:56 smithi079 ceph-mon[23158]: pgmap v2193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:56 smithi002 ceph-mon[17167]: pgmap v2193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:10:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3559.929435 secs 2022-03-21T15:10:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3559.929435 secs 2022-03-21T15:10:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3559.929435 secs 2022-03-21T15:10:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:10:58 smithi016 ceph-mon[22010]: pgmap v2194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:10:58 smithi079 ceph-mon[23158]: pgmap v2194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:10:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:10:58 smithi002 ceph-mon[17167]: pgmap v2194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:00 smithi016 ceph-mon[22010]: pgmap v2195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:00 smithi079 ceph-mon[23158]: pgmap v2195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:00 smithi002 ceph-mon[17167]: pgmap v2195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:02 smithi016 ceph-mon[22010]: pgmap v2196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3564.929567 secs 2022-03-21T15:11:03.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:02 smithi079 ceph-mon[23158]: pgmap v2196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3564.929567 secs 2022-03-21T15:11:03.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:02 smithi002 ceph-mon[17167]: pgmap v2196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3564.929567 secs 2022-03-21T15:11:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:04 smithi016 ceph-mon[22010]: pgmap v2197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:04 smithi079 ceph-mon[23158]: pgmap v2197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:04 smithi002 ceph-mon[17167]: pgmap v2197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:06 smithi016 ceph-mon[22010]: pgmap v2198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:06 smithi079 ceph-mon[23158]: pgmap v2198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:07.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:06 smithi002 ceph-mon[17167]: pgmap v2198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3569.929681 secs 2022-03-21T15:11:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3569.929681 secs 2022-03-21T15:11:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3569.929681 secs 2022-03-21T15:11:09.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:08 smithi079 ceph-mon[23158]: pgmap v2199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:08 smithi002 ceph-mon[17167]: pgmap v2199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:08 smithi016 ceph-mon[22010]: pgmap v2199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:10 smithi079 ceph-mon[23158]: pgmap v2200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:10 smithi002 ceph-mon[17167]: pgmap v2200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:10 smithi016 ceph-mon[22010]: pgmap v2200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:12 smithi079 ceph-mon[23158]: pgmap v2201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3574.929777 secs 2022-03-21T15:11:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:12 smithi002 ceph-mon[17167]: pgmap v2201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3574.929777 secs 2022-03-21T15:11:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:12 smithi016 ceph-mon[22010]: pgmap v2201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3574.929777 secs 2022-03-21T15:11:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:14 smithi079 ceph-mon[23158]: pgmap v2202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:14 smithi002 ceph-mon[17167]: pgmap v2202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:14 smithi016 ceph-mon[22010]: pgmap v2202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:16 smithi079 ceph-mon[23158]: pgmap v2203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:16 smithi002 ceph-mon[17167]: pgmap v2203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:16 smithi016 ceph-mon[22010]: pgmap v2203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3579.929886 secs 2022-03-21T15:11:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3579.929886 secs 2022-03-21T15:11:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3579.929886 secs 2022-03-21T15:11:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:18 smithi079 ceph-mon[23158]: pgmap v2204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:18 smithi002 ceph-mon[17167]: pgmap v2204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:18 smithi016 ceph-mon[22010]: pgmap v2204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:20 smithi079 ceph-mon[23158]: pgmap v2205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:11:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:11:21.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:20 smithi002 ceph-mon[17167]: pgmap v2205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:11:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:11:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:20 smithi016 ceph-mon[22010]: pgmap v2205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:11:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:11:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:22 smithi079 ceph-mon[23158]: pgmap v2206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3584.930008 secs 2022-03-21T15:11:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:22 smithi002 ceph-mon[17167]: pgmap v2206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3584.930008 secs 2022-03-21T15:11:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:22 smithi016 ceph-mon[22010]: pgmap v2206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3584.930008 secs 2022-03-21T15:11:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:24 smithi079 ceph-mon[23158]: pgmap v2207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:25.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:24 smithi002 ceph-mon[17167]: pgmap v2207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:24 smithi016 ceph-mon[22010]: pgmap v2207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:26 smithi079 ceph-mon[23158]: pgmap v2208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:26 smithi002 ceph-mon[17167]: pgmap v2208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:26 smithi016 ceph-mon[22010]: pgmap v2208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:28.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3589.930122 secs 2022-03-21T15:11:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3589.930122 secs 2022-03-21T15:11:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3589.930122 secs 2022-03-21T15:11:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:28 smithi079 ceph-mon[23158]: pgmap v2209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:28 smithi002 ceph-mon[17167]: pgmap v2209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:28 smithi016 ceph-mon[22010]: pgmap v2209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:30 smithi079 ceph-mon[23158]: pgmap v2210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:30 smithi002 ceph-mon[17167]: pgmap v2210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:30 smithi016 ceph-mon[22010]: pgmap v2210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:32 smithi079 ceph-mon[23158]: pgmap v2211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3594.930235 secs 2022-03-21T15:11:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:32 smithi002 ceph-mon[17167]: pgmap v2211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3594.930235 secs 2022-03-21T15:11:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:32 smithi016 ceph-mon[22010]: pgmap v2211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3594.930235 secs 2022-03-21T15:11:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:34 smithi079 ceph-mon[23158]: pgmap v2212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:34 smithi002 ceph-mon[17167]: pgmap v2212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:34 smithi016 ceph-mon[22010]: pgmap v2212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:36 smithi002 ceph-mon[17167]: pgmap v2213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:36 smithi016 ceph-mon[22010]: pgmap v2213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:36 smithi079 ceph-mon[23158]: pgmap v2213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3599.930359 secs 2022-03-21T15:11:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3599.930359 secs 2022-03-21T15:11:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3599.930359 secs 2022-03-21T15:11:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:38 smithi002 ceph-mon[17167]: pgmap v2214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:38 smithi016 ceph-mon[22010]: pgmap v2214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:38 smithi079 ceph-mon[23158]: pgmap v2214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:40 smithi002 ceph-mon[17167]: pgmap v2215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:40 smithi016 ceph-mon[22010]: pgmap v2215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:40 smithi079 ceph-mon[23158]: pgmap v2215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:42 smithi002 ceph-mon[17167]: pgmap v2216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3604.930494 secs 2022-03-21T15:11:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:42 smithi016 ceph-mon[22010]: pgmap v2216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3604.930494 secs 2022-03-21T15:11:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:42 smithi079 ceph-mon[23158]: pgmap v2216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3604.930494 secs 2022-03-21T15:11:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:44 smithi002 ceph-mon[17167]: pgmap v2217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:44 smithi016 ceph-mon[22010]: pgmap v2217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:44 smithi079 ceph-mon[23158]: pgmap v2217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:46 smithi002 ceph-mon[17167]: pgmap v2218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:46 smithi016 ceph-mon[22010]: pgmap v2218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:46 smithi079 ceph-mon[23158]: pgmap v2218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3609.930610 secs 2022-03-21T15:11:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3609.930610 secs 2022-03-21T15:11:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3609.930610 secs 2022-03-21T15:11:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:49 smithi002 ceph-mon[17167]: pgmap v2219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:11:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:11:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:11:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:49 smithi016 ceph-mon[22010]: pgmap v2219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:11:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:11:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:11:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:49 smithi079 ceph-mon[23158]: pgmap v2219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:11:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:11:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:11:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:11:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:50 smithi079 ceph-mon[23158]: pgmap v2220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:50.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:11:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:50 smithi002 ceph-mon[17167]: pgmap v2220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:11:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:50 smithi016 ceph-mon[22010]: pgmap v2220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:52 smithi079 ceph-mon[23158]: pgmap v2221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:52 smithi002 ceph-mon[17167]: pgmap v2221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:52 smithi016 ceph-mon[22010]: pgmap v2221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3614.930736 secs 2022-03-21T15:11:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3614.930736 secs 2022-03-21T15:11:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3614.930736 secs 2022-03-21T15:11:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:54 smithi079 ceph-mon[23158]: pgmap v2222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:55.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:54 smithi002 ceph-mon[17167]: pgmap v2222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:54 smithi016 ceph-mon[22010]: pgmap v2222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:56 smithi016 ceph-mon[22010]: pgmap v2223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:56 smithi079 ceph-mon[23158]: pgmap v2223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:56 smithi002 ceph-mon[17167]: pgmap v2223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:11:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3619.930852 secs 2022-03-21T15:11:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3619.930852 secs 2022-03-21T15:11:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3619.930852 secs 2022-03-21T15:11:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:11:58 smithi016 ceph-mon[22010]: pgmap v2224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:11:58 smithi079 ceph-mon[23158]: pgmap v2224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:11:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:11:58 smithi002 ceph-mon[17167]: pgmap v2224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:00 smithi016 ceph-mon[22010]: pgmap v2225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:00 smithi079 ceph-mon[23158]: pgmap v2225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:00 smithi002 ceph-mon[17167]: pgmap v2225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:02 smithi016 ceph-mon[22010]: pgmap v2226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3624.930966 secs 2022-03-21T15:12:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:02 smithi079 ceph-mon[23158]: pgmap v2226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3624.930966 secs 2022-03-21T15:12:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:02 smithi002 ceph-mon[17167]: pgmap v2226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3624.930966 secs 2022-03-21T15:12:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:04 smithi016 ceph-mon[22010]: pgmap v2227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:04 smithi079 ceph-mon[23158]: pgmap v2227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:04 smithi002 ceph-mon[17167]: pgmap v2227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:06 smithi079 ceph-mon[23158]: pgmap v2228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:06 smithi002 ceph-mon[17167]: pgmap v2228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:06 smithi016 ceph-mon[22010]: pgmap v2228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3629.931079 secs 2022-03-21T15:12:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3629.931079 secs 2022-03-21T15:12:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3629.931079 secs 2022-03-21T15:12:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:08 smithi079 ceph-mon[23158]: pgmap v2229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:08 smithi002 ceph-mon[17167]: pgmap v2229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:08 smithi016 ceph-mon[22010]: pgmap v2229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:10 smithi079 ceph-mon[23158]: pgmap v2230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:10 smithi002 ceph-mon[17167]: pgmap v2230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:10 smithi016 ceph-mon[22010]: pgmap v2230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:12 smithi079 ceph-mon[23158]: pgmap v2231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3634.931159 secs 2022-03-21T15:12:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:12 smithi002 ceph-mon[17167]: pgmap v2231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3634.931159 secs 2022-03-21T15:12:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:12 smithi016 ceph-mon[22010]: pgmap v2231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3634.931159 secs 2022-03-21T15:12:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:14 smithi079 ceph-mon[23158]: pgmap v2232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:14 smithi002 ceph-mon[17167]: pgmap v2232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:14 smithi016 ceph-mon[22010]: pgmap v2232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:16 smithi079 ceph-mon[23158]: pgmap v2233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:16 smithi002 ceph-mon[17167]: pgmap v2233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:16 smithi016 ceph-mon[22010]: pgmap v2233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3639.931275 secs 2022-03-21T15:12:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3639.931275 secs 2022-03-21T15:12:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3639.931275 secs 2022-03-21T15:12:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:18 smithi079 ceph-mon[23158]: pgmap v2234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:18 smithi002 ceph-mon[17167]: pgmap v2234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:18 smithi016 ceph-mon[22010]: pgmap v2234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:20 smithi079 ceph-mon[23158]: pgmap v2235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:12:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:20 smithi002 ceph-mon[17167]: pgmap v2235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:12:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:12:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:20 smithi016 ceph-mon[22010]: pgmap v2235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:12:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:12:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:22 smithi079 ceph-mon[23158]: pgmap v2236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3644.931394 secs 2022-03-21T15:12:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:22 smithi002 ceph-mon[17167]: pgmap v2236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3644.931394 secs 2022-03-21T15:12:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:22 smithi016 ceph-mon[22010]: pgmap v2236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3644.931394 secs 2022-03-21T15:12:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:24 smithi079 ceph-mon[23158]: pgmap v2237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:24 smithi002 ceph-mon[17167]: pgmap v2237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:24 smithi016 ceph-mon[22010]: pgmap v2237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:26 smithi079 ceph-mon[23158]: pgmap v2238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:26 smithi002 ceph-mon[17167]: pgmap v2238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:26 smithi016 ceph-mon[22010]: pgmap v2238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:28.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3649.931507 secs 2022-03-21T15:12:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3649.931507 secs 2022-03-21T15:12:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3649.931507 secs 2022-03-21T15:12:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:28 smithi079 ceph-mon[23158]: pgmap v2239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:28 smithi002 ceph-mon[17167]: pgmap v2239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:28 smithi016 ceph-mon[22010]: pgmap v2239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:30 smithi079 ceph-mon[23158]: pgmap v2240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:30 smithi002 ceph-mon[17167]: pgmap v2240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:30 smithi016 ceph-mon[22010]: pgmap v2240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:32 smithi002 ceph-mon[17167]: pgmap v2241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3654.931646 secs 2022-03-21T15:12:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:32 smithi016 ceph-mon[22010]: pgmap v2241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3654.931646 secs 2022-03-21T15:12:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:32 smithi079 ceph-mon[23158]: pgmap v2241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3654.931646 secs 2022-03-21T15:12:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:34 smithi002 ceph-mon[17167]: pgmap v2242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:34 smithi016 ceph-mon[22010]: pgmap v2242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:34 smithi079 ceph-mon[23158]: pgmap v2242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:36 smithi002 ceph-mon[17167]: pgmap v2243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:36 smithi016 ceph-mon[22010]: pgmap v2243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:36 smithi079 ceph-mon[23158]: pgmap v2243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3659.931740 secs 2022-03-21T15:12:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3659.931740 secs 2022-03-21T15:12:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3659.931740 secs 2022-03-21T15:12:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:38 smithi002 ceph-mon[17167]: pgmap v2244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:38 smithi016 ceph-mon[22010]: pgmap v2244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:38 smithi079 ceph-mon[23158]: pgmap v2244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:40 smithi002 ceph-mon[17167]: pgmap v2245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:40 smithi016 ceph-mon[22010]: pgmap v2245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:40 smithi079 ceph-mon[23158]: pgmap v2245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:42 smithi002 ceph-mon[17167]: pgmap v2246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3664.931857 secs 2022-03-21T15:12:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:42 smithi016 ceph-mon[22010]: pgmap v2246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3664.931857 secs 2022-03-21T15:12:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:42 smithi079 ceph-mon[23158]: pgmap v2246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3664.931857 secs 2022-03-21T15:12:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:44 smithi002 ceph-mon[17167]: pgmap v2247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:44 smithi016 ceph-mon[22010]: pgmap v2247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:44 smithi079 ceph-mon[23158]: pgmap v2247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:47 smithi002 ceph-mon[17167]: pgmap v2248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:47 smithi016 ceph-mon[22010]: pgmap v2248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:47 smithi079 ceph-mon[23158]: pgmap v2248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3669.931975 secs 2022-03-21T15:12:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3669.931975 secs 2022-03-21T15:12:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3669.931975 secs 2022-03-21T15:12:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:49 smithi002 ceph-mon[17167]: pgmap v2249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:49 smithi016 ceph-mon[22010]: pgmap v2249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:49 smithi079 ceph-mon[23158]: pgmap v2249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:12:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:12:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:12:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:50.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:12:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:12:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:12:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:12:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:12:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:12:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:50.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:51 smithi002 ceph-mon[17167]: pgmap v2250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:51 smithi016 ceph-mon[22010]: pgmap v2250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:51 smithi079 ceph-mon[23158]: pgmap v2250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:12:53.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:53 smithi002 ceph-mon[17167]: pgmap v2251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3674.932097 secs 2022-03-21T15:12:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:53 smithi016 ceph-mon[22010]: pgmap v2251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3674.932097 secs 2022-03-21T15:12:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:53 smithi079 ceph-mon[23158]: pgmap v2251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3674.932097 secs 2022-03-21T15:12:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:55 smithi002 ceph-mon[17167]: pgmap v2252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:55 smithi016 ceph-mon[22010]: pgmap v2252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:55 smithi079 ceph-mon[23158]: pgmap v2252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:57.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:57 smithi002 ceph-mon[17167]: pgmap v2253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:57 smithi016 ceph-mon[22010]: pgmap v2253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:57 smithi079 ceph-mon[23158]: pgmap v2253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:12:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3679.932225 secs 2022-03-21T15:12:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3679.932225 secs 2022-03-21T15:12:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3679.932225 secs 2022-03-21T15:12:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:12:59 smithi002 ceph-mon[17167]: pgmap v2254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:12:59 smithi016 ceph-mon[22010]: pgmap v2254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:12:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:12:59 smithi079 ceph-mon[23158]: pgmap v2254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:01 smithi002 ceph-mon[17167]: pgmap v2255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:01 smithi016 ceph-mon[22010]: pgmap v2255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:01 smithi079 ceph-mon[23158]: pgmap v2255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:03 smithi002 ceph-mon[17167]: pgmap v2256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3684.932339 secs 2022-03-21T15:13:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:03 smithi016 ceph-mon[22010]: pgmap v2256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3684.932339 secs 2022-03-21T15:13:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:03 smithi079 ceph-mon[23158]: pgmap v2256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3684.932339 secs 2022-03-21T15:13:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:05 smithi016 ceph-mon[22010]: pgmap v2257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:05 smithi079 ceph-mon[23158]: pgmap v2257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:05 smithi002 ceph-mon[17167]: pgmap v2257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:07 smithi016 ceph-mon[22010]: pgmap v2258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:07 smithi079 ceph-mon[23158]: pgmap v2258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:07 smithi002 ceph-mon[17167]: pgmap v2258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3689.932440 secs 2022-03-21T15:13:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3689.932440 secs 2022-03-21T15:13:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3689.932440 secs 2022-03-21T15:13:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:09 smithi016 ceph-mon[22010]: pgmap v2259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:09 smithi079 ceph-mon[23158]: pgmap v2259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:09 smithi002 ceph-mon[17167]: pgmap v2259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:11 smithi016 ceph-mon[22010]: pgmap v2260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:11 smithi079 ceph-mon[23158]: pgmap v2260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:11 smithi002 ceph-mon[17167]: pgmap v2260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:13 smithi016 ceph-mon[22010]: pgmap v2261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3694.932592 secs 2022-03-21T15:13:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:13 smithi079 ceph-mon[23158]: pgmap v2261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3694.932592 secs 2022-03-21T15:13:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:13 smithi002 ceph-mon[17167]: pgmap v2261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3694.932592 secs 2022-03-21T15:13:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:15 smithi016 ceph-mon[22010]: pgmap v2262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:15 smithi079 ceph-mon[23158]: pgmap v2262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:15 smithi002 ceph-mon[17167]: pgmap v2262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:17 smithi016 ceph-mon[22010]: pgmap v2263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:17 smithi079 ceph-mon[23158]: pgmap v2263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:17 smithi002 ceph-mon[17167]: pgmap v2263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3699.932716 secs 2022-03-21T15:13:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3699.932716 secs 2022-03-21T15:13:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3699.932716 secs 2022-03-21T15:13:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:19 smithi016 ceph-mon[22010]: pgmap v2264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:19 smithi079 ceph-mon[23158]: pgmap v2264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:19 smithi002 ceph-mon[17167]: pgmap v2264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:20 smithi079 ceph-mon[23158]: pgmap v2265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:13:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:13:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:20 smithi002 ceph-mon[17167]: pgmap v2265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:13:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:13:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:20 smithi016 ceph-mon[22010]: pgmap v2265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:13:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:13:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:22 smithi079 ceph-mon[23158]: pgmap v2266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3704.932861 secs 2022-03-21T15:13:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:22 smithi002 ceph-mon[17167]: pgmap v2266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3704.932861 secs 2022-03-21T15:13:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:22 smithi016 ceph-mon[22010]: pgmap v2266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3704.932861 secs 2022-03-21T15:13:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:24 smithi079 ceph-mon[23158]: pgmap v2267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:24 smithi002 ceph-mon[17167]: pgmap v2267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:24 smithi016 ceph-mon[22010]: pgmap v2267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:26 smithi079 ceph-mon[23158]: pgmap v2268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:27.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:26 smithi002 ceph-mon[17167]: pgmap v2268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:26 smithi016 ceph-mon[22010]: pgmap v2268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3709.932990 secs 2022-03-21T15:13:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3709.932990 secs 2022-03-21T15:13:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3709.932990 secs 2022-03-21T15:13:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:28 smithi079 ceph-mon[23158]: pgmap v2269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:28 smithi002 ceph-mon[17167]: pgmap v2269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:28 smithi016 ceph-mon[22010]: pgmap v2269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:30 smithi079 ceph-mon[23158]: pgmap v2270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:30 smithi002 ceph-mon[17167]: pgmap v2270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:30 smithi016 ceph-mon[22010]: pgmap v2270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:32 smithi079 ceph-mon[23158]: pgmap v2271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3714.933127 secs 2022-03-21T15:13:33.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:32 smithi002 ceph-mon[17167]: pgmap v2271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3714.933127 secs 2022-03-21T15:13:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:32 smithi016 ceph-mon[22010]: pgmap v2271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3714.933127 secs 2022-03-21T15:13:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:34 smithi079 ceph-mon[23158]: pgmap v2272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:34 smithi002 ceph-mon[17167]: pgmap v2272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:34 smithi016 ceph-mon[22010]: pgmap v2272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:36 smithi079 ceph-mon[23158]: pgmap v2273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:36 smithi002 ceph-mon[17167]: pgmap v2273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:36 smithi016 ceph-mon[22010]: pgmap v2273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3719.933255 secs 2022-03-21T15:13:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3719.933255 secs 2022-03-21T15:13:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3719.933255 secs 2022-03-21T15:13:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:38 smithi079 ceph-mon[23158]: pgmap v2274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:38 smithi002 ceph-mon[17167]: pgmap v2274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:38 smithi016 ceph-mon[22010]: pgmap v2274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:40 smithi002 ceph-mon[17167]: pgmap v2275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:40 smithi016 ceph-mon[22010]: pgmap v2275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:40 smithi079 ceph-mon[23158]: pgmap v2275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:42 smithi002 ceph-mon[17167]: pgmap v2276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3724.933392 secs 2022-03-21T15:13:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:42 smithi016 ceph-mon[22010]: pgmap v2276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3724.933392 secs 2022-03-21T15:13:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:42 smithi079 ceph-mon[23158]: pgmap v2276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3724.933392 secs 2022-03-21T15:13:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:44 smithi002 ceph-mon[17167]: pgmap v2277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:44 smithi016 ceph-mon[22010]: pgmap v2277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:44 smithi079 ceph-mon[23158]: pgmap v2277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:46 smithi002 ceph-mon[17167]: pgmap v2278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:46 smithi016 ceph-mon[22010]: pgmap v2278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:46 smithi079 ceph-mon[23158]: pgmap v2278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3729.933503 secs 2022-03-21T15:13:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3729.933503 secs 2022-03-21T15:13:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3729.933503 secs 2022-03-21T15:13:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:48 smithi002 ceph-mon[17167]: pgmap v2279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:48 smithi016 ceph-mon[22010]: pgmap v2279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:48 smithi079 ceph-mon[23158]: pgmap v2279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:50 smithi079 ceph-mon[23158]: pgmap v2280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:13:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:13:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:13:51.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:50 smithi016 ceph-mon[22010]: pgmap v2280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:51.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:13:51.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:13:51.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:13:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:50 smithi002 ceph-mon[17167]: pgmap v2280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:13:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:13:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:13:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:13:53.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:52 smithi002 ceph-mon[17167]: pgmap v2281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:13:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3734.933629 secs 2022-03-21T15:13:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:52 smithi016 ceph-mon[22010]: pgmap v2281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:13:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3734.933629 secs 2022-03-21T15:13:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:52 smithi079 ceph-mon[23158]: pgmap v2281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:13:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3734.933629 secs 2022-03-21T15:13:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:54 smithi002 ceph-mon[17167]: pgmap v2282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:54 smithi016 ceph-mon[22010]: pgmap v2282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:54 smithi079 ceph-mon[23158]: pgmap v2282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:56 smithi079 ceph-mon[23158]: pgmap v2283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:57.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:56 smithi002 ceph-mon[17167]: pgmap v2283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:56 smithi016 ceph-mon[22010]: pgmap v2283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:13:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3739.933775 secs 2022-03-21T15:13:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3739.933775 secs 2022-03-21T15:13:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3739.933775 secs 2022-03-21T15:13:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:13:58 smithi079 ceph-mon[23158]: pgmap v2284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:13:58 smithi002 ceph-mon[17167]: pgmap v2284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:13:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:13:58 smithi016 ceph-mon[22010]: pgmap v2284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:00 smithi079 ceph-mon[23158]: pgmap v2285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:14:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:00 smithi002 ceph-mon[17167]: pgmap v2285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:14:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:00 smithi016 ceph-mon[22010]: pgmap v2285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:14:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:02 smithi079 ceph-mon[23158]: pgmap v2286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3744.933909 secs 2022-03-21T15:14:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:02 smithi002 ceph-mon[17167]: pgmap v2286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3744.933909 secs 2022-03-21T15:14:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:02 smithi016 ceph-mon[22010]: pgmap v2286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3744.933909 secs 2022-03-21T15:14:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:04 smithi079 ceph-mon[23158]: pgmap v2287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:04 smithi002 ceph-mon[17167]: pgmap v2287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:04 smithi016 ceph-mon[22010]: pgmap v2287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:06 smithi079 ceph-mon[23158]: pgmap v2288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:06 smithi002 ceph-mon[17167]: pgmap v2288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:06 smithi016 ceph-mon[22010]: pgmap v2288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3749.934023 secs 2022-03-21T15:14:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3749.934023 secs 2022-03-21T15:14:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3749.934023 secs 2022-03-21T15:14:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:08 smithi002 ceph-mon[17167]: pgmap v2289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:08 smithi016 ceph-mon[22010]: pgmap v2289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:08 smithi079 ceph-mon[23158]: pgmap v2289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:11.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:10 smithi002 ceph-mon[17167]: pgmap v2290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:10 smithi016 ceph-mon[22010]: pgmap v2290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:10 smithi079 ceph-mon[23158]: pgmap v2290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:12 smithi002 ceph-mon[17167]: pgmap v2291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3754.934179 secs 2022-03-21T15:14:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:12 smithi016 ceph-mon[22010]: pgmap v2291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3754.934179 secs 2022-03-21T15:14:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:12 smithi079 ceph-mon[23158]: pgmap v2291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3754.934179 secs 2022-03-21T15:14:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:14 smithi002 ceph-mon[17167]: pgmap v2292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:15.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:14 smithi016 ceph-mon[22010]: pgmap v2292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:14 smithi079 ceph-mon[23158]: pgmap v2292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:17.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:16 smithi002 ceph-mon[17167]: pgmap v2293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:16 smithi016 ceph-mon[22010]: pgmap v2293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:16 smithi079 ceph-mon[23158]: pgmap v2293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3759.934297 secs 2022-03-21T15:14:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3759.934297 secs 2022-03-21T15:14:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3759.934297 secs 2022-03-21T15:14:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:18 smithi002 ceph-mon[17167]: pgmap v2294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:18 smithi016 ceph-mon[22010]: pgmap v2294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:18 smithi079 ceph-mon[23158]: pgmap v2294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:21 smithi002 ceph-mon[17167]: pgmap v2295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:14:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:14:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:21 smithi016 ceph-mon[22010]: pgmap v2295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:14:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:14:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:21 smithi079 ceph-mon[23158]: pgmap v2295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:14:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:14:23.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:23 smithi002 ceph-mon[17167]: pgmap v2296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3764.934381 secs 2022-03-21T15:14:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:23 smithi016 ceph-mon[22010]: pgmap v2296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3764.934381 secs 2022-03-21T15:14:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:23 smithi079 ceph-mon[23158]: pgmap v2296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3764.934381 secs 2022-03-21T15:14:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:25 smithi002 ceph-mon[17167]: pgmap v2297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:25 smithi016 ceph-mon[22010]: pgmap v2297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:25 smithi079 ceph-mon[23158]: pgmap v2297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:27 smithi002 ceph-mon[17167]: pgmap v2298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:27 smithi016 ceph-mon[22010]: pgmap v2298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:27 smithi079 ceph-mon[23158]: pgmap v2298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3769.934515 secs 2022-03-21T15:14:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3769.934515 secs 2022-03-21T15:14:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3769.934515 secs 2022-03-21T15:14:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:29 smithi002 ceph-mon[17167]: pgmap v2299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:29 smithi016 ceph-mon[22010]: pgmap v2299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:29 smithi079 ceph-mon[23158]: pgmap v2299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:31 smithi002 ceph-mon[17167]: pgmap v2300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:31 smithi016 ceph-mon[22010]: pgmap v2300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:31 smithi079 ceph-mon[23158]: pgmap v2300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:33 smithi002 ceph-mon[17167]: pgmap v2301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3774.934638 secs 2022-03-21T15:14:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:33 smithi016 ceph-mon[22010]: pgmap v2301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3774.934638 secs 2022-03-21T15:14:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:33 smithi079 ceph-mon[23158]: pgmap v2301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3774.934638 secs 2022-03-21T15:14:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:35 smithi002 ceph-mon[17167]: pgmap v2302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:35 smithi016 ceph-mon[22010]: pgmap v2302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:35 smithi079 ceph-mon[23158]: pgmap v2302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:37 smithi002 ceph-mon[17167]: pgmap v2303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:37 smithi016 ceph-mon[22010]: pgmap v2303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:37 smithi079 ceph-mon[23158]: pgmap v2303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3779.934762 secs 2022-03-21T15:14:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3779.934762 secs 2022-03-21T15:14:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3779.934762 secs 2022-03-21T15:14:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:39 smithi002 ceph-mon[17167]: pgmap v2304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:39 smithi016 ceph-mon[22010]: pgmap v2304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:39 smithi079 ceph-mon[23158]: pgmap v2304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:41 smithi002 ceph-mon[17167]: pgmap v2305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:41 smithi016 ceph-mon[22010]: pgmap v2305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:41 smithi079 ceph-mon[23158]: pgmap v2305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:43 smithi002 ceph-mon[17167]: pgmap v2306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3784.934887 secs 2022-03-21T15:14:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:43 smithi016 ceph-mon[22010]: pgmap v2306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3784.934887 secs 2022-03-21T15:14:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:43 smithi079 ceph-mon[23158]: pgmap v2306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3784.934887 secs 2022-03-21T15:14:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:45 smithi002 ceph-mon[17167]: pgmap v2307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:45.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:45 smithi016 ceph-mon[22010]: pgmap v2307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:45 smithi079 ceph-mon[23158]: pgmap v2307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:47 smithi002 ceph-mon[17167]: pgmap v2308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:47 smithi016 ceph-mon[22010]: pgmap v2308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:47 smithi079 ceph-mon[23158]: pgmap v2308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3789.935005 secs 2022-03-21T15:14:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3789.935005 secs 2022-03-21T15:14:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3789.935005 secs 2022-03-21T15:14:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:49 smithi002 ceph-mon[17167]: pgmap v2309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:49 smithi016 ceph-mon[22010]: pgmap v2309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:49 smithi079 ceph-mon[23158]: pgmap v2309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:51 smithi002 ceph-mon[17167]: pgmap v2310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:51 smithi016 ceph-mon[22010]: pgmap v2310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:51 smithi079 ceph-mon[23158]: pgmap v2310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:52.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:14:52.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:14:52.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:14:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:14:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:14:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:14:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:14:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:14:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:14:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:53 smithi002 ceph-mon[17167]: pgmap v2311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3794.935123 secs 2022-03-21T15:14:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:14:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:53 smithi016 ceph-mon[22010]: pgmap v2311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3794.935123 secs 2022-03-21T15:14:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:14:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:53 smithi079 ceph-mon[23158]: pgmap v2311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3794.935123 secs 2022-03-21T15:14:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:14:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:55 smithi002 ceph-mon[17167]: pgmap v2312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:55 smithi016 ceph-mon[22010]: pgmap v2312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:55 smithi079 ceph-mon[23158]: pgmap v2312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:57 smithi002 ceph-mon[17167]: pgmap v2313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:57 smithi016 ceph-mon[22010]: pgmap v2313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:57 smithi079 ceph-mon[23158]: pgmap v2313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:14:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3799.935220 secs 2022-03-21T15:14:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3799.935220 secs 2022-03-21T15:14:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3799.935220 secs 2022-03-21T15:14:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:14:59 smithi002 ceph-mon[17167]: pgmap v2314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:14:59 smithi016 ceph-mon[22010]: pgmap v2314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:14:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:14:59 smithi079 ceph-mon[23158]: pgmap v2314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:01 smithi002 ceph-mon[17167]: pgmap v2315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:01 smithi016 ceph-mon[22010]: pgmap v2315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:01 smithi079 ceph-mon[23158]: pgmap v2315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:03 smithi002 ceph-mon[17167]: pgmap v2316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3804.935361 secs 2022-03-21T15:15:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:03 smithi016 ceph-mon[22010]: pgmap v2316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3804.935361 secs 2022-03-21T15:15:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:03 smithi079 ceph-mon[23158]: pgmap v2316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3804.935361 secs 2022-03-21T15:15:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:05 smithi002 ceph-mon[17167]: pgmap v2317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:05 smithi016 ceph-mon[22010]: pgmap v2317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:05 smithi079 ceph-mon[23158]: pgmap v2317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:07 smithi002 ceph-mon[17167]: pgmap v2318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:07 smithi016 ceph-mon[22010]: pgmap v2318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:07 smithi079 ceph-mon[23158]: pgmap v2318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3809.935500 secs 2022-03-21T15:15:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3809.935500 secs 2022-03-21T15:15:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3809.935500 secs 2022-03-21T15:15:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:09 smithi002 ceph-mon[17167]: pgmap v2319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:09 smithi016 ceph-mon[22010]: pgmap v2319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:09 smithi079 ceph-mon[23158]: pgmap v2319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:11 smithi002 ceph-mon[17167]: pgmap v2320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:11 smithi016 ceph-mon[22010]: pgmap v2320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:11 smithi079 ceph-mon[23158]: pgmap v2320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:13 smithi016 ceph-mon[22010]: pgmap v2321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3814.935641 secs 2022-03-21T15:15:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:13 smithi079 ceph-mon[23158]: pgmap v2321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3814.935641 secs 2022-03-21T15:15:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:13 smithi002 ceph-mon[17167]: pgmap v2321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3814.935641 secs 2022-03-21T15:15:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:15 smithi016 ceph-mon[22010]: pgmap v2322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:15 smithi079 ceph-mon[23158]: pgmap v2322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:15 smithi002 ceph-mon[17167]: pgmap v2322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:17 smithi016 ceph-mon[22010]: pgmap v2323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:17 smithi079 ceph-mon[23158]: pgmap v2323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:17 smithi002 ceph-mon[17167]: pgmap v2323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3819.935703 secs 2022-03-21T15:15:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3819.935703 secs 2022-03-21T15:15:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3819.935703 secs 2022-03-21T15:15:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:19 smithi016 ceph-mon[22010]: pgmap v2324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:19 smithi079 ceph-mon[23158]: pgmap v2324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:19 smithi002 ceph-mon[17167]: pgmap v2324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:21 smithi016 ceph-mon[22010]: pgmap v2325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:15:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:15:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:21 smithi079 ceph-mon[23158]: pgmap v2325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:15:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:15:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:21 smithi002 ceph-mon[17167]: pgmap v2325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:15:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:15:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:23 smithi016 ceph-mon[22010]: pgmap v2326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3824.935773 secs 2022-03-21T15:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:23 smithi079 ceph-mon[23158]: pgmap v2326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3824.935773 secs 2022-03-21T15:15:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:23 smithi002 ceph-mon[17167]: pgmap v2326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3824.935773 secs 2022-03-21T15:15:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:25 smithi016 ceph-mon[22010]: pgmap v2327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:25 smithi079 ceph-mon[23158]: pgmap v2327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:25 smithi002 ceph-mon[17167]: pgmap v2327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:27 smithi016 ceph-mon[22010]: pgmap v2328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:27 smithi079 ceph-mon[23158]: pgmap v2328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:27 smithi002 ceph-mon[17167]: pgmap v2328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3829.935919 secs 2022-03-21T15:15:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3829.935919 secs 2022-03-21T15:15:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3829.935919 secs 2022-03-21T15:15:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:29 smithi016 ceph-mon[22010]: pgmap v2329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:29 smithi079 ceph-mon[23158]: pgmap v2329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:29 smithi002 ceph-mon[17167]: pgmap v2329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:31 smithi016 ceph-mon[22010]: pgmap v2330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:31 smithi079 ceph-mon[23158]: pgmap v2330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:31 smithi002 ceph-mon[17167]: pgmap v2330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:33 smithi079 ceph-mon[23158]: pgmap v2331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3834.936036 secs 2022-03-21T15:15:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:33 smithi002 ceph-mon[17167]: pgmap v2331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3834.936036 secs 2022-03-21T15:15:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:33 smithi016 ceph-mon[22010]: pgmap v2331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3834.936036 secs 2022-03-21T15:15:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:35 smithi079 ceph-mon[23158]: pgmap v2332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:35 smithi002 ceph-mon[17167]: pgmap v2332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:35 smithi016 ceph-mon[22010]: pgmap v2332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:37 smithi079 ceph-mon[23158]: pgmap v2333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:37 smithi002 ceph-mon[17167]: pgmap v2333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:37 smithi016 ceph-mon[22010]: pgmap v2333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3839.936167 secs 2022-03-21T15:15:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:38 smithi079 ceph-mon[23158]: pgmap v2334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3839.936167 secs 2022-03-21T15:15:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:38 smithi002 ceph-mon[17167]: pgmap v2334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3839.936167 secs 2022-03-21T15:15:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:38 smithi016 ceph-mon[22010]: pgmap v2334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:40 smithi002 ceph-mon[17167]: pgmap v2335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:40 smithi016 ceph-mon[22010]: pgmap v2335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:40 smithi079 ceph-mon[23158]: pgmap v2335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:42 smithi002 ceph-mon[17167]: pgmap v2336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3844.936288 secs 2022-03-21T15:15:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:42 smithi016 ceph-mon[22010]: pgmap v2336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3844.936288 secs 2022-03-21T15:15:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:42 smithi079 ceph-mon[23158]: pgmap v2336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3844.936288 secs 2022-03-21T15:15:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:44 smithi002 ceph-mon[17167]: pgmap v2337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:44 smithi016 ceph-mon[22010]: pgmap v2337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:44 smithi079 ceph-mon[23158]: pgmap v2337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:46 smithi002 ceph-mon[17167]: pgmap v2338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:46 smithi016 ceph-mon[22010]: pgmap v2338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:46 smithi079 ceph-mon[23158]: pgmap v2338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3849.936438 secs 2022-03-21T15:15:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3849.936438 secs 2022-03-21T15:15:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3849.936438 secs 2022-03-21T15:15:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:48 smithi002 ceph-mon[17167]: pgmap v2339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:48 smithi016 ceph-mon[22010]: pgmap v2339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:48 smithi079 ceph-mon[23158]: pgmap v2339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:51 smithi002 ceph-mon[17167]: pgmap v2340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:51 smithi016 ceph-mon[22010]: pgmap v2340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:51 smithi079 ceph-mon[23158]: pgmap v2340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:53 smithi002 ceph-mon[17167]: pgmap v2341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3854.936568 secs 2022-03-21T15:15:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:15:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:15:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:15:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:53 smithi016 ceph-mon[22010]: pgmap v2341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3854.936568 secs 2022-03-21T15:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:15:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:53 smithi079 ceph-mon[23158]: pgmap v2341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3854.936568 secs 2022-03-21T15:15:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:15:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:15:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:15:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:15:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:54 smithi002 ceph-mon[17167]: pgmap v2342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:15:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:54 smithi016 ceph-mon[22010]: pgmap v2342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:15:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:54 smithi079 ceph-mon[23158]: pgmap v2342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:56 smithi002 ceph-mon[17167]: pgmap v2343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:56 smithi016 ceph-mon[22010]: pgmap v2343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:56 smithi079 ceph-mon[23158]: pgmap v2343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:15:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3859.936656 secs 2022-03-21T15:15:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3859.936656 secs 2022-03-21T15:15:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3859.936656 secs 2022-03-21T15:15:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:15:58 smithi002 ceph-mon[17167]: pgmap v2344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:15:58 smithi016 ceph-mon[22010]: pgmap v2344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:15:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:15:58 smithi079 ceph-mon[23158]: pgmap v2344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:01 smithi002 ceph-mon[17167]: pgmap v2345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:01 smithi016 ceph-mon[22010]: pgmap v2345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:01 smithi079 ceph-mon[23158]: pgmap v2345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:03 smithi002 ceph-mon[17167]: pgmap v2346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3864.936776 secs 2022-03-21T15:16:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:03 smithi016 ceph-mon[22010]: pgmap v2346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3864.936776 secs 2022-03-21T15:16:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:03 smithi079 ceph-mon[23158]: pgmap v2346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3864.936776 secs 2022-03-21T15:16:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:05 smithi002 ceph-mon[17167]: pgmap v2347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:05 smithi016 ceph-mon[22010]: pgmap v2347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:05 smithi079 ceph-mon[23158]: pgmap v2347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:07 smithi002 ceph-mon[17167]: pgmap v2348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:07 smithi016 ceph-mon[22010]: pgmap v2348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:07 smithi079 ceph-mon[23158]: pgmap v2348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3869.936850 secs 2022-03-21T15:16:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3869.936850 secs 2022-03-21T15:16:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3869.936850 secs 2022-03-21T15:16:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:09 smithi002 ceph-mon[17167]: pgmap v2349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:09 smithi016 ceph-mon[22010]: pgmap v2349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:09 smithi079 ceph-mon[23158]: pgmap v2349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:11 smithi002 ceph-mon[17167]: pgmap v2350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:11 smithi016 ceph-mon[22010]: pgmap v2350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:11 smithi079 ceph-mon[23158]: pgmap v2350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:13.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:13 smithi002 ceph-mon[17167]: pgmap v2351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3874.936913 secs 2022-03-21T15:16:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:13 smithi016 ceph-mon[22010]: pgmap v2351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3874.936913 secs 2022-03-21T15:16:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:13 smithi079 ceph-mon[23158]: pgmap v2351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3874.936913 secs 2022-03-21T15:16:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:15 smithi002 ceph-mon[17167]: pgmap v2352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:15 smithi016 ceph-mon[22010]: pgmap v2352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:15 smithi079 ceph-mon[23158]: pgmap v2352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:17 smithi002 ceph-mon[17167]: pgmap v2353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:17 smithi016 ceph-mon[22010]: pgmap v2353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:17 smithi079 ceph-mon[23158]: pgmap v2353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3879.937026 secs 2022-03-21T15:16:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3879.937026 secs 2022-03-21T15:16:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3879.937026 secs 2022-03-21T15:16:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:19 smithi002 ceph-mon[17167]: pgmap v2354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:19 smithi016 ceph-mon[22010]: pgmap v2354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:19 smithi079 ceph-mon[23158]: pgmap v2354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:21 smithi002 ceph-mon[17167]: pgmap v2355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:16:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:16:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:21 smithi016 ceph-mon[22010]: pgmap v2355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:16:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:16:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:21 smithi079 ceph-mon[23158]: pgmap v2355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:16:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:16:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:23 smithi002 ceph-mon[17167]: pgmap v2356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3884.937167 secs 2022-03-21T15:16:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:23 smithi016 ceph-mon[22010]: pgmap v2356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3884.937167 secs 2022-03-21T15:16:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:23 smithi079 ceph-mon[23158]: pgmap v2356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3884.937167 secs 2022-03-21T15:16:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:25 smithi002 ceph-mon[17167]: pgmap v2357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:25 smithi016 ceph-mon[22010]: pgmap v2357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:25 smithi079 ceph-mon[23158]: pgmap v2357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:27 smithi002 ceph-mon[17167]: pgmap v2358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:27 smithi016 ceph-mon[22010]: pgmap v2358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:27 smithi079 ceph-mon[23158]: pgmap v2358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3889.937260 secs 2022-03-21T15:16:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3889.937260 secs 2022-03-21T15:16:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3889.937260 secs 2022-03-21T15:16:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:29 smithi002 ceph-mon[17167]: pgmap v2359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:29 smithi016 ceph-mon[22010]: pgmap v2359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:29 smithi079 ceph-mon[23158]: pgmap v2359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:31 smithi002 ceph-mon[17167]: pgmap v2360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:31 smithi016 ceph-mon[22010]: pgmap v2360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:31 smithi079 ceph-mon[23158]: pgmap v2360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:33 smithi002 ceph-mon[17167]: pgmap v2361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3894.937383 secs 2022-03-21T15:16:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:33 smithi016 ceph-mon[22010]: pgmap v2361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3894.937383 secs 2022-03-21T15:16:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:33 smithi079 ceph-mon[23158]: pgmap v2361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3894.937383 secs 2022-03-21T15:16:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:35 smithi002 ceph-mon[17167]: pgmap v2362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:35 smithi016 ceph-mon[22010]: pgmap v2362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:35 smithi079 ceph-mon[23158]: pgmap v2362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:37.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:37 smithi002 ceph-mon[17167]: pgmap v2363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:37 smithi016 ceph-mon[22010]: pgmap v2363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:37.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:37 smithi079 ceph-mon[23158]: pgmap v2363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3899.937513 secs 2022-03-21T15:16:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3899.937513 secs 2022-03-21T15:16:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3899.937513 secs 2022-03-21T15:16:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:39 smithi002 ceph-mon[17167]: pgmap v2364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:39 smithi016 ceph-mon[22010]: pgmap v2364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:39 smithi079 ceph-mon[23158]: pgmap v2364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:41 smithi002 ceph-mon[17167]: pgmap v2365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:41 smithi016 ceph-mon[22010]: pgmap v2365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:41 smithi079 ceph-mon[23158]: pgmap v2365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:43 smithi002 ceph-mon[17167]: pgmap v2366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3904.937617 secs 2022-03-21T15:16:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:43 smithi016 ceph-mon[22010]: pgmap v2366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3904.937617 secs 2022-03-21T15:16:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:43 smithi079 ceph-mon[23158]: pgmap v2366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3904.937617 secs 2022-03-21T15:16:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:45 smithi002 ceph-mon[17167]: pgmap v2367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:45 smithi016 ceph-mon[22010]: pgmap v2367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:45 smithi079 ceph-mon[23158]: pgmap v2367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:47 smithi002 ceph-mon[17167]: pgmap v2368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:47 smithi016 ceph-mon[22010]: pgmap v2368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:47 smithi079 ceph-mon[23158]: pgmap v2368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3909.937733 secs 2022-03-21T15:16:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3909.937733 secs 2022-03-21T15:16:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3909.937733 secs 2022-03-21T15:16:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:49 smithi016 ceph-mon[22010]: pgmap v2369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:49 smithi079 ceph-mon[23158]: pgmap v2369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:49 smithi002 ceph-mon[17167]: pgmap v2369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:51 smithi016 ceph-mon[22010]: pgmap v2370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:51 smithi079 ceph-mon[23158]: pgmap v2370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:51 smithi002 ceph-mon[17167]: pgmap v2370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:53 smithi016 ceph-mon[22010]: pgmap v2371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3914.937850 secs 2022-03-21T15:16:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:53 smithi079 ceph-mon[23158]: pgmap v2371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3914.937850 secs 2022-03-21T15:16:53.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:53 smithi002 ceph-mon[17167]: pgmap v2371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:53.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3914.937850 secs 2022-03-21T15:16:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:16:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:16:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:16:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:16:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:16:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:16:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:16:54.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:16:54.888 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:16:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:55 smithi016 ceph-mon[22010]: pgmap v2372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:16:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:55 smithi079 ceph-mon[23158]: pgmap v2372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:16:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:55 smithi002 ceph-mon[17167]: pgmap v2372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:16:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:57 smithi016 ceph-mon[22010]: pgmap v2373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:57 smithi079 ceph-mon[23158]: pgmap v2373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:57 smithi002 ceph-mon[17167]: pgmap v2373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:16:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3919.937963 secs 2022-03-21T15:16:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3919.937963 secs 2022-03-21T15:16:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3919.937963 secs 2022-03-21T15:16:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:16:59 smithi016 ceph-mon[22010]: pgmap v2374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:16:59 smithi079 ceph-mon[23158]: pgmap v2374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:16:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:16:59 smithi002 ceph-mon[17167]: pgmap v2374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:01 smithi016 ceph-mon[22010]: pgmap v2375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:01 smithi079 ceph-mon[23158]: pgmap v2375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:01 smithi002 ceph-mon[17167]: pgmap v2375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:03 smithi016 ceph-mon[22010]: pgmap v2376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3924.938068 secs 2022-03-21T15:17:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:03 smithi079 ceph-mon[23158]: pgmap v2376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3924.938068 secs 2022-03-21T15:17:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:03 smithi002 ceph-mon[17167]: pgmap v2376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3924.938068 secs 2022-03-21T15:17:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:05 smithi016 ceph-mon[22010]: pgmap v2377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:05 smithi079 ceph-mon[23158]: pgmap v2377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:05 smithi002 ceph-mon[17167]: pgmap v2377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:07 smithi016 ceph-mon[22010]: pgmap v2378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:07 smithi079 ceph-mon[23158]: pgmap v2378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:07 smithi002 ceph-mon[17167]: pgmap v2378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3929.938186 secs 2022-03-21T15:17:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3929.938186 secs 2022-03-21T15:17:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3929.938186 secs 2022-03-21T15:17:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:09 smithi079 ceph-mon[23158]: pgmap v2379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:09 smithi002 ceph-mon[17167]: pgmap v2379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:09 smithi016 ceph-mon[22010]: pgmap v2379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:11 smithi079 ceph-mon[23158]: pgmap v2380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:11 smithi002 ceph-mon[17167]: pgmap v2380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:11 smithi016 ceph-mon[22010]: pgmap v2380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:13 smithi079 ceph-mon[23158]: pgmap v2381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3934.938304 secs 2022-03-21T15:17:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:13 smithi002 ceph-mon[17167]: pgmap v2381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:13.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3934.938304 secs 2022-03-21T15:17:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:13 smithi016 ceph-mon[22010]: pgmap v2381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3934.938304 secs 2022-03-21T15:17:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:15 smithi079 ceph-mon[23158]: pgmap v2382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:15 smithi002 ceph-mon[17167]: pgmap v2382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:15 smithi016 ceph-mon[22010]: pgmap v2382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:17 smithi079 ceph-mon[23158]: pgmap v2383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:17 smithi002 ceph-mon[17167]: pgmap v2383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:17 smithi016 ceph-mon[22010]: pgmap v2383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3939.938405 secs 2022-03-21T15:17:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3939.938405 secs 2022-03-21T15:17:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3939.938405 secs 2022-03-21T15:17:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:19 smithi079 ceph-mon[23158]: pgmap v2384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:19 smithi002 ceph-mon[17167]: pgmap v2384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:19 smithi016 ceph-mon[22010]: pgmap v2384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:21 smithi079 ceph-mon[23158]: pgmap v2385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:17:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:17:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:21 smithi002 ceph-mon[17167]: pgmap v2385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:17:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:17:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:21 smithi016 ceph-mon[22010]: pgmap v2385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:17:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:17:22.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:22 smithi002 ceph-mon[17167]: pgmap v2386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:22 smithi016 ceph-mon[22010]: pgmap v2386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:22 smithi079 ceph-mon[23158]: pgmap v2386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3944.938556 secs 2022-03-21T15:17:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3944.938556 secs 2022-03-21T15:17:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3944.938556 secs 2022-03-21T15:17:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:24 smithi002 ceph-mon[17167]: pgmap v2387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:24 smithi016 ceph-mon[22010]: pgmap v2387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:24 smithi079 ceph-mon[23158]: pgmap v2387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:27 smithi002 ceph-mon[17167]: pgmap v2388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:27 smithi016 ceph-mon[22010]: pgmap v2388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:27 smithi079 ceph-mon[23158]: pgmap v2388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3949.938661 secs 2022-03-21T15:17:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3949.938661 secs 2022-03-21T15:17:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3949.938661 secs 2022-03-21T15:17:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:29 smithi002 ceph-mon[17167]: pgmap v2389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:29 smithi016 ceph-mon[22010]: pgmap v2389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:29 smithi079 ceph-mon[23158]: pgmap v2389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:31 smithi002 ceph-mon[17167]: pgmap v2390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:31 smithi016 ceph-mon[22010]: pgmap v2390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:31 smithi079 ceph-mon[23158]: pgmap v2390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:33 smithi002 ceph-mon[17167]: pgmap v2391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3954.938774 secs 2022-03-21T15:17:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:33 smithi016 ceph-mon[22010]: pgmap v2391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3954.938774 secs 2022-03-21T15:17:33.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:33 smithi079 ceph-mon[23158]: pgmap v2391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3954.938774 secs 2022-03-21T15:17:35.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:35 smithi002 ceph-mon[17167]: pgmap v2392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:35 smithi016 ceph-mon[22010]: pgmap v2392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:35 smithi079 ceph-mon[23158]: pgmap v2392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:37 smithi002 ceph-mon[17167]: pgmap v2393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:37 smithi016 ceph-mon[22010]: pgmap v2393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:37 smithi079 ceph-mon[23158]: pgmap v2393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3959.938891 secs 2022-03-21T15:17:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3959.938891 secs 2022-03-21T15:17:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3959.938891 secs 2022-03-21T15:17:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:39 smithi002 ceph-mon[17167]: pgmap v2394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:39 smithi016 ceph-mon[22010]: pgmap v2394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:39 smithi079 ceph-mon[23158]: pgmap v2394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:41 smithi002 ceph-mon[17167]: pgmap v2395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:41 smithi016 ceph-mon[22010]: pgmap v2395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:41 smithi079 ceph-mon[23158]: pgmap v2395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:43 smithi002 ceph-mon[17167]: pgmap v2396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3964.939009 secs 2022-03-21T15:17:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:43 smithi016 ceph-mon[22010]: pgmap v2396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3964.939009 secs 2022-03-21T15:17:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:43 smithi079 ceph-mon[23158]: pgmap v2396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3964.939009 secs 2022-03-21T15:17:45.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:45 smithi002 ceph-mon[17167]: pgmap v2397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:45.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:45 smithi016 ceph-mon[22010]: pgmap v2397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:45 smithi079 ceph-mon[23158]: pgmap v2397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:47 smithi002 ceph-mon[17167]: pgmap v2398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:47 smithi016 ceph-mon[22010]: pgmap v2398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:47.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:47 smithi079 ceph-mon[23158]: pgmap v2398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3969.939112 secs 2022-03-21T15:17:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3969.939112 secs 2022-03-21T15:17:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3969.939112 secs 2022-03-21T15:17:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:49 smithi002 ceph-mon[17167]: pgmap v2399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:49 smithi016 ceph-mon[22010]: pgmap v2399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:49 smithi079 ceph-mon[23158]: pgmap v2399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:51 smithi002 ceph-mon[17167]: pgmap v2400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:51 smithi016 ceph-mon[22010]: pgmap v2400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:51 smithi079 ceph-mon[23158]: pgmap v2400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:53 smithi002 ceph-mon[17167]: pgmap v2401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3974.939240 secs 2022-03-21T15:17:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:53 smithi016 ceph-mon[22010]: pgmap v2401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3974.939240 secs 2022-03-21T15:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:53 smithi079 ceph-mon[23158]: pgmap v2401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3974.939240 secs 2022-03-21T15:17:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: pgmap v2402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:17:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:17:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:17:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: pgmap v2402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: pgmap v2402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:17:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:17:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:17:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:55.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.372 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:17:56.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:17:56.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:17:56.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:17:56.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.374 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.374 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.374 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.374 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:17:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:17:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:17:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:17:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:17:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:17:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:17:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:17:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:17:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:17:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:17:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:17:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:57 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:17:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:57 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:17:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:57 smithi002 ceph-mon[17167]: pgmap v2403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:57 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:17:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:57 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:17:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:57 smithi016 ceph-mon[22010]: pgmap v2403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:57 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:17:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:57 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:17:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:57 smithi079 ceph-mon[23158]: pgmap v2403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:17:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3979.939354 secs 2022-03-21T15:17:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3979.939354 secs 2022-03-21T15:17:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3979.939354 secs 2022-03-21T15:17:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:17:59 smithi002 ceph-mon[17167]: pgmap v2404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:17:59 smithi016 ceph-mon[22010]: pgmap v2404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:17:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:17:59 smithi079 ceph-mon[23158]: pgmap v2404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:01 smithi002 ceph-mon[17167]: pgmap v2405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:01.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:01 smithi016 ceph-mon[22010]: pgmap v2405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:01 smithi079 ceph-mon[23158]: pgmap v2405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:03 smithi002 ceph-mon[17167]: pgmap v2406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3984.939491 secs 2022-03-21T15:18:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:03 smithi016 ceph-mon[22010]: pgmap v2406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3984.939491 secs 2022-03-21T15:18:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:03 smithi079 ceph-mon[23158]: pgmap v2406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3984.939491 secs 2022-03-21T15:18:05.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:05 smithi002 ceph-mon[17167]: pgmap v2407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:05 smithi016 ceph-mon[22010]: pgmap v2407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:05 smithi079 ceph-mon[23158]: pgmap v2407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:07 smithi002 ceph-mon[17167]: pgmap v2408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:07 smithi016 ceph-mon[22010]: pgmap v2408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:07 smithi079 ceph-mon[23158]: pgmap v2408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3989.939618 secs 2022-03-21T15:18:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3989.939618 secs 2022-03-21T15:18:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3989.939618 secs 2022-03-21T15:18:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:09 smithi016 ceph-mon[22010]: pgmap v2409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:09 smithi079 ceph-mon[23158]: pgmap v2409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:09 smithi002 ceph-mon[17167]: pgmap v2409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:11 smithi016 ceph-mon[22010]: pgmap v2410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:11 smithi079 ceph-mon[23158]: pgmap v2410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:11 smithi002 ceph-mon[17167]: pgmap v2410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:13 smithi016 ceph-mon[22010]: pgmap v2411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3994.939655 secs 2022-03-21T15:18:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:13 smithi079 ceph-mon[23158]: pgmap v2411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3994.939655 secs 2022-03-21T15:18:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:13 smithi002 ceph-mon[17167]: pgmap v2411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3994.939655 secs 2022-03-21T15:18:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:15 smithi016 ceph-mon[22010]: pgmap v2412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:15 smithi079 ceph-mon[23158]: pgmap v2412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:15 smithi002 ceph-mon[17167]: pgmap v2412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:17 smithi016 ceph-mon[22010]: pgmap v2413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:17 smithi079 ceph-mon[23158]: pgmap v2413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:17 smithi002 ceph-mon[17167]: pgmap v2413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 3999.939772 secs 2022-03-21T15:18:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 3999.939772 secs 2022-03-21T15:18:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 3999.939772 secs 2022-03-21T15:18:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:19 smithi016 ceph-mon[22010]: pgmap v2414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:19 smithi079 ceph-mon[23158]: pgmap v2414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:19 smithi002 ceph-mon[17167]: pgmap v2414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:21 smithi016 ceph-mon[22010]: pgmap v2415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:18:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:18:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:21 smithi079 ceph-mon[23158]: pgmap v2415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:18:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:21 smithi002 ceph-mon[17167]: pgmap v2415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:18:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:18:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:23 smithi016 ceph-mon[22010]: pgmap v2416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4004.939889 secs 2022-03-21T15:18:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:23 smithi079 ceph-mon[23158]: pgmap v2416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4004.939889 secs 2022-03-21T15:18:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:23 smithi002 ceph-mon[17167]: pgmap v2416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4004.939889 secs 2022-03-21T15:18:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:25 smithi016 ceph-mon[22010]: pgmap v2417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:25 smithi079 ceph-mon[23158]: pgmap v2417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:25 smithi002 ceph-mon[17167]: pgmap v2417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:27 smithi016 ceph-mon[22010]: pgmap v2418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:27 smithi079 ceph-mon[23158]: pgmap v2418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:27 smithi002 ceph-mon[17167]: pgmap v2418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4009.940010 secs 2022-03-21T15:18:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4009.940010 secs 2022-03-21T15:18:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4009.940010 secs 2022-03-21T15:18:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:29 smithi079 ceph-mon[23158]: pgmap v2419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:29 smithi002 ceph-mon[17167]: pgmap v2419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:29 smithi016 ceph-mon[22010]: pgmap v2419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:31 smithi079 ceph-mon[23158]: pgmap v2420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:31 smithi002 ceph-mon[17167]: pgmap v2420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:31 smithi016 ceph-mon[22010]: pgmap v2420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:33 smithi079 ceph-mon[23158]: pgmap v2421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4014.940111 secs 2022-03-21T15:18:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:33 smithi002 ceph-mon[17167]: pgmap v2421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4014.940111 secs 2022-03-21T15:18:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:33 smithi016 ceph-mon[22010]: pgmap v2421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4014.940111 secs 2022-03-21T15:18:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:35 smithi079 ceph-mon[23158]: pgmap v2422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:35 smithi002 ceph-mon[17167]: pgmap v2422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:35 smithi016 ceph-mon[22010]: pgmap v2422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:37 smithi079 ceph-mon[23158]: pgmap v2423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:37 smithi002 ceph-mon[17167]: pgmap v2423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:37 smithi016 ceph-mon[22010]: pgmap v2423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4019.940250 secs 2022-03-21T15:18:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4019.940250 secs 2022-03-21T15:18:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4019.940250 secs 2022-03-21T15:18:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:39 smithi079 ceph-mon[23158]: pgmap v2424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:39 smithi002 ceph-mon[17167]: pgmap v2424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:39 smithi016 ceph-mon[22010]: pgmap v2424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:41 smithi079 ceph-mon[23158]: pgmap v2425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:41 smithi002 ceph-mon[17167]: pgmap v2425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:41 smithi016 ceph-mon[22010]: pgmap v2425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:43 smithi079 ceph-mon[23158]: pgmap v2426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4024.940389 secs 2022-03-21T15:18:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:43 smithi002 ceph-mon[17167]: pgmap v2426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4024.940389 secs 2022-03-21T15:18:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:43 smithi016 ceph-mon[22010]: pgmap v2426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4024.940389 secs 2022-03-21T15:18:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:45 smithi079 ceph-mon[23158]: pgmap v2427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:45 smithi002 ceph-mon[17167]: pgmap v2427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:45 smithi016 ceph-mon[22010]: pgmap v2427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:47 smithi079 ceph-mon[23158]: pgmap v2428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:47 smithi002 ceph-mon[17167]: pgmap v2428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:47 smithi016 ceph-mon[22010]: pgmap v2428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4029.940497 secs 2022-03-21T15:18:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4029.940497 secs 2022-03-21T15:18:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4029.940497 secs 2022-03-21T15:18:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:49 smithi079 ceph-mon[23158]: pgmap v2429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:49 smithi002 ceph-mon[17167]: pgmap v2429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:49 smithi016 ceph-mon[22010]: pgmap v2429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:51 smithi002 ceph-mon[17167]: pgmap v2430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:51 smithi016 ceph-mon[22010]: pgmap v2430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:51 smithi079 ceph-mon[23158]: pgmap v2430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:53 smithi002 ceph-mon[17167]: pgmap v2431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4034.940622 secs 2022-03-21T15:18:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:53 smithi016 ceph-mon[22010]: pgmap v2431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4034.940622 secs 2022-03-21T15:18:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:53 smithi079 ceph-mon[23158]: pgmap v2431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4034.940622 secs 2022-03-21T15:18:55.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:55 smithi002 ceph-mon[17167]: pgmap v2432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:55 smithi016 ceph-mon[22010]: pgmap v2432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:55 smithi079 ceph-mon[23158]: pgmap v2432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:56.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:18:56.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:18:56.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:18:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:18:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:18:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:18:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:18:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:18:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:18:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:57 smithi002 ceph-mon[17167]: pgmap v2433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:18:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:57 smithi016 ceph-mon[22010]: pgmap v2433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:18:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:57 smithi079 ceph-mon[23158]: pgmap v2433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:18:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:18:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4039.940697 secs 2022-03-21T15:18:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4039.940697 secs 2022-03-21T15:18:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4039.940697 secs 2022-03-21T15:18:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:18:59 smithi002 ceph-mon[17167]: pgmap v2434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:18:59 smithi016 ceph-mon[22010]: pgmap v2434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:18:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:18:59 smithi079 ceph-mon[23158]: pgmap v2434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:01 smithi002 ceph-mon[17167]: pgmap v2435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:01 smithi016 ceph-mon[22010]: pgmap v2435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:01 smithi079 ceph-mon[23158]: pgmap v2435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:03 smithi002 ceph-mon[17167]: pgmap v2436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4044.940810 secs 2022-03-21T15:19:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:03 smithi016 ceph-mon[22010]: pgmap v2436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4044.940810 secs 2022-03-21T15:19:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:03 smithi079 ceph-mon[23158]: pgmap v2436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4044.940810 secs 2022-03-21T15:19:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:05 smithi002 ceph-mon[17167]: pgmap v2437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:05 smithi016 ceph-mon[22010]: pgmap v2437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:05 smithi079 ceph-mon[23158]: pgmap v2437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:07 smithi002 ceph-mon[17167]: pgmap v2438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:07 smithi016 ceph-mon[22010]: pgmap v2438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:07 smithi079 ceph-mon[23158]: pgmap v2438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4049.940869 secs 2022-03-21T15:19:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4049.940869 secs 2022-03-21T15:19:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4049.940869 secs 2022-03-21T15:19:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:09 smithi016 ceph-mon[22010]: pgmap v2439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:09 smithi079 ceph-mon[23158]: pgmap v2439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:09 smithi002 ceph-mon[17167]: pgmap v2439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:11 smithi016 ceph-mon[22010]: pgmap v2440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:11 smithi079 ceph-mon[23158]: pgmap v2440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:11 smithi002 ceph-mon[17167]: pgmap v2440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:13 smithi016 ceph-mon[22010]: pgmap v2441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4054.940954 secs 2022-03-21T15:19:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:13 smithi079 ceph-mon[23158]: pgmap v2441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4054.940954 secs 2022-03-21T15:19:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:13 smithi002 ceph-mon[17167]: pgmap v2441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4054.940954 secs 2022-03-21T15:19:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:15 smithi016 ceph-mon[22010]: pgmap v2442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:15 smithi079 ceph-mon[23158]: pgmap v2442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:15 smithi002 ceph-mon[17167]: pgmap v2442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:17 smithi016 ceph-mon[22010]: pgmap v2443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:17 smithi079 ceph-mon[23158]: pgmap v2443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:17 smithi002 ceph-mon[17167]: pgmap v2443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4059.941053 secs 2022-03-21T15:19:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4059.941053 secs 2022-03-21T15:19:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4059.941053 secs 2022-03-21T15:19:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:19 smithi016 ceph-mon[22010]: pgmap v2444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:19 smithi079 ceph-mon[23158]: pgmap v2444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:19 smithi002 ceph-mon[17167]: pgmap v2444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:21 smithi016 ceph-mon[22010]: pgmap v2445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:19:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:19:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:21 smithi079 ceph-mon[23158]: pgmap v2445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:19:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:19:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:21 smithi002 ceph-mon[17167]: pgmap v2445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:19:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:19:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:23 smithi016 ceph-mon[22010]: pgmap v2446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4064.941150 secs 2022-03-21T15:19:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:23 smithi079 ceph-mon[23158]: pgmap v2446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4064.941150 secs 2022-03-21T15:19:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:23 smithi002 ceph-mon[17167]: pgmap v2446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4064.941150 secs 2022-03-21T15:19:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:25 smithi016 ceph-mon[22010]: pgmap v2447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:25 smithi079 ceph-mon[23158]: pgmap v2447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:25 smithi002 ceph-mon[17167]: pgmap v2447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:27 smithi016 ceph-mon[22010]: pgmap v2448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:27 smithi079 ceph-mon[23158]: pgmap v2448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:27 smithi002 ceph-mon[17167]: pgmap v2448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4069.941271 secs 2022-03-21T15:19:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4069.941271 secs 2022-03-21T15:19:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4069.941271 secs 2022-03-21T15:19:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:29 smithi016 ceph-mon[22010]: pgmap v2449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:29 smithi079 ceph-mon[23158]: pgmap v2449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:29 smithi002 ceph-mon[17167]: pgmap v2449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:31 smithi079 ceph-mon[23158]: pgmap v2450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:31 smithi002 ceph-mon[17167]: pgmap v2450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:31 smithi016 ceph-mon[22010]: pgmap v2450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:33 smithi079 ceph-mon[23158]: pgmap v2451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4074.941403 secs 2022-03-21T15:19:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:33 smithi002 ceph-mon[17167]: pgmap v2451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4074.941403 secs 2022-03-21T15:19:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:33 smithi016 ceph-mon[22010]: pgmap v2451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4074.941403 secs 2022-03-21T15:19:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:35 smithi079 ceph-mon[23158]: pgmap v2452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:35 smithi002 ceph-mon[17167]: pgmap v2452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:35 smithi016 ceph-mon[22010]: pgmap v2452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:37 smithi079 ceph-mon[23158]: pgmap v2453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:37 smithi002 ceph-mon[17167]: pgmap v2453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:37 smithi016 ceph-mon[22010]: pgmap v2453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4079.941523 secs 2022-03-21T15:19:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4079.941523 secs 2022-03-21T15:19:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4079.941523 secs 2022-03-21T15:19:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:39 smithi079 ceph-mon[23158]: pgmap v2454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:39 smithi002 ceph-mon[17167]: pgmap v2454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:39 smithi016 ceph-mon[22010]: pgmap v2454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:41 smithi079 ceph-mon[23158]: pgmap v2455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:41 smithi002 ceph-mon[17167]: pgmap v2455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:41 smithi016 ceph-mon[22010]: pgmap v2455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:43 smithi079 ceph-mon[23158]: pgmap v2456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4084.941642 secs 2022-03-21T15:19:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:43 smithi002 ceph-mon[17167]: pgmap v2456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4084.941642 secs 2022-03-21T15:19:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:43 smithi016 ceph-mon[22010]: pgmap v2456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4084.941642 secs 2022-03-21T15:19:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:45 smithi079 ceph-mon[23158]: pgmap v2457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:45 smithi002 ceph-mon[17167]: pgmap v2457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:45 smithi016 ceph-mon[22010]: pgmap v2457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:47 smithi079 ceph-mon[23158]: pgmap v2458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:47 smithi002 ceph-mon[17167]: pgmap v2458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:47 smithi016 ceph-mon[22010]: pgmap v2458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4089.941751 secs 2022-03-21T15:19:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4089.941751 secs 2022-03-21T15:19:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4089.941751 secs 2022-03-21T15:19:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:49 smithi079 ceph-mon[23158]: pgmap v2459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:49 smithi002 ceph-mon[17167]: pgmap v2459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:49 smithi016 ceph-mon[22010]: pgmap v2459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:51 smithi079 ceph-mon[23158]: pgmap v2460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:51 smithi002 ceph-mon[17167]: pgmap v2460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:51 smithi016 ceph-mon[22010]: pgmap v2460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:53 smithi079 ceph-mon[23158]: pgmap v2461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4094.941872 secs 2022-03-21T15:19:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:53 smithi002 ceph-mon[17167]: pgmap v2461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4094.941872 secs 2022-03-21T15:19:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:53 smithi016 ceph-mon[22010]: pgmap v2461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4094.941872 secs 2022-03-21T15:19:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:55 smithi079 ceph-mon[23158]: pgmap v2462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:55 smithi002 ceph-mon[17167]: pgmap v2462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:55 smithi016 ceph-mon[22010]: pgmap v2462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:19:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:19:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:19:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:19:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:19:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:19:56.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:19:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:19:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:19:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:57 smithi002 ceph-mon[17167]: pgmap v2463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:19:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:57 smithi016 ceph-mon[22010]: pgmap v2463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:19:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:57 smithi079 ceph-mon[23158]: pgmap v2463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:19:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:19:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4099.942006 secs 2022-03-21T15:19:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:19:58 smithi002 ceph-mon[17167]: pgmap v2464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4099.942006 secs 2022-03-21T15:19:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:19:58 smithi016 ceph-mon[22010]: pgmap v2464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:19:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4099.942006 secs 2022-03-21T15:19:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:19:58 smithi079 ceph-mon[23158]: pgmap v2464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:00 smithi016 ceph-mon[22010]: pgmap v2465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:20:01.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:00 smithi079 ceph-mon[23158]: pgmap v2465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:20:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:00 smithi002 ceph-mon[17167]: pgmap v2465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:20:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:02 smithi016 ceph-mon[22010]: pgmap v2466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4104.942134 secs 2022-03-21T15:20:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:02 smithi079 ceph-mon[23158]: pgmap v2466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4104.942134 secs 2022-03-21T15:20:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:02 smithi002 ceph-mon[17167]: pgmap v2466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4104.942134 secs 2022-03-21T15:20:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:04 smithi016 ceph-mon[22010]: pgmap v2467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:04 smithi079 ceph-mon[23158]: pgmap v2467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:04 smithi002 ceph-mon[17167]: pgmap v2467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:07 smithi002 ceph-mon[17167]: pgmap v2468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:07 smithi016 ceph-mon[22010]: pgmap v2468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:07 smithi079 ceph-mon[23158]: pgmap v2468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4109.942270 secs 2022-03-21T15:20:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4109.942270 secs 2022-03-21T15:20:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4109.942270 secs 2022-03-21T15:20:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:09 smithi002 ceph-mon[17167]: pgmap v2469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:09 smithi016 ceph-mon[22010]: pgmap v2469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:09 smithi079 ceph-mon[23158]: pgmap v2469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:11 smithi002 ceph-mon[17167]: pgmap v2470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:11 smithi016 ceph-mon[22010]: pgmap v2470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:11 smithi079 ceph-mon[23158]: pgmap v2470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:13 smithi002 ceph-mon[17167]: pgmap v2471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4114.942389 secs 2022-03-21T15:20:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:13 smithi016 ceph-mon[22010]: pgmap v2471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4114.942389 secs 2022-03-21T15:20:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:13 smithi079 ceph-mon[23158]: pgmap v2471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4114.942389 secs 2022-03-21T15:20:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:15 smithi002 ceph-mon[17167]: pgmap v2472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:15 smithi016 ceph-mon[22010]: pgmap v2472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:15 smithi079 ceph-mon[23158]: pgmap v2472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:17 smithi002 ceph-mon[17167]: pgmap v2473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:17 smithi016 ceph-mon[22010]: pgmap v2473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:17 smithi079 ceph-mon[23158]: pgmap v2473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4119.942533 secs 2022-03-21T15:20:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4119.942533 secs 2022-03-21T15:20:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4119.942533 secs 2022-03-21T15:20:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:19 smithi016 ceph-mon[22010]: pgmap v2474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:19 smithi079 ceph-mon[23158]: pgmap v2474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:19 smithi002 ceph-mon[17167]: pgmap v2474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:21 smithi016 ceph-mon[22010]: pgmap v2475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:20:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:20:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:21 smithi079 ceph-mon[23158]: pgmap v2475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:20:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:20:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:21 smithi002 ceph-mon[17167]: pgmap v2475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:20:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:20:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:23 smithi016 ceph-mon[22010]: pgmap v2476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4124.942664 secs 2022-03-21T15:20:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:23 smithi079 ceph-mon[23158]: pgmap v2476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4124.942664 secs 2022-03-21T15:20:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:23 smithi002 ceph-mon[17167]: pgmap v2476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4124.942664 secs 2022-03-21T15:20:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:25 smithi016 ceph-mon[22010]: pgmap v2477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:25 smithi079 ceph-mon[23158]: pgmap v2477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:25 smithi002 ceph-mon[17167]: pgmap v2477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:27 smithi016 ceph-mon[22010]: pgmap v2478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:27 smithi079 ceph-mon[23158]: pgmap v2478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:27 smithi002 ceph-mon[17167]: pgmap v2478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4129.942757 secs 2022-03-21T15:20:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4129.942757 secs 2022-03-21T15:20:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4129.942757 secs 2022-03-21T15:20:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:29 smithi016 ceph-mon[22010]: pgmap v2479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:29 smithi079 ceph-mon[23158]: pgmap v2479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:29 smithi002 ceph-mon[17167]: pgmap v2479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:31 smithi016 ceph-mon[22010]: pgmap v2480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:31 smithi079 ceph-mon[23158]: pgmap v2480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:31 smithi002 ceph-mon[17167]: pgmap v2480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:33 smithi016 ceph-mon[22010]: pgmap v2481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4134.942899 secs 2022-03-21T15:20:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:33 smithi079 ceph-mon[23158]: pgmap v2481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4134.942899 secs 2022-03-21T15:20:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:33 smithi002 ceph-mon[17167]: pgmap v2481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4134.942899 secs 2022-03-21T15:20:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:35 smithi016 ceph-mon[22010]: pgmap v2482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:35 smithi079 ceph-mon[23158]: pgmap v2482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:35 smithi002 ceph-mon[17167]: pgmap v2482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:37 smithi016 ceph-mon[22010]: pgmap v2483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:37 smithi079 ceph-mon[23158]: pgmap v2483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:37 smithi002 ceph-mon[17167]: pgmap v2483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4139.943038 secs 2022-03-21T15:20:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4139.943038 secs 2022-03-21T15:20:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4139.943038 secs 2022-03-21T15:20:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:39 smithi079 ceph-mon[23158]: pgmap v2484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:39 smithi002 ceph-mon[17167]: pgmap v2484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:39 smithi016 ceph-mon[22010]: pgmap v2484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:41 smithi079 ceph-mon[23158]: pgmap v2485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:41 smithi002 ceph-mon[17167]: pgmap v2485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:41 smithi016 ceph-mon[22010]: pgmap v2485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:43 smithi079 ceph-mon[23158]: pgmap v2486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4144.943137 secs 2022-03-21T15:20:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:43 smithi002 ceph-mon[17167]: pgmap v2486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4144.943137 secs 2022-03-21T15:20:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:43 smithi016 ceph-mon[22010]: pgmap v2486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4144.943137 secs 2022-03-21T15:20:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:45 smithi079 ceph-mon[23158]: pgmap v2487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:45 smithi002 ceph-mon[17167]: pgmap v2487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:45 smithi016 ceph-mon[22010]: pgmap v2487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:47 smithi079 ceph-mon[23158]: pgmap v2488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:47 smithi002 ceph-mon[17167]: pgmap v2488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:47 smithi016 ceph-mon[22010]: pgmap v2488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4149.943274 secs 2022-03-21T15:20:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4149.943274 secs 2022-03-21T15:20:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4149.943274 secs 2022-03-21T15:20:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:49 smithi079 ceph-mon[23158]: pgmap v2489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:49 smithi002 ceph-mon[17167]: pgmap v2489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:49 smithi016 ceph-mon[22010]: pgmap v2489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:51 smithi079 ceph-mon[23158]: pgmap v2490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:51 smithi002 ceph-mon[17167]: pgmap v2490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:51 smithi016 ceph-mon[22010]: pgmap v2490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:53 smithi079 ceph-mon[23158]: pgmap v2491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4154.943409 secs 2022-03-21T15:20:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:53 smithi002 ceph-mon[17167]: pgmap v2491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4154.943409 secs 2022-03-21T15:20:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:53 smithi016 ceph-mon[22010]: pgmap v2491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4154.943409 secs 2022-03-21T15:20:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:55 smithi079 ceph-mon[23158]: pgmap v2492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:55 smithi002 ceph-mon[17167]: pgmap v2492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:55 smithi016 ceph-mon[22010]: pgmap v2492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:57.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:57 smithi002 ceph-mon[17167]: pgmap v2493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:20:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:20:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:20:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:57 smithi016 ceph-mon[22010]: pgmap v2493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:20:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:20:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:20:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:57 smithi079 ceph-mon[23158]: pgmap v2493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:20:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:20:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:20:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:20:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4159.943550 secs 2022-03-21T15:20:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4159.943550 secs 2022-03-21T15:20:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4159.943550 secs 2022-03-21T15:20:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:20:59 smithi079 ceph-mon[23158]: pgmap v2494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:59.523 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:20:59 smithi016 ceph-mon[22010]: pgmap v2494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:20:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:20:59 smithi002 ceph-mon[17167]: pgmap v2494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:00.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:00 smithi002 ceph-mon[17167]: pgmap v2495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:00 smithi016 ceph-mon[22010]: pgmap v2495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:00 smithi079 ceph-mon[23158]: pgmap v2495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:02.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:21:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:02 smithi002 ceph-mon[17167]: pgmap v2496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4164.943658 secs 2022-03-21T15:21:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:02 smithi016 ceph-mon[22010]: pgmap v2496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4164.943658 secs 2022-03-21T15:21:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:02 smithi079 ceph-mon[23158]: pgmap v2496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4164.943658 secs 2022-03-21T15:21:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:04 smithi002 ceph-mon[17167]: pgmap v2497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:04 smithi016 ceph-mon[22010]: pgmap v2497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:04 smithi079 ceph-mon[23158]: pgmap v2497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:07 smithi002 ceph-mon[17167]: pgmap v2498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:07 smithi016 ceph-mon[22010]: pgmap v2498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:07 smithi079 ceph-mon[23158]: pgmap v2498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4169.943761 secs 2022-03-21T15:21:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4169.943761 secs 2022-03-21T15:21:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4169.943761 secs 2022-03-21T15:21:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:09 smithi002 ceph-mon[17167]: pgmap v2499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:09 smithi016 ceph-mon[22010]: pgmap v2499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:09 smithi079 ceph-mon[23158]: pgmap v2499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:11 smithi002 ceph-mon[17167]: pgmap v2500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:11 smithi016 ceph-mon[22010]: pgmap v2500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:11 smithi079 ceph-mon[23158]: pgmap v2500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:13 smithi016 ceph-mon[22010]: pgmap v2501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4174.943826 secs 2022-03-21T15:21:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:13 smithi079 ceph-mon[23158]: pgmap v2501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4174.943826 secs 2022-03-21T15:21:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:13 smithi002 ceph-mon[17167]: pgmap v2501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4174.943826 secs 2022-03-21T15:21:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:15 smithi016 ceph-mon[22010]: pgmap v2502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:15 smithi079 ceph-mon[23158]: pgmap v2502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:15 smithi002 ceph-mon[17167]: pgmap v2502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:17 smithi016 ceph-mon[22010]: pgmap v2503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:17 smithi079 ceph-mon[23158]: pgmap v2503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:17 smithi002 ceph-mon[17167]: pgmap v2503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4179.943943 secs 2022-03-21T15:21:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4179.943943 secs 2022-03-21T15:21:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4179.943943 secs 2022-03-21T15:21:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:19 smithi016 ceph-mon[22010]: pgmap v2504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:19 smithi079 ceph-mon[23158]: pgmap v2504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:19 smithi002 ceph-mon[17167]: pgmap v2504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:21 smithi016 ceph-mon[22010]: pgmap v2505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:21:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:21:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:21 smithi079 ceph-mon[23158]: pgmap v2505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:21:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:21:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:21 smithi002 ceph-mon[17167]: pgmap v2505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:21:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:21:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:23 smithi016 ceph-mon[22010]: pgmap v2506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4184.944040 secs 2022-03-21T15:21:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:23 smithi079 ceph-mon[23158]: pgmap v2506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4184.944040 secs 2022-03-21T15:21:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:23 smithi002 ceph-mon[17167]: pgmap v2506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4184.944040 secs 2022-03-21T15:21:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:25 smithi016 ceph-mon[22010]: pgmap v2507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:25 smithi079 ceph-mon[23158]: pgmap v2507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:25 smithi002 ceph-mon[17167]: pgmap v2507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:27 smithi016 ceph-mon[22010]: pgmap v2508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:27 smithi079 ceph-mon[23158]: pgmap v2508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:27 smithi002 ceph-mon[17167]: pgmap v2508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4189.944184 secs 2022-03-21T15:21:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4189.944184 secs 2022-03-21T15:21:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4189.944184 secs 2022-03-21T15:21:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:29 smithi016 ceph-mon[22010]: pgmap v2509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:29 smithi079 ceph-mon[23158]: pgmap v2509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:29 smithi002 ceph-mon[17167]: pgmap v2509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:31 smithi016 ceph-mon[22010]: pgmap v2510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:31 smithi079 ceph-mon[23158]: pgmap v2510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:31 smithi002 ceph-mon[17167]: pgmap v2510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:33 smithi079 ceph-mon[23158]: pgmap v2511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4194.944301 secs 2022-03-21T15:21:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:33 smithi002 ceph-mon[17167]: pgmap v2511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4194.944301 secs 2022-03-21T15:21:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:33 smithi016 ceph-mon[22010]: pgmap v2511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4194.944301 secs 2022-03-21T15:21:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:35 smithi079 ceph-mon[23158]: pgmap v2512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:35 smithi002 ceph-mon[17167]: pgmap v2512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:35 smithi016 ceph-mon[22010]: pgmap v2512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:37 smithi079 ceph-mon[23158]: pgmap v2513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:37 smithi002 ceph-mon[17167]: pgmap v2513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:37 smithi016 ceph-mon[22010]: pgmap v2513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4199.944421 secs 2022-03-21T15:21:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4199.944421 secs 2022-03-21T15:21:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4199.944421 secs 2022-03-21T15:21:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:39 smithi079 ceph-mon[23158]: pgmap v2514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:39 smithi002 ceph-mon[17167]: pgmap v2514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:39 smithi016 ceph-mon[22010]: pgmap v2514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:41 smithi079 ceph-mon[23158]: pgmap v2515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:41 smithi002 ceph-mon[17167]: pgmap v2515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:41 smithi016 ceph-mon[22010]: pgmap v2515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:43 smithi079 ceph-mon[23158]: pgmap v2516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4204.944551 secs 2022-03-21T15:21:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:43 smithi002 ceph-mon[17167]: pgmap v2516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4204.944551 secs 2022-03-21T15:21:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:43 smithi016 ceph-mon[22010]: pgmap v2516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4204.944551 secs 2022-03-21T15:21:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:45 smithi079 ceph-mon[23158]: pgmap v2517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:45 smithi002 ceph-mon[17167]: pgmap v2517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:45 smithi016 ceph-mon[22010]: pgmap v2517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:47 smithi079 ceph-mon[23158]: pgmap v2518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:47 smithi002 ceph-mon[17167]: pgmap v2518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:47 smithi016 ceph-mon[22010]: pgmap v2518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:21:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4209.944678 secs 2022-03-21T15:21:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4209.944678 secs 2022-03-21T15:21:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4209.944678 secs 2022-03-21T15:21:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:49 smithi079 ceph-mon[23158]: pgmap v2519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:49 smithi002 ceph-mon[17167]: pgmap v2519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:49 smithi016 ceph-mon[22010]: pgmap v2519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:51 smithi079 ceph-mon[23158]: pgmap v2520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:51 smithi002 ceph-mon[17167]: pgmap v2520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:51 smithi016 ceph-mon[22010]: pgmap v2520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:53 smithi079 ceph-mon[23158]: pgmap v2521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4214.944750 secs 2022-03-21T15:21:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:53 smithi002 ceph-mon[17167]: pgmap v2521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4214.944750 secs 2022-03-21T15:21:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:53 smithi016 ceph-mon[22010]: pgmap v2521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4214.944750 secs 2022-03-21T15:21:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:55 smithi079 ceph-mon[23158]: pgmap v2522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:55 smithi002 ceph-mon[17167]: pgmap v2522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:55 smithi016 ceph-mon[22010]: pgmap v2522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:57 smithi079 ceph-mon[23158]: pgmap v2523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:57 smithi002 ceph-mon[17167]: pgmap v2523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:57 smithi016 ceph-mon[22010]: pgmap v2523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:21:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4219.944859 secs 2022-03-21T15:21:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4219.944859 secs 2022-03-21T15:21:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4219.944859 secs 2022-03-21T15:21:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:21:59 smithi002 ceph-mon[17167]: pgmap v2524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:21:59 smithi016 ceph-mon[22010]: pgmap v2524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:21:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:21:59 smithi079 ceph-mon[23158]: pgmap v2524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:01 smithi002 ceph-mon[17167]: pgmap v2525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:01 smithi016 ceph-mon[22010]: pgmap v2525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:01 smithi079 ceph-mon[23158]: pgmap v2525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:22:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:22:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:22:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:22:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:22:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:22:02.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:22:02.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:22:02.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:22:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:03 smithi002 ceph-mon[17167]: pgmap v2526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4224.945005 secs 2022-03-21T15:22:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:22:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:03 smithi016 ceph-mon[22010]: pgmap v2526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4224.945005 secs 2022-03-21T15:22:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:22:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:03 smithi079 ceph-mon[23158]: pgmap v2526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4224.945005 secs 2022-03-21T15:22:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:22:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:05 smithi002 ceph-mon[17167]: pgmap v2527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:05 smithi016 ceph-mon[22010]: pgmap v2527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:05 smithi079 ceph-mon[23158]: pgmap v2527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:07.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:07 smithi002 ceph-mon[17167]: pgmap v2528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:07 smithi016 ceph-mon[22010]: pgmap v2528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:07 smithi079 ceph-mon[23158]: pgmap v2528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4229.945142 secs 2022-03-21T15:22:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4229.945142 secs 2022-03-21T15:22:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4229.945142 secs 2022-03-21T15:22:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:09 smithi002 ceph-mon[17167]: pgmap v2529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:09 smithi016 ceph-mon[22010]: pgmap v2529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:09 smithi079 ceph-mon[23158]: pgmap v2529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:11 smithi016 ceph-mon[22010]: pgmap v2530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:11 smithi079 ceph-mon[23158]: pgmap v2530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:11 smithi002 ceph-mon[17167]: pgmap v2530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:13 smithi016 ceph-mon[22010]: pgmap v2531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4234.945262 secs 2022-03-21T15:22:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:13 smithi079 ceph-mon[23158]: pgmap v2531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4234.945262 secs 2022-03-21T15:22:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:13 smithi002 ceph-mon[17167]: pgmap v2531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4234.945262 secs 2022-03-21T15:22:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:15 smithi016 ceph-mon[22010]: pgmap v2532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:15 smithi079 ceph-mon[23158]: pgmap v2532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:15 smithi002 ceph-mon[17167]: pgmap v2532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:17 smithi016 ceph-mon[22010]: pgmap v2533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:17.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:17 smithi079 ceph-mon[23158]: pgmap v2533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:17 smithi002 ceph-mon[17167]: pgmap v2533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4239.945375 secs 2022-03-21T15:22:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4239.945375 secs 2022-03-21T15:22:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4239.945375 secs 2022-03-21T15:22:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:19 smithi016 ceph-mon[22010]: pgmap v2534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:19 smithi079 ceph-mon[23158]: pgmap v2534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:19 smithi002 ceph-mon[17167]: pgmap v2534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:21 smithi016 ceph-mon[22010]: pgmap v2535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:22:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:22:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:21 smithi079 ceph-mon[23158]: pgmap v2535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:22:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:22:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:21 smithi002 ceph-mon[17167]: pgmap v2535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:22:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:22:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:23 smithi016 ceph-mon[22010]: pgmap v2536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4244.945493 secs 2022-03-21T15:22:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:23 smithi079 ceph-mon[23158]: pgmap v2536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4244.945493 secs 2022-03-21T15:22:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:23 smithi002 ceph-mon[17167]: pgmap v2536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4244.945493 secs 2022-03-21T15:22:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:25 smithi016 ceph-mon[22010]: pgmap v2537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:25 smithi079 ceph-mon[23158]: pgmap v2537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:25 smithi002 ceph-mon[17167]: pgmap v2537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:27 smithi016 ceph-mon[22010]: pgmap v2538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:27 smithi079 ceph-mon[23158]: pgmap v2538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:27 smithi002 ceph-mon[17167]: pgmap v2538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4249.945633 secs 2022-03-21T15:22:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4249.945633 secs 2022-03-21T15:22:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4249.945633 secs 2022-03-21T15:22:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:29 smithi079 ceph-mon[23158]: pgmap v2539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:29 smithi002 ceph-mon[17167]: pgmap v2539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:29 smithi016 ceph-mon[22010]: pgmap v2539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:31 smithi079 ceph-mon[23158]: pgmap v2540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:31 smithi002 ceph-mon[17167]: pgmap v2540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:31 smithi016 ceph-mon[22010]: pgmap v2540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:33 smithi079 ceph-mon[23158]: pgmap v2541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4254.945712 secs 2022-03-21T15:22:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:33 smithi002 ceph-mon[17167]: pgmap v2541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4254.945712 secs 2022-03-21T15:22:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:33 smithi016 ceph-mon[22010]: pgmap v2541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4254.945712 secs 2022-03-21T15:22:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:35 smithi079 ceph-mon[23158]: pgmap v2542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:35 smithi002 ceph-mon[17167]: pgmap v2542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:35 smithi016 ceph-mon[22010]: pgmap v2542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:37 smithi079 ceph-mon[23158]: pgmap v2543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:37 smithi002 ceph-mon[17167]: pgmap v2543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:37 smithi016 ceph-mon[22010]: pgmap v2543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4259.945808 secs 2022-03-21T15:22:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4259.945808 secs 2022-03-21T15:22:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4259.945808 secs 2022-03-21T15:22:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:39 smithi079 ceph-mon[23158]: pgmap v2544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:39 smithi002 ceph-mon[17167]: pgmap v2544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:39 smithi016 ceph-mon[22010]: pgmap v2544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:41 smithi079 ceph-mon[23158]: pgmap v2545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:41 smithi002 ceph-mon[17167]: pgmap v2545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:41 smithi016 ceph-mon[22010]: pgmap v2545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:43 smithi079 ceph-mon[23158]: pgmap v2546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4264.945921 secs 2022-03-21T15:22:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:43 smithi002 ceph-mon[17167]: pgmap v2546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4264.945921 secs 2022-03-21T15:22:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:43 smithi016 ceph-mon[22010]: pgmap v2546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4264.945921 secs 2022-03-21T15:22:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:45 smithi079 ceph-mon[23158]: pgmap v2547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:45 smithi002 ceph-mon[17167]: pgmap v2547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:45 smithi016 ceph-mon[22010]: pgmap v2547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:47 smithi079 ceph-mon[23158]: pgmap v2548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:47 smithi002 ceph-mon[17167]: pgmap v2548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:47 smithi016 ceph-mon[22010]: pgmap v2548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4269.946038 secs 2022-03-21T15:22:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4269.946038 secs 2022-03-21T15:22:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4269.946038 secs 2022-03-21T15:22:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:49 smithi079 ceph-mon[23158]: pgmap v2549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:49 smithi002 ceph-mon[17167]: pgmap v2549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:49 smithi016 ceph-mon[22010]: pgmap v2549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:51 smithi079 ceph-mon[23158]: pgmap v2550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:51 smithi002 ceph-mon[17167]: pgmap v2550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:51 smithi016 ceph-mon[22010]: pgmap v2550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:53 smithi079 ceph-mon[23158]: pgmap v2551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4274.946173 secs 2022-03-21T15:22:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:53 smithi002 ceph-mon[17167]: pgmap v2551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4274.946173 secs 2022-03-21T15:22:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:53 smithi016 ceph-mon[22010]: pgmap v2551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4274.946173 secs 2022-03-21T15:22:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:55 smithi079 ceph-mon[23158]: pgmap v2552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:55 smithi002 ceph-mon[17167]: pgmap v2552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:55 smithi016 ceph-mon[22010]: pgmap v2552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:57 smithi002 ceph-mon[17167]: pgmap v2553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:57 smithi016 ceph-mon[22010]: pgmap v2553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:57 smithi079 ceph-mon[23158]: pgmap v2553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:22:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4279.946304 secs 2022-03-21T15:22:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4279.946304 secs 2022-03-21T15:22:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4279.946304 secs 2022-03-21T15:22:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:22:59 smithi002 ceph-mon[17167]: pgmap v2554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:22:59 smithi016 ceph-mon[22010]: pgmap v2554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:22:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:22:59 smithi079 ceph-mon[23158]: pgmap v2554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:01 smithi002 ceph-mon[17167]: pgmap v2555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:01 smithi016 ceph-mon[22010]: pgmap v2555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:01 smithi079 ceph-mon[23158]: pgmap v2555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: pgmap v2556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4284.946413 secs 2022-03-21T15:23:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:23:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:23:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:23:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: pgmap v2556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4284.946413 secs 2022-03-21T15:23:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:23:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:23:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:23:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: pgmap v2556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4284.946413 secs 2022-03-21T15:23:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:23:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:23:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:23:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:05 smithi002 ceph-mon[17167]: pgmap v2557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:05 smithi016 ceph-mon[22010]: pgmap v2557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:05 smithi079 ceph-mon[23158]: pgmap v2557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:23:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:07 smithi016 ceph-mon[22010]: pgmap v2558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:07 smithi079 ceph-mon[23158]: pgmap v2558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:07 smithi002 ceph-mon[17167]: pgmap v2558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4289.946519 secs 2022-03-21T15:23:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4289.946519 secs 2022-03-21T15:23:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4289.946519 secs 2022-03-21T15:23:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:09 smithi016 ceph-mon[22010]: pgmap v2559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:09 smithi079 ceph-mon[23158]: pgmap v2559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:09 smithi002 ceph-mon[17167]: pgmap v2559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:11 smithi016 ceph-mon[22010]: pgmap v2560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:11 smithi079 ceph-mon[23158]: pgmap v2560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:11 smithi002 ceph-mon[17167]: pgmap v2560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:13 smithi016 ceph-mon[22010]: pgmap v2561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4294.946598 secs 2022-03-21T15:23:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:13 smithi079 ceph-mon[23158]: pgmap v2561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4294.946598 secs 2022-03-21T15:23:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:13 smithi002 ceph-mon[17167]: pgmap v2561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4294.946598 secs 2022-03-21T15:23:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:15 smithi079 ceph-mon[23158]: pgmap v2562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:15 smithi002 ceph-mon[17167]: pgmap v2562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:15 smithi016 ceph-mon[22010]: pgmap v2562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:17 smithi079 ceph-mon[23158]: pgmap v2563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:17 smithi002 ceph-mon[17167]: pgmap v2563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:17 smithi016 ceph-mon[22010]: pgmap v2563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4299.946718 secs 2022-03-21T15:23:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4299.946718 secs 2022-03-21T15:23:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4299.946718 secs 2022-03-21T15:23:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:19 smithi079 ceph-mon[23158]: pgmap v2564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:19 smithi002 ceph-mon[17167]: pgmap v2564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:19 smithi016 ceph-mon[22010]: pgmap v2564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:21 smithi079 ceph-mon[23158]: pgmap v2565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:23:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:23:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:21 smithi002 ceph-mon[17167]: pgmap v2565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:23:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:23:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:21 smithi016 ceph-mon[22010]: pgmap v2565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:23:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:23:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:23 smithi079 ceph-mon[23158]: pgmap v2566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4304.946833 secs 2022-03-21T15:23:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:23 smithi002 ceph-mon[17167]: pgmap v2566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4304.946833 secs 2022-03-21T15:23:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:23 smithi016 ceph-mon[22010]: pgmap v2566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4304.946833 secs 2022-03-21T15:23:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:25 smithi079 ceph-mon[23158]: pgmap v2567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:25 smithi002 ceph-mon[17167]: pgmap v2567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:25 smithi016 ceph-mon[22010]: pgmap v2567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:27 smithi079 ceph-mon[23158]: pgmap v2568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:27 smithi002 ceph-mon[17167]: pgmap v2568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:27 smithi016 ceph-mon[22010]: pgmap v2568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4309.946952 secs 2022-03-21T15:23:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4309.946952 secs 2022-03-21T15:23:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4309.946952 secs 2022-03-21T15:23:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:29 smithi079 ceph-mon[23158]: pgmap v2569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:29 smithi002 ceph-mon[17167]: pgmap v2569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:29 smithi016 ceph-mon[22010]: pgmap v2569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:31 smithi079 ceph-mon[23158]: pgmap v2570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:31 smithi002 ceph-mon[17167]: pgmap v2570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:31 smithi016 ceph-mon[22010]: pgmap v2570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:33 smithi079 ceph-mon[23158]: pgmap v2571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4314.947081 secs 2022-03-21T15:23:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:33 smithi002 ceph-mon[17167]: pgmap v2571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4314.947081 secs 2022-03-21T15:23:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:33 smithi016 ceph-mon[22010]: pgmap v2571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4314.947081 secs 2022-03-21T15:23:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:35 smithi079 ceph-mon[23158]: pgmap v2572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:35 smithi002 ceph-mon[17167]: pgmap v2572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:35 smithi016 ceph-mon[22010]: pgmap v2572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:37 smithi079 ceph-mon[23158]: pgmap v2573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:37 smithi002 ceph-mon[17167]: pgmap v2573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:37 smithi016 ceph-mon[22010]: pgmap v2573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4319.947191 secs 2022-03-21T15:23:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4319.947191 secs 2022-03-21T15:23:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4319.947191 secs 2022-03-21T15:23:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:39 smithi079 ceph-mon[23158]: pgmap v2574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:39 smithi002 ceph-mon[17167]: pgmap v2574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:39 smithi016 ceph-mon[22010]: pgmap v2574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:41 smithi002 ceph-mon[17167]: pgmap v2575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:41 smithi016 ceph-mon[22010]: pgmap v2575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:41 smithi079 ceph-mon[23158]: pgmap v2575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:43 smithi002 ceph-mon[17167]: pgmap v2576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4324.947310 secs 2022-03-21T15:23:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:43 smithi016 ceph-mon[22010]: pgmap v2576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4324.947310 secs 2022-03-21T15:23:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:43 smithi079 ceph-mon[23158]: pgmap v2576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4324.947310 secs 2022-03-21T15:23:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:45 smithi002 ceph-mon[17167]: pgmap v2577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:45 smithi016 ceph-mon[22010]: pgmap v2577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:45 smithi079 ceph-mon[23158]: pgmap v2577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:47 smithi002 ceph-mon[17167]: pgmap v2578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:47 smithi016 ceph-mon[22010]: pgmap v2578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:47 smithi079 ceph-mon[23158]: pgmap v2578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4329.947462 secs 2022-03-21T15:23:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4329.947462 secs 2022-03-21T15:23:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4329.947462 secs 2022-03-21T15:23:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:49 smithi002 ceph-mon[17167]: pgmap v2579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:49 smithi016 ceph-mon[22010]: pgmap v2579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:49 smithi079 ceph-mon[23158]: pgmap v2579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:51 smithi002 ceph-mon[17167]: pgmap v2580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:51 smithi016 ceph-mon[22010]: pgmap v2580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:51 smithi079 ceph-mon[23158]: pgmap v2580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:53 smithi002 ceph-mon[17167]: pgmap v2581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4334.947579 secs 2022-03-21T15:23:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:53 smithi016 ceph-mon[22010]: pgmap v2581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4334.947579 secs 2022-03-21T15:23:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:53 smithi079 ceph-mon[23158]: pgmap v2581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4334.947579 secs 2022-03-21T15:23:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:55 smithi002 ceph-mon[17167]: pgmap v2582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:55 smithi016 ceph-mon[22010]: pgmap v2582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:55 smithi079 ceph-mon[23158]: pgmap v2582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:57 smithi002 ceph-mon[17167]: pgmap v2583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:57 smithi016 ceph-mon[22010]: pgmap v2583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:57 smithi079 ceph-mon[23158]: pgmap v2583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:23:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4339.947614 secs 2022-03-21T15:23:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4339.947614 secs 2022-03-21T15:23:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4339.947614 secs 2022-03-21T15:23:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:23:59 smithi002 ceph-mon[17167]: pgmap v2584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:23:59 smithi016 ceph-mon[22010]: pgmap v2584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:23:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:23:59 smithi079 ceph-mon[23158]: pgmap v2584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:01 smithi079 ceph-mon[23158]: pgmap v2585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:01 smithi002 ceph-mon[17167]: pgmap v2585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:01 smithi016 ceph-mon[22010]: pgmap v2585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:03 smithi079 ceph-mon[23158]: pgmap v2586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4344.947738 secs 2022-03-21T15:24:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:03 smithi002 ceph-mon[17167]: pgmap v2586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4344.947738 secs 2022-03-21T15:24:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:03 smithi016 ceph-mon[22010]: pgmap v2586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4344.947738 secs 2022-03-21T15:24:04.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:24:04.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:24:04.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:24:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:24:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:24:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:24:04.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:24:04.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:24:04.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:24:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:05 smithi079 ceph-mon[23158]: pgmap v2587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:24:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:05 smithi002 ceph-mon[17167]: pgmap v2587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:24:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:05 smithi016 ceph-mon[22010]: pgmap v2587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:24:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:07 smithi079 ceph-mon[23158]: pgmap v2588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:07.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:07 smithi002 ceph-mon[17167]: pgmap v2588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:07 smithi016 ceph-mon[22010]: pgmap v2588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4349.947884 secs 2022-03-21T15:24:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4349.947884 secs 2022-03-21T15:24:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4349.947884 secs 2022-03-21T15:24:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:09 smithi079 ceph-mon[23158]: pgmap v2589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:09 smithi002 ceph-mon[17167]: pgmap v2589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:09 smithi016 ceph-mon[22010]: pgmap v2589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:11 smithi079 ceph-mon[23158]: pgmap v2590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:11 smithi002 ceph-mon[17167]: pgmap v2590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:11 smithi016 ceph-mon[22010]: pgmap v2590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:13 smithi079 ceph-mon[23158]: pgmap v2591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4354.948001 secs 2022-03-21T15:24:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:13 smithi002 ceph-mon[17167]: pgmap v2591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4354.948001 secs 2022-03-21T15:24:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:13 smithi016 ceph-mon[22010]: pgmap v2591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4354.948001 secs 2022-03-21T15:24:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:15 smithi079 ceph-mon[23158]: pgmap v2592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:15 smithi002 ceph-mon[17167]: pgmap v2592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:15 smithi016 ceph-mon[22010]: pgmap v2592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:17 smithi079 ceph-mon[23158]: pgmap v2593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:17 smithi002 ceph-mon[17167]: pgmap v2593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:17 smithi016 ceph-mon[22010]: pgmap v2593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4359.948138 secs 2022-03-21T15:24:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4359.948138 secs 2022-03-21T15:24:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4359.948138 secs 2022-03-21T15:24:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:19 smithi079 ceph-mon[23158]: pgmap v2594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:19 smithi002 ceph-mon[17167]: pgmap v2594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:19 smithi016 ceph-mon[22010]: pgmap v2594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:21 smithi079 ceph-mon[23158]: pgmap v2595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:24:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:24:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:21 smithi002 ceph-mon[17167]: pgmap v2595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:24:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:24:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:21 smithi016 ceph-mon[22010]: pgmap v2595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:24:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:24:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:23 smithi079 ceph-mon[23158]: pgmap v2596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4364.948278 secs 2022-03-21T15:24:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:23 smithi002 ceph-mon[17167]: pgmap v2596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4364.948278 secs 2022-03-21T15:24:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:23 smithi016 ceph-mon[22010]: pgmap v2596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4364.948278 secs 2022-03-21T15:24:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:25 smithi002 ceph-mon[17167]: pgmap v2597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:25 smithi016 ceph-mon[22010]: pgmap v2597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:25 smithi079 ceph-mon[23158]: pgmap v2597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:27 smithi002 ceph-mon[17167]: pgmap v2598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:27 smithi016 ceph-mon[22010]: pgmap v2598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:27 smithi079 ceph-mon[23158]: pgmap v2598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4369.948404 secs 2022-03-21T15:24:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4369.948404 secs 2022-03-21T15:24:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4369.948404 secs 2022-03-21T15:24:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:29 smithi002 ceph-mon[17167]: pgmap v2599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:29 smithi016 ceph-mon[22010]: pgmap v2599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:29 smithi079 ceph-mon[23158]: pgmap v2599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:31 smithi002 ceph-mon[17167]: pgmap v2600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:31 smithi016 ceph-mon[22010]: pgmap v2600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:31 smithi079 ceph-mon[23158]: pgmap v2600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:33 smithi002 ceph-mon[17167]: pgmap v2601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4374.948521 secs 2022-03-21T15:24:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:33 smithi016 ceph-mon[22010]: pgmap v2601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4374.948521 secs 2022-03-21T15:24:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:33 smithi079 ceph-mon[23158]: pgmap v2601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4374.948521 secs 2022-03-21T15:24:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:35 smithi002 ceph-mon[17167]: pgmap v2602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:35 smithi016 ceph-mon[22010]: pgmap v2602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:35 smithi079 ceph-mon[23158]: pgmap v2602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:37 smithi002 ceph-mon[17167]: pgmap v2603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:37 smithi016 ceph-mon[22010]: pgmap v2603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:37 smithi079 ceph-mon[23158]: pgmap v2603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4379.948637 secs 2022-03-21T15:24:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4379.948637 secs 2022-03-21T15:24:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4379.948637 secs 2022-03-21T15:24:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:39 smithi002 ceph-mon[17167]: pgmap v2604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:39 smithi016 ceph-mon[22010]: pgmap v2604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:39 smithi079 ceph-mon[23158]: pgmap v2604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:41 smithi002 ceph-mon[17167]: pgmap v2605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:41 smithi016 ceph-mon[22010]: pgmap v2605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:41 smithi079 ceph-mon[23158]: pgmap v2605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:43 smithi002 ceph-mon[17167]: pgmap v2606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4384.948702 secs 2022-03-21T15:24:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:43 smithi016 ceph-mon[22010]: pgmap v2606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4384.948702 secs 2022-03-21T15:24:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:43 smithi079 ceph-mon[23158]: pgmap v2606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4384.948702 secs 2022-03-21T15:24:45.569 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:45 smithi002 ceph-mon[17167]: pgmap v2607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:45 smithi016 ceph-mon[22010]: pgmap v2607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:45 smithi079 ceph-mon[23158]: pgmap v2607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:47.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:47 smithi002 ceph-mon[17167]: pgmap v2608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:47 smithi016 ceph-mon[22010]: pgmap v2608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:47 smithi079 ceph-mon[23158]: pgmap v2608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4389.948819 secs 2022-03-21T15:24:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4389.948819 secs 2022-03-21T15:24:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4389.948819 secs 2022-03-21T15:24:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:49 smithi002 ceph-mon[17167]: pgmap v2609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:49 smithi016 ceph-mon[22010]: pgmap v2609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:49 smithi079 ceph-mon[23158]: pgmap v2609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:51 smithi079 ceph-mon[23158]: pgmap v2610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:51 smithi002 ceph-mon[17167]: pgmap v2610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:51 smithi016 ceph-mon[22010]: pgmap v2610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:53 smithi079 ceph-mon[23158]: pgmap v2611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4394.948896 secs 2022-03-21T15:24:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:53 smithi002 ceph-mon[17167]: pgmap v2611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4394.948896 secs 2022-03-21T15:24:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:53 smithi016 ceph-mon[22010]: pgmap v2611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4394.948896 secs 2022-03-21T15:24:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:55 smithi079 ceph-mon[23158]: pgmap v2612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:55 smithi002 ceph-mon[17167]: pgmap v2612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:55 smithi016 ceph-mon[22010]: pgmap v2612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:57 smithi079 ceph-mon[23158]: pgmap v2613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:57 smithi002 ceph-mon[17167]: pgmap v2613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:57 smithi016 ceph-mon[22010]: pgmap v2613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:24:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4399.949034 secs 2022-03-21T15:24:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4399.949034 secs 2022-03-21T15:24:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4399.949034 secs 2022-03-21T15:24:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:24:59 smithi079 ceph-mon[23158]: pgmap v2614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:24:59 smithi002 ceph-mon[17167]: pgmap v2614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:24:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:24:59 smithi016 ceph-mon[22010]: pgmap v2614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:01 smithi079 ceph-mon[23158]: pgmap v2615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:01 smithi002 ceph-mon[17167]: pgmap v2615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:01 smithi016 ceph-mon[22010]: pgmap v2615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:03 smithi079 ceph-mon[23158]: pgmap v2616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4404.949168 secs 2022-03-21T15:25:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:03 smithi002 ceph-mon[17167]: pgmap v2616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4404.949168 secs 2022-03-21T15:25:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:03 smithi016 ceph-mon[22010]: pgmap v2616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4404.949168 secs 2022-03-21T15:25:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:05 smithi002 ceph-mon[17167]: pgmap v2617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:25:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:25:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:25:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:05 smithi079 ceph-mon[23158]: pgmap v2617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:25:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:25:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:25:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:05 smithi016 ceph-mon[22010]: pgmap v2617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:25:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:25:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:06 smithi002 ceph-mon[17167]: pgmap v2618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:25:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:06 smithi016 ceph-mon[22010]: pgmap v2618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:25:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:06 smithi079 ceph-mon[23158]: pgmap v2618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4409.949284 secs 2022-03-21T15:25:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4409.949284 secs 2022-03-21T15:25:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4409.949284 secs 2022-03-21T15:25:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:08 smithi079 ceph-mon[23158]: pgmap v2619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:08 smithi002 ceph-mon[17167]: pgmap v2619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:08 smithi016 ceph-mon[22010]: pgmap v2619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:11 smithi079 ceph-mon[23158]: pgmap v2620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:11 smithi002 ceph-mon[17167]: pgmap v2620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:11 smithi016 ceph-mon[22010]: pgmap v2620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:13 smithi079 ceph-mon[23158]: pgmap v2621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4414.949399 secs 2022-03-21T15:25:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:13 smithi002 ceph-mon[17167]: pgmap v2621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4414.949399 secs 2022-03-21T15:25:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:13 smithi016 ceph-mon[22010]: pgmap v2621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4414.949399 secs 2022-03-21T15:25:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:15 smithi079 ceph-mon[23158]: pgmap v2622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:15 smithi002 ceph-mon[17167]: pgmap v2622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:15 smithi016 ceph-mon[22010]: pgmap v2622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:17 smithi079 ceph-mon[23158]: pgmap v2623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:17 smithi002 ceph-mon[17167]: pgmap v2623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:17 smithi016 ceph-mon[22010]: pgmap v2623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4419.949578 secs 2022-03-21T15:25:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4419.949578 secs 2022-03-21T15:25:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4419.949578 secs 2022-03-21T15:25:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:19 smithi079 ceph-mon[23158]: pgmap v2624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:19 smithi002 ceph-mon[17167]: pgmap v2624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:19 smithi016 ceph-mon[22010]: pgmap v2624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:21 smithi079 ceph-mon[23158]: pgmap v2625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:25:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:25:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:21 smithi002 ceph-mon[17167]: pgmap v2625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:25:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:25:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:21 smithi016 ceph-mon[22010]: pgmap v2625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:25:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:25:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:23 smithi002 ceph-mon[17167]: pgmap v2626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4424.949686 secs 2022-03-21T15:25:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:23 smithi016 ceph-mon[22010]: pgmap v2626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4424.949686 secs 2022-03-21T15:25:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:23 smithi079 ceph-mon[23158]: pgmap v2626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4424.949686 secs 2022-03-21T15:25:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:25 smithi002 ceph-mon[17167]: pgmap v2627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:25 smithi016 ceph-mon[22010]: pgmap v2627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:25 smithi079 ceph-mon[23158]: pgmap v2627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:27 smithi002 ceph-mon[17167]: pgmap v2628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:27 smithi016 ceph-mon[22010]: pgmap v2628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:27 smithi079 ceph-mon[23158]: pgmap v2628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4429.949765 secs 2022-03-21T15:25:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4429.949765 secs 2022-03-21T15:25:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4429.949765 secs 2022-03-21T15:25:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:29 smithi002 ceph-mon[17167]: pgmap v2629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:29 smithi016 ceph-mon[22010]: pgmap v2629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:29 smithi079 ceph-mon[23158]: pgmap v2629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:31 smithi002 ceph-mon[17167]: pgmap v2630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:31 smithi016 ceph-mon[22010]: pgmap v2630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:31 smithi079 ceph-mon[23158]: pgmap v2630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:33 smithi002 ceph-mon[17167]: pgmap v2631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4434.949882 secs 2022-03-21T15:25:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:33 smithi016 ceph-mon[22010]: pgmap v2631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4434.949882 secs 2022-03-21T15:25:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:33 smithi079 ceph-mon[23158]: pgmap v2631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4434.949882 secs 2022-03-21T15:25:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:35 smithi002 ceph-mon[17167]: pgmap v2632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:35 smithi016 ceph-mon[22010]: pgmap v2632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:35 smithi079 ceph-mon[23158]: pgmap v2632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:37 smithi002 ceph-mon[17167]: pgmap v2633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:37 smithi016 ceph-mon[22010]: pgmap v2633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:37 smithi079 ceph-mon[23158]: pgmap v2633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4439.949985 secs 2022-03-21T15:25:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4439.949985 secs 2022-03-21T15:25:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4439.949985 secs 2022-03-21T15:25:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:39 smithi002 ceph-mon[17167]: pgmap v2634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:39 smithi016 ceph-mon[22010]: pgmap v2634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:39 smithi079 ceph-mon[23158]: pgmap v2634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:41 smithi002 ceph-mon[17167]: pgmap v2635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:41 smithi016 ceph-mon[22010]: pgmap v2635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:41 smithi079 ceph-mon[23158]: pgmap v2635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:43 smithi002 ceph-mon[17167]: pgmap v2636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4444.950082 secs 2022-03-21T15:25:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:43 smithi016 ceph-mon[22010]: pgmap v2636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4444.950082 secs 2022-03-21T15:25:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:43 smithi079 ceph-mon[23158]: pgmap v2636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4444.950082 secs 2022-03-21T15:25:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:45 smithi002 ceph-mon[17167]: pgmap v2637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:45 smithi016 ceph-mon[22010]: pgmap v2637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:45 smithi079 ceph-mon[23158]: pgmap v2637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:47 smithi002 ceph-mon[17167]: pgmap v2638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:47 smithi016 ceph-mon[22010]: pgmap v2638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:47 smithi079 ceph-mon[23158]: pgmap v2638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4449.950204 secs 2022-03-21T15:25:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4449.950204 secs 2022-03-21T15:25:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4449.950204 secs 2022-03-21T15:25:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:49 smithi002 ceph-mon[17167]: pgmap v2639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:49 smithi016 ceph-mon[22010]: pgmap v2639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:49 smithi079 ceph-mon[23158]: pgmap v2639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:51 smithi002 ceph-mon[17167]: pgmap v2640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:51 smithi016 ceph-mon[22010]: pgmap v2640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:51 smithi079 ceph-mon[23158]: pgmap v2640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:52 smithi002 ceph-mon[17167]: pgmap v2641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:52 smithi016 ceph-mon[22010]: pgmap v2641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:52 smithi079 ceph-mon[23158]: pgmap v2641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4454.950324 secs 2022-03-21T15:25:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4454.950324 secs 2022-03-21T15:25:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4454.950324 secs 2022-03-21T15:25:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:54 smithi002 ceph-mon[17167]: pgmap v2642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:54 smithi016 ceph-mon[22010]: pgmap v2642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:54 smithi079 ceph-mon[23158]: pgmap v2642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:57 smithi079 ceph-mon[23158]: pgmap v2643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:57 smithi002 ceph-mon[17167]: pgmap v2643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:57 smithi016 ceph-mon[22010]: pgmap v2643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:25:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4459.950442 secs 2022-03-21T15:25:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4459.950442 secs 2022-03-21T15:25:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4459.950442 secs 2022-03-21T15:25:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:25:59 smithi079 ceph-mon[23158]: pgmap v2644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:25:59 smithi002 ceph-mon[17167]: pgmap v2644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:25:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:25:59 smithi016 ceph-mon[22010]: pgmap v2644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:01 smithi002 ceph-mon[17167]: pgmap v2645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:01 smithi016 ceph-mon[22010]: pgmap v2645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:01 smithi079 ceph-mon[23158]: pgmap v2645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:03 smithi002 ceph-mon[17167]: pgmap v2646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4464.950583 secs 2022-03-21T15:26:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:03 smithi016 ceph-mon[22010]: pgmap v2646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4464.950583 secs 2022-03-21T15:26:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:03 smithi079 ceph-mon[23158]: pgmap v2646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4464.950583 secs 2022-03-21T15:26:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:05 smithi002 ceph-mon[17167]: pgmap v2647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:05 smithi016 ceph-mon[22010]: pgmap v2647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:05 smithi079 ceph-mon[23158]: pgmap v2647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:26:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:26:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:26:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:26:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:26:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:26:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:26:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:26:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:26:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:07 smithi002 ceph-mon[17167]: pgmap v2648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:26:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:07 smithi016 ceph-mon[22010]: pgmap v2648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:26:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:07 smithi079 ceph-mon[23158]: pgmap v2648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:26:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4469.950712 secs 2022-03-21T15:26:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4469.950712 secs 2022-03-21T15:26:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4469.950712 secs 2022-03-21T15:26:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:09 smithi002 ceph-mon[17167]: pgmap v2649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:09 smithi016 ceph-mon[22010]: pgmap v2649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:09 smithi079 ceph-mon[23158]: pgmap v2649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:11 smithi002 ceph-mon[17167]: pgmap v2650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:11 smithi016 ceph-mon[22010]: pgmap v2650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:11 smithi079 ceph-mon[23158]: pgmap v2650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:13 smithi002 ceph-mon[17167]: pgmap v2651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4474.950834 secs 2022-03-21T15:26:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:13 smithi016 ceph-mon[22010]: pgmap v2651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4474.950834 secs 2022-03-21T15:26:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:13 smithi079 ceph-mon[23158]: pgmap v2651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4474.950834 secs 2022-03-21T15:26:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:15 smithi002 ceph-mon[17167]: pgmap v2652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:15 smithi016 ceph-mon[22010]: pgmap v2652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:17 smithi002 ceph-mon[17167]: pgmap v2653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:17 smithi016 ceph-mon[22010]: pgmap v2653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:15 smithi079 ceph-mon[23158]: pgmap v2652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:17 smithi079 ceph-mon[23158]: pgmap v2653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4479.950968 secs 2022-03-21T15:26:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4479.950968 secs 2022-03-21T15:26:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4479.950968 secs 2022-03-21T15:26:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:19 smithi002 ceph-mon[17167]: pgmap v2654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:19 smithi016 ceph-mon[22010]: pgmap v2654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:19 smithi079 ceph-mon[23158]: pgmap v2654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:21 smithi002 ceph-mon[17167]: pgmap v2655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:26:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:26:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:21 smithi016 ceph-mon[22010]: pgmap v2655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:26:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:26:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:21 smithi079 ceph-mon[23158]: pgmap v2655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:26:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:26:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:23 smithi002 ceph-mon[17167]: pgmap v2656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4484.951089 secs 2022-03-21T15:26:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:23 smithi016 ceph-mon[22010]: pgmap v2656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4484.951089 secs 2022-03-21T15:26:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:23 smithi079 ceph-mon[23158]: pgmap v2656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4484.951089 secs 2022-03-21T15:26:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:25 smithi002 ceph-mon[17167]: pgmap v2657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:25 smithi016 ceph-mon[22010]: pgmap v2657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:25 smithi079 ceph-mon[23158]: pgmap v2657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:27 smithi002 ceph-mon[17167]: pgmap v2658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:27 smithi016 ceph-mon[22010]: pgmap v2658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:27 smithi079 ceph-mon[23158]: pgmap v2658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4489.951186 secs 2022-03-21T15:26:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4489.951186 secs 2022-03-21T15:26:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4489.951186 secs 2022-03-21T15:26:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:29 smithi002 ceph-mon[17167]: pgmap v2659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:29 smithi016 ceph-mon[22010]: pgmap v2659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:29 smithi079 ceph-mon[23158]: pgmap v2659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:31 smithi002 ceph-mon[17167]: pgmap v2660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:31 smithi016 ceph-mon[22010]: pgmap v2660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:31 smithi079 ceph-mon[23158]: pgmap v2660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:33 smithi002 ceph-mon[17167]: pgmap v2661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4494.951315 secs 2022-03-21T15:26:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:33 smithi016 ceph-mon[22010]: pgmap v2661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4494.951315 secs 2022-03-21T15:26:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:33 smithi079 ceph-mon[23158]: pgmap v2661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4494.951315 secs 2022-03-21T15:26:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:35 smithi002 ceph-mon[17167]: pgmap v2662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:35 smithi016 ceph-mon[22010]: pgmap v2662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:35 smithi079 ceph-mon[23158]: pgmap v2662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:37 smithi002 ceph-mon[17167]: pgmap v2663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:37 smithi016 ceph-mon[22010]: pgmap v2663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:37 smithi079 ceph-mon[23158]: pgmap v2663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4499.951432 secs 2022-03-21T15:26:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4499.951432 secs 2022-03-21T15:26:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4499.951432 secs 2022-03-21T15:26:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:39 smithi002 ceph-mon[17167]: pgmap v2664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:39 smithi016 ceph-mon[22010]: pgmap v2664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:39 smithi079 ceph-mon[23158]: pgmap v2664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:41 smithi002 ceph-mon[17167]: pgmap v2665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:41 smithi016 ceph-mon[22010]: pgmap v2665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:41 smithi079 ceph-mon[23158]: pgmap v2665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:43 smithi002 ceph-mon[17167]: pgmap v2666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4504.951573 secs 2022-03-21T15:26:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:43 smithi016 ceph-mon[22010]: pgmap v2666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4504.951573 secs 2022-03-21T15:26:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:43 smithi079 ceph-mon[23158]: pgmap v2666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4504.951573 secs 2022-03-21T15:26:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:45 smithi002 ceph-mon[17167]: pgmap v2667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:45 smithi016 ceph-mon[22010]: pgmap v2667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:45 smithi079 ceph-mon[23158]: pgmap v2667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:47 smithi002 ceph-mon[17167]: pgmap v2668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:47 smithi016 ceph-mon[22010]: pgmap v2668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:47 smithi079 ceph-mon[23158]: pgmap v2668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4509.951622 secs 2022-03-21T15:26:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4509.951622 secs 2022-03-21T15:26:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4509.951622 secs 2022-03-21T15:26:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:49 smithi002 ceph-mon[17167]: pgmap v2669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:49 smithi016 ceph-mon[22010]: pgmap v2669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:49 smithi079 ceph-mon[23158]: pgmap v2669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:51 smithi002 ceph-mon[17167]: pgmap v2670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:51 smithi016 ceph-mon[22010]: pgmap v2670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:51 smithi079 ceph-mon[23158]: pgmap v2670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:53 smithi002 ceph-mon[17167]: pgmap v2671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4514.951720 secs 2022-03-21T15:26:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:53 smithi016 ceph-mon[22010]: pgmap v2671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4514.951720 secs 2022-03-21T15:26:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:53 smithi079 ceph-mon[23158]: pgmap v2671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4514.951720 secs 2022-03-21T15:26:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:55 smithi002 ceph-mon[17167]: pgmap v2672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:55 smithi016 ceph-mon[22010]: pgmap v2672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:55 smithi079 ceph-mon[23158]: pgmap v2672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:57 smithi002 ceph-mon[17167]: pgmap v2673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:57 smithi016 ceph-mon[22010]: pgmap v2673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:57 smithi079 ceph-mon[23158]: pgmap v2673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:26:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4519.951832 secs 2022-03-21T15:26:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4519.951832 secs 2022-03-21T15:26:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4519.951832 secs 2022-03-21T15:26:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:26:59 smithi002 ceph-mon[17167]: pgmap v2674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:26:59 smithi016 ceph-mon[22010]: pgmap v2674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:26:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:26:59 smithi079 ceph-mon[23158]: pgmap v2674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:01 smithi002 ceph-mon[17167]: pgmap v2675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:01.793 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:01 smithi079 ceph-mon[23158]: pgmap v2675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:01.794 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:01 smithi016 ceph-mon[22010]: pgmap v2675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:03 smithi002 ceph-mon[17167]: pgmap v2676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4524.951946 secs 2022-03-21T15:27:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:03 smithi016 ceph-mon[22010]: pgmap v2676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4524.951946 secs 2022-03-21T15:27:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:03 smithi079 ceph-mon[23158]: pgmap v2676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4524.951946 secs 2022-03-21T15:27:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:05 smithi002 ceph-mon[17167]: pgmap v2677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:05 smithi016 ceph-mon[22010]: pgmap v2677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:05 smithi079 ceph-mon[23158]: pgmap v2677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:07 smithi016 ceph-mon[22010]: pgmap v2678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:27:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:27:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:27:07.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:07 smithi079 ceph-mon[23158]: pgmap v2678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:27:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:27:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:27:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:07 smithi002 ceph-mon[17167]: pgmap v2678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:27:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:27:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:27:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4529.952056 secs 2022-03-21T15:27:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:27:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:08 smithi002 ceph-mon[17167]: pgmap v2679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4529.952056 secs 2022-03-21T15:27:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:27:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:08 smithi016 ceph-mon[22010]: pgmap v2679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:08.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4529.952056 secs 2022-03-21T15:27:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:27:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:08 smithi079 ceph-mon[23158]: pgmap v2679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:11 smithi002 ceph-mon[17167]: pgmap v2680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:11 smithi016 ceph-mon[22010]: pgmap v2680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:11 smithi079 ceph-mon[23158]: pgmap v2680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:13 smithi002 ceph-mon[17167]: pgmap v2681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4534.952180 secs 2022-03-21T15:27:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:13 smithi016 ceph-mon[22010]: pgmap v2681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4534.952180 secs 2022-03-21T15:27:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:13 smithi079 ceph-mon[23158]: pgmap v2681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4534.952180 secs 2022-03-21T15:27:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:15 smithi002 ceph-mon[17167]: pgmap v2682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:15 smithi016 ceph-mon[22010]: pgmap v2682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:15 smithi079 ceph-mon[23158]: pgmap v2682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:17 smithi002 ceph-mon[17167]: pgmap v2683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:17 smithi016 ceph-mon[22010]: pgmap v2683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:17 smithi079 ceph-mon[23158]: pgmap v2683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4539.952314 secs 2022-03-21T15:27:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4539.952314 secs 2022-03-21T15:27:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4539.952314 secs 2022-03-21T15:27:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:19 smithi002 ceph-mon[17167]: pgmap v2684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:19 smithi016 ceph-mon[22010]: pgmap v2684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:19 smithi079 ceph-mon[23158]: pgmap v2684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:21 smithi002 ceph-mon[17167]: pgmap v2685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:27:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:27:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:21 smithi016 ceph-mon[22010]: pgmap v2685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:27:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:27:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:21 smithi079 ceph-mon[23158]: pgmap v2685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:27:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:27:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:23 smithi002 ceph-mon[17167]: pgmap v2686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4544.952443 secs 2022-03-21T15:27:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:23 smithi016 ceph-mon[22010]: pgmap v2686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4544.952443 secs 2022-03-21T15:27:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:23 smithi079 ceph-mon[23158]: pgmap v2686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4544.952443 secs 2022-03-21T15:27:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:25 smithi002 ceph-mon[17167]: pgmap v2687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:25 smithi016 ceph-mon[22010]: pgmap v2687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:25 smithi079 ceph-mon[23158]: pgmap v2687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:27 smithi002 ceph-mon[17167]: pgmap v2688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:27 smithi016 ceph-mon[22010]: pgmap v2688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:27 smithi079 ceph-mon[23158]: pgmap v2688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4549.952599 secs 2022-03-21T15:27:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4549.952599 secs 2022-03-21T15:27:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4549.952599 secs 2022-03-21T15:27:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:29 smithi002 ceph-mon[17167]: pgmap v2689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:29 smithi016 ceph-mon[22010]: pgmap v2689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:29 smithi079 ceph-mon[23158]: pgmap v2689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:31 smithi002 ceph-mon[17167]: pgmap v2690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:31 smithi016 ceph-mon[22010]: pgmap v2690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:31 smithi079 ceph-mon[23158]: pgmap v2690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:33 smithi002 ceph-mon[17167]: pgmap v2691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4554.952702 secs 2022-03-21T15:27:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:33 smithi016 ceph-mon[22010]: pgmap v2691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4554.952702 secs 2022-03-21T15:27:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:33 smithi079 ceph-mon[23158]: pgmap v2691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4554.952702 secs 2022-03-21T15:27:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:35 smithi002 ceph-mon[17167]: pgmap v2692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:35 smithi016 ceph-mon[22010]: pgmap v2692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:35 smithi079 ceph-mon[23158]: pgmap v2692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:37 smithi002 ceph-mon[17167]: pgmap v2693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:37 smithi016 ceph-mon[22010]: pgmap v2693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:37 smithi079 ceph-mon[23158]: pgmap v2693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4559.952824 secs 2022-03-21T15:27:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4559.952824 secs 2022-03-21T15:27:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4559.952824 secs 2022-03-21T15:27:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:39 smithi002 ceph-mon[17167]: pgmap v2694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:39 smithi016 ceph-mon[22010]: pgmap v2694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:39 smithi079 ceph-mon[23158]: pgmap v2694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:41 smithi002 ceph-mon[17167]: pgmap v2695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:41 smithi016 ceph-mon[22010]: pgmap v2695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:41 smithi079 ceph-mon[23158]: pgmap v2695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:42 smithi002 ceph-mon[17167]: pgmap v2696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:42 smithi016 ceph-mon[22010]: pgmap v2696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:42 smithi079 ceph-mon[23158]: pgmap v2696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4564.952949 secs 2022-03-21T15:27:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4564.952949 secs 2022-03-21T15:27:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4564.952949 secs 2022-03-21T15:27:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:44 smithi002 ceph-mon[17167]: pgmap v2697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:44 smithi016 ceph-mon[22010]: pgmap v2697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:44 smithi079 ceph-mon[23158]: pgmap v2697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:47 smithi002 ceph-mon[17167]: pgmap v2698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:47 smithi016 ceph-mon[22010]: pgmap v2698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:47 smithi079 ceph-mon[23158]: pgmap v2698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4569.953040 secs 2022-03-21T15:27:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4569.953040 secs 2022-03-21T15:27:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4569.953040 secs 2022-03-21T15:27:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:49 smithi002 ceph-mon[17167]: pgmap v2699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:49 smithi016 ceph-mon[22010]: pgmap v2699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:49 smithi079 ceph-mon[23158]: pgmap v2699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:51 smithi002 ceph-mon[17167]: pgmap v2700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:51 smithi016 ceph-mon[22010]: pgmap v2700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:51 smithi079 ceph-mon[23158]: pgmap v2700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:53 smithi002 ceph-mon[17167]: pgmap v2701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4574.953135 secs 2022-03-21T15:27:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:53 smithi016 ceph-mon[22010]: pgmap v2701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4574.953135 secs 2022-03-21T15:27:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:53 smithi079 ceph-mon[23158]: pgmap v2701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4574.953135 secs 2022-03-21T15:27:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:55 smithi002 ceph-mon[17167]: pgmap v2702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:55 smithi016 ceph-mon[22010]: pgmap v2702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:55 smithi079 ceph-mon[23158]: pgmap v2702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:57 smithi002 ceph-mon[17167]: pgmap v2703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:57 smithi016 ceph-mon[22010]: pgmap v2703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:57 smithi079 ceph-mon[23158]: pgmap v2703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:27:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4579.953275 secs 2022-03-21T15:27:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4579.953275 secs 2022-03-21T15:27:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4579.953275 secs 2022-03-21T15:27:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:27:59 smithi002 ceph-mon[17167]: pgmap v2704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:27:59 smithi016 ceph-mon[22010]: pgmap v2704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:27:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:27:59 smithi079 ceph-mon[23158]: pgmap v2704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:01 smithi002 ceph-mon[17167]: pgmap v2705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:01 smithi016 ceph-mon[22010]: pgmap v2705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:01 smithi079 ceph-mon[23158]: pgmap v2705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:03 smithi002 ceph-mon[17167]: pgmap v2706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4584.953416 secs 2022-03-21T15:28:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:03 smithi016 ceph-mon[22010]: pgmap v2706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4584.953416 secs 2022-03-21T15:28:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:03 smithi079 ceph-mon[23158]: pgmap v2706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4584.953416 secs 2022-03-21T15:28:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:05 smithi002 ceph-mon[17167]: pgmap v2707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:05 smithi016 ceph-mon[22010]: pgmap v2707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:05 smithi079 ceph-mon[23158]: pgmap v2707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:07.550 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:07 smithi016 ceph-mon[22010]: pgmap v2708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:07.551 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:07 smithi079 ceph-mon[23158]: pgmap v2708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:07.584 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:07 smithi002 ceph-mon[17167]: pgmap v2708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4589.953575 secs 2022-03-21T15:28:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:28:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:28:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:28:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4589.953575 secs 2022-03-21T15:28:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:28:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4589.953575 secs 2022-03-21T15:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:28:09.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:28:09.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:28:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:28:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:28:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:28:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:28:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: pgmap v2709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:28:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:28:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: pgmap v2709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:28:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:28:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:28:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:28:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:28:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: pgmap v2709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:28:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:28:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:10 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:28:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:10 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:28:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:10 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:28:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:10 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:28:10.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:10 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:28:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:10 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:28:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:11 smithi002 ceph-mon[17167]: pgmap v2710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:11 smithi016 ceph-mon[22010]: pgmap v2710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:11 smithi079 ceph-mon[23158]: pgmap v2710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:12.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:12 smithi002 ceph-mon[17167]: pgmap v2711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:12 smithi016 ceph-mon[22010]: pgmap v2711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:12.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:12 smithi079 ceph-mon[23158]: pgmap v2711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4594.953661 secs 2022-03-21T15:28:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4594.953661 secs 2022-03-21T15:28:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4594.953661 secs 2022-03-21T15:28:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:14 smithi002 ceph-mon[17167]: pgmap v2712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:14 smithi016 ceph-mon[22010]: pgmap v2712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:14 smithi079 ceph-mon[23158]: pgmap v2712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:17 smithi002 ceph-mon[17167]: pgmap v2713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:17 smithi016 ceph-mon[22010]: pgmap v2713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:17 smithi079 ceph-mon[23158]: pgmap v2713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4599.953724 secs 2022-03-21T15:28:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4599.953724 secs 2022-03-21T15:28:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4599.953724 secs 2022-03-21T15:28:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:19 smithi002 ceph-mon[17167]: pgmap v2714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:19 smithi016 ceph-mon[22010]: pgmap v2714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:19 smithi079 ceph-mon[23158]: pgmap v2714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:21 smithi002 ceph-mon[17167]: pgmap v2715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:28:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:28:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:21 smithi016 ceph-mon[22010]: pgmap v2715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:28:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:28:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:21 smithi079 ceph-mon[23158]: pgmap v2715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:28:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:28:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:23 smithi002 ceph-mon[17167]: pgmap v2716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4604.953789 secs 2022-03-21T15:28:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:23 smithi016 ceph-mon[22010]: pgmap v2716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4604.953789 secs 2022-03-21T15:28:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:23 smithi079 ceph-mon[23158]: pgmap v2716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4604.953789 secs 2022-03-21T15:28:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:25 smithi002 ceph-mon[17167]: pgmap v2717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:25 smithi016 ceph-mon[22010]: pgmap v2717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:25 smithi079 ceph-mon[23158]: pgmap v2717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:27 smithi002 ceph-mon[17167]: pgmap v2718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:27 smithi016 ceph-mon[22010]: pgmap v2718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:27 smithi079 ceph-mon[23158]: pgmap v2718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4609.953855 secs 2022-03-21T15:28:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4609.953855 secs 2022-03-21T15:28:28.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4609.953855 secs 2022-03-21T15:28:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:29 smithi002 ceph-mon[17167]: pgmap v2719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:29 smithi016 ceph-mon[22010]: pgmap v2719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:29 smithi079 ceph-mon[23158]: pgmap v2719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:31 smithi002 ceph-mon[17167]: pgmap v2720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:31 smithi016 ceph-mon[22010]: pgmap v2720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:31 smithi079 ceph-mon[23158]: pgmap v2720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:33 smithi002 ceph-mon[17167]: pgmap v2721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4614.953965 secs 2022-03-21T15:28:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:33 smithi016 ceph-mon[22010]: pgmap v2721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4614.953965 secs 2022-03-21T15:28:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:33 smithi079 ceph-mon[23158]: pgmap v2721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4614.953965 secs 2022-03-21T15:28:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:35 smithi002 ceph-mon[17167]: pgmap v2722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:35 smithi016 ceph-mon[22010]: pgmap v2722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:35 smithi079 ceph-mon[23158]: pgmap v2722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:37 smithi002 ceph-mon[17167]: pgmap v2723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:37 smithi016 ceph-mon[22010]: pgmap v2723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:37 smithi079 ceph-mon[23158]: pgmap v2723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4619.954084 secs 2022-03-21T15:28:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4619.954084 secs 2022-03-21T15:28:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4619.954084 secs 2022-03-21T15:28:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:39 smithi002 ceph-mon[17167]: pgmap v2724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:39 smithi016 ceph-mon[22010]: pgmap v2724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:39 smithi079 ceph-mon[23158]: pgmap v2724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:41 smithi002 ceph-mon[17167]: pgmap v2725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:41 smithi016 ceph-mon[22010]: pgmap v2725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:41 smithi079 ceph-mon[23158]: pgmap v2725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:43 smithi002 ceph-mon[17167]: pgmap v2726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4624.954179 secs 2022-03-21T15:28:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:43 smithi016 ceph-mon[22010]: pgmap v2726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4624.954179 secs 2022-03-21T15:28:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:43 smithi079 ceph-mon[23158]: pgmap v2726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4624.954179 secs 2022-03-21T15:28:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:45 smithi002 ceph-mon[17167]: pgmap v2727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:45 smithi016 ceph-mon[22010]: pgmap v2727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:45 smithi079 ceph-mon[23158]: pgmap v2727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:47 smithi002 ceph-mon[17167]: pgmap v2728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:47 smithi016 ceph-mon[22010]: pgmap v2728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:47 smithi079 ceph-mon[23158]: pgmap v2728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4629.954294 secs 2022-03-21T15:28:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4629.954294 secs 2022-03-21T15:28:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4629.954294 secs 2022-03-21T15:28:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:49 smithi002 ceph-mon[17167]: pgmap v2729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:49 smithi016 ceph-mon[22010]: pgmap v2729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:49 smithi079 ceph-mon[23158]: pgmap v2729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:51 smithi002 ceph-mon[17167]: pgmap v2730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:51 smithi016 ceph-mon[22010]: pgmap v2730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:51 smithi079 ceph-mon[23158]: pgmap v2730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:53 smithi002 ceph-mon[17167]: pgmap v2731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4634.954419 secs 2022-03-21T15:28:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:53 smithi016 ceph-mon[22010]: pgmap v2731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4634.954419 secs 2022-03-21T15:28:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:53 smithi079 ceph-mon[23158]: pgmap v2731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4634.954419 secs 2022-03-21T15:28:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:55 smithi002 ceph-mon[17167]: pgmap v2732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:28:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:55 smithi016 ceph-mon[22010]: pgmap v2732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:28:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:55 smithi079 ceph-mon[23158]: pgmap v2732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:28:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:57 smithi002 ceph-mon[17167]: pgmap v2733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:57 smithi016 ceph-mon[22010]: pgmap v2733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:57 smithi079 ceph-mon[23158]: pgmap v2733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:28:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4639.954476 secs 2022-03-21T15:28:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4639.954476 secs 2022-03-21T15:28:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4639.954476 secs 2022-03-21T15:28:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:28:59 smithi002 ceph-mon[17167]: pgmap v2734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:28:59 smithi016 ceph-mon[22010]: pgmap v2734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:28:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:28:59 smithi079 ceph-mon[23158]: pgmap v2734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:01 smithi002 ceph-mon[17167]: pgmap v2735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:01 smithi016 ceph-mon[22010]: pgmap v2735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:01 smithi079 ceph-mon[23158]: pgmap v2735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:03 smithi002 ceph-mon[17167]: pgmap v2736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4644.954598 secs 2022-03-21T15:29:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:03 smithi016 ceph-mon[22010]: pgmap v2736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4644.954598 secs 2022-03-21T15:29:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:03 smithi079 ceph-mon[23158]: pgmap v2736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4644.954598 secs 2022-03-21T15:29:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:04 smithi016 ceph-mon[22010]: pgmap v2737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:04 smithi079 ceph-mon[23158]: pgmap v2737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:04 smithi002 ceph-mon[17167]: pgmap v2737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:07 smithi002 ceph-mon[17167]: pgmap v2738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:07 smithi016 ceph-mon[22010]: pgmap v2738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:07 smithi079 ceph-mon[23158]: pgmap v2738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4649.954713 secs 2022-03-21T15:29:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4649.954713 secs 2022-03-21T15:29:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4649.954713 secs 2022-03-21T15:29:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:09 smithi002 ceph-mon[17167]: pgmap v2739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:29:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:29:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:29:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:29:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:09 smithi016 ceph-mon[22010]: pgmap v2739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:29:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:29:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:29:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:29:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:09 smithi079 ceph-mon[23158]: pgmap v2739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:29:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:29:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:29:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:29:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:11 smithi002 ceph-mon[17167]: pgmap v2740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:11 smithi016 ceph-mon[22010]: pgmap v2740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:11 smithi079 ceph-mon[23158]: pgmap v2740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:13 smithi002 ceph-mon[17167]: pgmap v2741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4654.954837 secs 2022-03-21T15:29:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:13 smithi016 ceph-mon[22010]: pgmap v2741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4654.954837 secs 2022-03-21T15:29:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:13 smithi079 ceph-mon[23158]: pgmap v2741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4654.954837 secs 2022-03-21T15:29:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:15 smithi002 ceph-mon[17167]: pgmap v2742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:15 smithi016 ceph-mon[22010]: pgmap v2742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:15 smithi079 ceph-mon[23158]: pgmap v2742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:17 smithi002 ceph-mon[17167]: pgmap v2743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:17 smithi016 ceph-mon[22010]: pgmap v2743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:17 smithi079 ceph-mon[23158]: pgmap v2743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4659.954953 secs 2022-03-21T15:29:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4659.954953 secs 2022-03-21T15:29:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4659.954953 secs 2022-03-21T15:29:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:19 smithi002 ceph-mon[17167]: pgmap v2744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:19 smithi016 ceph-mon[22010]: pgmap v2744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:19 smithi079 ceph-mon[23158]: pgmap v2744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:21 smithi002 ceph-mon[17167]: pgmap v2745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:29:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:29:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:21 smithi016 ceph-mon[22010]: pgmap v2745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:29:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:29:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:21 smithi079 ceph-mon[23158]: pgmap v2745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:29:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:29:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:23 smithi002 ceph-mon[17167]: pgmap v2746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4664.955078 secs 2022-03-21T15:29:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:23 smithi016 ceph-mon[22010]: pgmap v2746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4664.955078 secs 2022-03-21T15:29:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:23 smithi079 ceph-mon[23158]: pgmap v2746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4664.955078 secs 2022-03-21T15:29:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:25 smithi002 ceph-mon[17167]: pgmap v2747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:25 smithi016 ceph-mon[22010]: pgmap v2747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:25 smithi079 ceph-mon[23158]: pgmap v2747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:26.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:26 smithi002 ceph-mon[17167]: pgmap v2748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:26 smithi016 ceph-mon[22010]: pgmap v2748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:26 smithi079 ceph-mon[23158]: pgmap v2748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4669.955235 secs 2022-03-21T15:29:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4669.955235 secs 2022-03-21T15:29:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4669.955235 secs 2022-03-21T15:29:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:28 smithi079 ceph-mon[23158]: pgmap v2749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:28 smithi002 ceph-mon[17167]: pgmap v2749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:28 smithi016 ceph-mon[22010]: pgmap v2749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:31 smithi002 ceph-mon[17167]: pgmap v2750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:31 smithi016 ceph-mon[22010]: pgmap v2750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:31 smithi079 ceph-mon[23158]: pgmap v2750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:33 smithi002 ceph-mon[17167]: pgmap v2751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4674.955331 secs 2022-03-21T15:29:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:33 smithi016 ceph-mon[22010]: pgmap v2751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4674.955331 secs 2022-03-21T15:29:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:33 smithi079 ceph-mon[23158]: pgmap v2751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4674.955331 secs 2022-03-21T15:29:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:35 smithi002 ceph-mon[17167]: pgmap v2752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:35 smithi016 ceph-mon[22010]: pgmap v2752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:35 smithi079 ceph-mon[23158]: pgmap v2752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:37 smithi002 ceph-mon[17167]: pgmap v2753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:37 smithi016 ceph-mon[22010]: pgmap v2753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:37 smithi079 ceph-mon[23158]: pgmap v2753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4679.955410 secs 2022-03-21T15:29:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4679.955410 secs 2022-03-21T15:29:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4679.955410 secs 2022-03-21T15:29:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:39 smithi002 ceph-mon[17167]: pgmap v2754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:39 smithi016 ceph-mon[22010]: pgmap v2754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:39 smithi079 ceph-mon[23158]: pgmap v2754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:41 smithi002 ceph-mon[17167]: pgmap v2755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:41 smithi016 ceph-mon[22010]: pgmap v2755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:41 smithi079 ceph-mon[23158]: pgmap v2755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:43 smithi002 ceph-mon[17167]: pgmap v2756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4684.955521 secs 2022-03-21T15:29:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:43 smithi016 ceph-mon[22010]: pgmap v2756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4684.955521 secs 2022-03-21T15:29:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:43 smithi079 ceph-mon[23158]: pgmap v2756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4684.955521 secs 2022-03-21T15:29:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:45 smithi002 ceph-mon[17167]: pgmap v2757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:45 smithi016 ceph-mon[22010]: pgmap v2757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:45 smithi079 ceph-mon[23158]: pgmap v2757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:47 smithi002 ceph-mon[17167]: pgmap v2758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:47 smithi016 ceph-mon[22010]: pgmap v2758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:47 smithi079 ceph-mon[23158]: pgmap v2758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4689.955617 secs 2022-03-21T15:29:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4689.955617 secs 2022-03-21T15:29:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4689.955617 secs 2022-03-21T15:29:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:49 smithi002 ceph-mon[17167]: pgmap v2759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:49 smithi016 ceph-mon[22010]: pgmap v2759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:49 smithi079 ceph-mon[23158]: pgmap v2759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:51 smithi002 ceph-mon[17167]: pgmap v2760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:51 smithi016 ceph-mon[22010]: pgmap v2760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:51 smithi079 ceph-mon[23158]: pgmap v2760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:53 smithi002 ceph-mon[17167]: pgmap v2761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4694.955741 secs 2022-03-21T15:29:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:53 smithi016 ceph-mon[22010]: pgmap v2761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4694.955741 secs 2022-03-21T15:29:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:53 smithi079 ceph-mon[23158]: pgmap v2761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4694.955741 secs 2022-03-21T15:29:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:55 smithi002 ceph-mon[17167]: pgmap v2762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:55 smithi016 ceph-mon[22010]: pgmap v2762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:55 smithi079 ceph-mon[23158]: pgmap v2762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:57.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:57 smithi002 ceph-mon[17167]: pgmap v2763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:57 smithi016 ceph-mon[22010]: pgmap v2763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:57 smithi079 ceph-mon[23158]: pgmap v2763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:29:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4699.955860 secs 2022-03-21T15:29:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4699.955860 secs 2022-03-21T15:29:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4699.955860 secs 2022-03-21T15:29:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:29:59 smithi002 ceph-mon[17167]: pgmap v2764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:29:59 smithi016 ceph-mon[22010]: pgmap v2764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:29:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:29:59 smithi079 ceph-mon[23158]: pgmap v2764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:30:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:30:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:30:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:01 smithi002 ceph-mon[17167]: pgmap v2765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:01 smithi016 ceph-mon[22010]: pgmap v2765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:01 smithi079 ceph-mon[23158]: pgmap v2765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:03 smithi002 ceph-mon[17167]: pgmap v2766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4704.955954 secs 2022-03-21T15:30:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:03 smithi016 ceph-mon[22010]: pgmap v2766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4704.955954 secs 2022-03-21T15:30:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:03 smithi079 ceph-mon[23158]: pgmap v2766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4704.955954 secs 2022-03-21T15:30:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:04 smithi016 ceph-mon[22010]: pgmap v2767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:04 smithi079 ceph-mon[23158]: pgmap v2767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:04 smithi002 ceph-mon[17167]: pgmap v2767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:07 smithi002 ceph-mon[17167]: pgmap v2768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:07 smithi016 ceph-mon[22010]: pgmap v2768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:07 smithi079 ceph-mon[23158]: pgmap v2768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4709.956077 secs 2022-03-21T15:30:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4709.956077 secs 2022-03-21T15:30:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4709.956077 secs 2022-03-21T15:30:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:09 smithi002 ceph-mon[17167]: pgmap v2769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:30:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:30:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:30:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:09 smithi016 ceph-mon[22010]: pgmap v2769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:30:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:30:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:30:09.669 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:09 smithi079 ceph-mon[23158]: pgmap v2769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:09.669 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:30:09.670 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:30:09.670 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:30:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:11 smithi002 ceph-mon[17167]: pgmap v2770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:30:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:11 smithi016 ceph-mon[22010]: pgmap v2770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:30:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:11 smithi079 ceph-mon[23158]: pgmap v2770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:30:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:13 smithi002 ceph-mon[17167]: pgmap v2771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4714.956197 secs 2022-03-21T15:30:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:13 smithi016 ceph-mon[22010]: pgmap v2771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4714.956197 secs 2022-03-21T15:30:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:13 smithi079 ceph-mon[23158]: pgmap v2771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4714.956197 secs 2022-03-21T15:30:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:15 smithi002 ceph-mon[17167]: pgmap v2772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:15 smithi016 ceph-mon[22010]: pgmap v2772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:15 smithi079 ceph-mon[23158]: pgmap v2772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:17.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:17 smithi002 ceph-mon[17167]: pgmap v2773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:17 smithi016 ceph-mon[22010]: pgmap v2773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:17 smithi079 ceph-mon[23158]: pgmap v2773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:30:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4719.956311 secs 2022-03-21T15:30:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4719.956311 secs 2022-03-21T15:30:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4719.956311 secs 2022-03-21T15:30:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:19 smithi002 ceph-mon[17167]: pgmap v2774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:19 smithi016 ceph-mon[22010]: pgmap v2774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:19 smithi079 ceph-mon[23158]: pgmap v2774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:21 smithi002 ceph-mon[17167]: pgmap v2775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:30:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:30:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:21 smithi016 ceph-mon[22010]: pgmap v2775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:30:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:30:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:21 smithi079 ceph-mon[23158]: pgmap v2775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:30:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:30:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:23 smithi002 ceph-mon[17167]: pgmap v2776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4724.956422 secs 2022-03-21T15:30:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:23 smithi016 ceph-mon[22010]: pgmap v2776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4724.956422 secs 2022-03-21T15:30:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:23 smithi079 ceph-mon[23158]: pgmap v2776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4724.956422 secs 2022-03-21T15:30:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:25 smithi002 ceph-mon[17167]: pgmap v2777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:25 smithi016 ceph-mon[22010]: pgmap v2777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:25 smithi079 ceph-mon[23158]: pgmap v2777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:27 smithi002 ceph-mon[17167]: pgmap v2778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:27 smithi016 ceph-mon[22010]: pgmap v2778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:27 smithi079 ceph-mon[23158]: pgmap v2778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4729.956539 secs 2022-03-21T15:30:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4729.956539 secs 2022-03-21T15:30:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4729.956539 secs 2022-03-21T15:30:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:29 smithi002 ceph-mon[17167]: pgmap v2779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:29 smithi016 ceph-mon[22010]: pgmap v2779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:29 smithi079 ceph-mon[23158]: pgmap v2779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:31 smithi002 ceph-mon[17167]: pgmap v2780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:31 smithi016 ceph-mon[22010]: pgmap v2780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:31 smithi079 ceph-mon[23158]: pgmap v2780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:33 smithi002 ceph-mon[17167]: pgmap v2781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4734.956670 secs 2022-03-21T15:30:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:33 smithi016 ceph-mon[22010]: pgmap v2781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4734.956670 secs 2022-03-21T15:30:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:33 smithi079 ceph-mon[23158]: pgmap v2781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 4 op/s 2022-03-21T15:30:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4734.956670 secs 2022-03-21T15:30:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:35 smithi002 ceph-mon[17167]: pgmap v2782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:35 smithi016 ceph-mon[22010]: pgmap v2782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:35 smithi079 ceph-mon[23158]: pgmap v2782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 3 op/s 2022-03-21T15:30:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:37 smithi002 ceph-mon[17167]: pgmap v2783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:37 smithi016 ceph-mon[22010]: pgmap v2783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:37 smithi079 ceph-mon[23158]: pgmap v2783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4739.956747 secs 2022-03-21T15:30:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4739.956747 secs 2022-03-21T15:30:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4739.956747 secs 2022-03-21T15:30:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:39 smithi002 ceph-mon[17167]: pgmap v2784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:39 smithi016 ceph-mon[22010]: pgmap v2784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:39 smithi079 ceph-mon[23158]: pgmap v2784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:41 smithi002 ceph-mon[17167]: pgmap v2785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:41 smithi016 ceph-mon[22010]: pgmap v2785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:41 smithi079 ceph-mon[23158]: pgmap v2785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:43 smithi002 ceph-mon[17167]: pgmap v2786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4744.956854 secs 2022-03-21T15:30:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:43 smithi016 ceph-mon[22010]: pgmap v2786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4744.956854 secs 2022-03-21T15:30:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:43 smithi079 ceph-mon[23158]: pgmap v2786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4744.956854 secs 2022-03-21T15:30:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:44 smithi016 ceph-mon[22010]: pgmap v2787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:44 smithi079 ceph-mon[23158]: pgmap v2787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:44.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:44 smithi002 ceph-mon[17167]: pgmap v2787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:47 smithi002 ceph-mon[17167]: pgmap v2788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:47 smithi016 ceph-mon[22010]: pgmap v2788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:47 smithi079 ceph-mon[23158]: pgmap v2788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4749.956976 secs 2022-03-21T15:30:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4749.956976 secs 2022-03-21T15:30:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4749.956976 secs 2022-03-21T15:30:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:49 smithi002 ceph-mon[17167]: pgmap v2789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:49 smithi016 ceph-mon[22010]: pgmap v2789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:49 smithi079 ceph-mon[23158]: pgmap v2789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:51 smithi002 ceph-mon[17167]: pgmap v2790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:51 smithi016 ceph-mon[22010]: pgmap v2790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:51 smithi079 ceph-mon[23158]: pgmap v2790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:53 smithi002 ceph-mon[17167]: pgmap v2791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4754.957084 secs 2022-03-21T15:30:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:53 smithi016 ceph-mon[22010]: pgmap v2791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4754.957084 secs 2022-03-21T15:30:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:53 smithi079 ceph-mon[23158]: pgmap v2791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4754.957084 secs 2022-03-21T15:30:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:55 smithi002 ceph-mon[17167]: pgmap v2792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:55 smithi016 ceph-mon[22010]: pgmap v2792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:55 smithi079 ceph-mon[23158]: pgmap v2792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:56 smithi016 ceph-mon[22010]: pgmap v2793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:56 smithi079 ceph-mon[23158]: pgmap v2793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:56 smithi002 ceph-mon[17167]: pgmap v2793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:30:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4759.957218 secs 2022-03-21T15:30:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4759.957218 secs 2022-03-21T15:30:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4759.957218 secs 2022-03-21T15:30:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:30:58 smithi002 ceph-mon[17167]: pgmap v2794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:30:58 smithi016 ceph-mon[22010]: pgmap v2794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:30:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:30:58 smithi079 ceph-mon[23158]: pgmap v2794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:01 smithi002 ceph-mon[17167]: pgmap v2795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:01 smithi016 ceph-mon[22010]: pgmap v2795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:01 smithi079 ceph-mon[23158]: pgmap v2795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:03 smithi002 ceph-mon[17167]: pgmap v2796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4764.957362 secs 2022-03-21T15:31:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:03 smithi016 ceph-mon[22010]: pgmap v2796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4764.957362 secs 2022-03-21T15:31:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:03 smithi079 ceph-mon[23158]: pgmap v2796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4764.957362 secs 2022-03-21T15:31:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:05 smithi002 ceph-mon[17167]: pgmap v2797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:05 smithi016 ceph-mon[22010]: pgmap v2797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:05 smithi079 ceph-mon[23158]: pgmap v2797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:07 smithi002 ceph-mon[17167]: pgmap v2798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:07 smithi016 ceph-mon[22010]: pgmap v2798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:07 smithi079 ceph-mon[23158]: pgmap v2798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4769.957472 secs 2022-03-21T15:31:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4769.957472 secs 2022-03-21T15:31:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4769.957472 secs 2022-03-21T15:31:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:09 smithi002 ceph-mon[17167]: pgmap v2799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:09 smithi016 ceph-mon[22010]: pgmap v2799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:09 smithi079 ceph-mon[23158]: pgmap v2799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:10 smithi016 ceph-mon[22010]: pgmap v2800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:31:10.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:10 smithi079 ceph-mon[23158]: pgmap v2800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:31:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:31:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:31:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:10 smithi002 ceph-mon[17167]: pgmap v2800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:31:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:31:10.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:31:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:13 smithi002 ceph-mon[17167]: pgmap v2801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4774.957565 secs 2022-03-21T15:31:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:13 smithi016 ceph-mon[22010]: pgmap v2801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4774.957565 secs 2022-03-21T15:31:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:13 smithi079 ceph-mon[23158]: pgmap v2801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4774.957565 secs 2022-03-21T15:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:14 smithi002 ceph-mon[17167]: pgmap v2802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:14 smithi016 ceph-mon[22010]: pgmap v2802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:14 smithi079 ceph-mon[23158]: pgmap v2802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:31:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:17 smithi002 ceph-mon[17167]: pgmap v2803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:17 smithi016 ceph-mon[22010]: pgmap v2803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:17 smithi079 ceph-mon[23158]: pgmap v2803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4779.957684 secs 2022-03-21T15:31:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4779.957684 secs 2022-03-21T15:31:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4779.957684 secs 2022-03-21T15:31:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:19 smithi002 ceph-mon[17167]: pgmap v2804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:19 smithi016 ceph-mon[22010]: pgmap v2804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:19 smithi079 ceph-mon[23158]: pgmap v2804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:21 smithi002 ceph-mon[17167]: pgmap v2805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:31:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:31:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:21 smithi016 ceph-mon[22010]: pgmap v2805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:31:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:31:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:21 smithi079 ceph-mon[23158]: pgmap v2805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:31:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:31:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:23 smithi002 ceph-mon[17167]: pgmap v2806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4784.957825 secs 2022-03-21T15:31:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:23 smithi016 ceph-mon[22010]: pgmap v2806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4784.957825 secs 2022-03-21T15:31:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:23 smithi079 ceph-mon[23158]: pgmap v2806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4784.957825 secs 2022-03-21T15:31:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:24 smithi002 ceph-mon[17167]: pgmap v2807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:24 smithi016 ceph-mon[22010]: pgmap v2807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:24 smithi079 ceph-mon[23158]: pgmap v2807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:27.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:27 smithi002 ceph-mon[17167]: pgmap v2808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:27 smithi016 ceph-mon[22010]: pgmap v2808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:27 smithi079 ceph-mon[23158]: pgmap v2808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4789.957935 secs 2022-03-21T15:31:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4789.957935 secs 2022-03-21T15:31:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4789.957935 secs 2022-03-21T15:31:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:29 smithi002 ceph-mon[17167]: pgmap v2809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:29 smithi016 ceph-mon[22010]: pgmap v2809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:29 smithi079 ceph-mon[23158]: pgmap v2809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:31 smithi002 ceph-mon[17167]: pgmap v2810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:31 smithi016 ceph-mon[22010]: pgmap v2810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:31 smithi079 ceph-mon[23158]: pgmap v2810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:32 smithi002 ceph-mon[17167]: pgmap v2811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:32 smithi016 ceph-mon[22010]: pgmap v2811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:32 smithi079 ceph-mon[23158]: pgmap v2811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4794.958056 secs 2022-03-21T15:31:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4794.958056 secs 2022-03-21T15:31:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4794.958056 secs 2022-03-21T15:31:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:34 smithi002 ceph-mon[17167]: pgmap v2812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:34 smithi016 ceph-mon[22010]: pgmap v2812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:34 smithi079 ceph-mon[23158]: pgmap v2812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:37 smithi002 ceph-mon[17167]: pgmap v2813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:37 smithi016 ceph-mon[22010]: pgmap v2813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:37 smithi079 ceph-mon[23158]: pgmap v2813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4799.958153 secs 2022-03-21T15:31:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4799.958153 secs 2022-03-21T15:31:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4799.958153 secs 2022-03-21T15:31:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:39 smithi002 ceph-mon[17167]: pgmap v2814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:39 smithi016 ceph-mon[22010]: pgmap v2814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:39 smithi079 ceph-mon[23158]: pgmap v2814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:41 smithi002 ceph-mon[17167]: pgmap v2815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:41 smithi016 ceph-mon[22010]: pgmap v2815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:41 smithi079 ceph-mon[23158]: pgmap v2815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:43 smithi002 ceph-mon[17167]: pgmap v2816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4804.958297 secs 2022-03-21T15:31:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:43 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 4804.843474 seconds ago 2022-03-21T15:31:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:43 smithi016 ceph-mon[22010]: pgmap v2816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4804.958297 secs 2022-03-21T15:31:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:43 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 4804.843474 seconds ago 2022-03-21T15:31:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:43 smithi079 ceph-mon[23158]: pgmap v2816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4804.958297 secs 2022-03-21T15:31:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:43 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 4804.843474 seconds ago 2022-03-21T15:31:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:45 smithi002 ceph-mon[17167]: pgmap v2817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:45 smithi016 ceph-mon[22010]: pgmap v2817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:45 smithi079 ceph-mon[23158]: pgmap v2817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:47 smithi002 ceph-mon[17167]: pgmap v2818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:47 smithi016 ceph-mon[22010]: pgmap v2818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:47 smithi079 ceph-mon[23158]: pgmap v2818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4809.958412 secs 2022-03-21T15:31:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:48 smithi002 ceph-mon[17167]: pgmap v2819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4809.958412 secs 2022-03-21T15:31:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:48 smithi016 ceph-mon[22010]: pgmap v2819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4809.958412 secs 2022-03-21T15:31:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:48 smithi079 ceph-mon[23158]: pgmap v2819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:51 smithi002 ceph-mon[17167]: pgmap v2820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:51 smithi016 ceph-mon[22010]: pgmap v2820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:51 smithi079 ceph-mon[23158]: pgmap v2820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:53 smithi002 ceph-mon[17167]: pgmap v2821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4814.958528 secs 2022-03-21T15:31:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:53 smithi016 ceph-mon[22010]: pgmap v2821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4814.958528 secs 2022-03-21T15:31:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:53 smithi079 ceph-mon[23158]: pgmap v2821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4814.958528 secs 2022-03-21T15:31:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:55 smithi002 ceph-mon[17167]: pgmap v2822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:55 smithi016 ceph-mon[22010]: pgmap v2822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:55 smithi079 ceph-mon[23158]: pgmap v2822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:56 smithi002 ceph-mon[17167]: pgmap v2823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:56 smithi016 ceph-mon[22010]: pgmap v2823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:56 smithi079 ceph-mon[23158]: pgmap v2823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:31:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4819.958623 secs 2022-03-21T15:31:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4819.958623 secs 2022-03-21T15:31:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4819.958623 secs 2022-03-21T15:31:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:31:58 smithi002 ceph-mon[17167]: pgmap v2824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:31:58 smithi016 ceph-mon[22010]: pgmap v2824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:31:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:31:58 smithi079 ceph-mon[23158]: pgmap v2824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:01 smithi002 ceph-mon[17167]: pgmap v2825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:01 smithi016 ceph-mon[22010]: pgmap v2825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:01 smithi079 ceph-mon[23158]: pgmap v2825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:03 smithi002 ceph-mon[17167]: pgmap v2826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4824.958761 secs 2022-03-21T15:32:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:03 smithi016 ceph-mon[22010]: pgmap v2826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:03.749 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4824.958761 secs 2022-03-21T15:32:03.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:03 smithi079 ceph-mon[23158]: pgmap v2826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:03.752 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4824.958761 secs 2022-03-21T15:32:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:04 smithi016 ceph-mon[22010]: pgmap v2827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:04 smithi079 ceph-mon[23158]: pgmap v2827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:04.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:04 smithi002 ceph-mon[17167]: pgmap v2827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:07 smithi002 ceph-mon[17167]: pgmap v2828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:07 smithi016 ceph-mon[22010]: pgmap v2828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:07 smithi079 ceph-mon[23158]: pgmap v2828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4829.958825 secs 2022-03-21T15:32:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4829.958825 secs 2022-03-21T15:32:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4829.958825 secs 2022-03-21T15:32:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:09 smithi002 ceph-mon[17167]: pgmap v2829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:09 smithi016 ceph-mon[22010]: pgmap v2829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:09 smithi079 ceph-mon[23158]: pgmap v2829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:11 smithi002 ceph-mon[17167]: pgmap v2830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:11 smithi016 ceph-mon[22010]: pgmap v2830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:11 smithi079 ceph-mon[23158]: pgmap v2830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:13 smithi002 ceph-mon[17167]: pgmap v2831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4834.958949 secs 2022-03-21T15:32:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:13 smithi016 ceph-mon[22010]: pgmap v2831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4834.958949 secs 2022-03-21T15:32:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:13 smithi079 ceph-mon[23158]: pgmap v2831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4834.958949 secs 2022-03-21T15:32:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:14 smithi002 ceph-mon[17167]: pgmap v2832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:14 smithi016 ceph-mon[22010]: pgmap v2832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:14 smithi079 ceph-mon[23158]: pgmap v2832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:32:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:32:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:32:15.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:32:15.569 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:32:15.569 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:32:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:32:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:32:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:32:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:16 smithi002 ceph-mon[17167]: pgmap v2833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:16 smithi016 ceph-mon[22010]: pgmap v2833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:16 smithi079 ceph-mon[23158]: pgmap v2833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:32:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4839.959058 secs 2022-03-21T15:32:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4839.959058 secs 2022-03-21T15:32:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4839.959058 secs 2022-03-21T15:32:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:18 smithi002 ceph-mon[17167]: pgmap v2834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:18 smithi016 ceph-mon[22010]: pgmap v2834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:18 smithi079 ceph-mon[23158]: pgmap v2834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:21 smithi002 ceph-mon[17167]: pgmap v2835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:32:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:32:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:21 smithi016 ceph-mon[22010]: pgmap v2835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:32:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:32:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:21 smithi079 ceph-mon[23158]: pgmap v2835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:32:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:32:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:22 smithi016 ceph-mon[22010]: pgmap v2836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:22.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:22 smithi079 ceph-mon[23158]: pgmap v2836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:22.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:22 smithi002 ceph-mon[17167]: pgmap v2836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4844.959174 secs 2022-03-21T15:32:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4844.959174 secs 2022-03-21T15:32:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4844.959174 secs 2022-03-21T15:32:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:24 smithi016 ceph-mon[22010]: pgmap v2837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:24 smithi079 ceph-mon[23158]: pgmap v2837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:24 smithi002 ceph-mon[17167]: pgmap v2837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:26 smithi016 ceph-mon[22010]: pgmap v2838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:26 smithi079 ceph-mon[23158]: pgmap v2838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:26 smithi002 ceph-mon[17167]: pgmap v2838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4849.959288 secs 2022-03-21T15:32:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4849.959288 secs 2022-03-21T15:32:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4849.959288 secs 2022-03-21T15:32:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:28 smithi002 ceph-mon[17167]: pgmap v2839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:28 smithi016 ceph-mon[22010]: pgmap v2839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:28 smithi079 ceph-mon[23158]: pgmap v2839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:31 smithi002 ceph-mon[17167]: pgmap v2840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:31 smithi016 ceph-mon[22010]: pgmap v2840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:31 smithi079 ceph-mon[23158]: pgmap v2840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:32.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:32 smithi002 ceph-mon[17167]: pgmap v2841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:32 smithi016 ceph-mon[22010]: pgmap v2841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:32 smithi079 ceph-mon[23158]: pgmap v2841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4854.959422 secs 2022-03-21T15:32:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4854.959422 secs 2022-03-21T15:32:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4854.959422 secs 2022-03-21T15:32:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:34 smithi016 ceph-mon[22010]: pgmap v2842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:34 smithi079 ceph-mon[23158]: pgmap v2842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:34.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:34 smithi002 ceph-mon[17167]: pgmap v2842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:36 smithi002 ceph-mon[17167]: pgmap v2843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:36 smithi016 ceph-mon[22010]: pgmap v2843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:36 smithi079 ceph-mon[23158]: pgmap v2843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4859.959554 secs 2022-03-21T15:32:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4859.959554 secs 2022-03-21T15:32:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4859.959554 secs 2022-03-21T15:32:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:38 smithi002 ceph-mon[17167]: pgmap v2844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:38 smithi016 ceph-mon[22010]: pgmap v2844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:38 smithi079 ceph-mon[23158]: pgmap v2844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:41 smithi016 ceph-mon[22010]: pgmap v2845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:41 smithi079 ceph-mon[23158]: pgmap v2845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:41 smithi002 ceph-mon[17167]: pgmap v2845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:43 smithi016 ceph-mon[22010]: pgmap v2846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4864.959611 secs 2022-03-21T15:32:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:43 smithi079 ceph-mon[23158]: pgmap v2846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4864.959611 secs 2022-03-21T15:32:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:43 smithi002 ceph-mon[17167]: pgmap v2846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4864.959611 secs 2022-03-21T15:32:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:45 smithi016 ceph-mon[22010]: pgmap v2847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:45 smithi079 ceph-mon[23158]: pgmap v2847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:45 smithi002 ceph-mon[17167]: pgmap v2847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:46 smithi016 ceph-mon[22010]: pgmap v2848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:46 smithi079 ceph-mon[23158]: pgmap v2848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:46 smithi002 ceph-mon[17167]: pgmap v2848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4869.959702 secs 2022-03-21T15:32:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4869.959702 secs 2022-03-21T15:32:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4869.959702 secs 2022-03-21T15:32:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:48 smithi002 ceph-mon[17167]: pgmap v2849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:48 smithi016 ceph-mon[22010]: pgmap v2849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:48 smithi079 ceph-mon[23158]: pgmap v2849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:51 smithi016 ceph-mon[22010]: pgmap v2850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:51 smithi079 ceph-mon[23158]: pgmap v2850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:51 smithi002 ceph-mon[17167]: pgmap v2850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:53 smithi016 ceph-mon[22010]: pgmap v2851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4874.959853 secs 2022-03-21T15:32:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:53 smithi079 ceph-mon[23158]: pgmap v2851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4874.959853 secs 2022-03-21T15:32:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:53 smithi002 ceph-mon[17167]: pgmap v2851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4874.959853 secs 2022-03-21T15:32:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:55 smithi016 ceph-mon[22010]: pgmap v2852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:55 smithi079 ceph-mon[23158]: pgmap v2852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:55 smithi002 ceph-mon[17167]: pgmap v2852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:56 smithi016 ceph-mon[22010]: pgmap v2853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:56 smithi079 ceph-mon[23158]: pgmap v2853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:56 smithi002 ceph-mon[17167]: pgmap v2853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:32:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4879.959962 secs 2022-03-21T15:32:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4879.959962 secs 2022-03-21T15:32:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4879.959962 secs 2022-03-21T15:32:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:32:58 smithi002 ceph-mon[17167]: pgmap v2854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:32:58 smithi016 ceph-mon[22010]: pgmap v2854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:32:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:32:58 smithi079 ceph-mon[23158]: pgmap v2854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:01 smithi016 ceph-mon[22010]: pgmap v2855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:01 smithi079 ceph-mon[23158]: pgmap v2855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:01 smithi002 ceph-mon[17167]: pgmap v2855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:03 smithi016 ceph-mon[22010]: pgmap v2856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4884.960072 secs 2022-03-21T15:33:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:03 smithi079 ceph-mon[23158]: pgmap v2856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4884.960072 secs 2022-03-21T15:33:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:03 smithi002 ceph-mon[17167]: pgmap v2856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4884.960072 secs 2022-03-21T15:33:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:05 smithi016 ceph-mon[22010]: pgmap v2857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:05 smithi079 ceph-mon[23158]: pgmap v2857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:05 smithi002 ceph-mon[17167]: pgmap v2857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:06 smithi016 ceph-mon[22010]: pgmap v2858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:06 smithi079 ceph-mon[23158]: pgmap v2858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:06 smithi002 ceph-mon[17167]: pgmap v2858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4889.960165 secs 2022-03-21T15:33:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4889.960165 secs 2022-03-21T15:33:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4889.960165 secs 2022-03-21T15:33:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:08 smithi002 ceph-mon[17167]: pgmap v2859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:08 smithi016 ceph-mon[22010]: pgmap v2859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:09 smithi079 ceph-mon[23158]: pgmap v2859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:11 smithi016 ceph-mon[22010]: pgmap v2860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:11 smithi079 ceph-mon[23158]: pgmap v2860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:11 smithi002 ceph-mon[17167]: pgmap v2860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:13 smithi016 ceph-mon[22010]: pgmap v2861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4894.960274 secs 2022-03-21T15:33:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:13 smithi079 ceph-mon[23158]: pgmap v2861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4894.960274 secs 2022-03-21T15:33:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:13 smithi002 ceph-mon[17167]: pgmap v2861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4894.960274 secs 2022-03-21T15:33:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:15 smithi016 ceph-mon[22010]: pgmap v2862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:15 smithi079 ceph-mon[23158]: pgmap v2862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:15 smithi002 ceph-mon[17167]: pgmap v2862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:16.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:16 smithi079 ceph-mon[23158]: pgmap v2863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:16.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:33:16.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:33:16.591 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:33:16.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:16 smithi016 ceph-mon[22010]: pgmap v2863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:16.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:33:16.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:33:16.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:33:16.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:16 smithi002 ceph-mon[17167]: pgmap v2863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:33:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:33:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:33:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4899.960401 secs 2022-03-21T15:33:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:18 smithi079 ceph-mon[23158]: pgmap v2864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4899.960401 secs 2022-03-21T15:33:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:18 smithi002 ceph-mon[17167]: pgmap v2864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4899.960401 secs 2022-03-21T15:33:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:33:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:18 smithi016 ceph-mon[22010]: pgmap v2864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:21 smithi016 ceph-mon[22010]: pgmap v2865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:33:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:33:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:21 smithi079 ceph-mon[23158]: pgmap v2865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:33:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:33:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:21 smithi002 ceph-mon[17167]: pgmap v2865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:33:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:33:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:23 smithi016 ceph-mon[22010]: pgmap v2866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4904.960575 secs 2022-03-21T15:33:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:23 smithi079 ceph-mon[23158]: pgmap v2866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4904.960575 secs 2022-03-21T15:33:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:23 smithi002 ceph-mon[17167]: pgmap v2866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4904.960575 secs 2022-03-21T15:33:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:25 smithi016 ceph-mon[22010]: pgmap v2867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:25 smithi079 ceph-mon[23158]: pgmap v2867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:25 smithi002 ceph-mon[17167]: pgmap v2867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:26 smithi016 ceph-mon[22010]: pgmap v2868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:26 smithi079 ceph-mon[23158]: pgmap v2868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:26 smithi002 ceph-mon[17167]: pgmap v2868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4909.960691 secs 2022-03-21T15:33:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4909.960691 secs 2022-03-21T15:33:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4909.960691 secs 2022-03-21T15:33:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:29 smithi002 ceph-mon[17167]: pgmap v2869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:29 smithi016 ceph-mon[22010]: pgmap v2869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:29 smithi079 ceph-mon[23158]: pgmap v2869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:31 smithi016 ceph-mon[22010]: pgmap v2870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:31 smithi079 ceph-mon[23158]: pgmap v2870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:31 smithi002 ceph-mon[17167]: pgmap v2870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:33 smithi016 ceph-mon[22010]: pgmap v2871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4914.960816 secs 2022-03-21T15:33:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:33 smithi079 ceph-mon[23158]: pgmap v2871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4914.960816 secs 2022-03-21T15:33:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:33 smithi002 ceph-mon[17167]: pgmap v2871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4914.960816 secs 2022-03-21T15:33:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:35 smithi016 ceph-mon[22010]: pgmap v2872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:35 smithi079 ceph-mon[23158]: pgmap v2872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:35 smithi002 ceph-mon[17167]: pgmap v2872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:36 smithi016 ceph-mon[22010]: pgmap v2873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:36 smithi079 ceph-mon[23158]: pgmap v2873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:36 smithi002 ceph-mon[17167]: pgmap v2873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4919.960935 secs 2022-03-21T15:33:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4919.960935 secs 2022-03-21T15:33:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4919.960935 secs 2022-03-21T15:33:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:39 smithi002 ceph-mon[17167]: pgmap v2874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:39 smithi016 ceph-mon[22010]: pgmap v2874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:39 smithi079 ceph-mon[23158]: pgmap v2874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:41 smithi016 ceph-mon[22010]: pgmap v2875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:41 smithi079 ceph-mon[23158]: pgmap v2875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:41 smithi002 ceph-mon[17167]: pgmap v2875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:43 smithi016 ceph-mon[22010]: pgmap v2876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4924.961061 secs 2022-03-21T15:33:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:43 smithi079 ceph-mon[23158]: pgmap v2876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4924.961061 secs 2022-03-21T15:33:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:43 smithi002 ceph-mon[17167]: pgmap v2876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4924.961061 secs 2022-03-21T15:33:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:45 smithi016 ceph-mon[22010]: pgmap v2877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:45 smithi079 ceph-mon[23158]: pgmap v2877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:45 smithi002 ceph-mon[17167]: pgmap v2877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:46 smithi016 ceph-mon[22010]: pgmap v2878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:46 smithi079 ceph-mon[23158]: pgmap v2878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:46 smithi002 ceph-mon[17167]: pgmap v2878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4929.961198 secs 2022-03-21T15:33:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4929.961198 secs 2022-03-21T15:33:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4929.961198 secs 2022-03-21T15:33:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:49 smithi002 ceph-mon[17167]: pgmap v2879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:49 smithi016 ceph-mon[22010]: pgmap v2879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:49 smithi079 ceph-mon[23158]: pgmap v2879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:51 smithi016 ceph-mon[22010]: pgmap v2880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:51 smithi079 ceph-mon[23158]: pgmap v2880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:51 smithi002 ceph-mon[17167]: pgmap v2880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:53 smithi016 ceph-mon[22010]: pgmap v2881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4934.961338 secs 2022-03-21T15:33:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:53 smithi079 ceph-mon[23158]: pgmap v2881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4934.961338 secs 2022-03-21T15:33:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:53 smithi002 ceph-mon[17167]: pgmap v2881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4934.961338 secs 2022-03-21T15:33:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:55 smithi016 ceph-mon[22010]: pgmap v2882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:55 smithi079 ceph-mon[23158]: pgmap v2882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:55 smithi002 ceph-mon[17167]: pgmap v2882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:56 smithi016 ceph-mon[22010]: pgmap v2883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:56 smithi079 ceph-mon[23158]: pgmap v2883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:56 smithi002 ceph-mon[17167]: pgmap v2883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:33:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4939.961474 secs 2022-03-21T15:33:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4939.961474 secs 2022-03-21T15:33:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4939.961474 secs 2022-03-21T15:33:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:33:59 smithi002 ceph-mon[17167]: pgmap v2884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:33:59 smithi016 ceph-mon[22010]: pgmap v2884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:33:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:33:59 smithi079 ceph-mon[23158]: pgmap v2884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:01 smithi016 ceph-mon[22010]: pgmap v2885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:01 smithi079 ceph-mon[23158]: pgmap v2885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:01 smithi002 ceph-mon[17167]: pgmap v2885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:03 smithi016 ceph-mon[22010]: pgmap v2886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4944.961593 secs 2022-03-21T15:34:03.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:03 smithi079 ceph-mon[23158]: pgmap v2886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4944.961593 secs 2022-03-21T15:34:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:03 smithi002 ceph-mon[17167]: pgmap v2886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4944.961593 secs 2022-03-21T15:34:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:05 smithi016 ceph-mon[22010]: pgmap v2887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:05 smithi079 ceph-mon[23158]: pgmap v2887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:05 smithi002 ceph-mon[17167]: pgmap v2887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:06 smithi016 ceph-mon[22010]: pgmap v2888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:06 smithi079 ceph-mon[23158]: pgmap v2888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:06 smithi002 ceph-mon[17167]: pgmap v2888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4949.961654 secs 2022-03-21T15:34:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4949.961654 secs 2022-03-21T15:34:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4949.961654 secs 2022-03-21T15:34:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:09 smithi002 ceph-mon[17167]: pgmap v2889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:09 smithi016 ceph-mon[22010]: pgmap v2889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:09 smithi079 ceph-mon[23158]: pgmap v2889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:11 smithi016 ceph-mon[22010]: pgmap v2890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:11 smithi079 ceph-mon[23158]: pgmap v2890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:11 smithi002 ceph-mon[17167]: pgmap v2890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:13 smithi016 ceph-mon[22010]: pgmap v2891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4954.961777 secs 2022-03-21T15:34:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:13 smithi079 ceph-mon[23158]: pgmap v2891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4954.961777 secs 2022-03-21T15:34:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:13 smithi002 ceph-mon[17167]: pgmap v2891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4954.961777 secs 2022-03-21T15:34:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:15 smithi016 ceph-mon[22010]: pgmap v2892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:15 smithi079 ceph-mon[23158]: pgmap v2892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:15 smithi002 ceph-mon[17167]: pgmap v2892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:16 smithi016 ceph-mon[22010]: pgmap v2893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:16 smithi079 ceph-mon[23158]: pgmap v2893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:16 smithi002 ceph-mon[17167]: pgmap v2893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:17.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:34:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:34:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:34:18.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:34:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:34:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:34:18.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:34:18.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:34:18.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:34:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4959.961862 secs 2022-03-21T15:34:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:18 smithi002 ceph-mon[17167]: pgmap v2894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:34:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4959.961862 secs 2022-03-21T15:34:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:18 smithi016 ceph-mon[22010]: pgmap v2894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:34:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4959.961862 secs 2022-03-21T15:34:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:18 smithi079 ceph-mon[23158]: pgmap v2894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:34:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:21 smithi016 ceph-mon[22010]: pgmap v2895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:34:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:34:21.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:21 smithi079 ceph-mon[23158]: pgmap v2895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:34:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:34:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:21 smithi002 ceph-mon[17167]: pgmap v2895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:34:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:34:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:23 smithi016 ceph-mon[22010]: pgmap v2896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4964.961981 secs 2022-03-21T15:34:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:23 smithi079 ceph-mon[23158]: pgmap v2896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4964.961981 secs 2022-03-21T15:34:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:23 smithi002 ceph-mon[17167]: pgmap v2896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4964.961981 secs 2022-03-21T15:34:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:25 smithi016 ceph-mon[22010]: pgmap v2897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:25 smithi079 ceph-mon[23158]: pgmap v2897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:25 smithi002 ceph-mon[17167]: pgmap v2897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:26 smithi016 ceph-mon[22010]: pgmap v2898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:26 smithi079 ceph-mon[23158]: pgmap v2898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:26 smithi002 ceph-mon[17167]: pgmap v2898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4969.962096 secs 2022-03-21T15:34:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4969.962096 secs 2022-03-21T15:34:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4969.962096 secs 2022-03-21T15:34:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:29 smithi002 ceph-mon[17167]: pgmap v2899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:29 smithi016 ceph-mon[22010]: pgmap v2899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:29 smithi079 ceph-mon[23158]: pgmap v2899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:31 smithi079 ceph-mon[23158]: pgmap v2900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:31 smithi002 ceph-mon[17167]: pgmap v2900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:31 smithi016 ceph-mon[22010]: pgmap v2900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:33 smithi079 ceph-mon[23158]: pgmap v2901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4974.962211 secs 2022-03-21T15:34:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:33 smithi002 ceph-mon[17167]: pgmap v2901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4974.962211 secs 2022-03-21T15:34:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:33 smithi016 ceph-mon[22010]: pgmap v2901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4974.962211 secs 2022-03-21T15:34:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:35 smithi079 ceph-mon[23158]: pgmap v2902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:35 smithi002 ceph-mon[17167]: pgmap v2902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:35 smithi016 ceph-mon[22010]: pgmap v2902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:36 smithi016 ceph-mon[22010]: pgmap v2903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:36 smithi079 ceph-mon[23158]: pgmap v2903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:36 smithi002 ceph-mon[17167]: pgmap v2903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4979.962331 secs 2022-03-21T15:34:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4979.962331 secs 2022-03-21T15:34:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4979.962331 secs 2022-03-21T15:34:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:39 smithi002 ceph-mon[17167]: pgmap v2904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:39 smithi016 ceph-mon[22010]: pgmap v2904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:39 smithi079 ceph-mon[23158]: pgmap v2904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:41 smithi016 ceph-mon[22010]: pgmap v2905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:41.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:41 smithi079 ceph-mon[23158]: pgmap v2905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:41 smithi002 ceph-mon[17167]: pgmap v2905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:43 smithi079 ceph-mon[23158]: pgmap v2906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4984.962394 secs 2022-03-21T15:34:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:43 smithi002 ceph-mon[17167]: pgmap v2906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4984.962394 secs 2022-03-21T15:34:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:43 smithi016 ceph-mon[22010]: pgmap v2906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4984.962394 secs 2022-03-21T15:34:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:45 smithi079 ceph-mon[23158]: pgmap v2907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:45 smithi002 ceph-mon[17167]: pgmap v2907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:45 smithi016 ceph-mon[22010]: pgmap v2907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:46 smithi016 ceph-mon[22010]: pgmap v2908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:46 smithi079 ceph-mon[23158]: pgmap v2908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:46 smithi002 ceph-mon[17167]: pgmap v2908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4989.962516 secs 2022-03-21T15:34:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4989.962516 secs 2022-03-21T15:34:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4989.962516 secs 2022-03-21T15:34:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:49 smithi002 ceph-mon[17167]: pgmap v2909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:49 smithi016 ceph-mon[22010]: pgmap v2909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:49 smithi079 ceph-mon[23158]: pgmap v2909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:51 smithi079 ceph-mon[23158]: pgmap v2910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:51 smithi002 ceph-mon[17167]: pgmap v2910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:51 smithi016 ceph-mon[22010]: pgmap v2910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:53 smithi079 ceph-mon[23158]: pgmap v2911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4994.962635 secs 2022-03-21T15:34:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:53 smithi002 ceph-mon[17167]: pgmap v2911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4994.962635 secs 2022-03-21T15:34:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:53 smithi016 ceph-mon[22010]: pgmap v2911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4994.962635 secs 2022-03-21T15:34:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:55 smithi079 ceph-mon[23158]: pgmap v2912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:55 smithi002 ceph-mon[17167]: pgmap v2912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:55 smithi016 ceph-mon[22010]: pgmap v2912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:56 smithi079 ceph-mon[23158]: pgmap v2913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:56 smithi002 ceph-mon[17167]: pgmap v2913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:56 smithi016 ceph-mon[22010]: pgmap v2913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:34:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 4999.962708 secs 2022-03-21T15:34:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 4999.962708 secs 2022-03-21T15:34:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 4999.962708 secs 2022-03-21T15:34:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:34:59 smithi002 ceph-mon[17167]: pgmap v2914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:34:59 smithi016 ceph-mon[22010]: pgmap v2914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:34:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:34:59 smithi079 ceph-mon[23158]: pgmap v2914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:01 smithi079 ceph-mon[23158]: pgmap v2915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:01 smithi002 ceph-mon[17167]: pgmap v2915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:01 smithi016 ceph-mon[22010]: pgmap v2915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:03 smithi079 ceph-mon[23158]: pgmap v2916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5004.962789 secs 2022-03-21T15:35:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:03 smithi002 ceph-mon[17167]: pgmap v2916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5004.962789 secs 2022-03-21T15:35:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:03 smithi016 ceph-mon[22010]: pgmap v2916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5004.962789 secs 2022-03-21T15:35:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:05 smithi079 ceph-mon[23158]: pgmap v2917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:05 smithi002 ceph-mon[17167]: pgmap v2917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:05 smithi016 ceph-mon[22010]: pgmap v2917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:06 smithi079 ceph-mon[23158]: pgmap v2918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:06 smithi002 ceph-mon[17167]: pgmap v2918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:06 smithi016 ceph-mon[22010]: pgmap v2918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5009.962902 secs 2022-03-21T15:35:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5009.962902 secs 2022-03-21T15:35:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5009.962902 secs 2022-03-21T15:35:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:09 smithi002 ceph-mon[17167]: pgmap v2919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:09 smithi016 ceph-mon[22010]: pgmap v2919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:09 smithi079 ceph-mon[23158]: pgmap v2919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:11 smithi079 ceph-mon[23158]: pgmap v2920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:11 smithi002 ceph-mon[17167]: pgmap v2920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:11 smithi016 ceph-mon[22010]: pgmap v2920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:13 smithi079 ceph-mon[23158]: pgmap v2921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5014.963023 secs 2022-03-21T15:35:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:13 smithi002 ceph-mon[17167]: pgmap v2921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5014.963023 secs 2022-03-21T15:35:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:13 smithi016 ceph-mon[22010]: pgmap v2921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5014.963023 secs 2022-03-21T15:35:15.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:15 smithi079 ceph-mon[23158]: pgmap v2922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:15 smithi002 ceph-mon[17167]: pgmap v2922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:15 smithi016 ceph-mon[22010]: pgmap v2922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:16 smithi079 ceph-mon[23158]: pgmap v2923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:16 smithi002 ceph-mon[17167]: pgmap v2923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:16 smithi016 ceph-mon[22010]: pgmap v2923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5019.963172 secs 2022-03-21T15:35:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5019.963172 secs 2022-03-21T15:35:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5019.963172 secs 2022-03-21T15:35:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:19 smithi002 ceph-mon[17167]: pgmap v2924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:35:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:35:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:35:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:19 smithi016 ceph-mon[22010]: pgmap v2924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:35:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:35:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:35:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:19 smithi079 ceph-mon[23158]: pgmap v2924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:35:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:35:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:35:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:35:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:20 smithi016 ceph-mon[22010]: pgmap v2925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:35:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:20 smithi079 ceph-mon[23158]: pgmap v2925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:20.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:35:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:20 smithi002 ceph-mon[17167]: pgmap v2925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:35:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:35:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:35:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:35:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:35:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:35:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:22 smithi079 ceph-mon[23158]: pgmap v2926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:22 smithi002 ceph-mon[17167]: pgmap v2926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:22 smithi016 ceph-mon[22010]: pgmap v2926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5024.963308 secs 2022-03-21T15:35:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5024.963308 secs 2022-03-21T15:35:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5024.963308 secs 2022-03-21T15:35:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:24 smithi079 ceph-mon[23158]: pgmap v2927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:24.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:24 smithi002 ceph-mon[17167]: pgmap v2927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:24 smithi016 ceph-mon[22010]: pgmap v2927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:26 smithi079 ceph-mon[23158]: pgmap v2928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:26 smithi002 ceph-mon[17167]: pgmap v2928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:26 smithi016 ceph-mon[22010]: pgmap v2928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5029.963440 secs 2022-03-21T15:35:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5029.963440 secs 2022-03-21T15:35:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5029.963440 secs 2022-03-21T15:35:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:29 smithi002 ceph-mon[17167]: pgmap v2929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:29 smithi016 ceph-mon[22010]: pgmap v2929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:29 smithi079 ceph-mon[23158]: pgmap v2929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:31 smithi079 ceph-mon[23158]: pgmap v2930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:31 smithi002 ceph-mon[17167]: pgmap v2930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:31 smithi016 ceph-mon[22010]: pgmap v2930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:33 smithi079 ceph-mon[23158]: pgmap v2931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5034.963585 secs 2022-03-21T15:35:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:33 smithi002 ceph-mon[17167]: pgmap v2931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:33.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5034.963585 secs 2022-03-21T15:35:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:33 smithi016 ceph-mon[22010]: pgmap v2931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5034.963585 secs 2022-03-21T15:35:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:35 smithi079 ceph-mon[23158]: pgmap v2932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:35 smithi002 ceph-mon[17167]: pgmap v2932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:35 smithi016 ceph-mon[22010]: pgmap v2932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:36 smithi079 ceph-mon[23158]: pgmap v2933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:36 smithi002 ceph-mon[17167]: pgmap v2933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:36 smithi016 ceph-mon[22010]: pgmap v2933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5039.963648 secs 2022-03-21T15:35:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5039.963648 secs 2022-03-21T15:35:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5039.963648 secs 2022-03-21T15:35:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:39 smithi002 ceph-mon[17167]: pgmap v2934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:39 smithi016 ceph-mon[22010]: pgmap v2934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:39 smithi079 ceph-mon[23158]: pgmap v2934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:41 smithi079 ceph-mon[23158]: pgmap v2935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:41 smithi002 ceph-mon[17167]: pgmap v2935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:41 smithi016 ceph-mon[22010]: pgmap v2935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:43 smithi079 ceph-mon[23158]: pgmap v2936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5044.963765 secs 2022-03-21T15:35:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:43 smithi002 ceph-mon[17167]: pgmap v2936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5044.963765 secs 2022-03-21T15:35:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:43 smithi016 ceph-mon[22010]: pgmap v2936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5044.963765 secs 2022-03-21T15:35:45.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:45 smithi079 ceph-mon[23158]: pgmap v2937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:45 smithi002 ceph-mon[17167]: pgmap v2937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:45 smithi016 ceph-mon[22010]: pgmap v2937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:46 smithi079 ceph-mon[23158]: pgmap v2938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:46 smithi002 ceph-mon[17167]: pgmap v2938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:46 smithi016 ceph-mon[22010]: pgmap v2938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5049.963875 secs 2022-03-21T15:35:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5049.963875 secs 2022-03-21T15:35:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5049.963875 secs 2022-03-21T15:35:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:49 smithi002 ceph-mon[17167]: pgmap v2939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:49 smithi016 ceph-mon[22010]: pgmap v2939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:49 smithi079 ceph-mon[23158]: pgmap v2939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:51 smithi079 ceph-mon[23158]: pgmap v2940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:51 smithi002 ceph-mon[17167]: pgmap v2940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:51 smithi016 ceph-mon[22010]: pgmap v2940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:53 smithi079 ceph-mon[23158]: pgmap v2941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5054.964001 secs 2022-03-21T15:35:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:53 smithi002 ceph-mon[17167]: pgmap v2941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5054.964001 secs 2022-03-21T15:35:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:53 smithi016 ceph-mon[22010]: pgmap v2941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:35:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5054.964001 secs 2022-03-21T15:35:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:55 smithi079 ceph-mon[23158]: pgmap v2942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:55 smithi002 ceph-mon[17167]: pgmap v2942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:55 smithi016 ceph-mon[22010]: pgmap v2942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:35:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:56 smithi079 ceph-mon[23158]: pgmap v2943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:56 smithi002 ceph-mon[17167]: pgmap v2943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:56 smithi016 ceph-mon[22010]: pgmap v2943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:35:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5059.964135 secs 2022-03-21T15:35:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5059.964135 secs 2022-03-21T15:35:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5059.964135 secs 2022-03-21T15:35:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:35:59 smithi002 ceph-mon[17167]: pgmap v2944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:35:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:35:59 smithi016 ceph-mon[22010]: pgmap v2944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:35:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:35:59 smithi079 ceph-mon[23158]: pgmap v2944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:01 smithi079 ceph-mon[23158]: pgmap v2945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:01 smithi002 ceph-mon[17167]: pgmap v2945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:01 smithi016 ceph-mon[22010]: pgmap v2945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:03 smithi079 ceph-mon[23158]: pgmap v2946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5064.964254 secs 2022-03-21T15:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:03 smithi002 ceph-mon[17167]: pgmap v2946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5064.964254 secs 2022-03-21T15:36:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:03 smithi016 ceph-mon[22010]: pgmap v2946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5064.964254 secs 2022-03-21T15:36:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:05 smithi079 ceph-mon[23158]: pgmap v2947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:05 smithi002 ceph-mon[17167]: pgmap v2947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:05 smithi016 ceph-mon[22010]: pgmap v2947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:06.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:06 smithi079 ceph-mon[23158]: pgmap v2948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:06 smithi002 ceph-mon[17167]: pgmap v2948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:06 smithi016 ceph-mon[22010]: pgmap v2948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5069.964384 secs 2022-03-21T15:36:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5069.964384 secs 2022-03-21T15:36:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5069.964384 secs 2022-03-21T15:36:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:09 smithi002 ceph-mon[17167]: pgmap v2949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:09 smithi016 ceph-mon[22010]: pgmap v2949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:09 smithi079 ceph-mon[23158]: pgmap v2949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:11 smithi079 ceph-mon[23158]: pgmap v2950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:11 smithi002 ceph-mon[17167]: pgmap v2950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:11 smithi016 ceph-mon[22010]: pgmap v2950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:13 smithi079 ceph-mon[23158]: pgmap v2951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5074.964503 secs 2022-03-21T15:36:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:13 smithi002 ceph-mon[17167]: pgmap v2951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5074.964503 secs 2022-03-21T15:36:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:13 smithi016 ceph-mon[22010]: pgmap v2951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5074.964503 secs 2022-03-21T15:36:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:15 smithi079 ceph-mon[23158]: pgmap v2952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:15 smithi002 ceph-mon[17167]: pgmap v2952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:15 smithi016 ceph-mon[22010]: pgmap v2952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:16 smithi079 ceph-mon[23158]: pgmap v2953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:16 smithi002 ceph-mon[17167]: pgmap v2953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:16 smithi016 ceph-mon[22010]: pgmap v2953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5079.964625 secs 2022-03-21T15:36:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5079.964625 secs 2022-03-21T15:36:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5079.964625 secs 2022-03-21T15:36:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:19 smithi002 ceph-mon[17167]: pgmap v2954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:19 smithi016 ceph-mon[22010]: pgmap v2954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:19.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:19 smithi079 ceph-mon[23158]: pgmap v2954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:36:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:36:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:36:20.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:36:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:36:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:36:20.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:36:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:36:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:36:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:21 smithi002 ceph-mon[17167]: pgmap v2955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:36:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:36:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:36:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:21 smithi016 ceph-mon[22010]: pgmap v2955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:36:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:36:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:36:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:21 smithi079 ceph-mon[23158]: pgmap v2955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:36:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:36:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:36:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:23 smithi002 ceph-mon[17167]: pgmap v2956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5084.964745 secs 2022-03-21T15:36:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:23 smithi016 ceph-mon[22010]: pgmap v2956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5084.964745 secs 2022-03-21T15:36:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:23 smithi079 ceph-mon[23158]: pgmap v2956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5084.964745 secs 2022-03-21T15:36:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:25 smithi002 ceph-mon[17167]: pgmap v2957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:25 smithi016 ceph-mon[22010]: pgmap v2957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:25 smithi079 ceph-mon[23158]: pgmap v2957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:26 smithi079 ceph-mon[23158]: pgmap v2958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:26 smithi002 ceph-mon[17167]: pgmap v2958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:26 smithi016 ceph-mon[22010]: pgmap v2958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5089.964839 secs 2022-03-21T15:36:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5089.964839 secs 2022-03-21T15:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5089.964839 secs 2022-03-21T15:36:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:29 smithi002 ceph-mon[17167]: pgmap v2959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:29 smithi016 ceph-mon[22010]: pgmap v2959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:29 smithi079 ceph-mon[23158]: pgmap v2959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:31 smithi079 ceph-mon[23158]: pgmap v2960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:31 smithi002 ceph-mon[17167]: pgmap v2960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:31 smithi016 ceph-mon[22010]: pgmap v2960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:33 smithi079 ceph-mon[23158]: pgmap v2961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5094.964905 secs 2022-03-21T15:36:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:33 smithi002 ceph-mon[17167]: pgmap v2961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5094.964905 secs 2022-03-21T15:36:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:33 smithi016 ceph-mon[22010]: pgmap v2961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5094.964905 secs 2022-03-21T15:36:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:35 smithi079 ceph-mon[23158]: pgmap v2962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:35 smithi002 ceph-mon[17167]: pgmap v2962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:35 smithi016 ceph-mon[22010]: pgmap v2962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:36 smithi079 ceph-mon[23158]: pgmap v2963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:36 smithi002 ceph-mon[17167]: pgmap v2963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:36 smithi016 ceph-mon[22010]: pgmap v2963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5099.965025 secs 2022-03-21T15:36:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5099.965025 secs 2022-03-21T15:36:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5099.965025 secs 2022-03-21T15:36:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:39 smithi002 ceph-mon[17167]: pgmap v2964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:39 smithi016 ceph-mon[22010]: pgmap v2964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:39 smithi079 ceph-mon[23158]: pgmap v2964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:41 smithi079 ceph-mon[23158]: pgmap v2965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:41 smithi002 ceph-mon[17167]: pgmap v2965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:41 smithi016 ceph-mon[22010]: pgmap v2965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:43 smithi079 ceph-mon[23158]: pgmap v2966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5104.965151 secs 2022-03-21T15:36:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:43 smithi002 ceph-mon[17167]: pgmap v2966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5104.965151 secs 2022-03-21T15:36:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:43 smithi016 ceph-mon[22010]: pgmap v2966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5104.965151 secs 2022-03-21T15:36:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:45 smithi079 ceph-mon[23158]: pgmap v2967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:45 smithi002 ceph-mon[17167]: pgmap v2967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:45 smithi016 ceph-mon[22010]: pgmap v2967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:46 smithi079 ceph-mon[23158]: pgmap v2968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:46 smithi002 ceph-mon[17167]: pgmap v2968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:46 smithi016 ceph-mon[22010]: pgmap v2968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5109.965263 secs 2022-03-21T15:36:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5109.965263 secs 2022-03-21T15:36:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5109.965263 secs 2022-03-21T15:36:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:49 smithi002 ceph-mon[17167]: pgmap v2969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:49 smithi016 ceph-mon[22010]: pgmap v2969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:49 smithi079 ceph-mon[23158]: pgmap v2969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:51 smithi079 ceph-mon[23158]: pgmap v2970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:51 smithi002 ceph-mon[17167]: pgmap v2970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:51 smithi016 ceph-mon[22010]: pgmap v2970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:53 smithi079 ceph-mon[23158]: pgmap v2971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5114.965407 secs 2022-03-21T15:36:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:53 smithi002 ceph-mon[17167]: pgmap v2971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5114.965407 secs 2022-03-21T15:36:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:53 smithi016 ceph-mon[22010]: pgmap v2971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:36:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5114.965407 secs 2022-03-21T15:36:55.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:55 smithi079 ceph-mon[23158]: pgmap v2972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:55 smithi002 ceph-mon[17167]: pgmap v2972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:55 smithi016 ceph-mon[22010]: pgmap v2972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:36:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:56 smithi079 ceph-mon[23158]: pgmap v2973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:56 smithi002 ceph-mon[17167]: pgmap v2973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:56 smithi016 ceph-mon[22010]: pgmap v2973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:36:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5119.965534 secs 2022-03-21T15:36:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5119.965534 secs 2022-03-21T15:36:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5119.965534 secs 2022-03-21T15:36:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:36:59 smithi002 ceph-mon[17167]: pgmap v2974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:36:59 smithi016 ceph-mon[22010]: pgmap v2974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:36:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:36:59 smithi079 ceph-mon[23158]: pgmap v2974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:01 smithi079 ceph-mon[23158]: pgmap v2975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:01 smithi002 ceph-mon[17167]: pgmap v2975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:01 smithi016 ceph-mon[22010]: pgmap v2975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:03 smithi079 ceph-mon[23158]: pgmap v2976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5124.965649 secs 2022-03-21T15:37:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:03 smithi002 ceph-mon[17167]: pgmap v2976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5124.965649 secs 2022-03-21T15:37:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:03 smithi016 ceph-mon[22010]: pgmap v2976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5124.965649 secs 2022-03-21T15:37:05.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:05 smithi079 ceph-mon[23158]: pgmap v2977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:05 smithi002 ceph-mon[17167]: pgmap v2977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:05 smithi016 ceph-mon[22010]: pgmap v2977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:06 smithi079 ceph-mon[23158]: pgmap v2978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:06 smithi002 ceph-mon[17167]: pgmap v2978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:06 smithi016 ceph-mon[22010]: pgmap v2978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5129.965768 secs 2022-03-21T15:37:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5129.965768 secs 2022-03-21T15:37:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5129.965768 secs 2022-03-21T15:37:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:09 smithi002 ceph-mon[17167]: pgmap v2979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:09 smithi016 ceph-mon[22010]: pgmap v2979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:09 smithi079 ceph-mon[23158]: pgmap v2979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:11 smithi079 ceph-mon[23158]: pgmap v2980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:11 smithi002 ceph-mon[17167]: pgmap v2980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:11 smithi016 ceph-mon[22010]: pgmap v2980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:13.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:13 smithi079 ceph-mon[23158]: pgmap v2981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5134.965894 secs 2022-03-21T15:37:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:13 smithi002 ceph-mon[17167]: pgmap v2981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5134.965894 secs 2022-03-21T15:37:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:13 smithi016 ceph-mon[22010]: pgmap v2981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5134.965894 secs 2022-03-21T15:37:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:15 smithi079 ceph-mon[23158]: pgmap v2982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:15 smithi002 ceph-mon[17167]: pgmap v2982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:15 smithi016 ceph-mon[22010]: pgmap v2982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:16 smithi079 ceph-mon[23158]: pgmap v2983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:16 smithi002 ceph-mon[17167]: pgmap v2983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:16 smithi016 ceph-mon[22010]: pgmap v2983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5139.966010 secs 2022-03-21T15:37:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5139.966010 secs 2022-03-21T15:37:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5139.966010 secs 2022-03-21T15:37:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:19 smithi002 ceph-mon[17167]: pgmap v2984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:19 smithi016 ceph-mon[22010]: pgmap v2984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:19 smithi079 ceph-mon[23158]: pgmap v2984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: pgmap v2985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:37:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:37:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:37:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:37:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:37:21.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: pgmap v2985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:21.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:37:21.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:37:21.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:37:21.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:37:21.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:37:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: pgmap v2985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:37:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:37:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:37:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:37:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:37:23.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:23 smithi016 ceph-mon[22010]: pgmap v2986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:23.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5144.966149 secs 2022-03-21T15:37:23.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:23 smithi079 ceph-mon[23158]: pgmap v2986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5144.966149 secs 2022-03-21T15:37:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:23 smithi002 ceph-mon[17167]: pgmap v2986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5144.966149 secs 2022-03-21T15:37:24.683 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:24 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T15:37:24.684 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:24.684 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:24 smithi002 ceph-mon[17167]: pgmap v2987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:24.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:24 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T15:37:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:24 smithi079 ceph-mon[23158]: pgmap v2987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:24 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T15:37:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:24 smithi016 ceph-mon[22010]: pgmap v2987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:26.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:26 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T15:37:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:26 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T15:37:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:26 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T15:37:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:37:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:27 smithi002 ceph-mon[17167]: pgmap v2988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:27 smithi016 ceph-mon[22010]: pgmap v2988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:27.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:27 smithi079 ceph-mon[23158]: pgmap v2988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5149.966251 secs 2022-03-21T15:37:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5149.966251 secs 2022-03-21T15:37:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5149.966251 secs 2022-03-21T15:37:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:29 smithi002 ceph-mon[17167]: pgmap v2989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:29.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:29 smithi016 ceph-mon[22010]: pgmap v2989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:29 smithi079 ceph-mon[23158]: pgmap v2989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:31 smithi079 ceph-mon[23158]: pgmap v2990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:31 smithi002 ceph-mon[17167]: pgmap v2990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:31 smithi016 ceph-mon[22010]: pgmap v2990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:33 smithi002 ceph-mon[17167]: pgmap v2991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5154.966376 secs 2022-03-21T15:37:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:33 smithi016 ceph-mon[22010]: pgmap v2991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5154.966376 secs 2022-03-21T15:37:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:33 smithi079 ceph-mon[23158]: pgmap v2991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5154.966376 secs 2022-03-21T15:37:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:35 smithi002 ceph-mon[17167]: pgmap v2992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:35 smithi016 ceph-mon[22010]: pgmap v2992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:35 smithi079 ceph-mon[23158]: pgmap v2992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:36 smithi079 ceph-mon[23158]: pgmap v2993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:36 smithi002 ceph-mon[17167]: pgmap v2993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:36 smithi016 ceph-mon[22010]: pgmap v2993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5159.966501 secs 2022-03-21T15:37:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5159.966501 secs 2022-03-21T15:37:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5159.966501 secs 2022-03-21T15:37:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:39 smithi002 ceph-mon[17167]: pgmap v2994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:39 smithi016 ceph-mon[22010]: pgmap v2994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:39 smithi079 ceph-mon[23158]: pgmap v2994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:41 smithi002 ceph-mon[17167]: pgmap v2995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:41 smithi016 ceph-mon[22010]: pgmap v2995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:41 smithi079 ceph-mon[23158]: pgmap v2995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:43 smithi002 ceph-mon[17167]: pgmap v2996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5164.966650 secs 2022-03-21T15:37:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:43 smithi016 ceph-mon[22010]: pgmap v2996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5164.966650 secs 2022-03-21T15:37:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:43 smithi079 ceph-mon[23158]: pgmap v2996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5164.966650 secs 2022-03-21T15:37:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:45 smithi002 ceph-mon[17167]: pgmap v2997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:45 smithi016 ceph-mon[22010]: pgmap v2997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:45 smithi079 ceph-mon[23158]: pgmap v2997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:46 smithi079 ceph-mon[23158]: pgmap v2998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:46 smithi002 ceph-mon[17167]: pgmap v2998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:46 smithi016 ceph-mon[22010]: pgmap v2998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5169.966711 secs 2022-03-21T15:37:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5169.966711 secs 2022-03-21T15:37:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5169.966711 secs 2022-03-21T15:37:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:49 smithi002 ceph-mon[17167]: pgmap v2999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:49 smithi016 ceph-mon[22010]: pgmap v2999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:49 smithi079 ceph-mon[23158]: pgmap v2999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:37:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:51 smithi002 ceph-mon[17167]: pgmap v3000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:51 smithi016 ceph-mon[22010]: pgmap v3000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:51 smithi079 ceph-mon[23158]: pgmap v3000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:37:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:53 smithi002 ceph-mon[17167]: pgmap v3001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5174.966830 secs 2022-03-21T15:37:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:53 smithi016 ceph-mon[22010]: pgmap v3001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5174.966830 secs 2022-03-21T15:37:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:53 smithi079 ceph-mon[23158]: pgmap v3001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:37:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5174.966830 secs 2022-03-21T15:37:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:55 smithi002 ceph-mon[17167]: pgmap v3002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:37:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:55 smithi016 ceph-mon[22010]: pgmap v3002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:37:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:55 smithi079 ceph-mon[23158]: pgmap v3002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:37:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:56 smithi002 ceph-mon[17167]: pgmap v3003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:56 smithi016 ceph-mon[22010]: pgmap v3003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:56 smithi079 ceph-mon[23158]: pgmap v3003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:37:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5179.966952 secs 2022-03-21T15:37:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5179.966952 secs 2022-03-21T15:37:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5179.966952 secs 2022-03-21T15:37:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:37:59 smithi002 ceph-mon[17167]: pgmap v3004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:37:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:37:59 smithi016 ceph-mon[22010]: pgmap v3004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:37:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:37:59 smithi079 ceph-mon[23158]: pgmap v3004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:38:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:01 smithi002 ceph-mon[17167]: pgmap v3005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:38:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:01 smithi016 ceph-mon[22010]: pgmap v3005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:38:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:01 smithi079 ceph-mon[23158]: pgmap v3005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:38:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:03 smithi002 ceph-mon[17167]: pgmap v3006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5184.967100 secs 2022-03-21T15:38:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:03 smithi016 ceph-mon[22010]: pgmap v3006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5184.967100 secs 2022-03-21T15:38:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:03 smithi079 ceph-mon[23158]: pgmap v3006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5184.967100 secs 2022-03-21T15:38:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:05 smithi002 ceph-mon[17167]: pgmap v3007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:05 smithi016 ceph-mon[22010]: pgmap v3007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:05 smithi079 ceph-mon[23158]: pgmap v3007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:06 smithi002 ceph-mon[17167]: pgmap v3008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:06 smithi016 ceph-mon[22010]: pgmap v3008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:06 smithi079 ceph-mon[23158]: pgmap v3008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5189.967217 secs 2022-03-21T15:38:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5189.967217 secs 2022-03-21T15:38:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5189.967217 secs 2022-03-21T15:38:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:09 smithi002 ceph-mon[17167]: pgmap v3009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:09.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:09 smithi016 ceph-mon[22010]: pgmap v3009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:09 smithi079 ceph-mon[23158]: pgmap v3009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:11 smithi002 ceph-mon[17167]: pgmap v3010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:11 smithi016 ceph-mon[22010]: pgmap v3010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:11 smithi079 ceph-mon[23158]: pgmap v3010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:13 smithi002 ceph-mon[17167]: pgmap v3011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5194.967356 secs 2022-03-21T15:38:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:13 smithi016 ceph-mon[22010]: pgmap v3011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5194.967356 secs 2022-03-21T15:38:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:13 smithi079 ceph-mon[23158]: pgmap v3011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5194.967356 secs 2022-03-21T15:38:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:15 smithi002 ceph-mon[17167]: pgmap v3012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:15 smithi016 ceph-mon[22010]: pgmap v3012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:15 smithi079 ceph-mon[23158]: pgmap v3012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:16 smithi002 ceph-mon[17167]: pgmap v3013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:16 smithi016 ceph-mon[22010]: pgmap v3013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:16 smithi079 ceph-mon[23158]: pgmap v3013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5199.967506 secs 2022-03-21T15:38:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5199.967506 secs 2022-03-21T15:38:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5199.967506 secs 2022-03-21T15:38:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:19 smithi002 ceph-mon[17167]: pgmap v3014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:19 smithi016 ceph-mon[22010]: pgmap v3014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:19 smithi079 ceph-mon[23158]: pgmap v3014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:21 smithi002 ceph-mon[17167]: pgmap v3015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:38:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:38:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:21 smithi016 ceph-mon[22010]: pgmap v3015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:38:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:38:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:21 smithi079 ceph-mon[23158]: pgmap v3015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:38:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:38:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:23 smithi002 ceph-mon[17167]: pgmap v3016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5204.967601 secs 2022-03-21T15:38:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:23 smithi016 ceph-mon[22010]: pgmap v3016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5204.967601 secs 2022-03-21T15:38:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:23 smithi079 ceph-mon[23158]: pgmap v3016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5204.967601 secs 2022-03-21T15:38:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:25 smithi016 ceph-mon[22010]: pgmap v3017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:38:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:38:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:38:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:25 smithi079 ceph-mon[23158]: pgmap v3017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:38:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:38:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:38:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:25 smithi002 ceph-mon[17167]: pgmap v3017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:38:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:38:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:38:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:38:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:38:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:38:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:38:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:26 smithi079 ceph-mon[23158]: pgmap v3018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:38:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:38:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:38:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:38:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:26 smithi002 ceph-mon[17167]: pgmap v3018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:38:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:38:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:26 smithi016 ceph-mon[22010]: pgmap v3018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:28.179 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5209.967701 secs 2022-03-21T15:38:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5209.967701 secs 2022-03-21T15:38:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5209.967701 secs 2022-03-21T15:38:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: pgmap v3019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T15:38:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:38:29.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:38:29.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: pgmap v3019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T15:38:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:38:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:38:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: pgmap v3019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T15:38:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:38:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:31 smithi002 ceph-mon[17167]: pgmap v3020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:31 smithi016 ceph-mon[22010]: pgmap v3020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:31 smithi079 ceph-mon[23158]: pgmap v3020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:33 smithi002 ceph-mon[17167]: pgmap v3021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5214.967830 secs 2022-03-21T15:38:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:33 smithi016 ceph-mon[22010]: pgmap v3021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5214.967830 secs 2022-03-21T15:38:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:33 smithi079 ceph-mon[23158]: pgmap v3021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5214.967830 secs 2022-03-21T15:38:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:35 smithi002 ceph-mon[17167]: pgmap v3022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:35 smithi016 ceph-mon[22010]: pgmap v3022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:35 smithi079 ceph-mon[23158]: pgmap v3022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:36 smithi002 ceph-mon[17167]: pgmap v3023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:36 smithi016 ceph-mon[22010]: pgmap v3023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:36 smithi079 ceph-mon[23158]: pgmap v3023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5219.967935 secs 2022-03-21T15:38:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5219.967935 secs 2022-03-21T15:38:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5219.967935 secs 2022-03-21T15:38:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:39 smithi002 ceph-mon[17167]: pgmap v3024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:39 smithi016 ceph-mon[22010]: pgmap v3024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:39 smithi079 ceph-mon[23158]: pgmap v3024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:41 smithi002 ceph-mon[17167]: pgmap v3025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:41 smithi016 ceph-mon[22010]: pgmap v3025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:41 smithi079 ceph-mon[23158]: pgmap v3025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:43 smithi002 ceph-mon[17167]: pgmap v3026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5224.968045 secs 2022-03-21T15:38:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:43 smithi016 ceph-mon[22010]: pgmap v3026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5224.968045 secs 2022-03-21T15:38:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:43 smithi079 ceph-mon[23158]: pgmap v3026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5224.968045 secs 2022-03-21T15:38:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:45 smithi002 ceph-mon[17167]: pgmap v3027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:45 smithi016 ceph-mon[22010]: pgmap v3027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:45 smithi079 ceph-mon[23158]: pgmap v3027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:46 smithi002 ceph-mon[17167]: pgmap v3028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:46 smithi016 ceph-mon[22010]: pgmap v3028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:46 smithi079 ceph-mon[23158]: pgmap v3028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5229.968139 secs 2022-03-21T15:38:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5229.968139 secs 2022-03-21T15:38:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5229.968139 secs 2022-03-21T15:38:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:49 smithi002 ceph-mon[17167]: pgmap v3029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:49 smithi016 ceph-mon[22010]: pgmap v3029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:49 smithi079 ceph-mon[23158]: pgmap v3029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:51 smithi002 ceph-mon[17167]: pgmap v3030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:51 smithi016 ceph-mon[22010]: pgmap v3030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:51 smithi079 ceph-mon[23158]: pgmap v3030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:53 smithi002 ceph-mon[17167]: pgmap v3031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5234.968255 secs 2022-03-21T15:38:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:53 smithi016 ceph-mon[22010]: pgmap v3031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5234.968255 secs 2022-03-21T15:38:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:53 smithi079 ceph-mon[23158]: pgmap v3031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:38:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5234.968255 secs 2022-03-21T15:38:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:55 smithi002 ceph-mon[17167]: pgmap v3032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:55 smithi016 ceph-mon[22010]: pgmap v3032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:55 smithi079 ceph-mon[23158]: pgmap v3032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:38:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:56 smithi002 ceph-mon[17167]: pgmap v3033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:56 smithi016 ceph-mon[22010]: pgmap v3033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:56 smithi079 ceph-mon[23158]: pgmap v3033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:38:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5239.968388 secs 2022-03-21T15:38:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5239.968388 secs 2022-03-21T15:38:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5239.968388 secs 2022-03-21T15:38:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:38:59 smithi002 ceph-mon[17167]: pgmap v3034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:38:59 smithi016 ceph-mon[22010]: pgmap v3034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:38:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:38:59 smithi079 ceph-mon[23158]: pgmap v3034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:01 smithi002 ceph-mon[17167]: pgmap v3035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:01 smithi016 ceph-mon[22010]: pgmap v3035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:01 smithi079 ceph-mon[23158]: pgmap v3035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:03 smithi002 ceph-mon[17167]: pgmap v3036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5244.968487 secs 2022-03-21T15:39:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:03 smithi016 ceph-mon[22010]: pgmap v3036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5244.968487 secs 2022-03-21T15:39:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:03 smithi079 ceph-mon[23158]: pgmap v3036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5244.968487 secs 2022-03-21T15:39:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:05 smithi002 ceph-mon[17167]: pgmap v3037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:05 smithi016 ceph-mon[22010]: pgmap v3037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:05 smithi079 ceph-mon[23158]: pgmap v3037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:06 smithi002 ceph-mon[17167]: pgmap v3038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:06 smithi016 ceph-mon[22010]: pgmap v3038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:06 smithi079 ceph-mon[23158]: pgmap v3038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5249.968637 secs 2022-03-21T15:39:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5249.968637 secs 2022-03-21T15:39:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5249.968637 secs 2022-03-21T15:39:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:09 smithi002 ceph-mon[17167]: pgmap v3039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:09 smithi016 ceph-mon[22010]: pgmap v3039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:09 smithi079 ceph-mon[23158]: pgmap v3039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:11 smithi002 ceph-mon[17167]: pgmap v3040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:11 smithi016 ceph-mon[22010]: pgmap v3040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:11 smithi079 ceph-mon[23158]: pgmap v3040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:13 smithi002 ceph-mon[17167]: pgmap v3041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5254.968758 secs 2022-03-21T15:39:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:13 smithi016 ceph-mon[22010]: pgmap v3041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5254.968758 secs 2022-03-21T15:39:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:13 smithi079 ceph-mon[23158]: pgmap v3041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5254.968758 secs 2022-03-21T15:39:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:15 smithi002 ceph-mon[17167]: pgmap v3042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:15 smithi016 ceph-mon[22010]: pgmap v3042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:15 smithi079 ceph-mon[23158]: pgmap v3042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:16 smithi002 ceph-mon[17167]: pgmap v3043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:16 smithi016 ceph-mon[22010]: pgmap v3043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:16 smithi079 ceph-mon[23158]: pgmap v3043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5259.968873 secs 2022-03-21T15:39:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5259.968873 secs 2022-03-21T15:39:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5259.968873 secs 2022-03-21T15:39:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:19 smithi002 ceph-mon[17167]: pgmap v3044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:19 smithi016 ceph-mon[22010]: pgmap v3044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:19 smithi079 ceph-mon[23158]: pgmap v3044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:21 smithi002 ceph-mon[17167]: pgmap v3045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:39:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:39:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:21 smithi016 ceph-mon[22010]: pgmap v3045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:39:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:39:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:21 smithi079 ceph-mon[23158]: pgmap v3045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:39:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:39:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:23 smithi002 ceph-mon[17167]: pgmap v3046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5264.968974 secs 2022-03-21T15:39:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:23 smithi016 ceph-mon[22010]: pgmap v3046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5264.968974 secs 2022-03-21T15:39:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:23 smithi079 ceph-mon[23158]: pgmap v3046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5264.968974 secs 2022-03-21T15:39:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:25 smithi002 ceph-mon[17167]: pgmap v3047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:25 smithi016 ceph-mon[22010]: pgmap v3047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:25 smithi079 ceph-mon[23158]: pgmap v3047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:26 smithi002 ceph-mon[17167]: pgmap v3048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:26 smithi016 ceph-mon[22010]: pgmap v3048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:26 smithi079 ceph-mon[23158]: pgmap v3048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:28.309 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5269.969083 secs 2022-03-21T15:39:28.313 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5269.969083 secs 2022-03-21T15:39:28.317 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5269.969083 secs 2022-03-21T15:39:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:29 smithi002 ceph-mon[17167]: pgmap v3049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:39:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:39:29.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:39:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:29 smithi016 ceph-mon[22010]: pgmap v3049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:39:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:39:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:39:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:29 smithi079 ceph-mon[23158]: pgmap v3049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:39:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:39:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:39:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:39:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:39:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:39:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:31 smithi002 ceph-mon[17167]: pgmap v3050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:31 smithi016 ceph-mon[22010]: pgmap v3050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:31 smithi079 ceph-mon[23158]: pgmap v3050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:33 smithi002 ceph-mon[17167]: pgmap v3051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5274.969204 secs 2022-03-21T15:39:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:33 smithi016 ceph-mon[22010]: pgmap v3051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5274.969204 secs 2022-03-21T15:39:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:33 smithi079 ceph-mon[23158]: pgmap v3051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5274.969204 secs 2022-03-21T15:39:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:35 smithi002 ceph-mon[17167]: pgmap v3052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:35 smithi016 ceph-mon[22010]: pgmap v3052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:35 smithi079 ceph-mon[23158]: pgmap v3052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:36 smithi002 ceph-mon[17167]: pgmap v3053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:36 smithi016 ceph-mon[22010]: pgmap v3053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:36 smithi079 ceph-mon[23158]: pgmap v3053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5279.969275 secs 2022-03-21T15:39:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5279.969275 secs 2022-03-21T15:39:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5279.969275 secs 2022-03-21T15:39:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:39 smithi002 ceph-mon[17167]: pgmap v3054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:39 smithi016 ceph-mon[22010]: pgmap v3054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:39 smithi079 ceph-mon[23158]: pgmap v3054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:41 smithi002 ceph-mon[17167]: pgmap v3055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:41.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:41 smithi016 ceph-mon[22010]: pgmap v3055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:41 smithi079 ceph-mon[23158]: pgmap v3055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:43 smithi002 ceph-mon[17167]: pgmap v3056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5284.969400 secs 2022-03-21T15:39:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:43 smithi016 ceph-mon[22010]: pgmap v3056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5284.969400 secs 2022-03-21T15:39:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:43 smithi079 ceph-mon[23158]: pgmap v3056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5284.969400 secs 2022-03-21T15:39:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:45 smithi002 ceph-mon[17167]: pgmap v3057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:45.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:45 smithi016 ceph-mon[22010]: pgmap v3057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:45 smithi079 ceph-mon[23158]: pgmap v3057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:46 smithi002 ceph-mon[17167]: pgmap v3058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:46 smithi016 ceph-mon[22010]: pgmap v3058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:46 smithi079 ceph-mon[23158]: pgmap v3058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5289.969537 secs 2022-03-21T15:39:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5289.969537 secs 2022-03-21T15:39:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5289.969537 secs 2022-03-21T15:39:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:49 smithi002 ceph-mon[17167]: pgmap v3059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:49 smithi016 ceph-mon[22010]: pgmap v3059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:49 smithi079 ceph-mon[23158]: pgmap v3059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:51 smithi002 ceph-mon[17167]: pgmap v3060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:51.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:51 smithi016 ceph-mon[22010]: pgmap v3060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:51 smithi079 ceph-mon[23158]: pgmap v3060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:53 smithi002 ceph-mon[17167]: pgmap v3061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5294.969576 secs 2022-03-21T15:39:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:53 smithi016 ceph-mon[22010]: pgmap v3061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5294.969576 secs 2022-03-21T15:39:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:53 smithi079 ceph-mon[23158]: pgmap v3061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:39:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5294.969576 secs 2022-03-21T15:39:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:55 smithi002 ceph-mon[17167]: pgmap v3062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:55 smithi016 ceph-mon[22010]: pgmap v3062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:55 smithi079 ceph-mon[23158]: pgmap v3062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:39:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:56 smithi002 ceph-mon[17167]: pgmap v3063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:56 smithi016 ceph-mon[22010]: pgmap v3063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:56 smithi079 ceph-mon[23158]: pgmap v3063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:39:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5299.969684 secs 2022-03-21T15:39:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5299.969684 secs 2022-03-21T15:39:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5299.969684 secs 2022-03-21T15:39:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:39:59 smithi002 ceph-mon[17167]: pgmap v3064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:59.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:39:59 smithi016 ceph-mon[22010]: pgmap v3064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:39:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:39:59 smithi079 ceph-mon[23158]: pgmap v3064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:00.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:40:00.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:40:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:40:01.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:01 smithi002 ceph-mon[17167]: pgmap v3065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:01 smithi016 ceph-mon[22010]: pgmap v3065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:01 smithi079 ceph-mon[23158]: pgmap v3065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:03 smithi002 ceph-mon[17167]: pgmap v3066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5304.969801 secs 2022-03-21T15:40:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:03 smithi016 ceph-mon[22010]: pgmap v3066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5304.969801 secs 2022-03-21T15:40:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:03 smithi079 ceph-mon[23158]: pgmap v3066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5304.969801 secs 2022-03-21T15:40:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:05 smithi002 ceph-mon[17167]: pgmap v3067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:05 smithi016 ceph-mon[22010]: pgmap v3067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:05 smithi079 ceph-mon[23158]: pgmap v3067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:06 smithi002 ceph-mon[17167]: pgmap v3068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:06 smithi016 ceph-mon[22010]: pgmap v3068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:06 smithi079 ceph-mon[23158]: pgmap v3068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5309.969911 secs 2022-03-21T15:40:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5309.969911 secs 2022-03-21T15:40:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5309.969911 secs 2022-03-21T15:40:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:09 smithi002 ceph-mon[17167]: pgmap v3069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:09 smithi016 ceph-mon[22010]: pgmap v3069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:09 smithi079 ceph-mon[23158]: pgmap v3069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:11 smithi002 ceph-mon[17167]: pgmap v3070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:11 smithi016 ceph-mon[22010]: pgmap v3070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:11 smithi079 ceph-mon[23158]: pgmap v3070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:13 smithi002 ceph-mon[17167]: pgmap v3071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5314.970039 secs 2022-03-21T15:40:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:13 smithi016 ceph-mon[22010]: pgmap v3071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5314.970039 secs 2022-03-21T15:40:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:13 smithi079 ceph-mon[23158]: pgmap v3071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5314.970039 secs 2022-03-21T15:40:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:15 smithi002 ceph-mon[17167]: pgmap v3072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:15 smithi016 ceph-mon[22010]: pgmap v3072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:15 smithi079 ceph-mon[23158]: pgmap v3072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:16 smithi002 ceph-mon[17167]: pgmap v3073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:16 smithi016 ceph-mon[22010]: pgmap v3073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:16 smithi079 ceph-mon[23158]: pgmap v3073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5319.970158 secs 2022-03-21T15:40:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5319.970158 secs 2022-03-21T15:40:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5319.970158 secs 2022-03-21T15:40:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:19 smithi002 ceph-mon[17167]: pgmap v3074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:19 smithi016 ceph-mon[22010]: pgmap v3074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:19 smithi079 ceph-mon[23158]: pgmap v3074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:21 smithi002 ceph-mon[17167]: pgmap v3075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:40:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:40:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:21 smithi016 ceph-mon[22010]: pgmap v3075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:40:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:40:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:21 smithi079 ceph-mon[23158]: pgmap v3075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:40:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:40:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:23 smithi002 ceph-mon[17167]: pgmap v3076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5324.970296 secs 2022-03-21T15:40:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:23 smithi016 ceph-mon[22010]: pgmap v3076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5324.970296 secs 2022-03-21T15:40:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:23 smithi079 ceph-mon[23158]: pgmap v3076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5324.970296 secs 2022-03-21T15:40:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:25 smithi002 ceph-mon[17167]: pgmap v3077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:25.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:25 smithi016 ceph-mon[22010]: pgmap v3077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:25 smithi079 ceph-mon[23158]: pgmap v3077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:26 smithi002 ceph-mon[17167]: pgmap v3078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:26.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:26 smithi016 ceph-mon[22010]: pgmap v3078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:26 smithi079 ceph-mon[23158]: pgmap v3078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5329.970412 secs 2022-03-21T15:40:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5329.970412 secs 2022-03-21T15:40:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5329.970412 secs 2022-03-21T15:40:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:29 smithi002 ceph-mon[17167]: pgmap v3079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:29.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:29 smithi016 ceph-mon[22010]: pgmap v3079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:29 smithi079 ceph-mon[23158]: pgmap v3079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:40:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:40:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:40:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:40:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:40:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:40:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:40:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:40:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:40:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:40:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:40:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:40:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:31 smithi002 ceph-mon[17167]: pgmap v3080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:31.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:31 smithi016 ceph-mon[22010]: pgmap v3080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:31 smithi079 ceph-mon[23158]: pgmap v3080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:33 smithi002 ceph-mon[17167]: pgmap v3081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5334.970534 secs 2022-03-21T15:40:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:33 smithi016 ceph-mon[22010]: pgmap v3081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5334.970534 secs 2022-03-21T15:40:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:33 smithi079 ceph-mon[23158]: pgmap v3081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5334.970534 secs 2022-03-21T15:40:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:35 smithi002 ceph-mon[17167]: pgmap v3082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:35 smithi016 ceph-mon[22010]: pgmap v3082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:35 smithi079 ceph-mon[23158]: pgmap v3082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:36 smithi002 ceph-mon[17167]: pgmap v3083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:36.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:36 smithi016 ceph-mon[22010]: pgmap v3083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:36 smithi079 ceph-mon[23158]: pgmap v3083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5339.970671 secs 2022-03-21T15:40:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5339.970671 secs 2022-03-21T15:40:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5339.970671 secs 2022-03-21T15:40:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:39 smithi002 ceph-mon[17167]: pgmap v3084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:39.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:39 smithi016 ceph-mon[22010]: pgmap v3084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:39 smithi079 ceph-mon[23158]: pgmap v3084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:41 smithi002 ceph-mon[17167]: pgmap v3085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:41 smithi016 ceph-mon[22010]: pgmap v3085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:41 smithi079 ceph-mon[23158]: pgmap v3085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:43 smithi002 ceph-mon[17167]: pgmap v3086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5344.970765 secs 2022-03-21T15:40:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:43 smithi016 ceph-mon[22010]: pgmap v3086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5344.970765 secs 2022-03-21T15:40:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:43 smithi079 ceph-mon[23158]: pgmap v3086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5344.970765 secs 2022-03-21T15:40:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:45 smithi002 ceph-mon[17167]: pgmap v3087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:45 smithi016 ceph-mon[22010]: pgmap v3087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:45 smithi079 ceph-mon[23158]: pgmap v3087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:46 smithi002 ceph-mon[17167]: pgmap v3088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:46 smithi016 ceph-mon[22010]: pgmap v3088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:46 smithi079 ceph-mon[23158]: pgmap v3088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5349.970891 secs 2022-03-21T15:40:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5349.970891 secs 2022-03-21T15:40:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5349.970891 secs 2022-03-21T15:40:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:49 smithi002 ceph-mon[17167]: pgmap v3089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:49 smithi016 ceph-mon[22010]: pgmap v3089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:49 smithi079 ceph-mon[23158]: pgmap v3089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:51 smithi002 ceph-mon[17167]: pgmap v3090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:51 smithi016 ceph-mon[22010]: pgmap v3090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:51 smithi079 ceph-mon[23158]: pgmap v3090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:53 smithi002 ceph-mon[17167]: pgmap v3091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5354.970966 secs 2022-03-21T15:40:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:53 smithi016 ceph-mon[22010]: pgmap v3091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5354.970966 secs 2022-03-21T15:40:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:53 smithi079 ceph-mon[23158]: pgmap v3091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:40:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5354.970966 secs 2022-03-21T15:40:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:55 smithi002 ceph-mon[17167]: pgmap v3092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:55 smithi016 ceph-mon[22010]: pgmap v3092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:55 smithi079 ceph-mon[23158]: pgmap v3092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:40:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:56 smithi002 ceph-mon[17167]: pgmap v3093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:56 smithi016 ceph-mon[22010]: pgmap v3093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:56 smithi079 ceph-mon[23158]: pgmap v3093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:40:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5359.971073 secs 2022-03-21T15:40:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5359.971073 secs 2022-03-21T15:40:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5359.971073 secs 2022-03-21T15:40:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:40:59 smithi002 ceph-mon[17167]: pgmap v3094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:40:59 smithi016 ceph-mon[22010]: pgmap v3094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:40:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:40:59 smithi079 ceph-mon[23158]: pgmap v3094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:01 smithi002 ceph-mon[17167]: pgmap v3095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:01 smithi016 ceph-mon[22010]: pgmap v3095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:01 smithi079 ceph-mon[23158]: pgmap v3095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:03 smithi002 ceph-mon[17167]: pgmap v3096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5364.971198 secs 2022-03-21T15:41:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:03 smithi016 ceph-mon[22010]: pgmap v3096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5364.971198 secs 2022-03-21T15:41:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:03 smithi079 ceph-mon[23158]: pgmap v3096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5364.971198 secs 2022-03-21T15:41:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:05 smithi002 ceph-mon[17167]: pgmap v3097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:05 smithi016 ceph-mon[22010]: pgmap v3097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:05 smithi079 ceph-mon[23158]: pgmap v3097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:06 smithi002 ceph-mon[17167]: pgmap v3098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:06 smithi016 ceph-mon[22010]: pgmap v3098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:06 smithi079 ceph-mon[23158]: pgmap v3098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5369.971292 secs 2022-03-21T15:41:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5369.971292 secs 2022-03-21T15:41:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5369.971292 secs 2022-03-21T15:41:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:09 smithi002 ceph-mon[17167]: pgmap v3099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:09.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:09 smithi016 ceph-mon[22010]: pgmap v3099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:09 smithi079 ceph-mon[23158]: pgmap v3099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:11 smithi002 ceph-mon[17167]: pgmap v3100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:11 smithi016 ceph-mon[22010]: pgmap v3100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:11 smithi079 ceph-mon[23158]: pgmap v3100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:13 smithi002 ceph-mon[17167]: pgmap v3101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5374.971427 secs 2022-03-21T15:41:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:13 smithi016 ceph-mon[22010]: pgmap v3101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5374.971427 secs 2022-03-21T15:41:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:13 smithi079 ceph-mon[23158]: pgmap v3101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5374.971427 secs 2022-03-21T15:41:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:15 smithi002 ceph-mon[17167]: pgmap v3102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:15.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:15 smithi016 ceph-mon[22010]: pgmap v3102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:15 smithi079 ceph-mon[23158]: pgmap v3102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:16 smithi002 ceph-mon[17167]: pgmap v3103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:16 smithi016 ceph-mon[22010]: pgmap v3103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:16 smithi079 ceph-mon[23158]: pgmap v3103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5379.971561 secs 2022-03-21T15:41:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5379.971561 secs 2022-03-21T15:41:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5379.971561 secs 2022-03-21T15:41:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:19 smithi002 ceph-mon[17167]: pgmap v3104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:19 smithi016 ceph-mon[22010]: pgmap v3104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:19 smithi079 ceph-mon[23158]: pgmap v3104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:21 smithi002 ceph-mon[17167]: pgmap v3105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:41:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:41:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:21 smithi016 ceph-mon[22010]: pgmap v3105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:41:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:41:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:21 smithi079 ceph-mon[23158]: pgmap v3105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:41:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:41:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:23 smithi002 ceph-mon[17167]: pgmap v3106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5384.971681 secs 2022-03-21T15:41:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:23 smithi016 ceph-mon[22010]: pgmap v3106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5384.971681 secs 2022-03-21T15:41:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:23 smithi079 ceph-mon[23158]: pgmap v3106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5384.971681 secs 2022-03-21T15:41:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:25 smithi002 ceph-mon[17167]: pgmap v3107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:25 smithi016 ceph-mon[22010]: pgmap v3107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:25 smithi079 ceph-mon[23158]: pgmap v3107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:26 smithi002 ceph-mon[17167]: pgmap v3108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:26 smithi016 ceph-mon[22010]: pgmap v3108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:26 smithi079 ceph-mon[23158]: pgmap v3108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5389.971772 secs 2022-03-21T15:41:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5389.971772 secs 2022-03-21T15:41:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5389.971772 secs 2022-03-21T15:41:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:29 smithi002 ceph-mon[17167]: pgmap v3109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:29 smithi016 ceph-mon[22010]: pgmap v3109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:29 smithi079 ceph-mon[23158]: pgmap v3109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:30.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:41:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:41:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:30 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:41:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:41:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:41:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:30 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:41:30.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:41:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:41:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:30 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:41:31.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:31 smithi002 ceph-mon[17167]: pgmap v3110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:31 smithi016 ceph-mon[22010]: pgmap v3110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:31.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:31 smithi079 ceph-mon[23158]: pgmap v3110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:33 smithi002 ceph-mon[17167]: pgmap v3111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5394.971897 secs 2022-03-21T15:41:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:33 smithi016 ceph-mon[22010]: pgmap v3111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5394.971897 secs 2022-03-21T15:41:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:33 smithi079 ceph-mon[23158]: pgmap v3111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5394.971897 secs 2022-03-21T15:41:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:34 smithi002 ceph-mon[17167]: pgmap v3112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:34 smithi016 ceph-mon[22010]: pgmap v3112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:34 smithi079 ceph-mon[23158]: pgmap v3112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:41:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:36 smithi002 ceph-mon[17167]: pgmap v3113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:36 smithi016 ceph-mon[22010]: pgmap v3113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:36 smithi079 ceph-mon[23158]: pgmap v3113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5399.972011 secs 2022-03-21T15:41:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5399.972011 secs 2022-03-21T15:41:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5399.972011 secs 2022-03-21T15:41:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:39 smithi002 ceph-mon[17167]: pgmap v3114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:39 smithi016 ceph-mon[22010]: pgmap v3114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:39 smithi079 ceph-mon[23158]: pgmap v3114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:41 smithi002 ceph-mon[17167]: pgmap v3115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:41 smithi016 ceph-mon[22010]: pgmap v3115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:41 smithi079 ceph-mon[23158]: pgmap v3115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:43 smithi002 ceph-mon[17167]: pgmap v3116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5404.972133 secs 2022-03-21T15:41:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:43 smithi016 ceph-mon[22010]: pgmap v3116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5404.972133 secs 2022-03-21T15:41:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:43 smithi079 ceph-mon[23158]: pgmap v3116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5404.972133 secs 2022-03-21T15:41:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:45 smithi002 ceph-mon[17167]: pgmap v3117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:45 smithi016 ceph-mon[22010]: pgmap v3117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:45 smithi079 ceph-mon[23158]: pgmap v3117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:46 smithi002 ceph-mon[17167]: pgmap v3118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:46 smithi016 ceph-mon[22010]: pgmap v3118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:46 smithi079 ceph-mon[23158]: pgmap v3118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5409.972264 secs 2022-03-21T15:41:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5409.972264 secs 2022-03-21T15:41:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5409.972264 secs 2022-03-21T15:41:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:49 smithi002 ceph-mon[17167]: pgmap v3119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:49 smithi016 ceph-mon[22010]: pgmap v3119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:49 smithi079 ceph-mon[23158]: pgmap v3119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:51 smithi002 ceph-mon[17167]: pgmap v3120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:51 smithi016 ceph-mon[22010]: pgmap v3120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:51 smithi079 ceph-mon[23158]: pgmap v3120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:53 smithi002 ceph-mon[17167]: pgmap v3121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5414.972390 secs 2022-03-21T15:41:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:53 smithi016 ceph-mon[22010]: pgmap v3121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5414.972390 secs 2022-03-21T15:41:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:53 smithi079 ceph-mon[23158]: pgmap v3121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:41:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5414.972390 secs 2022-03-21T15:41:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:55 smithi002 ceph-mon[17167]: pgmap v3122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:55.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:55 smithi016 ceph-mon[22010]: pgmap v3122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:55 smithi079 ceph-mon[23158]: pgmap v3122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:41:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:56 smithi002 ceph-mon[17167]: pgmap v3123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:56 smithi016 ceph-mon[22010]: pgmap v3123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:56 smithi079 ceph-mon[23158]: pgmap v3123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:41:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5419.972527 secs 2022-03-21T15:41:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5419.972527 secs 2022-03-21T15:41:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5419.972527 secs 2022-03-21T15:41:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:41:59 smithi002 ceph-mon[17167]: pgmap v3124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:59.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:41:59 smithi016 ceph-mon[22010]: pgmap v3124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:41:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:41:59 smithi079 ceph-mon[23158]: pgmap v3124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:01 smithi002 ceph-mon[17167]: pgmap v3125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:01 smithi016 ceph-mon[22010]: pgmap v3125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:01 smithi079 ceph-mon[23158]: pgmap v3125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:03 smithi002 ceph-mon[17167]: pgmap v3126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5424.972624 secs 2022-03-21T15:42:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:03 smithi016 ceph-mon[22010]: pgmap v3126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5424.972624 secs 2022-03-21T15:42:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:03 smithi079 ceph-mon[23158]: pgmap v3126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5424.972624 secs 2022-03-21T15:42:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:05 smithi002 ceph-mon[17167]: pgmap v3127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:05 smithi016 ceph-mon[22010]: pgmap v3127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:05 smithi079 ceph-mon[23158]: pgmap v3127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:06 smithi002 ceph-mon[17167]: pgmap v3128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:06 smithi016 ceph-mon[22010]: pgmap v3128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:06 smithi079 ceph-mon[23158]: pgmap v3128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5429.972707 secs 2022-03-21T15:42:08.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5429.972707 secs 2022-03-21T15:42:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5429.972707 secs 2022-03-21T15:42:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:09 smithi002 ceph-mon[17167]: pgmap v3129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:09 smithi016 ceph-mon[22010]: pgmap v3129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:09 smithi079 ceph-mon[23158]: pgmap v3129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:11 smithi002 ceph-mon[17167]: pgmap v3130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:11 smithi016 ceph-mon[22010]: pgmap v3130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:11 smithi079 ceph-mon[23158]: pgmap v3130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:13 smithi002 ceph-mon[17167]: pgmap v3131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5434.972832 secs 2022-03-21T15:42:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:13 smithi016 ceph-mon[22010]: pgmap v3131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5434.972832 secs 2022-03-21T15:42:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:13 smithi079 ceph-mon[23158]: pgmap v3131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5434.972832 secs 2022-03-21T15:42:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:15 smithi002 ceph-mon[17167]: pgmap v3132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:15 smithi016 ceph-mon[22010]: pgmap v3132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:15 smithi079 ceph-mon[23158]: pgmap v3132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:16 smithi002 ceph-mon[17167]: pgmap v3133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:16.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:16 smithi016 ceph-mon[22010]: pgmap v3133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:16 smithi079 ceph-mon[23158]: pgmap v3133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5439.972952 secs 2022-03-21T15:42:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5439.972952 secs 2022-03-21T15:42:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5439.972952 secs 2022-03-21T15:42:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:19 smithi002 ceph-mon[17167]: pgmap v3134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:19 smithi016 ceph-mon[22010]: pgmap v3134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:19 smithi079 ceph-mon[23158]: pgmap v3134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:21 smithi002 ceph-mon[17167]: pgmap v3135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:42:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:42:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:21 smithi016 ceph-mon[22010]: pgmap v3135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:42:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:42:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:21 smithi079 ceph-mon[23158]: pgmap v3135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:42:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:42:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:23 smithi002 ceph-mon[17167]: pgmap v3136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5444.973096 secs 2022-03-21T15:42:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:23 smithi016 ceph-mon[22010]: pgmap v3136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5444.973096 secs 2022-03-21T15:42:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:23 smithi079 ceph-mon[23158]: pgmap v3136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5444.973096 secs 2022-03-21T15:42:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:25 smithi002 ceph-mon[17167]: pgmap v3137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:25 smithi016 ceph-mon[22010]: pgmap v3137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:25 smithi079 ceph-mon[23158]: pgmap v3137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:26 smithi002 ceph-mon[17167]: pgmap v3138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:26 smithi016 ceph-mon[22010]: pgmap v3138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:26 smithi079 ceph-mon[23158]: pgmap v3138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5449.973230 secs 2022-03-21T15:42:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5449.973230 secs 2022-03-21T15:42:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5449.973230 secs 2022-03-21T15:42:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:29 smithi002 ceph-mon[17167]: pgmap v3139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:29 smithi016 ceph-mon[22010]: pgmap v3139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:29 smithi079 ceph-mon[23158]: pgmap v3139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:31 smithi002 ceph-mon[17167]: pgmap v3140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:31 smithi016 ceph-mon[22010]: pgmap v3140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:31 smithi079 ceph-mon[23158]: pgmap v3140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:33.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:33 smithi002 ceph-mon[17167]: pgmap v3141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5454.973345 secs 2022-03-21T15:42:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:33 smithi016 ceph-mon[22010]: pgmap v3141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5454.973345 secs 2022-03-21T15:42:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:33 smithi079 ceph-mon[23158]: pgmap v3141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5454.973345 secs 2022-03-21T15:42:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:35 smithi002 ceph-mon[17167]: pgmap v3142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:42:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:42:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:42:35.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:35 smithi016 ceph-mon[22010]: pgmap v3142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:42:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:42:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:42:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:35 smithi079 ceph-mon[23158]: pgmap v3142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:42:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:42:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:42:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:42:36.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:36 smithi002 ceph-mon[17167]: pgmap v3143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:42:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:36 smithi016 ceph-mon[22010]: pgmap v3143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:42:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:36 smithi079 ceph-mon[23158]: pgmap v3143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5459.973493 secs 2022-03-21T15:42:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5459.973493 secs 2022-03-21T15:42:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5459.973493 secs 2022-03-21T15:42:39.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:39 smithi002 ceph-mon[17167]: pgmap v3144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:39 smithi016 ceph-mon[22010]: pgmap v3144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:39 smithi079 ceph-mon[23158]: pgmap v3144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:41 smithi002 ceph-mon[17167]: pgmap v3145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:41.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:41 smithi016 ceph-mon[22010]: pgmap v3145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:41 smithi079 ceph-mon[23158]: pgmap v3145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:43 smithi002 ceph-mon[17167]: pgmap v3146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5464.973622 secs 2022-03-21T15:42:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:43 smithi016 ceph-mon[22010]: pgmap v3146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5464.973622 secs 2022-03-21T15:42:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:43 smithi079 ceph-mon[23158]: pgmap v3146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5464.973622 secs 2022-03-21T15:42:45.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:45 smithi002 ceph-mon[17167]: pgmap v3147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:45 smithi016 ceph-mon[22010]: pgmap v3147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:45 smithi079 ceph-mon[23158]: pgmap v3147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:46 smithi002 ceph-mon[17167]: pgmap v3148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:46 smithi016 ceph-mon[22010]: pgmap v3148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:46 smithi079 ceph-mon[23158]: pgmap v3148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5469.973767 secs 2022-03-21T15:42:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5469.973767 secs 2022-03-21T15:42:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5469.973767 secs 2022-03-21T15:42:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:49 smithi002 ceph-mon[17167]: pgmap v3149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:49.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:49 smithi016 ceph-mon[22010]: pgmap v3149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:49 smithi079 ceph-mon[23158]: pgmap v3149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:51 smithi002 ceph-mon[17167]: pgmap v3150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:51 smithi016 ceph-mon[22010]: pgmap v3150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:51 smithi079 ceph-mon[23158]: pgmap v3150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:53.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:53 smithi002 ceph-mon[17167]: pgmap v3151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5474.973890 secs 2022-03-21T15:42:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:53 smithi016 ceph-mon[22010]: pgmap v3151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5474.973890 secs 2022-03-21T15:42:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:53 smithi079 ceph-mon[23158]: pgmap v3151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:42:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5474.973890 secs 2022-03-21T15:42:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:55 smithi002 ceph-mon[17167]: pgmap v3152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:55 smithi016 ceph-mon[22010]: pgmap v3152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:55 smithi079 ceph-mon[23158]: pgmap v3152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:42:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:56 smithi002 ceph-mon[17167]: pgmap v3153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:56 smithi016 ceph-mon[22010]: pgmap v3153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:56 smithi079 ceph-mon[23158]: pgmap v3153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:42:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5479.974009 secs 2022-03-21T15:42:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5479.974009 secs 2022-03-21T15:42:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5479.974009 secs 2022-03-21T15:42:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:42:59 smithi002 ceph-mon[17167]: pgmap v3154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:42:59 smithi016 ceph-mon[22010]: pgmap v3154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:42:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:42:59 smithi079 ceph-mon[23158]: pgmap v3154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:01 smithi002 ceph-mon[17167]: pgmap v3155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:01 smithi016 ceph-mon[22010]: pgmap v3155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:01 smithi079 ceph-mon[23158]: pgmap v3155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:03 smithi002 ceph-mon[17167]: pgmap v3156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5484.974105 secs 2022-03-21T15:43:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:03 smithi016 ceph-mon[22010]: pgmap v3156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5484.974105 secs 2022-03-21T15:43:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:03 smithi079 ceph-mon[23158]: pgmap v3156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5484.974105 secs 2022-03-21T15:43:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:05 smithi002 ceph-mon[17167]: pgmap v3157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:05.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:05 smithi016 ceph-mon[22010]: pgmap v3157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:05 smithi079 ceph-mon[23158]: pgmap v3157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:06 smithi002 ceph-mon[17167]: pgmap v3158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:06 smithi016 ceph-mon[22010]: pgmap v3158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:06 smithi079 ceph-mon[23158]: pgmap v3158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:08.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5489.974221 secs 2022-03-21T15:43:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5489.974221 secs 2022-03-21T15:43:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5489.974221 secs 2022-03-21T15:43:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:09 smithi002 ceph-mon[17167]: pgmap v3159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:09.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:09 smithi016 ceph-mon[22010]: pgmap v3159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:09 smithi079 ceph-mon[23158]: pgmap v3159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:11 smithi002 ceph-mon[17167]: pgmap v3160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:11 smithi016 ceph-mon[22010]: pgmap v3160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:11 smithi079 ceph-mon[23158]: pgmap v3160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:13 smithi002 ceph-mon[17167]: pgmap v3161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5494.974283 secs 2022-03-21T15:43:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:13 smithi016 ceph-mon[22010]: pgmap v3161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5494.974283 secs 2022-03-21T15:43:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:13 smithi079 ceph-mon[23158]: pgmap v3161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5494.974283 secs 2022-03-21T15:43:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:15 smithi002 ceph-mon[17167]: pgmap v3162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:15 smithi016 ceph-mon[22010]: pgmap v3162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:15 smithi079 ceph-mon[23158]: pgmap v3162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:16.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:16 smithi002 ceph-mon[17167]: pgmap v3163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:16 smithi016 ceph-mon[22010]: pgmap v3163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:16 smithi079 ceph-mon[23158]: pgmap v3163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5499.974408 secs 2022-03-21T15:43:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5499.974408 secs 2022-03-21T15:43:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5499.974408 secs 2022-03-21T15:43:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:19 smithi002 ceph-mon[17167]: pgmap v3164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:19 smithi016 ceph-mon[22010]: pgmap v3164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:19 smithi079 ceph-mon[23158]: pgmap v3164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:21 smithi002 ceph-mon[17167]: pgmap v3165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:43:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:43:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:21 smithi016 ceph-mon[22010]: pgmap v3165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:43:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:43:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:21 smithi079 ceph-mon[23158]: pgmap v3165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:43:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:43:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:23 smithi002 ceph-mon[17167]: pgmap v3166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5504.974527 secs 2022-03-21T15:43:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:23 smithi016 ceph-mon[22010]: pgmap v3166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5504.974527 secs 2022-03-21T15:43:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:23 smithi079 ceph-mon[23158]: pgmap v3166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5504.974527 secs 2022-03-21T15:43:25.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:25 smithi002 ceph-mon[17167]: pgmap v3167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:25 smithi016 ceph-mon[22010]: pgmap v3167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:25 smithi079 ceph-mon[23158]: pgmap v3167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:26.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:26 smithi002 ceph-mon[17167]: pgmap v3168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:26 smithi016 ceph-mon[22010]: pgmap v3168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:26 smithi079 ceph-mon[23158]: pgmap v3168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5509.974641 secs 2022-03-21T15:43:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5509.974641 secs 2022-03-21T15:43:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5509.974641 secs 2022-03-21T15:43:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:29 smithi002 ceph-mon[17167]: pgmap v3169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:29 smithi016 ceph-mon[22010]: pgmap v3169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:29 smithi079 ceph-mon[23158]: pgmap v3169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:31 smithi002 ceph-mon[17167]: pgmap v3170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:31 smithi016 ceph-mon[22010]: pgmap v3170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:31 smithi079 ceph-mon[23158]: pgmap v3170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:33 smithi002 ceph-mon[17167]: pgmap v3171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5514.974768 secs 2022-03-21T15:43:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:33 smithi016 ceph-mon[22010]: pgmap v3171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5514.974768 secs 2022-03-21T15:43:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:33 smithi079 ceph-mon[23158]: pgmap v3171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5514.974768 secs 2022-03-21T15:43:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:35 smithi002 ceph-mon[17167]: pgmap v3172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:35.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:35 smithi016 ceph-mon[22010]: pgmap v3172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:35 smithi079 ceph-mon[23158]: pgmap v3172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:36.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:43:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:43:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:43:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:43:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:43:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:43:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:43:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:43:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:43:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:37.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:37 smithi002 ceph-mon[17167]: pgmap v3173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:37.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:37 smithi016 ceph-mon[22010]: pgmap v3173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:37 smithi079 ceph-mon[23158]: pgmap v3173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:43:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5519.974892 secs 2022-03-21T15:43:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5519.974892 secs 2022-03-21T15:43:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5519.974892 secs 2022-03-21T15:43:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:39 smithi002 ceph-mon[17167]: pgmap v3174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:39 smithi016 ceph-mon[22010]: pgmap v3174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:39 smithi079 ceph-mon[23158]: pgmap v3174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:41 smithi002 ceph-mon[17167]: pgmap v3175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:41 smithi016 ceph-mon[22010]: pgmap v3175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:41 smithi079 ceph-mon[23158]: pgmap v3175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:43 smithi016 ceph-mon[22010]: pgmap v3176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5524.975021 secs 2022-03-21T15:43:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:43 smithi079 ceph-mon[23158]: pgmap v3176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5524.975021 secs 2022-03-21T15:43:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:43 smithi002 ceph-mon[17167]: pgmap v3176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5524.975021 secs 2022-03-21T15:43:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:45 smithi016 ceph-mon[22010]: pgmap v3177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:45 smithi079 ceph-mon[23158]: pgmap v3177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:45 smithi002 ceph-mon[17167]: pgmap v3177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:46 smithi002 ceph-mon[17167]: pgmap v3178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:46 smithi016 ceph-mon[22010]: pgmap v3178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:46 smithi079 ceph-mon[23158]: pgmap v3178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5529.975138 secs 2022-03-21T15:43:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5529.975138 secs 2022-03-21T15:43:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5529.975138 secs 2022-03-21T15:43:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:49 smithi002 ceph-mon[17167]: pgmap v3179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:49 smithi016 ceph-mon[22010]: pgmap v3179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:49 smithi079 ceph-mon[23158]: pgmap v3179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:51 smithi002 ceph-mon[17167]: pgmap v3180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:51 smithi016 ceph-mon[22010]: pgmap v3180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:51 smithi079 ceph-mon[23158]: pgmap v3180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:53 smithi002 ceph-mon[17167]: pgmap v3181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5534.975280 secs 2022-03-21T15:43:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:53 smithi016 ceph-mon[22010]: pgmap v3181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5534.975280 secs 2022-03-21T15:43:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:53 smithi079 ceph-mon[23158]: pgmap v3181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:43:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5534.975280 secs 2022-03-21T15:43:55.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:55 smithi002 ceph-mon[17167]: pgmap v3182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:55.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:55 smithi016 ceph-mon[22010]: pgmap v3182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:55 smithi079 ceph-mon[23158]: pgmap v3182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:43:56.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:56 smithi002 ceph-mon[17167]: pgmap v3183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:56 smithi016 ceph-mon[22010]: pgmap v3183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:56 smithi079 ceph-mon[23158]: pgmap v3183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:43:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5539.975360 secs 2022-03-21T15:43:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5539.975360 secs 2022-03-21T15:43:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5539.975360 secs 2022-03-21T15:43:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:43:59 smithi002 ceph-mon[17167]: pgmap v3184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:59.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:43:59 smithi016 ceph-mon[22010]: pgmap v3184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:43:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:43:59 smithi079 ceph-mon[23158]: pgmap v3184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:01 smithi002 ceph-mon[17167]: pgmap v3185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:01 smithi016 ceph-mon[22010]: pgmap v3185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:01 smithi079 ceph-mon[23158]: pgmap v3185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:03.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5544.975501 secs 2022-03-21T15:44:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:03 smithi002 ceph-mon[17167]: pgmap v3186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5544.975501 secs 2022-03-21T15:44:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:03 smithi016 ceph-mon[22010]: pgmap v3186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5544.975501 secs 2022-03-21T15:44:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:03 smithi079 ceph-mon[23158]: pgmap v3186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:05.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:05 smithi002 ceph-mon[17167]: pgmap v3187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:05 smithi016 ceph-mon[22010]: pgmap v3187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:05 smithi079 ceph-mon[23158]: pgmap v3187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:06.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:06 smithi002 ceph-mon[17167]: pgmap v3188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:06.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:06 smithi016 ceph-mon[22010]: pgmap v3188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:06 smithi079 ceph-mon[23158]: pgmap v3188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5549.975636 secs 2022-03-21T15:44:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5549.975636 secs 2022-03-21T15:44:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5549.975636 secs 2022-03-21T15:44:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:09 smithi002 ceph-mon[17167]: pgmap v3189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:09 smithi016 ceph-mon[22010]: pgmap v3189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:09 smithi079 ceph-mon[23158]: pgmap v3189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:11.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:11 smithi002 ceph-mon[17167]: pgmap v3190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:11 smithi016 ceph-mon[22010]: pgmap v3190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:11 smithi079 ceph-mon[23158]: pgmap v3190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:13.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5554.975739 secs 2022-03-21T15:44:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:13 smithi002 ceph-mon[17167]: pgmap v3191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5554.975739 secs 2022-03-21T15:44:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:13 smithi016 ceph-mon[22010]: pgmap v3191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5554.975739 secs 2022-03-21T15:44:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:13 smithi079 ceph-mon[23158]: pgmap v3191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:15.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:15 smithi002 ceph-mon[17167]: pgmap v3192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:15.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:15 smithi016 ceph-mon[22010]: pgmap v3192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:15 smithi079 ceph-mon[23158]: pgmap v3192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:16 smithi016 ceph-mon[22010]: pgmap v3193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:16 smithi079 ceph-mon[23158]: pgmap v3193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:16 smithi002 ceph-mon[17167]: pgmap v3193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5559.975871 secs 2022-03-21T15:44:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5559.975871 secs 2022-03-21T15:44:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5559.975871 secs 2022-03-21T15:44:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:19 smithi002 ceph-mon[17167]: pgmap v3194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:19 smithi016 ceph-mon[22010]: pgmap v3194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:19 smithi079 ceph-mon[23158]: pgmap v3194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:21.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:21 smithi002 ceph-mon[17167]: pgmap v3195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:44:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:44:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:21 smithi016 ceph-mon[22010]: pgmap v3195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:44:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:44:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:21 smithi079 ceph-mon[23158]: pgmap v3195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:44:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:44:23.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5564.975971 secs 2022-03-21T15:44:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:23 smithi002 ceph-mon[17167]: pgmap v3196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5564.975971 secs 2022-03-21T15:44:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:23 smithi016 ceph-mon[22010]: pgmap v3196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5564.975971 secs 2022-03-21T15:44:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:23 smithi079 ceph-mon[23158]: pgmap v3196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:25 smithi002 ceph-mon[17167]: pgmap v3197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:25.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:25 smithi016 ceph-mon[22010]: pgmap v3197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:25 smithi079 ceph-mon[23158]: pgmap v3197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:26 smithi016 ceph-mon[22010]: pgmap v3198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:26 smithi079 ceph-mon[23158]: pgmap v3198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:26 smithi002 ceph-mon[17167]: pgmap v3198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5569.976092 secs 2022-03-21T15:44:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5569.976092 secs 2022-03-21T15:44:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5569.976092 secs 2022-03-21T15:44:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:29 smithi002 ceph-mon[17167]: pgmap v3199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:29 smithi016 ceph-mon[22010]: pgmap v3199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:29 smithi079 ceph-mon[23158]: pgmap v3199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:31.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:31 smithi002 ceph-mon[17167]: pgmap v3200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:31 smithi016 ceph-mon[22010]: pgmap v3200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:31 smithi079 ceph-mon[23158]: pgmap v3200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5574.976212 secs 2022-03-21T15:44:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:33 smithi002 ceph-mon[17167]: pgmap v3201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5574.976212 secs 2022-03-21T15:44:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:33 smithi016 ceph-mon[22010]: pgmap v3201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5574.976212 secs 2022-03-21T15:44:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:33 smithi079 ceph-mon[23158]: pgmap v3201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:35.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:35 smithi002 ceph-mon[17167]: pgmap v3202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:35 smithi016 ceph-mon[22010]: pgmap v3202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:35 smithi079 ceph-mon[23158]: pgmap v3202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:36 smithi016 ceph-mon[22010]: pgmap v3203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:36 smithi079 ceph-mon[23158]: pgmap v3203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:37.037 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:36 smithi002 ceph-mon[17167]: pgmap v3203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:37.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:44:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:44:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:44:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:44:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:44:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:44:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:44:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:44:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:44:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5579.976321 secs 2022-03-21T15:44:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:44:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:38 smithi002 ceph-mon[17167]: pgmap v3204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5579.976321 secs 2022-03-21T15:44:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:44:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:38 smithi016 ceph-mon[22010]: pgmap v3204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5579.976321 secs 2022-03-21T15:44:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:44:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:38 smithi079 ceph-mon[23158]: pgmap v3204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:41 smithi002 ceph-mon[17167]: pgmap v3205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:41.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:41 smithi016 ceph-mon[22010]: pgmap v3205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:41 smithi079 ceph-mon[23158]: pgmap v3205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5584.976451 secs 2022-03-21T15:44:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:43 smithi002 ceph-mon[17167]: pgmap v3206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5584.976451 secs 2022-03-21T15:44:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:43 smithi016 ceph-mon[22010]: pgmap v3206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5584.976451 secs 2022-03-21T15:44:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:43 smithi079 ceph-mon[23158]: pgmap v3206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:45 smithi016 ceph-mon[22010]: pgmap v3207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:45 smithi079 ceph-mon[23158]: pgmap v3207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:45 smithi002 ceph-mon[17167]: pgmap v3207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:46 smithi016 ceph-mon[22010]: pgmap v3208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:46 smithi079 ceph-mon[23158]: pgmap v3208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:46 smithi002 ceph-mon[17167]: pgmap v3208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5589.976568 secs 2022-03-21T15:44:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5589.976568 secs 2022-03-21T15:44:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5589.976568 secs 2022-03-21T15:44:49.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:49 smithi002 ceph-mon[17167]: pgmap v3209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:49 smithi016 ceph-mon[22010]: pgmap v3209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:49 smithi079 ceph-mon[23158]: pgmap v3209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:51 smithi002 ceph-mon[17167]: pgmap v3210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:51.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:51 smithi016 ceph-mon[22010]: pgmap v3210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:51 smithi079 ceph-mon[23158]: pgmap v3210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5594.976668 secs 2022-03-21T15:44:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:53 smithi016 ceph-mon[22010]: pgmap v3211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5594.976668 secs 2022-03-21T15:44:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:53 smithi079 ceph-mon[23158]: pgmap v3211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5594.976668 secs 2022-03-21T15:44:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:53 smithi002 ceph-mon[17167]: pgmap v3211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:44:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:55 smithi016 ceph-mon[22010]: pgmap v3212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:55 smithi079 ceph-mon[23158]: pgmap v3212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:55 smithi002 ceph-mon[17167]: pgmap v3212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:44:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:56 smithi016 ceph-mon[22010]: pgmap v3213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:56 smithi079 ceph-mon[23158]: pgmap v3213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:56 smithi002 ceph-mon[17167]: pgmap v3213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:44:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5599.976788 secs 2022-03-21T15:44:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5599.976788 secs 2022-03-21T15:44:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5599.976788 secs 2022-03-21T15:44:59.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:44:59 smithi002 ceph-mon[17167]: pgmap v3214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:44:59 smithi016 ceph-mon[22010]: pgmap v3214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:44:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:44:59 smithi079 ceph-mon[23158]: pgmap v3214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:01 smithi016 ceph-mon[22010]: pgmap v3215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:01 smithi079 ceph-mon[23158]: pgmap v3215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:01 smithi002 ceph-mon[17167]: pgmap v3215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5604.976887 secs 2022-03-21T15:45:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:03 smithi016 ceph-mon[22010]: pgmap v3216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5604.976887 secs 2022-03-21T15:45:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:03 smithi079 ceph-mon[23158]: pgmap v3216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5604.976887 secs 2022-03-21T15:45:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:03 smithi002 ceph-mon[17167]: pgmap v3216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:05 smithi016 ceph-mon[22010]: pgmap v3217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:05 smithi079 ceph-mon[23158]: pgmap v3217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:05 smithi002 ceph-mon[17167]: pgmap v3217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:06 smithi016 ceph-mon[22010]: pgmap v3218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:06 smithi079 ceph-mon[23158]: pgmap v3218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:06 smithi002 ceph-mon[17167]: pgmap v3218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5609.977004 secs 2022-03-21T15:45:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5609.977004 secs 2022-03-21T15:45:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5609.977004 secs 2022-03-21T15:45:09.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:09 smithi002 ceph-mon[17167]: pgmap v3219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:09 smithi016 ceph-mon[22010]: pgmap v3219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:09 smithi079 ceph-mon[23158]: pgmap v3219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:11 smithi016 ceph-mon[22010]: pgmap v3220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:11 smithi079 ceph-mon[23158]: pgmap v3220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:11 smithi002 ceph-mon[17167]: pgmap v3220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5614.977127 secs 2022-03-21T15:45:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:13 smithi016 ceph-mon[22010]: pgmap v3221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5614.977127 secs 2022-03-21T15:45:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:13 smithi079 ceph-mon[23158]: pgmap v3221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5614.977127 secs 2022-03-21T15:45:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:13 smithi002 ceph-mon[17167]: pgmap v3221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:15 smithi016 ceph-mon[22010]: pgmap v3222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:15 smithi079 ceph-mon[23158]: pgmap v3222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:15 smithi002 ceph-mon[17167]: pgmap v3222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:16 smithi016 ceph-mon[22010]: pgmap v3223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:16 smithi079 ceph-mon[23158]: pgmap v3223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:16 smithi002 ceph-mon[17167]: pgmap v3223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5619.977248 secs 2022-03-21T15:45:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5619.977248 secs 2022-03-21T15:45:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5619.977248 secs 2022-03-21T15:45:19.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:19 smithi002 ceph-mon[17167]: pgmap v3224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:19 smithi016 ceph-mon[22010]: pgmap v3224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:19 smithi079 ceph-mon[23158]: pgmap v3224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:21 smithi016 ceph-mon[22010]: pgmap v3225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:45:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:45:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:21 smithi079 ceph-mon[23158]: pgmap v3225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:45:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:45:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:21 smithi002 ceph-mon[17167]: pgmap v3225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:45:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:45:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5624.977369 secs 2022-03-21T15:45:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:23 smithi016 ceph-mon[22010]: pgmap v3226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5624.977369 secs 2022-03-21T15:45:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:23 smithi079 ceph-mon[23158]: pgmap v3226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5624.977369 secs 2022-03-21T15:45:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:23 smithi002 ceph-mon[17167]: pgmap v3226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:25 smithi016 ceph-mon[22010]: pgmap v3227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:25 smithi079 ceph-mon[23158]: pgmap v3227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:25 smithi002 ceph-mon[17167]: pgmap v3227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:26 smithi016 ceph-mon[22010]: pgmap v3228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:26 smithi079 ceph-mon[23158]: pgmap v3228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:26 smithi002 ceph-mon[17167]: pgmap v3228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:28.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5629.977495 secs 2022-03-21T15:45:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5629.977495 secs 2022-03-21T15:45:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5629.977495 secs 2022-03-21T15:45:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:29 smithi002 ceph-mon[17167]: pgmap v3229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:29 smithi016 ceph-mon[22010]: pgmap v3229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:29 smithi079 ceph-mon[23158]: pgmap v3229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:31 smithi016 ceph-mon[22010]: pgmap v3230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:31 smithi079 ceph-mon[23158]: pgmap v3230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:31 smithi002 ceph-mon[17167]: pgmap v3230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5634.977565 secs 2022-03-21T15:45:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:33 smithi016 ceph-mon[22010]: pgmap v3231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5634.977565 secs 2022-03-21T15:45:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:33 smithi079 ceph-mon[23158]: pgmap v3231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5634.977565 secs 2022-03-21T15:45:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:33 smithi002 ceph-mon[17167]: pgmap v3231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:35 smithi016 ceph-mon[22010]: pgmap v3232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:35 smithi079 ceph-mon[23158]: pgmap v3232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:35 smithi002 ceph-mon[17167]: pgmap v3232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:36 smithi016 ceph-mon[22010]: pgmap v3233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:36 smithi079 ceph-mon[23158]: pgmap v3233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:36 smithi002 ceph-mon[17167]: pgmap v3233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:45:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:45:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:45:38.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:45:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:45:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:45:38.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:45:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:45:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:45:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5639.977702 secs 2022-03-21T15:45:39.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:38 smithi002 ceph-mon[17167]: pgmap v3234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:45:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5639.977702 secs 2022-03-21T15:45:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:38 smithi016 ceph-mon[22010]: pgmap v3234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:45:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5639.977702 secs 2022-03-21T15:45:39.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:38 smithi079 ceph-mon[23158]: pgmap v3234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:45:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:41 smithi016 ceph-mon[22010]: pgmap v3235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:41 smithi079 ceph-mon[23158]: pgmap v3235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:41 smithi002 ceph-mon[17167]: pgmap v3235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5644.977861 secs 2022-03-21T15:45:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:43 smithi016 ceph-mon[22010]: pgmap v3236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5644.977861 secs 2022-03-21T15:45:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:43 smithi079 ceph-mon[23158]: pgmap v3236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5644.977861 secs 2022-03-21T15:45:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:43 smithi002 ceph-mon[17167]: pgmap v3236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:45 smithi016 ceph-mon[22010]: pgmap v3237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:45 smithi079 ceph-mon[23158]: pgmap v3237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:45.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:45 smithi002 ceph-mon[17167]: pgmap v3237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:46 smithi016 ceph-mon[22010]: pgmap v3238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:46 smithi079 ceph-mon[23158]: pgmap v3238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:46 smithi002 ceph-mon[17167]: pgmap v3238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:48.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5649.978000 secs 2022-03-21T15:45:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5649.978000 secs 2022-03-21T15:45:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5649.978000 secs 2022-03-21T15:45:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:49 smithi016 ceph-mon[22010]: pgmap v3239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:49 smithi079 ceph-mon[23158]: pgmap v3239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:49 smithi002 ceph-mon[17167]: pgmap v3239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:51 smithi016 ceph-mon[22010]: pgmap v3240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:51 smithi079 ceph-mon[23158]: pgmap v3240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:51 smithi002 ceph-mon[17167]: pgmap v3240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5654.978111 secs 2022-03-21T15:45:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:53 smithi016 ceph-mon[22010]: pgmap v3241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5654.978111 secs 2022-03-21T15:45:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:53 smithi079 ceph-mon[23158]: pgmap v3241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5654.978111 secs 2022-03-21T15:45:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:53 smithi002 ceph-mon[17167]: pgmap v3241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:45:55.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:55 smithi016 ceph-mon[22010]: pgmap v3242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:55 smithi079 ceph-mon[23158]: pgmap v3242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:55 smithi002 ceph-mon[17167]: pgmap v3242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:45:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:56 smithi016 ceph-mon[22010]: pgmap v3243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:56 smithi079 ceph-mon[23158]: pgmap v3243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:56 smithi002 ceph-mon[17167]: pgmap v3243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:45:58.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5659.978231 secs 2022-03-21T15:45:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5659.978231 secs 2022-03-21T15:45:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5659.978231 secs 2022-03-21T15:45:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:45:59 smithi016 ceph-mon[22010]: pgmap v3244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:45:59 smithi079 ceph-mon[23158]: pgmap v3244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:45:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:45:59 smithi002 ceph-mon[17167]: pgmap v3244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:01 smithi016 ceph-mon[22010]: pgmap v3245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:01 smithi079 ceph-mon[23158]: pgmap v3245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:01 smithi002 ceph-mon[17167]: pgmap v3245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5664.978355 secs 2022-03-21T15:46:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:03 smithi016 ceph-mon[22010]: pgmap v3246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5664.978355 secs 2022-03-21T15:46:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:03 smithi079 ceph-mon[23158]: pgmap v3246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5664.978355 secs 2022-03-21T15:46:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:03 smithi002 ceph-mon[17167]: pgmap v3246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:05 smithi016 ceph-mon[22010]: pgmap v3247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:05 smithi079 ceph-mon[23158]: pgmap v3247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:05 smithi002 ceph-mon[17167]: pgmap v3247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:06 smithi016 ceph-mon[22010]: pgmap v3248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:06 smithi079 ceph-mon[23158]: pgmap v3248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:06 smithi002 ceph-mon[17167]: pgmap v3248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:08.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5669.978473 secs 2022-03-21T15:46:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5669.978473 secs 2022-03-21T15:46:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5669.978473 secs 2022-03-21T15:46:09.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:09 smithi016 ceph-mon[22010]: pgmap v3249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:09 smithi079 ceph-mon[23158]: pgmap v3249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:09 smithi002 ceph-mon[17167]: pgmap v3249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:11 smithi016 ceph-mon[22010]: pgmap v3250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:11 smithi079 ceph-mon[23158]: pgmap v3250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:11 smithi002 ceph-mon[17167]: pgmap v3250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5674.978604 secs 2022-03-21T15:46:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:13 smithi016 ceph-mon[22010]: pgmap v3251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5674.978604 secs 2022-03-21T15:46:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:13 smithi079 ceph-mon[23158]: pgmap v3251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5674.978604 secs 2022-03-21T15:46:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:13 smithi002 ceph-mon[17167]: pgmap v3251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:15 smithi016 ceph-mon[22010]: pgmap v3252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:15 smithi079 ceph-mon[23158]: pgmap v3252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:15 smithi002 ceph-mon[17167]: pgmap v3252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:16 smithi016 ceph-mon[22010]: pgmap v3253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:16 smithi079 ceph-mon[23158]: pgmap v3253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:16 smithi002 ceph-mon[17167]: pgmap v3253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:18.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5679.978724 secs 2022-03-21T15:46:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5679.978724 secs 2022-03-21T15:46:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5679.978724 secs 2022-03-21T15:46:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:19 smithi016 ceph-mon[22010]: pgmap v3254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:19 smithi079 ceph-mon[23158]: pgmap v3254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:19 smithi002 ceph-mon[17167]: pgmap v3254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:21 smithi016 ceph-mon[22010]: pgmap v3255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:46:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:46:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:21 smithi079 ceph-mon[23158]: pgmap v3255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:46:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:46:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:21 smithi002 ceph-mon[17167]: pgmap v3255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:46:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:46:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5684.978805 secs 2022-03-21T15:46:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:23 smithi016 ceph-mon[22010]: pgmap v3256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5684.978805 secs 2022-03-21T15:46:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:23 smithi079 ceph-mon[23158]: pgmap v3256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5684.978805 secs 2022-03-21T15:46:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:23 smithi002 ceph-mon[17167]: pgmap v3256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:25 smithi016 ceph-mon[22010]: pgmap v3257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:25 smithi079 ceph-mon[23158]: pgmap v3257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:25 smithi002 ceph-mon[17167]: pgmap v3257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:26 smithi016 ceph-mon[22010]: pgmap v3258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:26 smithi079 ceph-mon[23158]: pgmap v3258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:26 smithi002 ceph-mon[17167]: pgmap v3258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5689.978938 secs 2022-03-21T15:46:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5689.978938 secs 2022-03-21T15:46:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5689.978938 secs 2022-03-21T15:46:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:29 smithi016 ceph-mon[22010]: pgmap v3259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:46:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:29 smithi079 ceph-mon[23158]: pgmap v3259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:46:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:29 smithi002 ceph-mon[17167]: pgmap v3259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T15:46:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:31 smithi016 ceph-mon[22010]: pgmap v3260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:31 smithi079 ceph-mon[23158]: pgmap v3260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:31 smithi002 ceph-mon[17167]: pgmap v3260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:46:33.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5694.979060 secs 2022-03-21T15:46:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:33 smithi016 ceph-mon[22010]: pgmap v3261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5694.979060 secs 2022-03-21T15:46:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:33 smithi079 ceph-mon[23158]: pgmap v3261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5694.979060 secs 2022-03-21T15:46:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:33 smithi002 ceph-mon[17167]: pgmap v3261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:35 smithi016 ceph-mon[22010]: pgmap v3262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:35 smithi079 ceph-mon[23158]: pgmap v3262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:35 smithi002 ceph-mon[17167]: pgmap v3262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:36 smithi016 ceph-mon[22010]: pgmap v3263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:36 smithi079 ceph-mon[23158]: pgmap v3263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:36 smithi002 ceph-mon[17167]: pgmap v3263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5699.979186 secs 2022-03-21T15:46:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5699.979186 secs 2022-03-21T15:46:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5699.979186 secs 2022-03-21T15:46:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:39 smithi016 ceph-mon[22010]: pgmap v3264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:46:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:46:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:46:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:39 smithi079 ceph-mon[23158]: pgmap v3264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:46:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:46:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:46:39.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:39 smithi002 ceph-mon[17167]: pgmap v3264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:39.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:46:39.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:46:39.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:46:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:46:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:40 smithi002 ceph-mon[17167]: pgmap v3265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:46:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:40 smithi016 ceph-mon[22010]: pgmap v3265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:46:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:40 smithi079 ceph-mon[23158]: pgmap v3265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5704.979326 secs 2022-03-21T15:46:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:43 smithi016 ceph-mon[22010]: pgmap v3266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5704.979326 secs 2022-03-21T15:46:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:43 smithi079 ceph-mon[23158]: pgmap v3266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5704.979326 secs 2022-03-21T15:46:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:43 smithi002 ceph-mon[17167]: pgmap v3266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:45 smithi016 ceph-mon[22010]: pgmap v3267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:45 smithi079 ceph-mon[23158]: pgmap v3267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:45 smithi002 ceph-mon[17167]: pgmap v3267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:46 smithi079 ceph-mon[23158]: pgmap v3268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:46 smithi002 ceph-mon[17167]: pgmap v3268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:46 smithi016 ceph-mon[22010]: pgmap v3268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5709.979446 secs 2022-03-21T15:46:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5709.979446 secs 2022-03-21T15:46:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5709.979446 secs 2022-03-21T15:46:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:49 smithi016 ceph-mon[22010]: pgmap v3269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:49 smithi079 ceph-mon[23158]: pgmap v3269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:49 smithi002 ceph-mon[17167]: pgmap v3269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:51 smithi016 ceph-mon[22010]: pgmap v3270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:51 smithi079 ceph-mon[23158]: pgmap v3270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:51 smithi002 ceph-mon[17167]: pgmap v3270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:53.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5714.979589 secs 2022-03-21T15:46:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:53 smithi016 ceph-mon[22010]: pgmap v3271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5714.979589 secs 2022-03-21T15:46:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:53 smithi079 ceph-mon[23158]: pgmap v3271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5714.979589 secs 2022-03-21T15:46:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:53 smithi002 ceph-mon[17167]: pgmap v3271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:46:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:55 smithi079 ceph-mon[23158]: pgmap v3272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:55 smithi002 ceph-mon[17167]: pgmap v3272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:55 smithi016 ceph-mon[22010]: pgmap v3272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:46:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:56 smithi079 ceph-mon[23158]: pgmap v3273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:56 smithi002 ceph-mon[17167]: pgmap v3273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:56 smithi016 ceph-mon[22010]: pgmap v3273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:46:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5719.979730 secs 2022-03-21T15:46:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5719.979730 secs 2022-03-21T15:46:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5719.979730 secs 2022-03-21T15:46:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:46:59 smithi016 ceph-mon[22010]: pgmap v3274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:46:59 smithi079 ceph-mon[23158]: pgmap v3274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:46:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:46:59 smithi002 ceph-mon[17167]: pgmap v3274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:01 smithi016 ceph-mon[22010]: pgmap v3275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:01 smithi079 ceph-mon[23158]: pgmap v3275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:01 smithi002 ceph-mon[17167]: pgmap v3275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5724.979858 secs 2022-03-21T15:47:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:03 smithi016 ceph-mon[22010]: pgmap v3276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5724.979858 secs 2022-03-21T15:47:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:03 smithi079 ceph-mon[23158]: pgmap v3276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5724.979858 secs 2022-03-21T15:47:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:03 smithi002 ceph-mon[17167]: pgmap v3276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:05 smithi016 ceph-mon[22010]: pgmap v3277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:05 smithi079 ceph-mon[23158]: pgmap v3277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:05 smithi002 ceph-mon[17167]: pgmap v3277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:06 smithi079 ceph-mon[23158]: pgmap v3278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:06 smithi002 ceph-mon[17167]: pgmap v3278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:06 smithi016 ceph-mon[22010]: pgmap v3278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5729.979979 secs 2022-03-21T15:47:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5729.979979 secs 2022-03-21T15:47:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5729.979979 secs 2022-03-21T15:47:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:09 smithi016 ceph-mon[22010]: pgmap v3279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:09 smithi079 ceph-mon[23158]: pgmap v3279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:09 smithi002 ceph-mon[17167]: pgmap v3279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:11 smithi016 ceph-mon[22010]: pgmap v3280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:11 smithi079 ceph-mon[23158]: pgmap v3280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:11 smithi002 ceph-mon[17167]: pgmap v3280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5734.980101 secs 2022-03-21T15:47:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:13 smithi079 ceph-mon[23158]: pgmap v3281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5734.980101 secs 2022-03-21T15:47:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:13 smithi002 ceph-mon[17167]: pgmap v3281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5734.980101 secs 2022-03-21T15:47:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:13 smithi016 ceph-mon[22010]: pgmap v3281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:15 smithi079 ceph-mon[23158]: pgmap v3282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:15 smithi002 ceph-mon[17167]: pgmap v3282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:15 smithi016 ceph-mon[22010]: pgmap v3282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:16 smithi079 ceph-mon[23158]: pgmap v3283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:16 smithi002 ceph-mon[17167]: pgmap v3283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:16 smithi016 ceph-mon[22010]: pgmap v3283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5739.980217 secs 2022-03-21T15:47:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5739.980217 secs 2022-03-21T15:47:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5739.980217 secs 2022-03-21T15:47:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:19 smithi016 ceph-mon[22010]: pgmap v3284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T15:47:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:19 smithi079 ceph-mon[23158]: pgmap v3284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T15:47:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:19 smithi002 ceph-mon[17167]: pgmap v3284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T15:47:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:21 smithi016 ceph-mon[22010]: pgmap v3285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:47:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:47:21.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:21 smithi079 ceph-mon[23158]: pgmap v3285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:47:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:47:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:21 smithi002 ceph-mon[17167]: pgmap v3285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:47:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:47:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:47:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5744.980357 secs 2022-03-21T15:47:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:23 smithi079 ceph-mon[23158]: pgmap v3286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5744.980357 secs 2022-03-21T15:47:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:23 smithi002 ceph-mon[17167]: pgmap v3286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5744.980357 secs 2022-03-21T15:47:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:23 smithi016 ceph-mon[22010]: pgmap v3286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:25 smithi079 ceph-mon[23158]: pgmap v3287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:25 smithi002 ceph-mon[17167]: pgmap v3287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:25 smithi016 ceph-mon[22010]: pgmap v3287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:26 smithi079 ceph-mon[23158]: pgmap v3288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:26 smithi002 ceph-mon[17167]: pgmap v3288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:26 smithi016 ceph-mon[22010]: pgmap v3288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5749.980464 secs 2022-03-21T15:47:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5749.980464 secs 2022-03-21T15:47:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5749.980464 secs 2022-03-21T15:47:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:29 smithi016 ceph-mon[22010]: pgmap v3289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:29 smithi079 ceph-mon[23158]: pgmap v3289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:29 smithi002 ceph-mon[17167]: pgmap v3289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:31 smithi079 ceph-mon[23158]: pgmap v3290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:31 smithi002 ceph-mon[17167]: pgmap v3290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:31 smithi016 ceph-mon[22010]: pgmap v3290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5754.980567 secs 2022-03-21T15:47:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:33 smithi079 ceph-mon[23158]: pgmap v3291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5754.980567 secs 2022-03-21T15:47:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:33 smithi002 ceph-mon[17167]: pgmap v3291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5754.980567 secs 2022-03-21T15:47:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:33 smithi016 ceph-mon[22010]: pgmap v3291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:35 smithi079 ceph-mon[23158]: pgmap v3292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:35 smithi002 ceph-mon[17167]: pgmap v3292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:35 smithi016 ceph-mon[22010]: pgmap v3292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:36 smithi079 ceph-mon[23158]: pgmap v3293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:36 smithi002 ceph-mon[17167]: pgmap v3293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:36 smithi016 ceph-mon[22010]: pgmap v3293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:38.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5759.980693 secs 2022-03-21T15:47:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5759.980693 secs 2022-03-21T15:47:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5759.980693 secs 2022-03-21T15:47:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:39 smithi016 ceph-mon[22010]: pgmap v3294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:39 smithi079 ceph-mon[23158]: pgmap v3294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:39 smithi002 ceph-mon[17167]: pgmap v3294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:40.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:47:40.409 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:47:40.409 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:47:40.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:47:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:47:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:47:40.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:47:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:47:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:47:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:41 smithi079 ceph-mon[23158]: pgmap v3295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:47:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:41 smithi002 ceph-mon[17167]: pgmap v3295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:47:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:41 smithi016 ceph-mon[22010]: pgmap v3295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:47:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5764.980819 secs 2022-03-21T15:47:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: slow request 5764.876048 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: slow request 5764.664864 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: slow request 5764.977458 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: slow request 5764.928320 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: slow request 5764.903060 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:43 smithi079 ceph-mon[23158]: pgmap v3296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5764.980819 secs 2022-03-21T15:47:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: slow request 5764.876048 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: slow request 5764.664864 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: slow request 5764.977458 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: slow request 5764.928320 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: slow request 5764.903060 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:43 smithi002 ceph-mon[17167]: pgmap v3296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5764.980819 secs 2022-03-21T15:47:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: slow request 5764.876048 seconds old, received at 2022-03-21T14:11:37.420463+0000: client_request(client.14964:6185 create #0x10000000265/file1176 2022-03-21T14:11:37.418896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: slow request 5764.664864 seconds old, received at 2022-03-21T14:11:37.631647+0000: client_request(client.14964:6248 create #0x10000000264/file1183 2022-03-21T14:11:37.629891+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: slow request 5764.977458 seconds old, received at 2022-03-21T14:11:37.319053+0000: client_request(client.14964:6058 create #0x10000000265/file1144 2022-03-21T14:11:37.317899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: slow request 5764.928320 seconds old, received at 2022-03-21T14:11:37.368191+0000: client_request(client.14964:6123 create #0x10000000265/file1160 2022-03-21T14:11:37.366897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: slow request 5764.903060 seconds old, received at 2022-03-21T14:11:37.393451+0000: client_request(client.14964:6155 create #0x10000000265/file1168 2022-03-21T14:11:37.391897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:43 smithi016 ceph-mon[22010]: pgmap v3296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:45 smithi079 ceph-mon[23158]: pgmap v3297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:45 smithi002 ceph-mon[17167]: pgmap v3297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:45 smithi016 ceph-mon[22010]: pgmap v3297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:46 smithi079 ceph-mon[23158]: pgmap v3298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:46 smithi002 ceph-mon[17167]: pgmap v3298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:46 smithi016 ceph-mon[22010]: pgmap v3298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5769.980895 secs 2022-03-21T15:47:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: slow request 5769.872856 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: slow request 5769.952188 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: slow request 5769.869898 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: slow request 5769.974094 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:48 smithi016 ceph-mon[22010]: slow request 5769.925119 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5769.980895 secs 2022-03-21T15:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: slow request 5769.872856 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: slow request 5769.952188 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: slow request 5769.869898 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: slow request 5769.974094 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:48 smithi079 ceph-mon[23158]: slow request 5769.925119 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5769.980895 secs 2022-03-21T15:47:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: slow request 5769.872856 seconds old, received at 2022-03-21T14:11:37.423730+0000: client_request(client.14964:6187 create #0x10000000265/file1177 2022-03-21T14:11:37.421896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: slow request 5769.952188 seconds old, received at 2022-03-21T14:11:37.344399+0000: client_request(client.14964:6092 create #0x10000000265/file1152 2022-03-21T14:11:37.342898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: slow request 5769.869898 seconds old, received at 2022-03-21T14:11:37.426689+0000: client_request(client.14964:6189 create #0x10000000265/file1178 2022-03-21T14:11:37.424896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: slow request 5769.974094 seconds old, received at 2022-03-21T14:11:37.322493+0000: client_request(client.14964:6062 create #0x10000000265/file1145 2022-03-21T14:11:37.320898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:48 smithi002 ceph-mon[17167]: slow request 5769.925119 seconds old, received at 2022-03-21T14:11:37.371468+0000: client_request(client.14964:6127 create #0x10000000265/file1161 2022-03-21T14:11:37.369897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:49 smithi016 ceph-mon[22010]: pgmap v3299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:49 smithi079 ceph-mon[23158]: pgmap v3299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:49 smithi002 ceph-mon[17167]: pgmap v3299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:51 smithi079 ceph-mon[23158]: pgmap v3300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:51 smithi002 ceph-mon[17167]: pgmap v3300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:51 smithi016 ceph-mon[22010]: pgmap v3300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5774.981022 secs 2022-03-21T15:47:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: slow request 5774.867006 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: slow request 5774.949590 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: slow request 5774.899165 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: slow request 5774.864260 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: slow request 5774.971043 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:53 smithi079 ceph-mon[23158]: pgmap v3301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5774.981022 secs 2022-03-21T15:47:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: slow request 5774.867006 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: slow request 5774.949590 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: slow request 5774.899165 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: slow request 5774.864260 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: slow request 5774.971043 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:53 smithi002 ceph-mon[17167]: pgmap v3301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5774.981022 secs 2022-03-21T15:47:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: slow request 5774.867006 seconds old, received at 2022-03-21T14:11:37.429708+0000: client_request(client.14964:6191 create #0x10000000265/file1179 2022-03-21T14:11:37.427896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: slow request 5774.949590 seconds old, received at 2022-03-21T14:11:37.347124+0000: client_request(client.14964:6096 create #0x10000000265/file1153 2022-03-21T14:11:37.345898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: slow request 5774.899165 seconds old, received at 2022-03-21T14:11:37.397549+0000: client_request(client.14964:6160 create #0x10000000265/file1169 2022-03-21T14:11:37.395897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: slow request 5774.864260 seconds old, received at 2022-03-21T14:11:37.432454+0000: client_request(client.14964:6193 create #0x10000000265/file1180 2022-03-21T14:11:37.430896+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: slow request 5774.971043 seconds old, received at 2022-03-21T14:11:37.325671+0000: client_request(client.14964:6066 create #0x10000000265/file1146 2022-03-21T14:11:37.323898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:53 smithi016 ceph-mon[22010]: pgmap v3301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:47:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:55 smithi079 ceph-mon[23158]: pgmap v3302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:55 smithi002 ceph-mon[17167]: pgmap v3302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:55 smithi016 ceph-mon[22010]: pgmap v3302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:47:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:56 smithi079 ceph-mon[23158]: pgmap v3303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:56 smithi002 ceph-mon[17167]: pgmap v3303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:56 smithi016 ceph-mon[22010]: pgmap v3303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:47:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5779.981136 secs 2022-03-21T15:47:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: slow request 5779.946893 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: slow request 5779.922040 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: slow request 5779.895127 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: slow request 5779.968239 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:58 smithi016 ceph-mon[22010]: slow request 5779.943825 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5779.981136 secs 2022-03-21T15:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: slow request 5779.946893 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: slow request 5779.922040 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: slow request 5779.895127 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: slow request 5779.968239 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:58 smithi079 ceph-mon[23158]: slow request 5779.943825 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5779.981136 secs 2022-03-21T15:47:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: slow request 5779.946893 seconds old, received at 2022-03-21T14:11:37.349934+0000: client_request(client.14964:6099 create #0x10000000265/file1154 2022-03-21T14:11:37.347898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: slow request 5779.922040 seconds old, received at 2022-03-21T14:11:37.374788+0000: client_request(client.14964:6131 create #0x10000000265/file1162 2022-03-21T14:11:37.372897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: slow request 5779.895127 seconds old, received at 2022-03-21T14:11:37.401700+0000: client_request(client.14964:6164 create #0x10000000265/file1170 2022-03-21T14:11:37.399897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: slow request 5779.968239 seconds old, received at 2022-03-21T14:11:37.328588+0000: client_request(client.14964:6070 create #0x10000000265/file1147 2022-03-21T14:11:37.326898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:58 smithi002 ceph-mon[17167]: slow request 5779.943825 seconds old, received at 2022-03-21T14:11:37.353002+0000: client_request(client.14964:6103 create #0x10000000265/file1155 2022-03-21T14:11:37.351898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:47:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:47:59 smithi016 ceph-mon[22010]: pgmap v3304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:47:59 smithi079 ceph-mon[23158]: pgmap v3304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:47:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:47:59 smithi002 ceph-mon[17167]: pgmap v3304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:01 smithi079 ceph-mon[23158]: pgmap v3305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:01 smithi002 ceph-mon[17167]: pgmap v3305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:01 smithi016 ceph-mon[22010]: pgmap v3305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5784.981266 secs 2022-03-21T15:48:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: slow request 5784.919365 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: slow request 5784.892520 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: slow request 5784.916403 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: slow request 5784.964983 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: slow request 5784.941280 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:03 smithi079 ceph-mon[23158]: pgmap v3306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5784.981266 secs 2022-03-21T15:48:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: slow request 5784.919365 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: slow request 5784.892520 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: slow request 5784.916403 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: slow request 5784.964983 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: slow request 5784.941280 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:03 smithi002 ceph-mon[17167]: pgmap v3306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5784.981266 secs 2022-03-21T15:48:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: slow request 5784.919365 seconds old, received at 2022-03-21T14:11:37.377592+0000: client_request(client.14964:6135 create #0x10000000265/file1163 2022-03-21T14:11:37.375897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: slow request 5784.892520 seconds old, received at 2022-03-21T14:11:37.404437+0000: client_request(client.14964:6168 create #0x10000000265/file1171 2022-03-21T14:11:37.402897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: slow request 5784.916403 seconds old, received at 2022-03-21T14:11:37.380554+0000: client_request(client.14964:6138 create #0x10000000265/file1164 2022-03-21T14:11:37.378897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: slow request 5784.964983 seconds old, received at 2022-03-21T14:11:37.331974+0000: client_request(client.14964:6075 create #0x10000000265/file1148 2022-03-21T14:11:37.330898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: slow request 5784.941280 seconds old, received at 2022-03-21T14:11:37.355677+0000: client_request(client.14964:6107 create #0x10000000265/file1156 2022-03-21T14:11:37.353898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:03 smithi016 ceph-mon[22010]: pgmap v3306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:05 smithi079 ceph-mon[23158]: pgmap v3307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:05 smithi002 ceph-mon[17167]: pgmap v3307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:05 smithi016 ceph-mon[22010]: pgmap v3307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:06 smithi079 ceph-mon[23158]: pgmap v3308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:06 smithi002 ceph-mon[17167]: pgmap v3308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:06 smithi016 ceph-mon[22010]: pgmap v3308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5789.981389 secs 2022-03-21T15:48:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: slow request 5789.889451 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: slow request 5789.938555 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: slow request 5789.913451 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: slow request 5789.962046 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:08 smithi016 ceph-mon[22010]: slow request 5789.885689 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5789.981389 secs 2022-03-21T15:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: slow request 5789.889451 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: slow request 5789.938555 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: slow request 5789.913451 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: slow request 5789.962046 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:08 smithi079 ceph-mon[23158]: slow request 5789.885689 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5789.981389 secs 2022-03-21T15:48:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: slow request 5789.889451 seconds old, received at 2022-03-21T14:11:37.407630+0000: client_request(client.14964:6172 create #0x10000000265/file1172 2022-03-21T14:11:37.405896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: slow request 5789.938555 seconds old, received at 2022-03-21T14:11:37.358525+0000: client_request(client.14964:6110 create #0x10000000265/file1157 2022-03-21T14:11:37.356898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: slow request 5789.913451 seconds old, received at 2022-03-21T14:11:37.383629+0000: client_request(client.14964:6142 create #0x10000000265/file1165 2022-03-21T14:11:37.381897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: slow request 5789.962046 seconds old, received at 2022-03-21T14:11:37.335034+0000: client_request(client.14964:6079 create #0x10000000265/file1149 2022-03-21T14:11:37.332898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:08 smithi002 ceph-mon[17167]: slow request 5789.885689 seconds old, received at 2022-03-21T14:11:37.411392+0000: client_request(client.14964:6176 create #0x10000000265/file1173 2022-03-21T14:11:37.409896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:09 smithi016 ceph-mon[22010]: pgmap v3309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:09 smithi079 ceph-mon[23158]: pgmap v3309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:09 smithi002 ceph-mon[17167]: pgmap v3309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:11 smithi079 ceph-mon[23158]: pgmap v3310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:11 smithi002 ceph-mon[17167]: pgmap v3310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:11 smithi016 ceph-mon[22010]: pgmap v3310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5794.981514 secs 2022-03-21T15:48:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: slow request 5794.865406 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:48:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: slow request 5794.882811 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: slow request 5794.958712 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: slow request 5794.935310 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: slow request 5794.910253 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:13 smithi079 ceph-mon[23158]: pgmap v3311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5794.981514 secs 2022-03-21T15:48:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: slow request 5794.865406 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:48:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: slow request 5794.882811 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: slow request 5794.958712 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: slow request 5794.935310 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: slow request 5794.910253 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:13 smithi002 ceph-mon[17167]: pgmap v3311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5794.981514 secs 2022-03-21T15:48:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: slow request 5794.865406 seconds old, received at 2022-03-21T14:11:37.431799+0000: internal op fragmentdir:mds.0:23 currently dispatched 2022-03-21T15:48:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: slow request 5794.882811 seconds old, received at 2022-03-21T14:11:37.414394+0000: client_request(client.14964:6178 create #0x10000000265/file1174 2022-03-21T14:11:37.412896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: slow request 5794.958712 seconds old, received at 2022-03-21T14:11:37.338493+0000: client_request(client.14964:6083 create #0x10000000265/file1150 2022-03-21T14:11:37.336898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: slow request 5794.935310 seconds old, received at 2022-03-21T14:11:37.361895+0000: client_request(client.14964:6115 create #0x10000000265/file1158 2022-03-21T14:11:37.359898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: slow request 5794.910253 seconds old, received at 2022-03-21T14:11:37.386952+0000: client_request(client.14964:6147 create #0x10000000265/file1166 2022-03-21T14:11:37.385897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:13 smithi016 ceph-mon[22010]: pgmap v3311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:15 smithi079 ceph-mon[23158]: pgmap v3312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:15 smithi002 ceph-mon[17167]: pgmap v3312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:15 smithi016 ceph-mon[22010]: pgmap v3312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:16 smithi079 ceph-mon[23158]: pgmap v3313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:16 smithi002 ceph-mon[17167]: pgmap v3313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:16 smithi016 ceph-mon[22010]: pgmap v3313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: 40 slow requests, 5 included below; oldest blocked for > 5799.981596 secs 2022-03-21T15:48:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: slow request 5799.879571 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: slow request 5799.981595 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: slow request 5799.932372 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: slow request 5799.907422 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:18 smithi016 ceph-mon[22010]: slow request 5799.955869 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: 40 slow requests, 5 included below; oldest blocked for > 5799.981596 secs 2022-03-21T15:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: slow request 5799.879571 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: slow request 5799.981595 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: slow request 5799.932372 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: slow request 5799.907422 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:18 smithi079 ceph-mon[23158]: slow request 5799.955869 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: 40 slow requests, 5 included below; oldest blocked for > 5799.981596 secs 2022-03-21T15:48:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: slow request 5799.879571 seconds old, received at 2022-03-21T14:11:37.417716+0000: client_request(client.14964:6181 create #0x10000000265/file1175 2022-03-21T14:11:37.415896+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: slow request 5799.981595 seconds old, received at 2022-03-21T14:11:37.315693+0000: client_request(client.14964:6054 create #0x10000000265/file1143 2022-03-21T14:11:37.313899+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: slow request 5799.932372 seconds old, received at 2022-03-21T14:11:37.364916+0000: client_request(client.14964:6119 create #0x10000000265/file1159 2022-03-21T14:11:37.363897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: slow request 5799.907422 seconds old, received at 2022-03-21T14:11:37.389866+0000: client_request(client.14964:6151 create #0x10000000265/file1167 2022-03-21T14:11:37.387897+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:18 smithi002 ceph-mon[17167]: slow request 5799.955869 seconds old, received at 2022-03-21T14:11:37.341419+0000: client_request(client.14964:6088 create #0x10000000265/file1151 2022-03-21T14:11:37.339898+0000 ASYNC caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) currently failed to authpin, dir is being fragmented 2022-03-21T15:48:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:19 smithi016 ceph-mon[22010]: pgmap v3314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:19 smithi079 ceph-mon[23158]: pgmap v3314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:19 smithi002 ceph-mon[17167]: pgmap v3314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:21 smithi079 ceph-mon[23158]: pgmap v3315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:48:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:48:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:21 smithi002 ceph-mon[17167]: pgmap v3315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:48:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:48:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:21 smithi016 ceph-mon[22010]: pgmap v3315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:48:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:48:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5804.981681 secs 2022-03-21T15:48:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:23 smithi079 ceph-mon[23158]: pgmap v3316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5804.981681 secs 2022-03-21T15:48:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:23 smithi002 ceph-mon[17167]: pgmap v3316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5804.981681 secs 2022-03-21T15:48:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:23 smithi016 ceph-mon[22010]: pgmap v3316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:25 smithi079 ceph-mon[23158]: pgmap v3317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:25 smithi002 ceph-mon[17167]: pgmap v3317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:25 smithi016 ceph-mon[22010]: pgmap v3317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:26 smithi079 ceph-mon[23158]: pgmap v3318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:26 smithi002 ceph-mon[17167]: pgmap v3318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:26 smithi016 ceph-mon[22010]: pgmap v3318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5809.981791 secs 2022-03-21T15:48:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5809.981791 secs 2022-03-21T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5809.981791 secs 2022-03-21T15:48:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:29 smithi016 ceph-mon[22010]: pgmap v3319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:29 smithi079 ceph-mon[23158]: pgmap v3319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:29 smithi002 ceph-mon[17167]: pgmap v3319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:31 smithi079 ceph-mon[23158]: pgmap v3320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:31 smithi002 ceph-mon[17167]: pgmap v3320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:31 smithi016 ceph-mon[22010]: pgmap v3320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5814.981907 secs 2022-03-21T15:48:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:33 smithi079 ceph-mon[23158]: pgmap v3321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5814.981907 secs 2022-03-21T15:48:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:33 smithi002 ceph-mon[17167]: pgmap v3321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5814.981907 secs 2022-03-21T15:48:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:33 smithi016 ceph-mon[22010]: pgmap v3321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:35 smithi079 ceph-mon[23158]: pgmap v3322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:35 smithi002 ceph-mon[17167]: pgmap v3322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:35 smithi016 ceph-mon[22010]: pgmap v3322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:36 smithi079 ceph-mon[23158]: pgmap v3323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:36 smithi002 ceph-mon[17167]: pgmap v3323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:36 smithi016 ceph-mon[22010]: pgmap v3323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5819.982050 secs 2022-03-21T15:48:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5819.982050 secs 2022-03-21T15:48:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5819.982050 secs 2022-03-21T15:48:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:39 smithi016 ceph-mon[22010]: pgmap v3324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:39 smithi079 ceph-mon[23158]: pgmap v3324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:39 smithi002 ceph-mon[17167]: pgmap v3324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:48:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:48:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:48:40.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:48:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:48:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:48:40.875 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:48:40.875 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:48:40.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:48:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: pgmap v3325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:48:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: pgmap v3325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:48:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: pgmap v3325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:48:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:48:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:48:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:48:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:48:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:48:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:48:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:42 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:48:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:48:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:48:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:48:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:48:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:48:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:42 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:48:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:48:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:48:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:48:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:48:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:48:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:42 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:48:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5824.982147 secs 2022-03-21T15:48:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:43 smithi002 ceph-mon[17167]: pgmap v3326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5824.982147 secs 2022-03-21T15:48:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:43 smithi016 ceph-mon[22010]: pgmap v3326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5824.982147 secs 2022-03-21T15:48:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:43 smithi079 ceph-mon[23158]: pgmap v3326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:45 smithi002 ceph-mon[17167]: pgmap v3327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:45 smithi016 ceph-mon[22010]: pgmap v3327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:45 smithi079 ceph-mon[23158]: pgmap v3327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:46 smithi079 ceph-mon[23158]: pgmap v3328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:46 smithi002 ceph-mon[17167]: pgmap v3328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:46 smithi016 ceph-mon[22010]: pgmap v3328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5829.982280 secs 2022-03-21T15:48:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5829.982280 secs 2022-03-21T15:48:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5829.982280 secs 2022-03-21T15:48:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:49 smithi016 ceph-mon[22010]: pgmap v3329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:49 smithi079 ceph-mon[23158]: pgmap v3329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:49 smithi002 ceph-mon[17167]: pgmap v3329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:51 smithi079 ceph-mon[23158]: pgmap v3330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:51 smithi002 ceph-mon[17167]: pgmap v3330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:51 smithi016 ceph-mon[22010]: pgmap v3330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5834.982428 secs 2022-03-21T15:48:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:53 smithi079 ceph-mon[23158]: pgmap v3331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5834.982428 secs 2022-03-21T15:48:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:53 smithi002 ceph-mon[17167]: pgmap v3331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5834.982428 secs 2022-03-21T15:48:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:53 smithi016 ceph-mon[22010]: pgmap v3331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:48:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:55 smithi079 ceph-mon[23158]: pgmap v3332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:55 smithi002 ceph-mon[17167]: pgmap v3332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:55 smithi016 ceph-mon[22010]: pgmap v3332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:48:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:56 smithi079 ceph-mon[23158]: pgmap v3333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:56 smithi002 ceph-mon[17167]: pgmap v3333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:56 smithi016 ceph-mon[22010]: pgmap v3333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:48:58.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5839.982593 secs 2022-03-21T15:48:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5839.982593 secs 2022-03-21T15:48:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5839.982593 secs 2022-03-21T15:48:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:48:59 smithi016 ceph-mon[22010]: pgmap v3334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:48:59 smithi079 ceph-mon[23158]: pgmap v3334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:48:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:48:59 smithi002 ceph-mon[17167]: pgmap v3334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:01 smithi079 ceph-mon[23158]: pgmap v3335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:01 smithi002 ceph-mon[17167]: pgmap v3335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:01 smithi016 ceph-mon[22010]: pgmap v3335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5844.982641 secs 2022-03-21T15:49:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:03 smithi079 ceph-mon[23158]: pgmap v3336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5844.982641 secs 2022-03-21T15:49:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:03 smithi002 ceph-mon[17167]: pgmap v3336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5844.982641 secs 2022-03-21T15:49:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:03 smithi016 ceph-mon[22010]: pgmap v3336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:05 smithi079 ceph-mon[23158]: pgmap v3337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:05 smithi002 ceph-mon[17167]: pgmap v3337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:05 smithi016 ceph-mon[22010]: pgmap v3337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:06 smithi079 ceph-mon[23158]: pgmap v3338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:06 smithi002 ceph-mon[17167]: pgmap v3338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:06 smithi016 ceph-mon[22010]: pgmap v3338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5849.982751 secs 2022-03-21T15:49:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5849.982751 secs 2022-03-21T15:49:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5849.982751 secs 2022-03-21T15:49:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:09 smithi016 ceph-mon[22010]: pgmap v3339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:09 smithi079 ceph-mon[23158]: pgmap v3339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:09 smithi002 ceph-mon[17167]: pgmap v3339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:11 smithi079 ceph-mon[23158]: pgmap v3340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:11 smithi002 ceph-mon[17167]: pgmap v3340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:11 smithi016 ceph-mon[22010]: pgmap v3340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:13.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5854.982869 secs 2022-03-21T15:49:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:13 smithi079 ceph-mon[23158]: pgmap v3341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5854.982869 secs 2022-03-21T15:49:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:13 smithi002 ceph-mon[17167]: pgmap v3341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5854.982869 secs 2022-03-21T15:49:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:13 smithi016 ceph-mon[22010]: pgmap v3341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:15 smithi079 ceph-mon[23158]: pgmap v3342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:15 smithi002 ceph-mon[17167]: pgmap v3342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:15 smithi016 ceph-mon[22010]: pgmap v3342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:16.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:16 smithi079 ceph-mon[23158]: pgmap v3343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:16 smithi002 ceph-mon[17167]: pgmap v3343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:16 smithi016 ceph-mon[22010]: pgmap v3343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5859.982975 secs 2022-03-21T15:49:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5859.982975 secs 2022-03-21T15:49:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5859.982975 secs 2022-03-21T15:49:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:19 smithi016 ceph-mon[22010]: pgmap v3344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:19 smithi079 ceph-mon[23158]: pgmap v3344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:19 smithi002 ceph-mon[17167]: pgmap v3344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:21 smithi079 ceph-mon[23158]: pgmap v3345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:49:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:21 smithi002 ceph-mon[17167]: pgmap v3345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:49:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:49:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:21 smithi016 ceph-mon[22010]: pgmap v3345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:49:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:49:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5864.983082 secs 2022-03-21T15:49:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:23 smithi079 ceph-mon[23158]: pgmap v3346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5864.983082 secs 2022-03-21T15:49:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:23 smithi002 ceph-mon[17167]: pgmap v3346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5864.983082 secs 2022-03-21T15:49:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:23 smithi016 ceph-mon[22010]: pgmap v3346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:25.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:25 smithi079 ceph-mon[23158]: pgmap v3347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:25 smithi002 ceph-mon[17167]: pgmap v3347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:25 smithi016 ceph-mon[22010]: pgmap v3347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:26 smithi002 ceph-mon[17167]: pgmap v3348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:26 smithi016 ceph-mon[22010]: pgmap v3348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:26 smithi079 ceph-mon[23158]: pgmap v3348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5869.983217 secs 2022-03-21T15:49:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5869.983217 secs 2022-03-21T15:49:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5869.983217 secs 2022-03-21T15:49:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:29 smithi016 ceph-mon[22010]: pgmap v3349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:29 smithi079 ceph-mon[23158]: pgmap v3349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:29 smithi002 ceph-mon[17167]: pgmap v3349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:31.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:31 smithi079 ceph-mon[23158]: pgmap v3350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:31 smithi002 ceph-mon[17167]: pgmap v3350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:31 smithi016 ceph-mon[22010]: pgmap v3350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5874.983367 secs 2022-03-21T15:49:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:33 smithi079 ceph-mon[23158]: pgmap v3351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5874.983367 secs 2022-03-21T15:49:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:33 smithi002 ceph-mon[17167]: pgmap v3351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5874.983367 secs 2022-03-21T15:49:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:33 smithi016 ceph-mon[22010]: pgmap v3351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:35 smithi079 ceph-mon[23158]: pgmap v3352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:35 smithi002 ceph-mon[17167]: pgmap v3352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:35 smithi016 ceph-mon[22010]: pgmap v3352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:36 smithi002 ceph-mon[17167]: pgmap v3353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:36 smithi016 ceph-mon[22010]: pgmap v3353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:36 smithi079 ceph-mon[23158]: pgmap v3353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5879.983490 secs 2022-03-21T15:49:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5879.983490 secs 2022-03-21T15:49:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5879.983490 secs 2022-03-21T15:49:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:39 smithi016 ceph-mon[22010]: pgmap v3354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:39 smithi079 ceph-mon[23158]: pgmap v3354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:39 smithi002 ceph-mon[17167]: pgmap v3354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:41.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:41 smithi002 ceph-mon[17167]: pgmap v3355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:49:41.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:49:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:49:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:41 smithi016 ceph-mon[22010]: pgmap v3355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:49:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:49:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:49:41.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:41 smithi079 ceph-mon[23158]: pgmap v3355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:49:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:49:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:49:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:49:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5884.983623 secs 2022-03-21T15:49:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:43 smithi016 ceph-mon[22010]: pgmap v3356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:43.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:49:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5884.983623 secs 2022-03-21T15:49:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:43 smithi079 ceph-mon[23158]: pgmap v3356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:43.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:49:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5884.983623 secs 2022-03-21T15:49:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:43 smithi002 ceph-mon[17167]: pgmap v3356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:45 smithi079 ceph-mon[23158]: pgmap v3357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:45 smithi002 ceph-mon[17167]: pgmap v3357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:45 smithi016 ceph-mon[22010]: pgmap v3357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:46 smithi002 ceph-mon[17167]: pgmap v3358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:46 smithi016 ceph-mon[22010]: pgmap v3358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:46 smithi079 ceph-mon[23158]: pgmap v3358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:48.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5889.983700 secs 2022-03-21T15:49:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5889.983700 secs 2022-03-21T15:49:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5889.983700 secs 2022-03-21T15:49:49.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:49 smithi016 ceph-mon[22010]: pgmap v3359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:49 smithi079 ceph-mon[23158]: pgmap v3359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:49 smithi002 ceph-mon[17167]: pgmap v3359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:51.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:51 smithi079 ceph-mon[23158]: pgmap v3360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:51 smithi002 ceph-mon[17167]: pgmap v3360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:51 smithi016 ceph-mon[22010]: pgmap v3360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:53.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5894.983824 secs 2022-03-21T15:49:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:53 smithi079 ceph-mon[23158]: pgmap v3361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5894.983824 secs 2022-03-21T15:49:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:53 smithi002 ceph-mon[17167]: pgmap v3361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5894.983824 secs 2022-03-21T15:49:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:53 smithi016 ceph-mon[22010]: pgmap v3361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:49:55.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:55 smithi079 ceph-mon[23158]: pgmap v3362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:55 smithi002 ceph-mon[17167]: pgmap v3362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:55 smithi016 ceph-mon[22010]: pgmap v3362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:49:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:56 smithi002 ceph-mon[17167]: pgmap v3363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:56 smithi016 ceph-mon[22010]: pgmap v3363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:56 smithi079 ceph-mon[23158]: pgmap v3363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:49:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5899.983915 secs 2022-03-21T15:49:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5899.983915 secs 2022-03-21T15:49:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5899.983915 secs 2022-03-21T15:49:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:49:59 smithi016 ceph-mon[22010]: pgmap v3364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:49:59 smithi079 ceph-mon[23158]: pgmap v3364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:49:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:49:59 smithi002 ceph-mon[17167]: pgmap v3364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:00.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:50:00.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:50:00.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T15:50:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:01 smithi016 ceph-mon[22010]: pgmap v3365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:01.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:01 smithi079 ceph-mon[23158]: pgmap v3365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:01.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:01 smithi002 ceph-mon[17167]: pgmap v3365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:03.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5904.983996 secs 2022-03-21T15:50:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:03 smithi079 ceph-mon[23158]: pgmap v3366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5904.983996 secs 2022-03-21T15:50:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:03 smithi002 ceph-mon[17167]: pgmap v3366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5904.983996 secs 2022-03-21T15:50:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:03 smithi016 ceph-mon[22010]: pgmap v3366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:05.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:05 smithi079 ceph-mon[23158]: pgmap v3367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:05 smithi002 ceph-mon[17167]: pgmap v3367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:05 smithi016 ceph-mon[22010]: pgmap v3367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:06 smithi002 ceph-mon[17167]: pgmap v3368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:06 smithi016 ceph-mon[22010]: pgmap v3368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:06 smithi079 ceph-mon[23158]: pgmap v3368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5909.984111 secs 2022-03-21T15:50:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5909.984111 secs 2022-03-21T15:50:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5909.984111 secs 2022-03-21T15:50:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:09 smithi016 ceph-mon[22010]: pgmap v3369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:09 smithi079 ceph-mon[23158]: pgmap v3369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:09 smithi002 ceph-mon[17167]: pgmap v3369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:11 smithi079 ceph-mon[23158]: pgmap v3370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:11 smithi002 ceph-mon[17167]: pgmap v3370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:11 smithi016 ceph-mon[22010]: pgmap v3370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5914.984262 secs 2022-03-21T15:50:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:13 smithi002 ceph-mon[17167]: pgmap v3371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5914.984262 secs 2022-03-21T15:50:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:13 smithi016 ceph-mon[22010]: pgmap v3371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5914.984262 secs 2022-03-21T15:50:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:13 smithi079 ceph-mon[23158]: pgmap v3371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:15 smithi002 ceph-mon[17167]: pgmap v3372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:15 smithi016 ceph-mon[22010]: pgmap v3372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:15 smithi079 ceph-mon[23158]: pgmap v3372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:16 smithi002 ceph-mon[17167]: pgmap v3373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:16 smithi016 ceph-mon[22010]: pgmap v3373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:16 smithi079 ceph-mon[23158]: pgmap v3373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:18.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5919.984375 secs 2022-03-21T15:50:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5919.984375 secs 2022-03-21T15:50:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5919.984375 secs 2022-03-21T15:50:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:19 smithi016 ceph-mon[22010]: pgmap v3374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:19 smithi079 ceph-mon[23158]: pgmap v3374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:19 smithi002 ceph-mon[17167]: pgmap v3374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:21 smithi002 ceph-mon[17167]: pgmap v3375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:50:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:21 smithi016 ceph-mon[22010]: pgmap v3375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:50:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:21 smithi079 ceph-mon[23158]: pgmap v3375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:50:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:50:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5924.984492 secs 2022-03-21T15:50:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:23 smithi002 ceph-mon[17167]: pgmap v3376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5924.984492 secs 2022-03-21T15:50:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:23 smithi016 ceph-mon[22010]: pgmap v3376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5924.984492 secs 2022-03-21T15:50:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:23 smithi079 ceph-mon[23158]: pgmap v3376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:25 smithi002 ceph-mon[17167]: pgmap v3377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:25 smithi016 ceph-mon[22010]: pgmap v3377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:25 smithi079 ceph-mon[23158]: pgmap v3377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:26 smithi002 ceph-mon[17167]: pgmap v3378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:26 smithi016 ceph-mon[22010]: pgmap v3378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:26 smithi079 ceph-mon[23158]: pgmap v3378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5929.984594 secs 2022-03-21T15:50:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5929.984594 secs 2022-03-21T15:50:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5929.984594 secs 2022-03-21T15:50:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:29 smithi016 ceph-mon[22010]: pgmap v3379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:29 smithi079 ceph-mon[23158]: pgmap v3379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:29 smithi002 ceph-mon[17167]: pgmap v3379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:31 smithi002 ceph-mon[17167]: pgmap v3380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:31 smithi016 ceph-mon[22010]: pgmap v3380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:31 smithi079 ceph-mon[23158]: pgmap v3380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5934.984679 secs 2022-03-21T15:50:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:33 smithi002 ceph-mon[17167]: pgmap v3381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5934.984679 secs 2022-03-21T15:50:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:33 smithi016 ceph-mon[22010]: pgmap v3381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5934.984679 secs 2022-03-21T15:50:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:33 smithi079 ceph-mon[23158]: pgmap v3381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:35 smithi002 ceph-mon[17167]: pgmap v3382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:35 smithi016 ceph-mon[22010]: pgmap v3382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:35 smithi079 ceph-mon[23158]: pgmap v3382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:36 smithi002 ceph-mon[17167]: pgmap v3383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:36 smithi016 ceph-mon[22010]: pgmap v3383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:36 smithi079 ceph-mon[23158]: pgmap v3383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:38.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5939.984814 secs 2022-03-21T15:50:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5939.984814 secs 2022-03-21T15:50:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5939.984814 secs 2022-03-21T15:50:39.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:39 smithi016 ceph-mon[22010]: pgmap v3384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:39 smithi079 ceph-mon[23158]: pgmap v3384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:39 smithi002 ceph-mon[17167]: pgmap v3384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:41 smithi002 ceph-mon[17167]: pgmap v3385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:41 smithi016 ceph-mon[22010]: pgmap v3385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:41 smithi079 ceph-mon[23158]: pgmap v3385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:50:42.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:50:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:50:42.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:50:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:50:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:50:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:50:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:50:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:50:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5944.984906 secs 2022-03-21T15:50:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:43 smithi002 ceph-mon[17167]: pgmap v3386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:50:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5944.984906 secs 2022-03-21T15:50:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:43 smithi016 ceph-mon[22010]: pgmap v3386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:50:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5944.984906 secs 2022-03-21T15:50:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:43 smithi079 ceph-mon[23158]: pgmap v3386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:50:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:45 smithi002 ceph-mon[17167]: pgmap v3387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:45 smithi016 ceph-mon[22010]: pgmap v3387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:45 smithi079 ceph-mon[23158]: pgmap v3387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:46 smithi002 ceph-mon[17167]: pgmap v3388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:46 smithi016 ceph-mon[22010]: pgmap v3388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:46 smithi079 ceph-mon[23158]: pgmap v3388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5949.985023 secs 2022-03-21T15:50:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5949.985023 secs 2022-03-21T15:50:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5949.985023 secs 2022-03-21T15:50:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:49 smithi016 ceph-mon[22010]: pgmap v3389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:49 smithi079 ceph-mon[23158]: pgmap v3389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:49 smithi002 ceph-mon[17167]: pgmap v3389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:51 smithi002 ceph-mon[17167]: pgmap v3390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:51 smithi016 ceph-mon[22010]: pgmap v3390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:51 smithi079 ceph-mon[23158]: pgmap v3390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5954.985144 secs 2022-03-21T15:50:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:53 smithi002 ceph-mon[17167]: pgmap v3391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5954.985144 secs 2022-03-21T15:50:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:53 smithi016 ceph-mon[22010]: pgmap v3391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5954.985144 secs 2022-03-21T15:50:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:53 smithi079 ceph-mon[23158]: pgmap v3391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:50:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:55 smithi002 ceph-mon[17167]: pgmap v3392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:55 smithi016 ceph-mon[22010]: pgmap v3392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:55 smithi079 ceph-mon[23158]: pgmap v3392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:50:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:56 smithi002 ceph-mon[17167]: pgmap v3393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:56 smithi016 ceph-mon[22010]: pgmap v3393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:56 smithi079 ceph-mon[23158]: pgmap v3393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:50:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5959.985260 secs 2022-03-21T15:50:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5959.985260 secs 2022-03-21T15:50:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5959.985260 secs 2022-03-21T15:50:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:50:59 smithi079 ceph-mon[23158]: pgmap v3394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:50:59 smithi002 ceph-mon[17167]: pgmap v3394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:50:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:50:59 smithi016 ceph-mon[22010]: pgmap v3394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:01 smithi002 ceph-mon[17167]: pgmap v3395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:01 smithi016 ceph-mon[22010]: pgmap v3395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:01 smithi079 ceph-mon[23158]: pgmap v3395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5964.985361 secs 2022-03-21T15:51:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:03 smithi002 ceph-mon[17167]: pgmap v3396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5964.985361 secs 2022-03-21T15:51:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:03 smithi016 ceph-mon[22010]: pgmap v3396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5964.985361 secs 2022-03-21T15:51:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:03 smithi079 ceph-mon[23158]: pgmap v3396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:05 smithi002 ceph-mon[17167]: pgmap v3397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:05 smithi016 ceph-mon[22010]: pgmap v3397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:05 smithi079 ceph-mon[23158]: pgmap v3397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:06 smithi002 ceph-mon[17167]: pgmap v3398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:06 smithi016 ceph-mon[22010]: pgmap v3398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:06 smithi079 ceph-mon[23158]: pgmap v3398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5969.985505 secs 2022-03-21T15:51:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5969.985505 secs 2022-03-21T15:51:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5969.985505 secs 2022-03-21T15:51:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:09 smithi079 ceph-mon[23158]: pgmap v3399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:09 smithi002 ceph-mon[17167]: pgmap v3399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:09 smithi016 ceph-mon[22010]: pgmap v3399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:11 smithi002 ceph-mon[17167]: pgmap v3400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:11 smithi016 ceph-mon[22010]: pgmap v3400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:11 smithi079 ceph-mon[23158]: pgmap v3400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5974.985624 secs 2022-03-21T15:51:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:13 smithi002 ceph-mon[17167]: pgmap v3401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5974.985624 secs 2022-03-21T15:51:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:13 smithi016 ceph-mon[22010]: pgmap v3401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5974.985624 secs 2022-03-21T15:51:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:13 smithi079 ceph-mon[23158]: pgmap v3401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:15 smithi002 ceph-mon[17167]: pgmap v3402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:15 smithi016 ceph-mon[22010]: pgmap v3402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:15 smithi079 ceph-mon[23158]: pgmap v3402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:16 smithi002 ceph-mon[17167]: pgmap v3403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:16 smithi016 ceph-mon[22010]: pgmap v3403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:16 smithi079 ceph-mon[23158]: pgmap v3403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5979.985742 secs 2022-03-21T15:51:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5979.985742 secs 2022-03-21T15:51:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5979.985742 secs 2022-03-21T15:51:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:19 smithi079 ceph-mon[23158]: pgmap v3404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:19 smithi002 ceph-mon[17167]: pgmap v3404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:19 smithi016 ceph-mon[22010]: pgmap v3404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:21 smithi002 ceph-mon[17167]: pgmap v3405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:51:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:21 smithi016 ceph-mon[22010]: pgmap v3405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:51:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:21 smithi079 ceph-mon[23158]: pgmap v3405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:51:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:51:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5984.985853 secs 2022-03-21T15:51:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:23 smithi002 ceph-mon[17167]: pgmap v3406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5984.985853 secs 2022-03-21T15:51:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:23 smithi016 ceph-mon[22010]: pgmap v3406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5984.985853 secs 2022-03-21T15:51:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:23 smithi079 ceph-mon[23158]: pgmap v3406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:25 smithi002 ceph-mon[17167]: pgmap v3407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:25 smithi016 ceph-mon[22010]: pgmap v3407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:25 smithi079 ceph-mon[23158]: pgmap v3407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:26 smithi002 ceph-mon[17167]: pgmap v3408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:26 smithi016 ceph-mon[22010]: pgmap v3408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:26 smithi079 ceph-mon[23158]: pgmap v3408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5989.985940 secs 2022-03-21T15:51:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5989.985940 secs 2022-03-21T15:51:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5989.985940 secs 2022-03-21T15:51:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:29 smithi079 ceph-mon[23158]: pgmap v3409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:29 smithi002 ceph-mon[17167]: pgmap v3409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:29 smithi016 ceph-mon[22010]: pgmap v3409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:31 smithi002 ceph-mon[17167]: pgmap v3410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:31 smithi016 ceph-mon[22010]: pgmap v3410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:31 smithi079 ceph-mon[23158]: pgmap v3410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5994.986056 secs 2022-03-21T15:51:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:33 smithi002 ceph-mon[17167]: pgmap v3411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5994.986056 secs 2022-03-21T15:51:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:33 smithi016 ceph-mon[22010]: pgmap v3411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5994.986056 secs 2022-03-21T15:51:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:33 smithi079 ceph-mon[23158]: pgmap v3411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:35 smithi002 ceph-mon[17167]: pgmap v3412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:35 smithi016 ceph-mon[22010]: pgmap v3412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:35 smithi079 ceph-mon[23158]: pgmap v3412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:36 smithi002 ceph-mon[17167]: pgmap v3413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:36 smithi016 ceph-mon[22010]: pgmap v3413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:36 smithi079 ceph-mon[23158]: pgmap v3413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 5999.986170 secs 2022-03-21T15:51:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 5999.986170 secs 2022-03-21T15:51:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 5999.986170 secs 2022-03-21T15:51:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:39 smithi079 ceph-mon[23158]: pgmap v3414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:39 smithi002 ceph-mon[17167]: pgmap v3414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:39 smithi016 ceph-mon[22010]: pgmap v3414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:41 smithi002 ceph-mon[17167]: pgmap v3415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:41 smithi016 ceph-mon[22010]: pgmap v3415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:41 smithi079 ceph-mon[23158]: pgmap v3415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6004.986293 secs 2022-03-21T15:51:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:43 smithi002 ceph-mon[17167]: pgmap v3416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:51:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:51:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:51:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6004.986293 secs 2022-03-21T15:51:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:43 smithi016 ceph-mon[22010]: pgmap v3416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:51:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:51:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:51:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6004.986293 secs 2022-03-21T15:51:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:43 smithi079 ceph-mon[23158]: pgmap v3416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:51:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:51:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:51:45.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:45 smithi016 ceph-mon[22010]: pgmap v3417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:45 smithi002 ceph-mon[17167]: pgmap v3417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:45 smithi079 ceph-mon[23158]: pgmap v3417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:46.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:46 smithi002 ceph-mon[17167]: pgmap v3418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:46.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:46 smithi016 ceph-mon[22010]: pgmap v3418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:46.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:46 smithi079 ceph-mon[23158]: pgmap v3418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:48.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:51:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6009.986405 secs 2022-03-21T15:51:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:48 smithi002 ceph-mon[17167]: pgmap v3419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:49.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6009.986405 secs 2022-03-21T15:51:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:48 smithi016 ceph-mon[22010]: pgmap v3419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6009.986405 secs 2022-03-21T15:51:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:48 smithi079 ceph-mon[23158]: pgmap v3419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:51.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:51 smithi016 ceph-mon[22010]: pgmap v3420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:51 smithi002 ceph-mon[17167]: pgmap v3420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:51 smithi079 ceph-mon[23158]: pgmap v3420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6014.986538 secs 2022-03-21T15:51:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:53 smithi002 ceph-mon[17167]: pgmap v3421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6014.986538 secs 2022-03-21T15:51:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:53 smithi016 ceph-mon[22010]: pgmap v3421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6014.986538 secs 2022-03-21T15:51:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:53 smithi079 ceph-mon[23158]: pgmap v3421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:51:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:55 smithi016 ceph-mon[22010]: pgmap v3422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:55 smithi002 ceph-mon[17167]: pgmap v3422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:55 smithi079 ceph-mon[23158]: pgmap v3422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:51:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:56 smithi002 ceph-mon[17167]: pgmap v3423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:56 smithi016 ceph-mon[22010]: pgmap v3423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:56 smithi079 ceph-mon[23158]: pgmap v3423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:51:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6019.986653 secs 2022-03-21T15:51:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6019.986653 secs 2022-03-21T15:51:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6019.986653 secs 2022-03-21T15:51:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:51:59 smithi079 ceph-mon[23158]: pgmap v3424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:51:59 smithi002 ceph-mon[17167]: pgmap v3424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:51:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:51:59 smithi016 ceph-mon[22010]: pgmap v3424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:01 smithi002 ceph-mon[17167]: pgmap v3425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:01 smithi016 ceph-mon[22010]: pgmap v3425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:01 smithi079 ceph-mon[23158]: pgmap v3425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6024.986770 secs 2022-03-21T15:52:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:03 smithi002 ceph-mon[17167]: pgmap v3426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6024.986770 secs 2022-03-21T15:52:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:03 smithi016 ceph-mon[22010]: pgmap v3426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6024.986770 secs 2022-03-21T15:52:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:03 smithi079 ceph-mon[23158]: pgmap v3426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:05 smithi002 ceph-mon[17167]: pgmap v3427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:05 smithi016 ceph-mon[22010]: pgmap v3427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:05 smithi079 ceph-mon[23158]: pgmap v3427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:06 smithi002 ceph-mon[17167]: pgmap v3428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:06 smithi016 ceph-mon[22010]: pgmap v3428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:06 smithi079 ceph-mon[23158]: pgmap v3428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6029.986881 secs 2022-03-21T15:52:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6029.986881 secs 2022-03-21T15:52:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6029.986881 secs 2022-03-21T15:52:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:09 smithi079 ceph-mon[23158]: pgmap v3429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:09 smithi002 ceph-mon[17167]: pgmap v3429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:09 smithi016 ceph-mon[22010]: pgmap v3429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:11 smithi002 ceph-mon[17167]: pgmap v3430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:11 smithi016 ceph-mon[22010]: pgmap v3430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:11 smithi079 ceph-mon[23158]: pgmap v3430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6034.986991 secs 2022-03-21T15:52:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:13 smithi002 ceph-mon[17167]: pgmap v3431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6034.986991 secs 2022-03-21T15:52:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:13 smithi016 ceph-mon[22010]: pgmap v3431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6034.986991 secs 2022-03-21T15:52:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:13 smithi079 ceph-mon[23158]: pgmap v3431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:15 smithi002 ceph-mon[17167]: pgmap v3432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:15 smithi016 ceph-mon[22010]: pgmap v3432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:15 smithi079 ceph-mon[23158]: pgmap v3432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:16 smithi002 ceph-mon[17167]: pgmap v3433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:16 smithi016 ceph-mon[22010]: pgmap v3433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:16 smithi079 ceph-mon[23158]: pgmap v3433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6039.987133 secs 2022-03-21T15:52:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6039.987133 secs 2022-03-21T15:52:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6039.987133 secs 2022-03-21T15:52:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:19 smithi079 ceph-mon[23158]: pgmap v3434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:19 smithi002 ceph-mon[17167]: pgmap v3434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:19 smithi016 ceph-mon[22010]: pgmap v3434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:21 smithi002 ceph-mon[17167]: pgmap v3435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:52:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:52:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:21 smithi016 ceph-mon[22010]: pgmap v3435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:52:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:52:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:21 smithi079 ceph-mon[23158]: pgmap v3435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:52:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:52:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6044.987245 secs 2022-03-21T15:52:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:23 smithi002 ceph-mon[17167]: pgmap v3436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6044.987245 secs 2022-03-21T15:52:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:23 smithi016 ceph-mon[22010]: pgmap v3436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6044.987245 secs 2022-03-21T15:52:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:23 smithi079 ceph-mon[23158]: pgmap v3436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:25 smithi002 ceph-mon[17167]: pgmap v3437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:25 smithi016 ceph-mon[22010]: pgmap v3437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:25 smithi079 ceph-mon[23158]: pgmap v3437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:26 smithi002 ceph-mon[17167]: pgmap v3438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:26 smithi016 ceph-mon[22010]: pgmap v3438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:26 smithi079 ceph-mon[23158]: pgmap v3438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6049.987355 secs 2022-03-21T15:52:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6049.987355 secs 2022-03-21T15:52:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6049.987355 secs 2022-03-21T15:52:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:29 smithi079 ceph-mon[23158]: pgmap v3439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:29 smithi002 ceph-mon[17167]: pgmap v3439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:29 smithi016 ceph-mon[22010]: pgmap v3439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:31 smithi002 ceph-mon[17167]: pgmap v3440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:31 smithi016 ceph-mon[22010]: pgmap v3440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:31 smithi079 ceph-mon[23158]: pgmap v3440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6054.987483 secs 2022-03-21T15:52:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:33 smithi002 ceph-mon[17167]: pgmap v3441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6054.987483 secs 2022-03-21T15:52:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:33 smithi016 ceph-mon[22010]: pgmap v3441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6054.987483 secs 2022-03-21T15:52:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:33 smithi079 ceph-mon[23158]: pgmap v3441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:35 smithi002 ceph-mon[17167]: pgmap v3442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:35 smithi016 ceph-mon[22010]: pgmap v3442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:35 smithi079 ceph-mon[23158]: pgmap v3442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:36 smithi002 ceph-mon[17167]: pgmap v3443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:36 smithi016 ceph-mon[22010]: pgmap v3443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:36 smithi079 ceph-mon[23158]: pgmap v3443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6059.987601 secs 2022-03-21T15:52:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6059.987601 secs 2022-03-21T15:52:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6059.987601 secs 2022-03-21T15:52:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:39 smithi079 ceph-mon[23158]: pgmap v3444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:39 smithi002 ceph-mon[17167]: pgmap v3444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:39 smithi016 ceph-mon[22010]: pgmap v3444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:41 smithi002 ceph-mon[17167]: pgmap v3445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:41 smithi016 ceph-mon[22010]: pgmap v3445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:41 smithi079 ceph-mon[23158]: pgmap v3445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6064.987682 secs 2022-03-21T15:52:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:43 smithi002 ceph-mon[17167]: pgmap v3446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6064.987682 secs 2022-03-21T15:52:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:43 smithi016 ceph-mon[22010]: pgmap v3446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6064.987682 secs 2022-03-21T15:52:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:43 smithi079 ceph-mon[23158]: pgmap v3446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:45 smithi002 ceph-mon[17167]: pgmap v3447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:45 smithi016 ceph-mon[22010]: pgmap v3447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:45 smithi079 ceph-mon[23158]: pgmap v3447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:46 smithi002 ceph-mon[17167]: pgmap v3448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:46 smithi016 ceph-mon[22010]: pgmap v3448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:46 smithi079 ceph-mon[23158]: pgmap v3448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:52:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:52:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:52:48.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:52:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:52:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:52:48.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:52:48.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:52:48.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:52:49.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6069.987789 secs 2022-03-21T15:52:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:48 smithi002 ceph-mon[17167]: pgmap v3449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:52:49.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6069.987789 secs 2022-03-21T15:52:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:48 smithi016 ceph-mon[22010]: pgmap v3449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:52:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6069.987789 secs 2022-03-21T15:52:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:48 smithi079 ceph-mon[23158]: pgmap v3449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:52:51.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:51 smithi002 ceph-mon[17167]: pgmap v3450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:51 smithi016 ceph-mon[22010]: pgmap v3450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:51 smithi079 ceph-mon[23158]: pgmap v3450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6074.987907 secs 2022-03-21T15:52:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:53 smithi002 ceph-mon[17167]: pgmap v3451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6074.987907 secs 2022-03-21T15:52:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:53 smithi016 ceph-mon[22010]: pgmap v3451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6074.987907 secs 2022-03-21T15:52:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:53 smithi079 ceph-mon[23158]: pgmap v3451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:52:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:55 smithi002 ceph-mon[17167]: pgmap v3452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:55 smithi016 ceph-mon[22010]: pgmap v3452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:55 smithi079 ceph-mon[23158]: pgmap v3452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:52:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:56 smithi002 ceph-mon[17167]: pgmap v3453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:56 smithi016 ceph-mon[22010]: pgmap v3453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:56 smithi079 ceph-mon[23158]: pgmap v3453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:52:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6079.988041 secs 2022-03-21T15:52:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6079.988041 secs 2022-03-21T15:52:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6079.988041 secs 2022-03-21T15:52:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:52:59 smithi079 ceph-mon[23158]: pgmap v3454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:52:59 smithi002 ceph-mon[17167]: pgmap v3454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:52:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:52:59 smithi016 ceph-mon[22010]: pgmap v3454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:01 smithi002 ceph-mon[17167]: pgmap v3455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:01 smithi016 ceph-mon[22010]: pgmap v3455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:01 smithi079 ceph-mon[23158]: pgmap v3455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6084.988163 secs 2022-03-21T15:53:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:03 smithi002 ceph-mon[17167]: pgmap v3456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6084.988163 secs 2022-03-21T15:53:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:03 smithi016 ceph-mon[22010]: pgmap v3456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6084.988163 secs 2022-03-21T15:53:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:03 smithi079 ceph-mon[23158]: pgmap v3456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:05 smithi002 ceph-mon[17167]: pgmap v3457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:05 smithi016 ceph-mon[22010]: pgmap v3457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:05 smithi079 ceph-mon[23158]: pgmap v3457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:06 smithi002 ceph-mon[17167]: pgmap v3458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:06 smithi016 ceph-mon[22010]: pgmap v3458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:06 smithi079 ceph-mon[23158]: pgmap v3458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6089.988280 secs 2022-03-21T15:53:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6089.988280 secs 2022-03-21T15:53:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6089.988280 secs 2022-03-21T15:53:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:09 smithi079 ceph-mon[23158]: pgmap v3459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:09 smithi002 ceph-mon[17167]: pgmap v3459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:09 smithi016 ceph-mon[22010]: pgmap v3459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:11 smithi002 ceph-mon[17167]: pgmap v3460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:11 smithi016 ceph-mon[22010]: pgmap v3460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:11 smithi079 ceph-mon[23158]: pgmap v3460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6094.988391 secs 2022-03-21T15:53:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:13 smithi002 ceph-mon[17167]: pgmap v3461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6094.988391 secs 2022-03-21T15:53:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:13 smithi016 ceph-mon[22010]: pgmap v3461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6094.988391 secs 2022-03-21T15:53:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:13 smithi079 ceph-mon[23158]: pgmap v3461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:15 smithi002 ceph-mon[17167]: pgmap v3462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:15 smithi016 ceph-mon[22010]: pgmap v3462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:15 smithi079 ceph-mon[23158]: pgmap v3462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:16 smithi002 ceph-mon[17167]: pgmap v3463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:16 smithi016 ceph-mon[22010]: pgmap v3463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:16 smithi079 ceph-mon[23158]: pgmap v3463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6099.988527 secs 2022-03-21T15:53:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6099.988527 secs 2022-03-21T15:53:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6099.988527 secs 2022-03-21T15:53:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:19 smithi079 ceph-mon[23158]: pgmap v3464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:19 smithi002 ceph-mon[17167]: pgmap v3464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:19 smithi016 ceph-mon[22010]: pgmap v3464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:21.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:21 smithi002 ceph-mon[17167]: pgmap v3465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:53:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:53:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:21 smithi016 ceph-mon[22010]: pgmap v3465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:53:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:53:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:21 smithi079 ceph-mon[23158]: pgmap v3465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:53:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:53:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6104.988650 secs 2022-03-21T15:53:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:23 smithi002 ceph-mon[17167]: pgmap v3466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6104.988650 secs 2022-03-21T15:53:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:23 smithi016 ceph-mon[22010]: pgmap v3466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6104.988650 secs 2022-03-21T15:53:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:23 smithi079 ceph-mon[23158]: pgmap v3466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:25 smithi002 ceph-mon[17167]: pgmap v3467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:25 smithi016 ceph-mon[22010]: pgmap v3467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:25 smithi079 ceph-mon[23158]: pgmap v3467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:26 smithi002 ceph-mon[17167]: pgmap v3468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:26 smithi016 ceph-mon[22010]: pgmap v3468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:26 smithi079 ceph-mon[23158]: pgmap v3468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6109.988738 secs 2022-03-21T15:53:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6109.988738 secs 2022-03-21T15:53:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6109.988738 secs 2022-03-21T15:53:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:29 smithi079 ceph-mon[23158]: pgmap v3469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:29 smithi002 ceph-mon[17167]: pgmap v3469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:29 smithi016 ceph-mon[22010]: pgmap v3469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:31 smithi002 ceph-mon[17167]: pgmap v3470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:31 smithi016 ceph-mon[22010]: pgmap v3470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:31 smithi079 ceph-mon[23158]: pgmap v3470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6114.988889 secs 2022-03-21T15:53:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:33 smithi002 ceph-mon[17167]: pgmap v3471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6114.988889 secs 2022-03-21T15:53:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:33 smithi016 ceph-mon[22010]: pgmap v3471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6114.988889 secs 2022-03-21T15:53:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:33 smithi079 ceph-mon[23158]: pgmap v3471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:35 smithi002 ceph-mon[17167]: pgmap v3472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:35 smithi016 ceph-mon[22010]: pgmap v3472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:35 smithi079 ceph-mon[23158]: pgmap v3472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:36 smithi002 ceph-mon[17167]: pgmap v3473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:36 smithi016 ceph-mon[22010]: pgmap v3473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:36 smithi079 ceph-mon[23158]: pgmap v3473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T15:53:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6119.989021 secs 2022-03-21T15:53:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6119.989021 secs 2022-03-21T15:53:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6119.989021 secs 2022-03-21T15:53:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:39 smithi079 ceph-mon[23158]: pgmap v3474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:39 smithi002 ceph-mon[17167]: pgmap v3474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:39.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:39 smithi016 ceph-mon[22010]: pgmap v3474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:41 smithi002 ceph-mon[17167]: pgmap v3475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:41 smithi016 ceph-mon[22010]: pgmap v3475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:41 smithi079 ceph-mon[23158]: pgmap v3475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:43.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6124.989161 secs 2022-03-21T15:53:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:43 smithi002 ceph-mon[17167]: pgmap v3476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6124.989161 secs 2022-03-21T15:53:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:43 smithi016 ceph-mon[22010]: pgmap v3476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6124.989161 secs 2022-03-21T15:53:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:43 smithi079 ceph-mon[23158]: pgmap v3476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:45.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:45 smithi002 ceph-mon[17167]: pgmap v3477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:45 smithi016 ceph-mon[22010]: pgmap v3477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:45 smithi079 ceph-mon[23158]: pgmap v3477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:46 smithi002 ceph-mon[17167]: pgmap v3478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:46 smithi016 ceph-mon[22010]: pgmap v3478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:46 smithi079 ceph-mon[23158]: pgmap v3478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6129.989239 secs 2022-03-21T15:53:48.561 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6129.989239 secs 2022-03-21T15:53:48.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6129.989239 secs 2022-03-21T15:53:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: pgmap v3479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:53:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:53:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:53:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: pgmap v3479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:53:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:53:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:53:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: pgmap v3479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T15:53:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:53:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:53:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:53:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:51.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:51 smithi002 ceph-mon[17167]: pgmap v3480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:51 smithi016 ceph-mon[22010]: pgmap v3480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:53:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:51 smithi079 ceph-mon[23158]: pgmap v3480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6134.989359 secs 2022-03-21T15:53:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:53 smithi002 ceph-mon[17167]: pgmap v3481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6134.989359 secs 2022-03-21T15:53:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:53 smithi016 ceph-mon[22010]: pgmap v3481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6134.989359 secs 2022-03-21T15:53:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:53 smithi079 ceph-mon[23158]: pgmap v3481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:55 smithi002 ceph-mon[17167]: pgmap v3482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:55 smithi016 ceph-mon[22010]: pgmap v3482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:55 smithi079 ceph-mon[23158]: pgmap v3482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:56 smithi002 ceph-mon[17167]: pgmap v3483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:56 smithi016 ceph-mon[22010]: pgmap v3483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:56 smithi079 ceph-mon[23158]: pgmap v3483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:53:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6139.989483 secs 2022-03-21T15:53:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6139.989483 secs 2022-03-21T15:53:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6139.989483 secs 2022-03-21T15:53:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:53:59 smithi079 ceph-mon[23158]: pgmap v3484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:59.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:53:59 smithi002 ceph-mon[17167]: pgmap v3484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:53:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:53:59 smithi016 ceph-mon[22010]: pgmap v3484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:01.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:01 smithi002 ceph-mon[17167]: pgmap v3485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:01 smithi016 ceph-mon[22010]: pgmap v3485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:01 smithi079 ceph-mon[23158]: pgmap v3485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:03.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6144.989615 secs 2022-03-21T15:54:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:03 smithi002 ceph-mon[17167]: pgmap v3486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6144.989615 secs 2022-03-21T15:54:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:03 smithi016 ceph-mon[22010]: pgmap v3486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6144.989615 secs 2022-03-21T15:54:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:03 smithi079 ceph-mon[23158]: pgmap v3486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:05 smithi002 ceph-mon[17167]: pgmap v3487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:05 smithi016 ceph-mon[22010]: pgmap v3487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:05 smithi079 ceph-mon[23158]: pgmap v3487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:06 smithi002 ceph-mon[17167]: pgmap v3488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:06 smithi016 ceph-mon[22010]: pgmap v3488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:06 smithi079 ceph-mon[23158]: pgmap v3488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6149.989710 secs 2022-03-21T15:54:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6149.989710 secs 2022-03-21T15:54:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6149.989710 secs 2022-03-21T15:54:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:09 smithi079 ceph-mon[23158]: pgmap v3489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:09 smithi002 ceph-mon[17167]: pgmap v3489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:09 smithi016 ceph-mon[22010]: pgmap v3489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:11.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:11 smithi002 ceph-mon[17167]: pgmap v3490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:11 smithi016 ceph-mon[22010]: pgmap v3490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:11 smithi079 ceph-mon[23158]: pgmap v3490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6154.989791 secs 2022-03-21T15:54:13.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:13 smithi002 ceph-mon[17167]: pgmap v3491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6154.989791 secs 2022-03-21T15:54:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:13 smithi016 ceph-mon[22010]: pgmap v3491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6154.989791 secs 2022-03-21T15:54:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:13 smithi079 ceph-mon[23158]: pgmap v3491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:15.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:15 smithi002 ceph-mon[17167]: pgmap v3492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:15 smithi016 ceph-mon[22010]: pgmap v3492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:15 smithi079 ceph-mon[23158]: pgmap v3492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T15:54:17.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:16 smithi002 ceph-mon[17167]: pgmap v3493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:16 smithi016 ceph-mon[22010]: pgmap v3493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:16 smithi079 ceph-mon[23158]: pgmap v3493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6159.989899 secs 2022-03-21T15:54:18.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6159.989899 secs 2022-03-21T15:54:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6159.989899 secs 2022-03-21T15:54:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:19 smithi079 ceph-mon[23158]: pgmap v3494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:19.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:19 smithi002 ceph-mon[17167]: pgmap v3494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:19 smithi016 ceph-mon[22010]: pgmap v3494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:21 smithi002 ceph-mon[17167]: pgmap v3495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:54:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:54:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:21 smithi016 ceph-mon[22010]: pgmap v3495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:54:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:54:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:21 smithi079 ceph-mon[23158]: pgmap v3495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:54:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:54:23.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6164.990035 secs 2022-03-21T15:54:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:23 smithi002 ceph-mon[17167]: pgmap v3496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6164.990035 secs 2022-03-21T15:54:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:23 smithi016 ceph-mon[22010]: pgmap v3496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6164.990035 secs 2022-03-21T15:54:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:23 smithi079 ceph-mon[23158]: pgmap v3496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:25.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:25 smithi002 ceph-mon[17167]: pgmap v3497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:25 smithi016 ceph-mon[22010]: pgmap v3497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:25.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:25 smithi079 ceph-mon[23158]: pgmap v3497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:27.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:26 smithi002 ceph-mon[17167]: pgmap v3498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:26 smithi016 ceph-mon[22010]: pgmap v3498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:26 smithi079 ceph-mon[23158]: pgmap v3498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6169.990156 secs 2022-03-21T15:54:28.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6169.990156 secs 2022-03-21T15:54:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6169.990156 secs 2022-03-21T15:54:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:29 smithi079 ceph-mon[23158]: pgmap v3499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:29 smithi002 ceph-mon[17167]: pgmap v3499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:29 smithi016 ceph-mon[22010]: pgmap v3499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:31.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:31 smithi002 ceph-mon[17167]: pgmap v3500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:31.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:31 smithi016 ceph-mon[22010]: pgmap v3500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:31 smithi079 ceph-mon[23158]: pgmap v3500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:33.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6174.990288 secs 2022-03-21T15:54:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:33 smithi002 ceph-mon[17167]: pgmap v3501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6174.990288 secs 2022-03-21T15:54:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:33 smithi016 ceph-mon[22010]: pgmap v3501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6174.990288 secs 2022-03-21T15:54:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:33 smithi079 ceph-mon[23158]: pgmap v3501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:35.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:35 smithi002 ceph-mon[17167]: pgmap v3502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:35 smithi016 ceph-mon[22010]: pgmap v3502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:35 smithi079 ceph-mon[23158]: pgmap v3502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:37.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:36 smithi002 ceph-mon[17167]: pgmap v3503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:37.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:36 smithi016 ceph-mon[22010]: pgmap v3503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:36 smithi079 ceph-mon[23158]: pgmap v3503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6179.990392 secs 2022-03-21T15:54:38.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6179.990392 secs 2022-03-21T15:54:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6179.990392 secs 2022-03-21T15:54:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:39 smithi079 ceph-mon[23158]: pgmap v3504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:39.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:39 smithi002 ceph-mon[17167]: pgmap v3504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:39.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:39 smithi016 ceph-mon[22010]: pgmap v3504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:41.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:41 smithi002 ceph-mon[17167]: pgmap v3505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:41 smithi016 ceph-mon[22010]: pgmap v3505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:41 smithi079 ceph-mon[23158]: pgmap v3505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6184.990525 secs 2022-03-21T15:54:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:43 smithi002 ceph-mon[17167]: pgmap v3506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6184.990525 secs 2022-03-21T15:54:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:43 smithi016 ceph-mon[22010]: pgmap v3506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6184.990525 secs 2022-03-21T15:54:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:43 smithi079 ceph-mon[23158]: pgmap v3506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:45.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:45 smithi002 ceph-mon[17167]: pgmap v3507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:45 smithi016 ceph-mon[22010]: pgmap v3507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:45 smithi079 ceph-mon[23158]: pgmap v3507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:47.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:46 smithi002 ceph-mon[17167]: pgmap v3508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:46 smithi016 ceph-mon[22010]: pgmap v3508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:46 smithi079 ceph-mon[23158]: pgmap v3508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6189.990639 secs 2022-03-21T15:54:48.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6189.990639 secs 2022-03-21T15:54:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6189.990639 secs 2022-03-21T15:54:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:49 smithi079 ceph-mon[23158]: pgmap v3509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:49.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:49 smithi002 ceph-mon[17167]: pgmap v3509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:49 smithi016 ceph-mon[22010]: pgmap v3509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:54:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:54:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:54:50.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:54:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:54:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:54:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:54:50.445 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:54:50.445 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:54:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:51 smithi079 ceph-mon[23158]: pgmap v3510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:54:51.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:51 smithi002 ceph-mon[17167]: pgmap v3510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:54:51.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:51 smithi016 ceph-mon[22010]: pgmap v3510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:54:53.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6194.990754 secs 2022-03-21T15:54:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:53 smithi002 ceph-mon[17167]: pgmap v3511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6194.990754 secs 2022-03-21T15:54:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:53 smithi016 ceph-mon[22010]: pgmap v3511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6194.990754 secs 2022-03-21T15:54:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:53 smithi079 ceph-mon[23158]: pgmap v3511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:55.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:55 smithi002 ceph-mon[17167]: pgmap v3512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:55 smithi016 ceph-mon[22010]: pgmap v3512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:55 smithi079 ceph-mon[23158]: pgmap v3512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:57.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:56 smithi002 ceph-mon[17167]: pgmap v3513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:56 smithi016 ceph-mon[22010]: pgmap v3513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:56 smithi079 ceph-mon[23158]: pgmap v3513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:54:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6199.990825 secs 2022-03-21T15:54:58.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6199.990825 secs 2022-03-21T15:54:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6199.990825 secs 2022-03-21T15:54:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:54:59 smithi079 ceph-mon[23158]: pgmap v3514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:54:59 smithi002 ceph-mon[17167]: pgmap v3514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:54:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:54:59 smithi016 ceph-mon[22010]: pgmap v3514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:01 smithi002 ceph-mon[17167]: pgmap v3515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:01 smithi016 ceph-mon[22010]: pgmap v3515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:01 smithi079 ceph-mon[23158]: pgmap v3515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6204.990942 secs 2022-03-21T15:55:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:03 smithi002 ceph-mon[17167]: pgmap v3516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6204.990942 secs 2022-03-21T15:55:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:03 smithi016 ceph-mon[22010]: pgmap v3516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6204.990942 secs 2022-03-21T15:55:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:03 smithi079 ceph-mon[23158]: pgmap v3516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:05.865 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:05 smithi002 ceph-mon[17167]: pgmap v3517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:05 smithi016 ceph-mon[22010]: pgmap v3517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:05 smithi079 ceph-mon[23158]: pgmap v3517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:07.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:06 smithi002 ceph-mon[17167]: pgmap v3518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:06 smithi016 ceph-mon[22010]: pgmap v3518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:06 smithi079 ceph-mon[23158]: pgmap v3518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6209.991053 secs 2022-03-21T15:55:08.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6209.991053 secs 2022-03-21T15:55:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6209.991053 secs 2022-03-21T15:55:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:09 smithi079 ceph-mon[23158]: pgmap v3519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:09 smithi002 ceph-mon[17167]: pgmap v3519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:09 smithi016 ceph-mon[22010]: pgmap v3519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:11 smithi002 ceph-mon[17167]: pgmap v3520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:11 smithi016 ceph-mon[22010]: pgmap v3520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:11 smithi079 ceph-mon[23158]: pgmap v3520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6214.991177 secs 2022-03-21T15:55:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:13 smithi002 ceph-mon[17167]: pgmap v3521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6214.991177 secs 2022-03-21T15:55:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:13 smithi016 ceph-mon[22010]: pgmap v3521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6214.991177 secs 2022-03-21T15:55:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:13 smithi079 ceph-mon[23158]: pgmap v3521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:15.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:15 smithi002 ceph-mon[17167]: pgmap v3522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:15 smithi016 ceph-mon[22010]: pgmap v3522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:15 smithi079 ceph-mon[23158]: pgmap v3522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:16 smithi002 ceph-mon[17167]: pgmap v3523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:17.247 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:16 smithi079 ceph-mon[23158]: pgmap v3523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:17.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:16 smithi016 ceph-mon[22010]: pgmap v3523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6219.991309 secs 2022-03-21T15:55:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6219.991309 secs 2022-03-21T15:55:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6219.991309 secs 2022-03-21T15:55:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:19 smithi079 ceph-mon[23158]: pgmap v3524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:19 smithi002 ceph-mon[17167]: pgmap v3524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:19 smithi016 ceph-mon[22010]: pgmap v3524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:21 smithi002 ceph-mon[17167]: pgmap v3525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:55:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:55:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:21 smithi016 ceph-mon[22010]: pgmap v3525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:55:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:21 smithi079 ceph-mon[23158]: pgmap v3525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:55:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:55:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6224.991427 secs 2022-03-21T15:55:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:23 smithi002 ceph-mon[17167]: pgmap v3526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6224.991427 secs 2022-03-21T15:55:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:23 smithi016 ceph-mon[22010]: pgmap v3526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6224.991427 secs 2022-03-21T15:55:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:23 smithi079 ceph-mon[23158]: pgmap v3526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:25 smithi002 ceph-mon[17167]: pgmap v3527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:25 smithi016 ceph-mon[22010]: pgmap v3527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:25 smithi079 ceph-mon[23158]: pgmap v3527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:26 smithi002 ceph-mon[17167]: pgmap v3528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:26 smithi016 ceph-mon[22010]: pgmap v3528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:26 smithi079 ceph-mon[23158]: pgmap v3528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6229.991557 secs 2022-03-21T15:55:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6229.991557 secs 2022-03-21T15:55:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6229.991557 secs 2022-03-21T15:55:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:29 smithi079 ceph-mon[23158]: pgmap v3529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:29.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:29 smithi002 ceph-mon[17167]: pgmap v3529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:29 smithi016 ceph-mon[22010]: pgmap v3529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:31.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:31 smithi002 ceph-mon[17167]: pgmap v3530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:31 smithi016 ceph-mon[22010]: pgmap v3530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:31 smithi079 ceph-mon[23158]: pgmap v3530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6234.991691 secs 2022-03-21T15:55:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:33 smithi002 ceph-mon[17167]: pgmap v3531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6234.991691 secs 2022-03-21T15:55:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:33 smithi016 ceph-mon[22010]: pgmap v3531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6234.991691 secs 2022-03-21T15:55:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:33 smithi079 ceph-mon[23158]: pgmap v3531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:35 smithi002 ceph-mon[17167]: pgmap v3532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:35 smithi016 ceph-mon[22010]: pgmap v3532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:35 smithi079 ceph-mon[23158]: pgmap v3532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:36 smithi002 ceph-mon[17167]: pgmap v3533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:36 smithi016 ceph-mon[22010]: pgmap v3533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:36 smithi079 ceph-mon[23158]: pgmap v3533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6239.991801 secs 2022-03-21T15:55:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6239.991801 secs 2022-03-21T15:55:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6239.991801 secs 2022-03-21T15:55:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:39 smithi079 ceph-mon[23158]: pgmap v3534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:39 smithi002 ceph-mon[17167]: pgmap v3534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:39 smithi016 ceph-mon[22010]: pgmap v3534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:41.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:41 smithi002 ceph-mon[17167]: pgmap v3535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:41 smithi016 ceph-mon[22010]: pgmap v3535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:41 smithi079 ceph-mon[23158]: pgmap v3535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6244.991916 secs 2022-03-21T15:55:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:43 smithi002 ceph-mon[17167]: pgmap v3536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6244.991916 secs 2022-03-21T15:55:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:43 smithi016 ceph-mon[22010]: pgmap v3536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6244.991916 secs 2022-03-21T15:55:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:43 smithi079 ceph-mon[23158]: pgmap v3536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:45.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:45 smithi002 ceph-mon[17167]: pgmap v3537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:45 smithi016 ceph-mon[22010]: pgmap v3537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:45 smithi079 ceph-mon[23158]: pgmap v3537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:46 smithi002 ceph-mon[17167]: pgmap v3538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:47.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:46 smithi016 ceph-mon[22010]: pgmap v3538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:46 smithi079 ceph-mon[23158]: pgmap v3538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6249.992048 secs 2022-03-21T15:55:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6249.992048 secs 2022-03-21T15:55:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6249.992048 secs 2022-03-21T15:55:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:49 smithi079 ceph-mon[23158]: pgmap v3539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:49 smithi002 ceph-mon[17167]: pgmap v3539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:49 smithi016 ceph-mon[22010]: pgmap v3539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:51.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:51 smithi002 ceph-mon[17167]: pgmap v3540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:51.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:55:51.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:55:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:55:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:51 smithi016 ceph-mon[22010]: pgmap v3540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:55:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:55:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:55:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:51 smithi079 ceph-mon[23158]: pgmap v3540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:55:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:55:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:55:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:55:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:52 smithi002 ceph-mon[17167]: pgmap v3541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:55:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:52 smithi016 ceph-mon[22010]: pgmap v3541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:55:53.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:52 smithi079 ceph-mon[23158]: pgmap v3541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6254.992170 secs 2022-03-21T15:55:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6254.992170 secs 2022-03-21T15:55:54.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6254.992170 secs 2022-03-21T15:55:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:54 smithi002 ceph-mon[17167]: pgmap v3542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:54 smithi016 ceph-mon[22010]: pgmap v3542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:55.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:54 smithi079 ceph-mon[23158]: pgmap v3542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:56 smithi002 ceph-mon[17167]: pgmap v3543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:57.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:56 smithi016 ceph-mon[22010]: pgmap v3543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:56 smithi079 ceph-mon[23158]: pgmap v3543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:55:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6259.992282 secs 2022-03-21T15:55:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6259.992282 secs 2022-03-21T15:55:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6259.992282 secs 2022-03-21T15:55:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:55:59 smithi079 ceph-mon[23158]: pgmap v3544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:55:59 smithi002 ceph-mon[17167]: pgmap v3544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:55:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:55:59 smithi016 ceph-mon[22010]: pgmap v3544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:01 smithi002 ceph-mon[17167]: pgmap v3545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:01 smithi016 ceph-mon[22010]: pgmap v3545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:01 smithi079 ceph-mon[23158]: pgmap v3545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6264.992400 secs 2022-03-21T15:56:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:03 smithi002 ceph-mon[17167]: pgmap v3546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6264.992400 secs 2022-03-21T15:56:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:03 smithi016 ceph-mon[22010]: pgmap v3546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6264.992400 secs 2022-03-21T15:56:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:03 smithi079 ceph-mon[23158]: pgmap v3546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:05 smithi002 ceph-mon[17167]: pgmap v3547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:05 smithi016 ceph-mon[22010]: pgmap v3547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:05 smithi079 ceph-mon[23158]: pgmap v3547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:06 smithi002 ceph-mon[17167]: pgmap v3548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:06 smithi016 ceph-mon[22010]: pgmap v3548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:06 smithi079 ceph-mon[23158]: pgmap v3548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6269.992513 secs 2022-03-21T15:56:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6269.992513 secs 2022-03-21T15:56:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6269.992513 secs 2022-03-21T15:56:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:09 smithi079 ceph-mon[23158]: pgmap v3549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:09.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:09 smithi002 ceph-mon[17167]: pgmap v3549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:09 smithi016 ceph-mon[22010]: pgmap v3549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:11 smithi002 ceph-mon[17167]: pgmap v3550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:11 smithi016 ceph-mon[22010]: pgmap v3550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:11 smithi079 ceph-mon[23158]: pgmap v3550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6274.992616 secs 2022-03-21T15:56:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:13 smithi002 ceph-mon[17167]: pgmap v3551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6274.992616 secs 2022-03-21T15:56:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:13 smithi016 ceph-mon[22010]: pgmap v3551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6274.992616 secs 2022-03-21T15:56:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:13 smithi079 ceph-mon[23158]: pgmap v3551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:15.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:15 smithi002 ceph-mon[17167]: pgmap v3552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:15 smithi016 ceph-mon[22010]: pgmap v3552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:15 smithi079 ceph-mon[23158]: pgmap v3552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:16 smithi016 ceph-mon[22010]: pgmap v3553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:16 smithi079 ceph-mon[23158]: pgmap v3553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:16 smithi002 ceph-mon[17167]: pgmap v3553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6279.992717 secs 2022-03-21T15:56:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6279.992717 secs 2022-03-21T15:56:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6279.992717 secs 2022-03-21T15:56:19.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:19 smithi079 ceph-mon[23158]: pgmap v3554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:19 smithi002 ceph-mon[17167]: pgmap v3554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:19 smithi016 ceph-mon[22010]: pgmap v3554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:21 smithi002 ceph-mon[17167]: pgmap v3555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:56:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:56:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:21 smithi016 ceph-mon[22010]: pgmap v3555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:56:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:56:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:21 smithi079 ceph-mon[23158]: pgmap v3555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:56:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:56:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6284.992823 secs 2022-03-21T15:56:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:23 smithi002 ceph-mon[17167]: pgmap v3556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6284.992823 secs 2022-03-21T15:56:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:23 smithi016 ceph-mon[22010]: pgmap v3556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6284.992823 secs 2022-03-21T15:56:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:23 smithi079 ceph-mon[23158]: pgmap v3556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:25 smithi002 ceph-mon[17167]: pgmap v3557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:25 smithi016 ceph-mon[22010]: pgmap v3557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:25 smithi079 ceph-mon[23158]: pgmap v3557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:27.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:26 smithi016 ceph-mon[22010]: pgmap v3558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:26 smithi079 ceph-mon[23158]: pgmap v3558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:26 smithi002 ceph-mon[17167]: pgmap v3558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6289.992937 secs 2022-03-21T15:56:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6289.992937 secs 2022-03-21T15:56:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6289.992937 secs 2022-03-21T15:56:29.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:29 smithi079 ceph-mon[23158]: pgmap v3559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:29 smithi002 ceph-mon[17167]: pgmap v3559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:29 smithi016 ceph-mon[22010]: pgmap v3559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:31.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:31 smithi002 ceph-mon[17167]: pgmap v3560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:31 smithi016 ceph-mon[22010]: pgmap v3560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:31 smithi079 ceph-mon[23158]: pgmap v3560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6294.993055 secs 2022-03-21T15:56:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:33 smithi002 ceph-mon[17167]: pgmap v3561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6294.993055 secs 2022-03-21T15:56:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:33 smithi016 ceph-mon[22010]: pgmap v3561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6294.993055 secs 2022-03-21T15:56:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:33 smithi079 ceph-mon[23158]: pgmap v3561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:35.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:35 smithi002 ceph-mon[17167]: pgmap v3562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:35.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:35 smithi016 ceph-mon[22010]: pgmap v3562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:35 smithi079 ceph-mon[23158]: pgmap v3562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:36 smithi016 ceph-mon[22010]: pgmap v3563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:36 smithi079 ceph-mon[23158]: pgmap v3563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:36 smithi002 ceph-mon[17167]: pgmap v3563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6299.993174 secs 2022-03-21T15:56:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6299.993174 secs 2022-03-21T15:56:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6299.993174 secs 2022-03-21T15:56:39.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:39 smithi079 ceph-mon[23158]: pgmap v3564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:39 smithi002 ceph-mon[17167]: pgmap v3564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:39.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:39 smithi016 ceph-mon[22010]: pgmap v3564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:41 smithi002 ceph-mon[17167]: pgmap v3565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:41 smithi016 ceph-mon[22010]: pgmap v3565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:41 smithi079 ceph-mon[23158]: pgmap v3565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6304.993310 secs 2022-03-21T15:56:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:43 smithi002 ceph-mon[17167]: pgmap v3566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:43.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6304.993310 secs 2022-03-21T15:56:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:43 smithi016 ceph-mon[22010]: pgmap v3566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6304.993310 secs 2022-03-21T15:56:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:43 smithi079 ceph-mon[23158]: pgmap v3566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:45.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:45 smithi002 ceph-mon[17167]: pgmap v3567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:45.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:45 smithi016 ceph-mon[22010]: pgmap v3567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:45 smithi079 ceph-mon[23158]: pgmap v3567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:46 smithi016 ceph-mon[22010]: pgmap v3568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:46 smithi079 ceph-mon[23158]: pgmap v3568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:46 smithi002 ceph-mon[17167]: pgmap v3568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6309.993424 secs 2022-03-21T15:56:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6309.993424 secs 2022-03-21T15:56:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6309.993424 secs 2022-03-21T15:56:49.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:49 smithi079 ceph-mon[23158]: pgmap v3569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:49 smithi002 ceph-mon[17167]: pgmap v3569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:49 smithi016 ceph-mon[22010]: pgmap v3569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:51.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:51 smithi002 ceph-mon[17167]: pgmap v3570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:51.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:51 smithi016 ceph-mon[22010]: pgmap v3570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:51 smithi079 ceph-mon[23158]: pgmap v3570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:56:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:56:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:56:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:56:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:56:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:56:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:56:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:56:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:56:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6314.993573 secs 2022-03-21T15:56:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:53 smithi002 ceph-mon[17167]: pgmap v3571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6314.993573 secs 2022-03-21T15:56:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:53 smithi016 ceph-mon[22010]: pgmap v3571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6314.993573 secs 2022-03-21T15:56:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:53 smithi079 ceph-mon[23158]: pgmap v3571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:55.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:55 smithi002 ceph-mon[17167]: pgmap v3572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:55 smithi016 ceph-mon[22010]: pgmap v3572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:55 smithi079 ceph-mon[23158]: pgmap v3572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:56 smithi016 ceph-mon[22010]: pgmap v3573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:56 smithi079 ceph-mon[23158]: pgmap v3573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:56 smithi002 ceph-mon[17167]: pgmap v3573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:56:58.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6319.993691 secs 2022-03-21T15:56:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6319.993691 secs 2022-03-21T15:56:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6319.993691 secs 2022-03-21T15:56:59.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:56:59 smithi079 ceph-mon[23158]: pgmap v3574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:56:59 smithi002 ceph-mon[17167]: pgmap v3574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:56:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:56:59 smithi016 ceph-mon[22010]: pgmap v3574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:01.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:01 smithi002 ceph-mon[17167]: pgmap v3575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:01 smithi016 ceph-mon[22010]: pgmap v3575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:01 smithi079 ceph-mon[23158]: pgmap v3575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6324.993813 secs 2022-03-21T15:57:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:03 smithi002 ceph-mon[17167]: pgmap v3576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6324.993813 secs 2022-03-21T15:57:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:03 smithi016 ceph-mon[22010]: pgmap v3576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6324.993813 secs 2022-03-21T15:57:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:03 smithi079 ceph-mon[23158]: pgmap v3576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:05 smithi002 ceph-mon[17167]: pgmap v3577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:05 smithi016 ceph-mon[22010]: pgmap v3577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:05 smithi079 ceph-mon[23158]: pgmap v3577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:06 smithi016 ceph-mon[22010]: pgmap v3578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:06 smithi079 ceph-mon[23158]: pgmap v3578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:06 smithi002 ceph-mon[17167]: pgmap v3578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:08.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6329.993909 secs 2022-03-21T15:57:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6329.993909 secs 2022-03-21T15:57:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6329.993909 secs 2022-03-21T15:57:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:09 smithi079 ceph-mon[23158]: pgmap v3579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:09 smithi002 ceph-mon[17167]: pgmap v3579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:09 smithi016 ceph-mon[22010]: pgmap v3579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:11.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:11 smithi002 ceph-mon[17167]: pgmap v3580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:11 smithi016 ceph-mon[22010]: pgmap v3580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:11 smithi079 ceph-mon[23158]: pgmap v3580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6334.994018 secs 2022-03-21T15:57:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:13 smithi002 ceph-mon[17167]: pgmap v3581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6334.994018 secs 2022-03-21T15:57:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:13 smithi016 ceph-mon[22010]: pgmap v3581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6334.994018 secs 2022-03-21T15:57:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:13 smithi079 ceph-mon[23158]: pgmap v3581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:15 smithi016 ceph-mon[22010]: pgmap v3582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:15 smithi079 ceph-mon[23158]: pgmap v3582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:16.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:15 smithi002 ceph-mon[17167]: pgmap v3582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:16 smithi016 ceph-mon[22010]: pgmap v3583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:16 smithi079 ceph-mon[23158]: pgmap v3583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:16 smithi002 ceph-mon[17167]: pgmap v3583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:18.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6339.994154 secs 2022-03-21T15:57:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6339.994154 secs 2022-03-21T15:57:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6339.994154 secs 2022-03-21T15:57:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:19 smithi002 ceph-mon[17167]: pgmap v3584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:19 smithi016 ceph-mon[22010]: pgmap v3584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:19 smithi079 ceph-mon[23158]: pgmap v3584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:21 smithi002 ceph-mon[17167]: pgmap v3585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:57:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:57:21.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:21 smithi016 ceph-mon[22010]: pgmap v3585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:57:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:57:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:21 smithi079 ceph-mon[23158]: pgmap v3585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:57:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:57:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6344.994277 secs 2022-03-21T15:57:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:23 smithi016 ceph-mon[22010]: pgmap v3586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6344.994277 secs 2022-03-21T15:57:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:23 smithi079 ceph-mon[23158]: pgmap v3586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6344.994277 secs 2022-03-21T15:57:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:23 smithi002 ceph-mon[17167]: pgmap v3586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:25 smithi016 ceph-mon[22010]: pgmap v3587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:25 smithi079 ceph-mon[23158]: pgmap v3587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:26.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:25 smithi002 ceph-mon[17167]: pgmap v3587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:26 smithi016 ceph-mon[22010]: pgmap v3588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:26 smithi079 ceph-mon[23158]: pgmap v3588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:26 smithi002 ceph-mon[17167]: pgmap v3588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:28.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6349.994391 secs 2022-03-21T15:57:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6349.994391 secs 2022-03-21T15:57:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6349.994391 secs 2022-03-21T15:57:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:29 smithi002 ceph-mon[17167]: pgmap v3589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:29 smithi016 ceph-mon[22010]: pgmap v3589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:29 smithi079 ceph-mon[23158]: pgmap v3589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:31.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:31 smithi002 ceph-mon[17167]: pgmap v3590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:31.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:31 smithi016 ceph-mon[22010]: pgmap v3590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:31 smithi079 ceph-mon[23158]: pgmap v3590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6354.994505 secs 2022-03-21T15:57:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:33 smithi016 ceph-mon[22010]: pgmap v3591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6354.994505 secs 2022-03-21T15:57:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:33 smithi079 ceph-mon[23158]: pgmap v3591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6354.994505 secs 2022-03-21T15:57:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:33 smithi002 ceph-mon[17167]: pgmap v3591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:35 smithi016 ceph-mon[22010]: pgmap v3592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:35 smithi079 ceph-mon[23158]: pgmap v3592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:36.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:35 smithi002 ceph-mon[17167]: pgmap v3592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:36 smithi016 ceph-mon[22010]: pgmap v3593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:36 smithi079 ceph-mon[23158]: pgmap v3593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:36 smithi002 ceph-mon[17167]: pgmap v3593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:38.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6359.994622 secs 2022-03-21T15:57:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6359.994622 secs 2022-03-21T15:57:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6359.994622 secs 2022-03-21T15:57:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:39 smithi002 ceph-mon[17167]: pgmap v3594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:39 smithi016 ceph-mon[22010]: pgmap v3594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:39 smithi079 ceph-mon[23158]: pgmap v3594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:41 smithi016 ceph-mon[22010]: pgmap v3595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:41 smithi079 ceph-mon[23158]: pgmap v3595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:42.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:41 smithi002 ceph-mon[17167]: pgmap v3595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6364.994728 secs 2022-03-21T15:57:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:43 smithi016 ceph-mon[22010]: pgmap v3596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6364.994728 secs 2022-03-21T15:57:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:43 smithi079 ceph-mon[23158]: pgmap v3596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6364.994728 secs 2022-03-21T15:57:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:43 smithi002 ceph-mon[17167]: pgmap v3596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:45 smithi016 ceph-mon[22010]: pgmap v3597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:45.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:45 smithi079 ceph-mon[23158]: pgmap v3597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:46.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:45 smithi002 ceph-mon[17167]: pgmap v3597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:46 smithi016 ceph-mon[22010]: pgmap v3598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:46 smithi079 ceph-mon[23158]: pgmap v3598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:46 smithi002 ceph-mon[17167]: pgmap v3598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:48.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6369.994847 secs 2022-03-21T15:57:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6369.994847 secs 2022-03-21T15:57:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6369.994847 secs 2022-03-21T15:57:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:49 smithi002 ceph-mon[17167]: pgmap v3599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:49 smithi016 ceph-mon[22010]: pgmap v3599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:49 smithi079 ceph-mon[23158]: pgmap v3599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:51 smithi016 ceph-mon[22010]: pgmap v3600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:51 smithi079 ceph-mon[23158]: pgmap v3600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:52.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:51 smithi002 ceph-mon[17167]: pgmap v3600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:57:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:57:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:57:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:57:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:57:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:57:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:57:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:57:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:57:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6374.994948 secs 2022-03-21T15:57:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:53 smithi002 ceph-mon[17167]: pgmap v3601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:57:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6374.994948 secs 2022-03-21T15:57:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:53 smithi016 ceph-mon[22010]: pgmap v3601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:57:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6374.994948 secs 2022-03-21T15:57:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:53 smithi079 ceph-mon[23158]: pgmap v3601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:57:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:55 smithi016 ceph-mon[22010]: pgmap v3602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:55 smithi079 ceph-mon[23158]: pgmap v3602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:56.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:55 smithi002 ceph-mon[17167]: pgmap v3602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:56 smithi016 ceph-mon[22010]: pgmap v3603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:56 smithi079 ceph-mon[23158]: pgmap v3603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:56 smithi002 ceph-mon[17167]: pgmap v3603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:57:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6379.995062 secs 2022-03-21T15:57:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6379.995062 secs 2022-03-21T15:57:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6379.995062 secs 2022-03-21T15:57:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:57:59 smithi002 ceph-mon[17167]: pgmap v3604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:57:59 smithi016 ceph-mon[22010]: pgmap v3604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:57:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:57:59 smithi079 ceph-mon[23158]: pgmap v3604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:01 smithi016 ceph-mon[22010]: pgmap v3605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:01 smithi079 ceph-mon[23158]: pgmap v3605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:01 smithi002 ceph-mon[17167]: pgmap v3605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6384.995184 secs 2022-03-21T15:58:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:03 smithi016 ceph-mon[22010]: pgmap v3606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6384.995184 secs 2022-03-21T15:58:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:03 smithi079 ceph-mon[23158]: pgmap v3606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6384.995184 secs 2022-03-21T15:58:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:03 smithi002 ceph-mon[17167]: pgmap v3606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:05 smithi016 ceph-mon[22010]: pgmap v3607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:05 smithi079 ceph-mon[23158]: pgmap v3607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:06.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:05 smithi002 ceph-mon[17167]: pgmap v3607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:06 smithi016 ceph-mon[22010]: pgmap v3608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:06 smithi079 ceph-mon[23158]: pgmap v3608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:06 smithi002 ceph-mon[17167]: pgmap v3608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6389.995298 secs 2022-03-21T15:58:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6389.995298 secs 2022-03-21T15:58:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6389.995298 secs 2022-03-21T15:58:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:09 smithi002 ceph-mon[17167]: pgmap v3609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:09 smithi016 ceph-mon[22010]: pgmap v3609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:09.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:09 smithi079 ceph-mon[23158]: pgmap v3609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:11.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:11 smithi016 ceph-mon[22010]: pgmap v3610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:11.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:11 smithi079 ceph-mon[23158]: pgmap v3610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:12.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:11 smithi002 ceph-mon[17167]: pgmap v3610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:13.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6394.995414 secs 2022-03-21T15:58:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:13 smithi016 ceph-mon[22010]: pgmap v3611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6394.995414 secs 2022-03-21T15:58:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:13 smithi079 ceph-mon[23158]: pgmap v3611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6394.995414 secs 2022-03-21T15:58:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:13 smithi002 ceph-mon[17167]: pgmap v3611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:15.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:15 smithi016 ceph-mon[22010]: pgmap v3612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:15 smithi079 ceph-mon[23158]: pgmap v3612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:16.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:15 smithi002 ceph-mon[17167]: pgmap v3612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:17.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:16 smithi016 ceph-mon[22010]: pgmap v3613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:16 smithi079 ceph-mon[23158]: pgmap v3613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:16 smithi002 ceph-mon[17167]: pgmap v3613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6399.995533 secs 2022-03-21T15:58:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6399.995533 secs 2022-03-21T15:58:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6399.995533 secs 2022-03-21T15:58:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:19 smithi002 ceph-mon[17167]: pgmap v3614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:19 smithi016 ceph-mon[22010]: pgmap v3614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:19 smithi079 ceph-mon[23158]: pgmap v3614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:21 smithi016 ceph-mon[22010]: pgmap v3615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:58:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:58:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:21 smithi079 ceph-mon[23158]: pgmap v3615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:58:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:58:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:21 smithi002 ceph-mon[17167]: pgmap v3615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:58:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:58:23.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6404.995652 secs 2022-03-21T15:58:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:23 smithi016 ceph-mon[22010]: pgmap v3616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6404.995652 secs 2022-03-21T15:58:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:23 smithi079 ceph-mon[23158]: pgmap v3616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6404.995652 secs 2022-03-21T15:58:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:23 smithi002 ceph-mon[17167]: pgmap v3616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:25.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:25 smithi016 ceph-mon[22010]: pgmap v3617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:25 smithi079 ceph-mon[23158]: pgmap v3617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:26.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:25 smithi002 ceph-mon[17167]: pgmap v3617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:26 smithi079 ceph-mon[23158]: pgmap v3618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:26 smithi002 ceph-mon[17167]: pgmap v3618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:26 smithi016 ceph-mon[22010]: pgmap v3618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6409.995721 secs 2022-03-21T15:58:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6409.995721 secs 2022-03-21T15:58:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6409.995721 secs 2022-03-21T15:58:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:29 smithi002 ceph-mon[17167]: pgmap v3619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:29 smithi016 ceph-mon[22010]: pgmap v3619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:29 smithi079 ceph-mon[23158]: pgmap v3619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:31 smithi016 ceph-mon[22010]: pgmap v3620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:31.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:31 smithi079 ceph-mon[23158]: pgmap v3620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:32.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:31 smithi002 ceph-mon[17167]: pgmap v3620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6414.995840 secs 2022-03-21T15:58:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:33 smithi016 ceph-mon[22010]: pgmap v3621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6414.995840 secs 2022-03-21T15:58:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:33 smithi079 ceph-mon[23158]: pgmap v3621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6414.995840 secs 2022-03-21T15:58:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:33 smithi002 ceph-mon[17167]: pgmap v3621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:34 smithi079 ceph-mon[23158]: pgmap v3622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:34 smithi002 ceph-mon[17167]: pgmap v3622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:34 smithi016 ceph-mon[22010]: pgmap v3622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:36 smithi079 ceph-mon[23158]: pgmap v3623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:36 smithi002 ceph-mon[17167]: pgmap v3623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:36 smithi016 ceph-mon[22010]: pgmap v3623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6419.995953 secs 2022-03-21T15:58:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6419.995953 secs 2022-03-21T15:58:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6419.995953 secs 2022-03-21T15:58:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:39 smithi002 ceph-mon[17167]: pgmap v3624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:39 smithi016 ceph-mon[22010]: pgmap v3624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:39 smithi079 ceph-mon[23158]: pgmap v3624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:41.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:41 smithi016 ceph-mon[22010]: pgmap v3625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:41.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:41 smithi079 ceph-mon[23158]: pgmap v3625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:42.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:41 smithi002 ceph-mon[17167]: pgmap v3625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6424.996062 secs 2022-03-21T15:58:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:43 smithi016 ceph-mon[22010]: pgmap v3626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6424.996062 secs 2022-03-21T15:58:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:43 smithi079 ceph-mon[23158]: pgmap v3626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6424.996062 secs 2022-03-21T15:58:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:43 smithi002 ceph-mon[17167]: pgmap v3626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:44 smithi016 ceph-mon[22010]: pgmap v3627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:44 smithi079 ceph-mon[23158]: pgmap v3627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:44 smithi002 ceph-mon[17167]: pgmap v3627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:46 smithi079 ceph-mon[23158]: pgmap v3628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:46 smithi002 ceph-mon[17167]: pgmap v3628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:47.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:46 smithi016 ceph-mon[22010]: pgmap v3628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6429.996178 secs 2022-03-21T15:58:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6429.996178 secs 2022-03-21T15:58:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6429.996178 secs 2022-03-21T15:58:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:49 smithi002 ceph-mon[17167]: pgmap v3629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:49 smithi016 ceph-mon[22010]: pgmap v3629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:49 smithi079 ceph-mon[23158]: pgmap v3629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:51 smithi016 ceph-mon[22010]: pgmap v3630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:51 smithi079 ceph-mon[23158]: pgmap v3630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:52.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:51 smithi002 ceph-mon[17167]: pgmap v3630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:52 smithi079 ceph-mon[23158]: pgmap v3631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:52 smithi002 ceph-mon[17167]: pgmap v3631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:52 smithi016 ceph-mon[22010]: pgmap v3631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6434.996295 secs 2022-03-21T15:58:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:58:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:58:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:53 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:58:53.900 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6434.996295 secs 2022-03-21T15:58:53.901 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:58:53.901 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:58:53.901 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:53 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:58:53.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6434.996295 secs 2022-03-21T15:58:53.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:58:53.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:58:53.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:53 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:58:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:58:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:58:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:58:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:58:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:58:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:58:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:54 smithi079 ceph-mon[23158]: pgmap v3632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:58:55.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:58:55.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:58:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:58:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:58:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:58:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:58:55.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:54 smithi002 ceph-mon[17167]: pgmap v3632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T15:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T15:58:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T15:58:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T15:58:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T15:58:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T15:58:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T15:58:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:58:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:54 smithi016 ceph-mon[22010]: pgmap v3632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:57.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:56 smithi079 ceph-mon[23158]: pgmap v3633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:56 smithi002 ceph-mon[17167]: pgmap v3633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:57.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:56 smithi016 ceph-mon[22010]: pgmap v3633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:58:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6439.996416 secs 2022-03-21T15:58:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6439.996416 secs 2022-03-21T15:58:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6439.996416 secs 2022-03-21T15:58:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:58:59 smithi002 ceph-mon[17167]: pgmap v3634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:58:59 smithi016 ceph-mon[22010]: pgmap v3634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:58:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:58:59 smithi079 ceph-mon[23158]: pgmap v3634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:01.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:01 smithi016 ceph-mon[22010]: pgmap v3635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:01 smithi079 ceph-mon[23158]: pgmap v3635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:02.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:01 smithi002 ceph-mon[17167]: pgmap v3635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:02 smithi016 ceph-mon[22010]: pgmap v3636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:02 smithi079 ceph-mon[23158]: pgmap v3636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:02 smithi002 ceph-mon[17167]: pgmap v3636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6444.996580 secs 2022-03-21T15:59:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6444.996580 secs 2022-03-21T15:59:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6444.996580 secs 2022-03-21T15:59:04.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:04 smithi016 ceph-mon[22010]: pgmap v3637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:04 smithi079 ceph-mon[23158]: pgmap v3637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:04 smithi002 ceph-mon[17167]: pgmap v3637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:06 smithi079 ceph-mon[23158]: pgmap v3638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:06 smithi002 ceph-mon[17167]: pgmap v3638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:06 smithi016 ceph-mon[22010]: pgmap v3638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6449.996696 secs 2022-03-21T15:59:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6449.996696 secs 2022-03-21T15:59:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6449.996696 secs 2022-03-21T15:59:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:09 smithi002 ceph-mon[17167]: pgmap v3639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:09 smithi016 ceph-mon[22010]: pgmap v3639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:09.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:09 smithi079 ceph-mon[23158]: pgmap v3639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:10.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:10 smithi079 ceph-mon[23158]: pgmap v3640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:10 smithi002 ceph-mon[17167]: pgmap v3640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:10 smithi016 ceph-mon[22010]: pgmap v3640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:12 smithi079 ceph-mon[23158]: pgmap v3641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:12 smithi002 ceph-mon[17167]: pgmap v3641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:12 smithi016 ceph-mon[22010]: pgmap v3641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6454.996802 secs 2022-03-21T15:59:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6454.996802 secs 2022-03-21T15:59:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6454.996802 secs 2022-03-21T15:59:14.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:14 smithi079 ceph-mon[23158]: pgmap v3642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:14 smithi002 ceph-mon[17167]: pgmap v3642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:14 smithi016 ceph-mon[22010]: pgmap v3642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:17.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:16 smithi079 ceph-mon[23158]: pgmap v3643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:16 smithi002 ceph-mon[17167]: pgmap v3643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:16 smithi016 ceph-mon[22010]: pgmap v3643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6459.996908 secs 2022-03-21T15:59:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6459.996908 secs 2022-03-21T15:59:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6459.996908 secs 2022-03-21T15:59:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:19 smithi002 ceph-mon[17167]: pgmap v3644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:19 smithi016 ceph-mon[22010]: pgmap v3644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:19 smithi079 ceph-mon[23158]: pgmap v3644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:20 smithi079 ceph-mon[23158]: pgmap v3645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:20 smithi002 ceph-mon[17167]: pgmap v3645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:21.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:20 smithi016 ceph-mon[22010]: pgmap v3645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:59:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:59:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:59:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:59:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T15:59:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T15:59:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:22 smithi079 ceph-mon[23158]: pgmap v3646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:22 smithi002 ceph-mon[17167]: pgmap v3646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:22 smithi016 ceph-mon[22010]: pgmap v3646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6464.997056 secs 2022-03-21T15:59:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6464.997056 secs 2022-03-21T15:59:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6464.997056 secs 2022-03-21T15:59:25.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:24 smithi079 ceph-mon[23158]: pgmap v3647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:59:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:24 smithi002 ceph-mon[17167]: pgmap v3647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:59:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:24 smithi016 ceph-mon[22010]: pgmap v3647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T15:59:27.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:26 smithi079 ceph-mon[23158]: pgmap v3648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:26 smithi002 ceph-mon[17167]: pgmap v3648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:27.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:26 smithi016 ceph-mon[22010]: pgmap v3648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6469.997172 secs 2022-03-21T15:59:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6469.997172 secs 2022-03-21T15:59:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6469.997172 secs 2022-03-21T15:59:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:29 smithi002 ceph-mon[17167]: pgmap v3649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:29 smithi016 ceph-mon[22010]: pgmap v3649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:29 smithi079 ceph-mon[23158]: pgmap v3649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:30.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:30 smithi079 ceph-mon[23158]: pgmap v3650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:30 smithi002 ceph-mon[17167]: pgmap v3650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:30 smithi016 ceph-mon[22010]: pgmap v3650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:32 smithi079 ceph-mon[23158]: pgmap v3651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:32 smithi002 ceph-mon[17167]: pgmap v3651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:32 smithi016 ceph-mon[22010]: pgmap v3651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6474.997292 secs 2022-03-21T15:59:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6474.997292 secs 2022-03-21T15:59:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6474.997292 secs 2022-03-21T15:59:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:34 smithi079 ceph-mon[23158]: pgmap v3652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:34 smithi002 ceph-mon[17167]: pgmap v3652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:34 smithi016 ceph-mon[22010]: pgmap v3652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:36 smithi079 ceph-mon[23158]: pgmap v3653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:36 smithi002 ceph-mon[17167]: pgmap v3653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:37.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:36 smithi016 ceph-mon[22010]: pgmap v3653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6479.997349 secs 2022-03-21T15:59:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6479.997349 secs 2022-03-21T15:59:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6479.997349 secs 2022-03-21T15:59:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:39 smithi002 ceph-mon[17167]: pgmap v3654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:39 smithi016 ceph-mon[22010]: pgmap v3654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:39 smithi079 ceph-mon[23158]: pgmap v3654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:40 smithi079 ceph-mon[23158]: pgmap v3655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:40 smithi002 ceph-mon[17167]: pgmap v3655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:40 smithi016 ceph-mon[22010]: pgmap v3655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:42 smithi079 ceph-mon[23158]: pgmap v3656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:42 smithi002 ceph-mon[17167]: pgmap v3656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:42 smithi016 ceph-mon[22010]: pgmap v3656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6484.997464 secs 2022-03-21T15:59:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6484.997464 secs 2022-03-21T15:59:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6484.997464 secs 2022-03-21T15:59:44.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:44 smithi079 ceph-mon[23158]: pgmap v3657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:44 smithi002 ceph-mon[17167]: pgmap v3657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:44 smithi016 ceph-mon[22010]: pgmap v3657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:46 smithi079 ceph-mon[23158]: pgmap v3658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:46 smithi002 ceph-mon[17167]: pgmap v3658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:47.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:46 smithi016 ceph-mon[22010]: pgmap v3658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6489.997578 secs 2022-03-21T15:59:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6489.997578 secs 2022-03-21T15:59:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6489.997578 secs 2022-03-21T15:59:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:49 smithi002 ceph-mon[17167]: pgmap v3659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:49 smithi016 ceph-mon[22010]: pgmap v3659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:49 smithi079 ceph-mon[23158]: pgmap v3659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:50.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:50 smithi079 ceph-mon[23158]: pgmap v3660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:50 smithi002 ceph-mon[17167]: pgmap v3660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:51.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:50 smithi016 ceph-mon[22010]: pgmap v3660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:52.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:52 smithi079 ceph-mon[23158]: pgmap v3661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:52 smithi002 ceph-mon[17167]: pgmap v3661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:52 smithi016 ceph-mon[22010]: pgmap v3661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6494.997703 secs 2022-03-21T15:59:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6494.997703 secs 2022-03-21T15:59:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6494.997703 secs 2022-03-21T15:59:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:59:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:59:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:54 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:59:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:54 smithi079 ceph-mon[23158]: pgmap v3662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:59:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:59:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:54 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:59:55.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:54 smithi002 ceph-mon[17167]: pgmap v3662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T15:59:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T15:59:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:54 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T15:59:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:54 smithi016 ceph-mon[22010]: pgmap v3662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:59:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:59:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T15:59:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:57 smithi079 ceph-mon[23158]: pgmap v3663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:57 smithi002 ceph-mon[17167]: pgmap v3663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:57 smithi016 ceph-mon[22010]: pgmap v3663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T15:59:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6499.997762 secs 2022-03-21T15:59:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6499.997762 secs 2022-03-21T15:59:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6499.997762 secs 2022-03-21T15:59:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 15:59:59 smithi002 ceph-mon[17167]: pgmap v3664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 15:59:59 smithi016 ceph-mon[22010]: pgmap v3664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T15:59:59.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 15:59:59 smithi079 ceph-mon[23158]: pgmap v3664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:00:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:00:00.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:00:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:01 smithi002 ceph-mon[17167]: pgmap v3665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:01.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:01 smithi016 ceph-mon[22010]: pgmap v3665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:01.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:01 smithi079 ceph-mon[23158]: pgmap v3665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:02 smithi079 ceph-mon[23158]: pgmap v3666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:02 smithi002 ceph-mon[17167]: pgmap v3666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:02 smithi016 ceph-mon[22010]: pgmap v3666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6504.997876 secs 2022-03-21T16:00:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6504.997876 secs 2022-03-21T16:00:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6504.997876 secs 2022-03-21T16:00:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:04 smithi079 ceph-mon[23158]: pgmap v3667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:04 smithi002 ceph-mon[17167]: pgmap v3667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:04 smithi016 ceph-mon[22010]: pgmap v3667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:07.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:06 smithi079 ceph-mon[23158]: pgmap v3668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:06 smithi002 ceph-mon[17167]: pgmap v3668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:06 smithi016 ceph-mon[22010]: pgmap v3668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6509.997989 secs 2022-03-21T16:00:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6509.997989 secs 2022-03-21T16:00:08.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6509.997989 secs 2022-03-21T16:00:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:09 smithi002 ceph-mon[17167]: pgmap v3669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:09 smithi016 ceph-mon[22010]: pgmap v3669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:09.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:09 smithi079 ceph-mon[23158]: pgmap v3669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:10 smithi079 ceph-mon[23158]: pgmap v3670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:10 smithi002 ceph-mon[17167]: pgmap v3670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:10 smithi016 ceph-mon[22010]: pgmap v3670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:12 smithi079 ceph-mon[23158]: pgmap v3671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:12 smithi002 ceph-mon[17167]: pgmap v3671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:12 smithi016 ceph-mon[22010]: pgmap v3671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6514.998139 secs 2022-03-21T16:00:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6514.998139 secs 2022-03-21T16:00:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6514.998139 secs 2022-03-21T16:00:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:14 smithi079 ceph-mon[23158]: pgmap v3672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:14 smithi002 ceph-mon[17167]: pgmap v3672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:14 smithi016 ceph-mon[22010]: pgmap v3672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:16 smithi079 ceph-mon[23158]: pgmap v3673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:16 smithi002 ceph-mon[17167]: pgmap v3673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:17.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:16 smithi016 ceph-mon[22010]: pgmap v3673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6519.998253 secs 2022-03-21T16:00:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6519.998253 secs 2022-03-21T16:00:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6519.998253 secs 2022-03-21T16:00:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:19 smithi002 ceph-mon[17167]: pgmap v3674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:19 smithi016 ceph-mon[22010]: pgmap v3674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:19.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:19 smithi079 ceph-mon[23158]: pgmap v3674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:20 smithi079 ceph-mon[23158]: pgmap v3675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:20 smithi002 ceph-mon[17167]: pgmap v3675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:20 smithi016 ceph-mon[22010]: pgmap v3675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:22.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:00:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:00:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:00:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:00:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:00:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:00:23.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:22 smithi079 ceph-mon[23158]: pgmap v3676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:22 smithi002 ceph-mon[17167]: pgmap v3676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:22 smithi016 ceph-mon[22010]: pgmap v3676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:24.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6524.998374 secs 2022-03-21T16:00:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6524.998374 secs 2022-03-21T16:00:24.455 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6524.998374 secs 2022-03-21T16:00:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:24 smithi002 ceph-mon[17167]: pgmap v3677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:24 smithi016 ceph-mon[22010]: pgmap v3677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:25.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:24 smithi079 ceph-mon[23158]: pgmap v3677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:26 smithi079 ceph-mon[23158]: pgmap v3678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:26 smithi002 ceph-mon[17167]: pgmap v3678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:27.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:26 smithi016 ceph-mon[22010]: pgmap v3678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6529.998490 secs 2022-03-21T16:00:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6529.998490 secs 2022-03-21T16:00:28.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6529.998490 secs 2022-03-21T16:00:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:29 smithi002 ceph-mon[17167]: pgmap v3679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:29 smithi016 ceph-mon[22010]: pgmap v3679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:29 smithi079 ceph-mon[23158]: pgmap v3679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:30 smithi079 ceph-mon[23158]: pgmap v3680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:30 smithi002 ceph-mon[17167]: pgmap v3680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:30 smithi016 ceph-mon[22010]: pgmap v3680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:32 smithi079 ceph-mon[23158]: pgmap v3681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:32 smithi002 ceph-mon[17167]: pgmap v3681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:32 smithi016 ceph-mon[22010]: pgmap v3681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6534.998604 secs 2022-03-21T16:00:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6534.998604 secs 2022-03-21T16:00:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6534.998604 secs 2022-03-21T16:00:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:34 smithi079 ceph-mon[23158]: pgmap v3682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:34 smithi002 ceph-mon[17167]: pgmap v3682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:34 smithi016 ceph-mon[22010]: pgmap v3682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:37.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:36 smithi079 ceph-mon[23158]: pgmap v3683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:36 smithi002 ceph-mon[17167]: pgmap v3683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:37.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:36 smithi016 ceph-mon[22010]: pgmap v3683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6539.998668 secs 2022-03-21T16:00:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6539.998668 secs 2022-03-21T16:00:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6539.998668 secs 2022-03-21T16:00:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:39 smithi002 ceph-mon[17167]: pgmap v3684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:39.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:39 smithi016 ceph-mon[22010]: pgmap v3684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:39 smithi079 ceph-mon[23158]: pgmap v3684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:40 smithi079 ceph-mon[23158]: pgmap v3685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:40 smithi002 ceph-mon[17167]: pgmap v3685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:40 smithi016 ceph-mon[22010]: pgmap v3685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:42 smithi079 ceph-mon[23158]: pgmap v3686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:42 smithi002 ceph-mon[17167]: pgmap v3686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:42 smithi016 ceph-mon[22010]: pgmap v3686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6544.998790 secs 2022-03-21T16:00:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6544.998790 secs 2022-03-21T16:00:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6544.998790 secs 2022-03-21T16:00:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:44 smithi002 ceph-mon[17167]: pgmap v3687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:44 smithi016 ceph-mon[22010]: pgmap v3687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:44 smithi079 ceph-mon[23158]: pgmap v3687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:47.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:46 smithi079 ceph-mon[23158]: pgmap v3688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:46 smithi002 ceph-mon[17167]: pgmap v3688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:46 smithi016 ceph-mon[22010]: pgmap v3688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6549.998887 secs 2022-03-21T16:00:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6549.998887 secs 2022-03-21T16:00:48.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6549.998887 secs 2022-03-21T16:00:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:49 smithi002 ceph-mon[17167]: pgmap v3689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:49 smithi016 ceph-mon[22010]: pgmap v3689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:49 smithi079 ceph-mon[23158]: pgmap v3689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:50 smithi079 ceph-mon[23158]: pgmap v3690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:50 smithi002 ceph-mon[17167]: pgmap v3690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:51.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:50 smithi016 ceph-mon[22010]: pgmap v3690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:52 smithi079 ceph-mon[23158]: pgmap v3691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:52 smithi002 ceph-mon[17167]: pgmap v3691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:52 smithi016 ceph-mon[22010]: pgmap v3691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6554.999001 secs 2022-03-21T16:00:54.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6554.999001 secs 2022-03-21T16:00:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6554.999001 secs 2022-03-21T16:00:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:54 smithi002 ceph-mon[17167]: pgmap v3692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:54 smithi016 ceph-mon[22010]: pgmap v3692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:55.217 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:54 smithi079 ceph-mon[23158]: pgmap v3692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:00:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:00:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:55 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:00:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:00:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:00:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:55 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:00:56.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:00:56.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:00:56.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:55 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:00:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:00:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:57 smithi002 ceph-mon[17167]: pgmap v3693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:00:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:57 smithi016 ceph-mon[22010]: pgmap v3693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:00:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:57 smithi079 ceph-mon[23158]: pgmap v3693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:00:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6559.999143 secs 2022-03-21T16:00:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6559.999143 secs 2022-03-21T16:00:58.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6559.999143 secs 2022-03-21T16:00:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:00:59 smithi002 ceph-mon[17167]: pgmap v3694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:00:59 smithi016 ceph-mon[22010]: pgmap v3694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:00:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:00:59 smithi079 ceph-mon[23158]: pgmap v3694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:00 smithi079 ceph-mon[23158]: pgmap v3695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:00 smithi002 ceph-mon[17167]: pgmap v3695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:00 smithi016 ceph-mon[22010]: pgmap v3695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:02 smithi079 ceph-mon[23158]: pgmap v3696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:02 smithi002 ceph-mon[17167]: pgmap v3696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:02 smithi016 ceph-mon[22010]: pgmap v3696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6564.999225 secs 2022-03-21T16:01:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6564.999225 secs 2022-03-21T16:01:04.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6564.999225 secs 2022-03-21T16:01:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:04 smithi002 ceph-mon[17167]: pgmap v3697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:04 smithi016 ceph-mon[22010]: pgmap v3697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:04 smithi079 ceph-mon[23158]: pgmap v3697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:06 smithi002 ceph-mon[17167]: pgmap v3698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:06 smithi016 ceph-mon[22010]: pgmap v3698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:07.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:06 smithi079 ceph-mon[23158]: pgmap v3698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6569.999346 secs 2022-03-21T16:01:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6569.999346 secs 2022-03-21T16:01:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6569.999346 secs 2022-03-21T16:01:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:09 smithi002 ceph-mon[17167]: pgmap v3699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:09 smithi016 ceph-mon[22010]: pgmap v3699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:09.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:09 smithi079 ceph-mon[23158]: pgmap v3699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:10 smithi079 ceph-mon[23158]: pgmap v3700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:10 smithi002 ceph-mon[17167]: pgmap v3700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:10 smithi016 ceph-mon[22010]: pgmap v3700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:12 smithi079 ceph-mon[23158]: pgmap v3701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:12 smithi002 ceph-mon[17167]: pgmap v3701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:12 smithi016 ceph-mon[22010]: pgmap v3701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6574.999463 secs 2022-03-21T16:01:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6574.999463 secs 2022-03-21T16:01:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6574.999463 secs 2022-03-21T16:01:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:14 smithi002 ceph-mon[17167]: pgmap v3702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:14 smithi016 ceph-mon[22010]: pgmap v3702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:14 smithi079 ceph-mon[23158]: pgmap v3702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:16 smithi002 ceph-mon[17167]: pgmap v3703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:16 smithi016 ceph-mon[22010]: pgmap v3703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:16 smithi079 ceph-mon[23158]: pgmap v3703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6579.999580 secs 2022-03-21T16:01:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6579.999580 secs 2022-03-21T16:01:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6579.999580 secs 2022-03-21T16:01:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:19 smithi002 ceph-mon[17167]: pgmap v3704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:19 smithi016 ceph-mon[22010]: pgmap v3704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:19 smithi079 ceph-mon[23158]: pgmap v3704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:20 smithi079 ceph-mon[23158]: pgmap v3705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:20 smithi002 ceph-mon[17167]: pgmap v3705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:20 smithi016 ceph-mon[22010]: pgmap v3705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:01:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:01:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:01:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:01:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:01:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:01:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:22 smithi002 ceph-mon[17167]: pgmap v3706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:22 smithi016 ceph-mon[22010]: pgmap v3706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:22 smithi079 ceph-mon[23158]: pgmap v3706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6584.999724 secs 2022-03-21T16:01:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6584.999724 secs 2022-03-21T16:01:24.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6584.999724 secs 2022-03-21T16:01:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:24 smithi002 ceph-mon[17167]: pgmap v3707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:25.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:24 smithi016 ceph-mon[22010]: pgmap v3707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:24 smithi079 ceph-mon[23158]: pgmap v3707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:26 smithi002 ceph-mon[17167]: pgmap v3708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:26 smithi016 ceph-mon[22010]: pgmap v3708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:26 smithi079 ceph-mon[23158]: pgmap v3708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6589.999798 secs 2022-03-21T16:01:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6589.999798 secs 2022-03-21T16:01:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6589.999798 secs 2022-03-21T16:01:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:29 smithi002 ceph-mon[17167]: pgmap v3709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:29 smithi016 ceph-mon[22010]: pgmap v3709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:29 smithi079 ceph-mon[23158]: pgmap v3709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:31.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:30 smithi002 ceph-mon[17167]: pgmap v3710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:30 smithi016 ceph-mon[22010]: pgmap v3710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:30 smithi079 ceph-mon[23158]: pgmap v3710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:32 smithi002 ceph-mon[17167]: pgmap v3711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:32 smithi016 ceph-mon[22010]: pgmap v3711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:32 smithi079 ceph-mon[23158]: pgmap v3711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6594.999908 secs 2022-03-21T16:01:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6594.999908 secs 2022-03-21T16:01:34.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6594.999908 secs 2022-03-21T16:01:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:34 smithi002 ceph-mon[17167]: pgmap v3712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:34 smithi016 ceph-mon[22010]: pgmap v3712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:34 smithi079 ceph-mon[23158]: pgmap v3712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:36 smithi002 ceph-mon[17167]: pgmap v3713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:37.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:36 smithi016 ceph-mon[22010]: pgmap v3713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:36 smithi079 ceph-mon[23158]: pgmap v3713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6599.999991 secs 2022-03-21T16:01:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6599.999991 secs 2022-03-21T16:01:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6599.999991 secs 2022-03-21T16:01:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:39 smithi002 ceph-mon[17167]: pgmap v3714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:39 smithi016 ceph-mon[22010]: pgmap v3714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:39 smithi079 ceph-mon[23158]: pgmap v3714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:40 smithi002 ceph-mon[17167]: pgmap v3715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:41.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:40 smithi016 ceph-mon[22010]: pgmap v3715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:41.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:40 smithi079 ceph-mon[23158]: pgmap v3715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:42 smithi002 ceph-mon[17167]: pgmap v3716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:42 smithi016 ceph-mon[22010]: pgmap v3716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:42 smithi079 ceph-mon[23158]: pgmap v3716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6605.000112 secs 2022-03-21T16:01:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6605.000112 secs 2022-03-21T16:01:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6605.000112 secs 2022-03-21T16:01:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:44 smithi002 ceph-mon[17167]: pgmap v3717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:44 smithi016 ceph-mon[22010]: pgmap v3717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:44 smithi079 ceph-mon[23158]: pgmap v3717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:46 smithi002 ceph-mon[17167]: pgmap v3718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:47.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:46 smithi016 ceph-mon[22010]: pgmap v3718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:46 smithi079 ceph-mon[23158]: pgmap v3718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6610.000230 secs 2022-03-21T16:01:48.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6610.000230 secs 2022-03-21T16:01:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6610.000230 secs 2022-03-21T16:01:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:49 smithi002 ceph-mon[17167]: pgmap v3719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:49 smithi016 ceph-mon[22010]: pgmap v3719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:49 smithi079 ceph-mon[23158]: pgmap v3719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:50 smithi002 ceph-mon[17167]: pgmap v3720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:51.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:50 smithi016 ceph-mon[22010]: pgmap v3720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:50 smithi079 ceph-mon[23158]: pgmap v3720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:52 smithi002 ceph-mon[17167]: pgmap v3721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:52 smithi016 ceph-mon[22010]: pgmap v3721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:52 smithi079 ceph-mon[23158]: pgmap v3721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6615.000323 secs 2022-03-21T16:01:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6615.000323 secs 2022-03-21T16:01:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6615.000323 secs 2022-03-21T16:01:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:54 smithi002 ceph-mon[17167]: pgmap v3722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:54 smithi016 ceph-mon[22010]: pgmap v3722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:54 smithi079 ceph-mon[23158]: pgmap v3722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:01:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:01:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:01:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:56 smithi002 ceph-mon[17167]: pgmap v3723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:57.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:01:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:01:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:01:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:56 smithi016 ceph-mon[22010]: pgmap v3723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:01:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:01:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:01:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:56 smithi079 ceph-mon[23158]: pgmap v3723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:01:58.611 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6620.000399 secs 2022-03-21T16:01:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6620.000399 secs 2022-03-21T16:01:58.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6620.000399 secs 2022-03-21T16:01:59.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:01:59 smithi016 ceph-mon[22010]: pgmap v3724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:59.915 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.915 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.916 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:01:59 smithi002 ceph-mon[17167]: pgmap v3724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:01:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:01:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:01:59 smithi079 ceph-mon[23158]: pgmap v3724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:00 smithi002 ceph-mon[17167]: pgmap v3725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:00 smithi016 ceph-mon[22010]: pgmap v3725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:00 smithi079 ceph-mon[23158]: pgmap v3725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:02:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:02 smithi002 ceph-mon[17167]: pgmap v3726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:03.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:02 smithi016 ceph-mon[22010]: pgmap v3726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:02 smithi079 ceph-mon[23158]: pgmap v3726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6625.000514 secs 2022-03-21T16:02:04.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6625.000514 secs 2022-03-21T16:02:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6625.000514 secs 2022-03-21T16:02:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:04 smithi002 ceph-mon[17167]: pgmap v3727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:04 smithi016 ceph-mon[22010]: pgmap v3727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:04 smithi079 ceph-mon[23158]: pgmap v3727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:06 smithi002 ceph-mon[17167]: pgmap v3728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:06 smithi016 ceph-mon[22010]: pgmap v3728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:06 smithi079 ceph-mon[23158]: pgmap v3728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6630.000630 secs 2022-03-21T16:02:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6630.000630 secs 2022-03-21T16:02:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6630.000630 secs 2022-03-21T16:02:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:09 smithi002 ceph-mon[17167]: pgmap v3729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:09.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:09 smithi016 ceph-mon[22010]: pgmap v3729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:09 smithi079 ceph-mon[23158]: pgmap v3729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:10 smithi002 ceph-mon[17167]: pgmap v3730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:10 smithi016 ceph-mon[22010]: pgmap v3730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:10 smithi079 ceph-mon[23158]: pgmap v3730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:12 smithi002 ceph-mon[17167]: pgmap v3731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:12 smithi016 ceph-mon[22010]: pgmap v3731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:12 smithi079 ceph-mon[23158]: pgmap v3731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6635.000752 secs 2022-03-21T16:02:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6635.000752 secs 2022-03-21T16:02:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6635.000752 secs 2022-03-21T16:02:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:14 smithi002 ceph-mon[17167]: pgmap v3732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:14 smithi016 ceph-mon[22010]: pgmap v3732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:14 smithi079 ceph-mon[23158]: pgmap v3732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:17 smithi002 ceph-mon[17167]: pgmap v3733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:17.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:17 smithi016 ceph-mon[22010]: pgmap v3733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:17 smithi079 ceph-mon[23158]: pgmap v3733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6640.000910 secs 2022-03-21T16:02:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6640.000910 secs 2022-03-21T16:02:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6640.000910 secs 2022-03-21T16:02:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:19 smithi002 ceph-mon[17167]: pgmap v3734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:19 smithi016 ceph-mon[22010]: pgmap v3734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:19 smithi079 ceph-mon[23158]: pgmap v3734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:20 smithi002 ceph-mon[17167]: pgmap v3735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:20 smithi016 ceph-mon[22010]: pgmap v3735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:20 smithi079 ceph-mon[23158]: pgmap v3735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:02:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:02:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:02:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:02:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:02:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:02:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:23 smithi002 ceph-mon[17167]: pgmap v3736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:23 smithi016 ceph-mon[22010]: pgmap v3736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:23 smithi079 ceph-mon[23158]: pgmap v3736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:24 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6645.001031 secs 2022-03-21T16:02:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6645.001031 secs 2022-03-21T16:02:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6645.001031 secs 2022-03-21T16:02:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:25 smithi002 ceph-mon[17167]: pgmap v3737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:25.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:25 smithi016 ceph-mon[22010]: pgmap v3737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:25 smithi079 ceph-mon[23158]: pgmap v3737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:27 smithi002 ceph-mon[17167]: pgmap v3738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:27.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:27 smithi016 ceph-mon[22010]: pgmap v3738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:27 smithi079 ceph-mon[23158]: pgmap v3738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6650.001145 secs 2022-03-21T16:02:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6650.001145 secs 2022-03-21T16:02:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6650.001145 secs 2022-03-21T16:02:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:29 smithi002 ceph-mon[17167]: pgmap v3739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:29 smithi016 ceph-mon[22010]: pgmap v3739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:29 smithi079 ceph-mon[23158]: pgmap v3739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:30 smithi002 ceph-mon[17167]: pgmap v3740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:30 smithi016 ceph-mon[22010]: pgmap v3740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:30 smithi079 ceph-mon[23158]: pgmap v3740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:32 smithi002 ceph-mon[17167]: pgmap v3741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:32 smithi016 ceph-mon[22010]: pgmap v3741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:32 smithi079 ceph-mon[23158]: pgmap v3741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6655.001260 secs 2022-03-21T16:02:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6655.001260 secs 2022-03-21T16:02:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6655.001260 secs 2022-03-21T16:02:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:34 smithi002 ceph-mon[17167]: pgmap v3742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:34 smithi016 ceph-mon[22010]: pgmap v3742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:34 smithi079 ceph-mon[23158]: pgmap v3742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:37 smithi002 ceph-mon[17167]: pgmap v3743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:37.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:37 smithi016 ceph-mon[22010]: pgmap v3743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:37.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:37 smithi079 ceph-mon[23158]: pgmap v3743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6660.001374 secs 2022-03-21T16:02:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6660.001374 secs 2022-03-21T16:02:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6660.001374 secs 2022-03-21T16:02:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:39 smithi002 ceph-mon[17167]: pgmap v3744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:39.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:39 smithi016 ceph-mon[22010]: pgmap v3744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:39 smithi079 ceph-mon[23158]: pgmap v3744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:40 smithi002 ceph-mon[17167]: pgmap v3745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:40 smithi016 ceph-mon[22010]: pgmap v3745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:40 smithi079 ceph-mon[23158]: pgmap v3745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:42 smithi002 ceph-mon[17167]: pgmap v3746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:42 smithi016 ceph-mon[22010]: pgmap v3746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:42 smithi079 ceph-mon[23158]: pgmap v3746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6665.001485 secs 2022-03-21T16:02:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6665.001485 secs 2022-03-21T16:02:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6665.001485 secs 2022-03-21T16:02:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:44 smithi002 ceph-mon[17167]: pgmap v3747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:44 smithi016 ceph-mon[22010]: pgmap v3747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:44 smithi079 ceph-mon[23158]: pgmap v3747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:02:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:47 smithi002 ceph-mon[17167]: pgmap v3748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:47 smithi016 ceph-mon[22010]: pgmap v3748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:47 smithi079 ceph-mon[23158]: pgmap v3748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6670.001603 secs 2022-03-21T16:02:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6670.001603 secs 2022-03-21T16:02:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6670.001603 secs 2022-03-21T16:02:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:49 smithi002 ceph-mon[17167]: pgmap v3749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:49 smithi016 ceph-mon[22010]: pgmap v3749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:49 smithi079 ceph-mon[23158]: pgmap v3749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:50 smithi002 ceph-mon[17167]: pgmap v3750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:50 smithi016 ceph-mon[22010]: pgmap v3750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:50 smithi079 ceph-mon[23158]: pgmap v3750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:52 smithi002 ceph-mon[17167]: pgmap v3751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:52 smithi016 ceph-mon[22010]: pgmap v3751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:52 smithi079 ceph-mon[23158]: pgmap v3751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6675.001730 secs 2022-03-21T16:02:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6675.001730 secs 2022-03-21T16:02:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6675.001730 secs 2022-03-21T16:02:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:54 smithi002 ceph-mon[17167]: pgmap v3752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:54 smithi016 ceph-mon[22010]: pgmap v3752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:54 smithi079 ceph-mon[23158]: pgmap v3752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:57 smithi002 ceph-mon[17167]: pgmap v3753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:57.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:57 smithi016 ceph-mon[22010]: pgmap v3753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:57 smithi079 ceph-mon[23158]: pgmap v3753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:02:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6680.001866 secs 2022-03-21T16:02:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6680.001866 secs 2022-03-21T16:02:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6680.001866 secs 2022-03-21T16:02:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:02:59 smithi002 ceph-mon[17167]: pgmap v3754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:02:59 smithi016 ceph-mon[22010]: pgmap v3754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:02:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:02:59 smithi079 ceph-mon[23158]: pgmap v3754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:00 smithi002 ceph-mon[17167]: pgmap v3755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:03:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:03:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:03:01.147 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:00 smithi079 ceph-mon[23158]: pgmap v3755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:01.147 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:03:01.147 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:03:01.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:03:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:00 smithi016 ceph-mon[22010]: pgmap v3755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:03:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:03:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:03:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6685.001979 secs 2022-03-21T16:03:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:03 smithi002 ceph-mon[17167]: pgmap v3756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:03.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6685.001979 secs 2022-03-21T16:03:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:03 smithi016 ceph-mon[22010]: pgmap v3756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6685.001979 secs 2022-03-21T16:03:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:03 smithi079 ceph-mon[23158]: pgmap v3756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:04 smithi002 ceph-mon[17167]: pgmap v3757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:05.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:04 smithi016 ceph-mon[22010]: pgmap v3757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:04 smithi079 ceph-mon[23158]: pgmap v3757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:07 smithi002 ceph-mon[17167]: pgmap v3758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:07 smithi016 ceph-mon[22010]: pgmap v3758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:07 smithi079 ceph-mon[23158]: pgmap v3758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6690.002097 secs 2022-03-21T16:03:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6690.002097 secs 2022-03-21T16:03:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6690.002097 secs 2022-03-21T16:03:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:09 smithi002 ceph-mon[17167]: pgmap v3759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:09 smithi016 ceph-mon[22010]: pgmap v3759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:09 smithi079 ceph-mon[23158]: pgmap v3759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:10 smithi002 ceph-mon[17167]: pgmap v3760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:10 smithi016 ceph-mon[22010]: pgmap v3760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:10 smithi079 ceph-mon[23158]: pgmap v3760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:12 smithi002 ceph-mon[17167]: pgmap v3761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:13.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:12 smithi016 ceph-mon[22010]: pgmap v3761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:12 smithi079 ceph-mon[23158]: pgmap v3761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6695.002214 secs 2022-03-21T16:03:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6695.002214 secs 2022-03-21T16:03:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6695.002214 secs 2022-03-21T16:03:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:14 smithi002 ceph-mon[17167]: pgmap v3762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:14 smithi016 ceph-mon[22010]: pgmap v3762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:14 smithi079 ceph-mon[23158]: pgmap v3762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:17 smithi002 ceph-mon[17167]: pgmap v3763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:17 smithi016 ceph-mon[22010]: pgmap v3763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:17 smithi079 ceph-mon[23158]: pgmap v3763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6700.002312 secs 2022-03-21T16:03:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6700.002312 secs 2022-03-21T16:03:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6700.002312 secs 2022-03-21T16:03:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:19 smithi002 ceph-mon[17167]: pgmap v3764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:19 smithi016 ceph-mon[22010]: pgmap v3764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:19 smithi079 ceph-mon[23158]: pgmap v3764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:20 smithi002 ceph-mon[17167]: pgmap v3765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:20 smithi016 ceph-mon[22010]: pgmap v3765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:20 smithi079 ceph-mon[23158]: pgmap v3765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:03:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:03:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:03:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:03:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:03:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:03:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:23 smithi002 ceph-mon[17167]: pgmap v3766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:23.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:23 smithi016 ceph-mon[22010]: pgmap v3766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:23 smithi079 ceph-mon[23158]: pgmap v3766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:24 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6705.002408 secs 2022-03-21T16:03:24.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6705.002408 secs 2022-03-21T16:03:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6705.002408 secs 2022-03-21T16:03:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:25 smithi002 ceph-mon[17167]: pgmap v3767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:25.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:25 smithi016 ceph-mon[22010]: pgmap v3767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:25 smithi079 ceph-mon[23158]: pgmap v3767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:27 smithi002 ceph-mon[17167]: pgmap v3768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:27 smithi016 ceph-mon[22010]: pgmap v3768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:27 smithi079 ceph-mon[23158]: pgmap v3768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6710.002533 secs 2022-03-21T16:03:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6710.002533 secs 2022-03-21T16:03:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6710.002533 secs 2022-03-21T16:03:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:29 smithi002 ceph-mon[17167]: pgmap v3769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:29.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:29 smithi016 ceph-mon[22010]: pgmap v3769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:29 smithi079 ceph-mon[23158]: pgmap v3769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:30 smithi002 ceph-mon[17167]: pgmap v3770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:31.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:30 smithi016 ceph-mon[22010]: pgmap v3770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:30 smithi079 ceph-mon[23158]: pgmap v3770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:32 smithi002 ceph-mon[17167]: pgmap v3771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:32 smithi016 ceph-mon[22010]: pgmap v3771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:32 smithi079 ceph-mon[23158]: pgmap v3771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6715.002658 secs 2022-03-21T16:03:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6715.002658 secs 2022-03-21T16:03:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6715.002658 secs 2022-03-21T16:03:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:34 smithi002 ceph-mon[17167]: pgmap v3772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:34 smithi016 ceph-mon[22010]: pgmap v3772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:34 smithi079 ceph-mon[23158]: pgmap v3772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:37 smithi002 ceph-mon[17167]: pgmap v3773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:37 smithi016 ceph-mon[22010]: pgmap v3773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:37 smithi079 ceph-mon[23158]: pgmap v3773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6720.002778 secs 2022-03-21T16:03:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6720.002778 secs 2022-03-21T16:03:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6720.002778 secs 2022-03-21T16:03:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:39 smithi002 ceph-mon[17167]: pgmap v3774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:39 smithi016 ceph-mon[22010]: pgmap v3774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:39 smithi079 ceph-mon[23158]: pgmap v3774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:40 smithi002 ceph-mon[17167]: pgmap v3775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:41.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:40 smithi016 ceph-mon[22010]: pgmap v3775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:40 smithi079 ceph-mon[23158]: pgmap v3775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:42 smithi002 ceph-mon[17167]: pgmap v3776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:42 smithi016 ceph-mon[22010]: pgmap v3776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:42 smithi079 ceph-mon[23158]: pgmap v3776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6725.002871 secs 2022-03-21T16:03:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6725.002871 secs 2022-03-21T16:03:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6725.002871 secs 2022-03-21T16:03:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:44 smithi002 ceph-mon[17167]: pgmap v3777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:45.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:44 smithi016 ceph-mon[22010]: pgmap v3777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:44 smithi079 ceph-mon[23158]: pgmap v3777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:47 smithi002 ceph-mon[17167]: pgmap v3778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:47.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:47 smithi016 ceph-mon[22010]: pgmap v3778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:47 smithi079 ceph-mon[23158]: pgmap v3778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6730.002988 secs 2022-03-21T16:03:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6730.002988 secs 2022-03-21T16:03:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6730.002988 secs 2022-03-21T16:03:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:49 smithi002 ceph-mon[17167]: pgmap v3779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:49.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:49 smithi016 ceph-mon[22010]: pgmap v3779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:49 smithi079 ceph-mon[23158]: pgmap v3779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:50 smithi002 ceph-mon[17167]: pgmap v3780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:51.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:50 smithi016 ceph-mon[22010]: pgmap v3780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:50 smithi079 ceph-mon[23158]: pgmap v3780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:52 smithi002 ceph-mon[17167]: pgmap v3781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:53.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:52 smithi016 ceph-mon[22010]: pgmap v3781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:52 smithi079 ceph-mon[23158]: pgmap v3781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6735.003065 secs 2022-03-21T16:03:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6735.003065 secs 2022-03-21T16:03:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6735.003065 secs 2022-03-21T16:03:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:54 smithi002 ceph-mon[17167]: pgmap v3782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:54 smithi016 ceph-mon[22010]: pgmap v3782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:54 smithi079 ceph-mon[23158]: pgmap v3782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:57 smithi002 ceph-mon[17167]: pgmap v3783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:57 smithi016 ceph-mon[22010]: pgmap v3783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:57 smithi079 ceph-mon[23158]: pgmap v3783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:03:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6740.003160 secs 2022-03-21T16:03:58.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6740.003160 secs 2022-03-21T16:03:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6740.003160 secs 2022-03-21T16:03:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:03:59 smithi002 ceph-mon[17167]: pgmap v3784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:03:59 smithi016 ceph-mon[22010]: pgmap v3784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:03:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:03:59 smithi079 ceph-mon[23158]: pgmap v3784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:00 smithi002 ceph-mon[17167]: pgmap v3785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:00 smithi016 ceph-mon[22010]: pgmap v3785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:00 smithi079 ceph-mon[23158]: pgmap v3785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:04:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:04:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:04:02.549 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:04:02.550 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:04:02.550 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:04:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:04:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:04:02.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:04:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6745.003227 secs 2022-03-21T16:04:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: pgmap v3786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6745.003227 secs 2022-03-21T16:04:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: pgmap v3786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6745.003227 secs 2022-03-21T16:04:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: pgmap v3786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:04 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:04 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:04 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:04:05.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:05 smithi016 ceph-mon[22010]: pgmap v3787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:05 smithi079 ceph-mon[23158]: pgmap v3787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:06.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:05 smithi002 ceph-mon[17167]: pgmap v3787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:07 smithi002 ceph-mon[17167]: pgmap v3788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:07 smithi016 ceph-mon[22010]: pgmap v3788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:07 smithi079 ceph-mon[23158]: pgmap v3788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6750.003289 secs 2022-03-21T16:04:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6750.003289 secs 2022-03-21T16:04:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6750.003289 secs 2022-03-21T16:04:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:09 smithi002 ceph-mon[17167]: pgmap v3789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:09 smithi016 ceph-mon[22010]: pgmap v3789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:09 smithi079 ceph-mon[23158]: pgmap v3789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:10 smithi002 ceph-mon[17167]: pgmap v3790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:11.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:10 smithi016 ceph-mon[22010]: pgmap v3790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:10 smithi079 ceph-mon[23158]: pgmap v3790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:12 smithi002 ceph-mon[17167]: pgmap v3791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:12 smithi016 ceph-mon[22010]: pgmap v3791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:12 smithi079 ceph-mon[23158]: pgmap v3791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6755.003425 secs 2022-03-21T16:04:14.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6755.003425 secs 2022-03-21T16:04:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6755.003425 secs 2022-03-21T16:04:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:14 smithi002 ceph-mon[17167]: pgmap v3792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:14 smithi016 ceph-mon[22010]: pgmap v3792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:14 smithi079 ceph-mon[23158]: pgmap v3792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:17 smithi002 ceph-mon[17167]: pgmap v3793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:17 smithi016 ceph-mon[22010]: pgmap v3793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:17 smithi079 ceph-mon[23158]: pgmap v3793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6760.003561 secs 2022-03-21T16:04:18.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6760.003561 secs 2022-03-21T16:04:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6760.003561 secs 2022-03-21T16:04:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:19 smithi002 ceph-mon[17167]: pgmap v3794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:19.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:19 smithi016 ceph-mon[22010]: pgmap v3794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:19 smithi079 ceph-mon[23158]: pgmap v3794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:20 smithi002 ceph-mon[17167]: pgmap v3795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:20 smithi016 ceph-mon[22010]: pgmap v3795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:20 smithi079 ceph-mon[23158]: pgmap v3795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:04:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:04:22.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:04:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:04:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:04:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:04:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:22 smithi002 ceph-mon[17167]: pgmap v3796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:23.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:22 smithi016 ceph-mon[22010]: pgmap v3796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:22 smithi079 ceph-mon[23158]: pgmap v3796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6765.003682 secs 2022-03-21T16:04:24.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6765.003682 secs 2022-03-21T16:04:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6765.003682 secs 2022-03-21T16:04:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:24 smithi002 ceph-mon[17167]: pgmap v3797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:24 smithi016 ceph-mon[22010]: pgmap v3797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:24 smithi079 ceph-mon[23158]: pgmap v3797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:27 smithi002 ceph-mon[17167]: pgmap v3798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:27.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:27 smithi016 ceph-mon[22010]: pgmap v3798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:27 smithi079 ceph-mon[23158]: pgmap v3798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6770.003796 secs 2022-03-21T16:04:28.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6770.003796 secs 2022-03-21T16:04:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6770.003796 secs 2022-03-21T16:04:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:29 smithi002 ceph-mon[17167]: pgmap v3799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:29 smithi016 ceph-mon[22010]: pgmap v3799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:29 smithi079 ceph-mon[23158]: pgmap v3799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:30 smithi002 ceph-mon[17167]: pgmap v3800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:30 smithi016 ceph-mon[22010]: pgmap v3800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:30 smithi079 ceph-mon[23158]: pgmap v3800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:32 smithi002 ceph-mon[17167]: pgmap v3801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:33.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:32 smithi016 ceph-mon[22010]: pgmap v3801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:32 smithi079 ceph-mon[23158]: pgmap v3801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6775.003907 secs 2022-03-21T16:04:34.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6775.003907 secs 2022-03-21T16:04:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6775.003907 secs 2022-03-21T16:04:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:34 smithi002 ceph-mon[17167]: pgmap v3802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:34 smithi016 ceph-mon[22010]: pgmap v3802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:34 smithi079 ceph-mon[23158]: pgmap v3802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:37 smithi002 ceph-mon[17167]: pgmap v3803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:37.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:37 smithi016 ceph-mon[22010]: pgmap v3803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:37 smithi079 ceph-mon[23158]: pgmap v3803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6780.004045 secs 2022-03-21T16:04:38.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6780.004045 secs 2022-03-21T16:04:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6780.004045 secs 2022-03-21T16:04:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:39 smithi002 ceph-mon[17167]: pgmap v3804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:39.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:39 smithi016 ceph-mon[22010]: pgmap v3804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:39 smithi079 ceph-mon[23158]: pgmap v3804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:41.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:40 smithi002 ceph-mon[17167]: pgmap v3805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:40 smithi016 ceph-mon[22010]: pgmap v3805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:41.550 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:40 smithi079 ceph-mon[23158]: pgmap v3805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:42 smithi002 ceph-mon[17167]: pgmap v3806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:43.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:42 smithi016 ceph-mon[22010]: pgmap v3806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:42 smithi079 ceph-mon[23158]: pgmap v3806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6785.004161 secs 2022-03-21T16:04:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6785.004161 secs 2022-03-21T16:04:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6785.004161 secs 2022-03-21T16:04:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:44 smithi002 ceph-mon[17167]: pgmap v3807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:44 smithi016 ceph-mon[22010]: pgmap v3807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:44 smithi079 ceph-mon[23158]: pgmap v3807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:47 smithi002 ceph-mon[17167]: pgmap v3808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:47.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:47 smithi016 ceph-mon[22010]: pgmap v3808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:47 smithi079 ceph-mon[23158]: pgmap v3808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6790.004227 secs 2022-03-21T16:04:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6790.004227 secs 2022-03-21T16:04:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6790.004227 secs 2022-03-21T16:04:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:49 smithi002 ceph-mon[17167]: pgmap v3809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:49 smithi016 ceph-mon[22010]: pgmap v3809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:49 smithi079 ceph-mon[23158]: pgmap v3809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:50 smithi002 ceph-mon[17167]: pgmap v3810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:51.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:50 smithi016 ceph-mon[22010]: pgmap v3810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:50 smithi079 ceph-mon[23158]: pgmap v3810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:52 smithi002 ceph-mon[17167]: pgmap v3811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:52 smithi016 ceph-mon[22010]: pgmap v3811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:52 smithi079 ceph-mon[23158]: pgmap v3811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6795.004381 secs 2022-03-21T16:04:54.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6795.004381 secs 2022-03-21T16:04:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6795.004381 secs 2022-03-21T16:04:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:54 smithi002 ceph-mon[17167]: pgmap v3812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:55.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:54 smithi016 ceph-mon[22010]: pgmap v3812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:54 smithi079 ceph-mon[23158]: pgmap v3812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:57 smithi002 ceph-mon[17167]: pgmap v3813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:57.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:57 smithi016 ceph-mon[22010]: pgmap v3813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:57 smithi079 ceph-mon[23158]: pgmap v3813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:04:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6800.004524 secs 2022-03-21T16:04:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6800.004524 secs 2022-03-21T16:04:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6800.004524 secs 2022-03-21T16:04:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:04:59 smithi002 ceph-mon[17167]: pgmap v3814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:59.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:04:59 smithi016 ceph-mon[22010]: pgmap v3814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:04:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:04:59 smithi079 ceph-mon[23158]: pgmap v3814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:00 smithi002 ceph-mon[17167]: pgmap v3815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:01.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:00 smithi016 ceph-mon[22010]: pgmap v3815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:00 smithi079 ceph-mon[23158]: pgmap v3815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:02 smithi002 ceph-mon[17167]: pgmap v3816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:02 smithi016 ceph-mon[22010]: pgmap v3816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:02 smithi079 ceph-mon[23158]: pgmap v3816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6805.004637 secs 2022-03-21T16:05:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:05:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:05:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:03 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:05:04.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6805.004637 secs 2022-03-21T16:05:04.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:05:04.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:05:04.077 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:03 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:05:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6805.004637 secs 2022-03-21T16:05:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:05:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:05:04.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:03 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:05:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:05:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:05 smithi002 ceph-mon[17167]: pgmap v3817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:05:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:05 smithi016 ceph-mon[22010]: pgmap v3817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:05:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:05 smithi079 ceph-mon[23158]: pgmap v3817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:07 smithi002 ceph-mon[17167]: pgmap v3818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:07.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:07 smithi016 ceph-mon[22010]: pgmap v3818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:07 smithi079 ceph-mon[23158]: pgmap v3818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6810.004757 secs 2022-03-21T16:05:08.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6810.004757 secs 2022-03-21T16:05:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6810.004757 secs 2022-03-21T16:05:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:09 smithi002 ceph-mon[17167]: pgmap v3819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:09 smithi016 ceph-mon[22010]: pgmap v3819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:09 smithi079 ceph-mon[23158]: pgmap v3819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:12 smithi002 ceph-mon[17167]: pgmap v3820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:12 smithi016 ceph-mon[22010]: pgmap v3820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:12 smithi079 ceph-mon[23158]: pgmap v3820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:14 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6815.004870 secs 2022-03-21T16:05:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:14 smithi002 ceph-mon[17167]: pgmap v3821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:14 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6815.004870 secs 2022-03-21T16:05:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:14 smithi016 ceph-mon[22010]: pgmap v3821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:14 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6815.004870 secs 2022-03-21T16:05:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:14 smithi079 ceph-mon[23158]: pgmap v3821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:16 smithi002 ceph-mon[17167]: pgmap v3822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:16.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:16 smithi016 ceph-mon[22010]: pgmap v3822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:16 smithi079 ceph-mon[23158]: pgmap v3822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:18 smithi002 ceph-mon[17167]: pgmap v3823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:18 smithi016 ceph-mon[22010]: pgmap v3823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:18 smithi079 ceph-mon[23158]: pgmap v3823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:19 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6820.004985 secs 2022-03-21T16:05:19.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:19 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6820.004985 secs 2022-03-21T16:05:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:19 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6820.004985 secs 2022-03-21T16:05:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:20 smithi002 ceph-mon[17167]: pgmap v3824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:20.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:20 smithi016 ceph-mon[22010]: pgmap v3824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:20 smithi079 ceph-mon[23158]: pgmap v3824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:05:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:05:21.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:05:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:05:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:05:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:05:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:22 smithi002 ceph-mon[17167]: pgmap v3825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:22.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:22 smithi016 ceph-mon[22010]: pgmap v3825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:22 smithi079 ceph-mon[23158]: pgmap v3825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:24 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6825.005124 secs 2022-03-21T16:05:24.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:24 smithi002 ceph-mon[17167]: pgmap v3826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6825.005124 secs 2022-03-21T16:05:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:24 smithi016 ceph-mon[22010]: pgmap v3826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6825.005124 secs 2022-03-21T16:05:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:24 smithi079 ceph-mon[23158]: pgmap v3826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:26 smithi002 ceph-mon[17167]: pgmap v3827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:26.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:26 smithi016 ceph-mon[22010]: pgmap v3827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:26 smithi079 ceph-mon[23158]: pgmap v3827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:28 smithi002 ceph-mon[17167]: pgmap v3828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:28.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:28 smithi016 ceph-mon[22010]: pgmap v3828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:28 smithi079 ceph-mon[23158]: pgmap v3828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:29 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6830.005245 secs 2022-03-21T16:05:29.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:29 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6830.005245 secs 2022-03-21T16:05:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:29 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6830.005245 secs 2022-03-21T16:05:30.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:30 smithi002 ceph-mon[17167]: pgmap v3829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:30.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:30 smithi016 ceph-mon[22010]: pgmap v3829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:30 smithi079 ceph-mon[23158]: pgmap v3829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:32.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:32 smithi002 ceph-mon[17167]: pgmap v3830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:32.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:32 smithi016 ceph-mon[22010]: pgmap v3830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:32 smithi079 ceph-mon[23158]: pgmap v3830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:34.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:34 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6835.005365 secs 2022-03-21T16:05:34.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:34 smithi002 ceph-mon[17167]: pgmap v3831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:34 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6835.005365 secs 2022-03-21T16:05:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:34 smithi016 ceph-mon[22010]: pgmap v3831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:34 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6835.005365 secs 2022-03-21T16:05:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:34 smithi079 ceph-mon[23158]: pgmap v3831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:36 smithi016 ceph-mon[22010]: pgmap v3832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:36 smithi079 ceph-mon[23158]: pgmap v3832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:36 smithi002 ceph-mon[17167]: pgmap v3832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:38 smithi016 ceph-mon[22010]: pgmap v3833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:38 smithi079 ceph-mon[23158]: pgmap v3833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:38 smithi002 ceph-mon[17167]: pgmap v3833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:39 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6840.005483 secs 2022-03-21T16:05:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:39 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6840.005483 secs 2022-03-21T16:05:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:39 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6840.005483 secs 2022-03-21T16:05:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:40 smithi016 ceph-mon[22010]: pgmap v3834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:40 smithi079 ceph-mon[23158]: pgmap v3834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:40 smithi002 ceph-mon[17167]: pgmap v3834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:42 smithi016 ceph-mon[22010]: pgmap v3835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:42 smithi079 ceph-mon[23158]: pgmap v3835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:42 smithi002 ceph-mon[17167]: pgmap v3835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:44 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6845.005581 secs 2022-03-21T16:05:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:44 smithi016 ceph-mon[22010]: pgmap v3836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:44 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6845.005581 secs 2022-03-21T16:05:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:44 smithi079 ceph-mon[23158]: pgmap v3836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:44 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6845.005581 secs 2022-03-21T16:05:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:44 smithi002 ceph-mon[17167]: pgmap v3836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:46 smithi016 ceph-mon[22010]: pgmap v3837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:46 smithi079 ceph-mon[23158]: pgmap v3837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:46 smithi002 ceph-mon[17167]: pgmap v3837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:48 smithi016 ceph-mon[22010]: pgmap v3838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:48 smithi079 ceph-mon[23158]: pgmap v3838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:48 smithi002 ceph-mon[17167]: pgmap v3838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:49 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6850.005721 secs 2022-03-21T16:05:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:49 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6850.005721 secs 2022-03-21T16:05:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:49 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6850.005721 secs 2022-03-21T16:05:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:50 smithi016 ceph-mon[22010]: pgmap v3839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:50 smithi079 ceph-mon[23158]: pgmap v3839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:50 smithi002 ceph-mon[17167]: pgmap v3839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:52 smithi016 ceph-mon[22010]: pgmap v3840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:52 smithi079 ceph-mon[23158]: pgmap v3840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:52 smithi002 ceph-mon[17167]: pgmap v3840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:54 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6855.005875 secs 2022-03-21T16:05:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:54 smithi016 ceph-mon[22010]: pgmap v3841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:54 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6855.005875 secs 2022-03-21T16:05:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:54 smithi079 ceph-mon[23158]: pgmap v3841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:54 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6855.005875 secs 2022-03-21T16:05:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:54 smithi002 ceph-mon[17167]: pgmap v3841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:56 smithi016 ceph-mon[22010]: pgmap v3842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:56 smithi079 ceph-mon[23158]: pgmap v3842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:56 smithi002 ceph-mon[17167]: pgmap v3842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:05:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:58 smithi016 ceph-mon[22010]: pgmap v3843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:58 smithi079 ceph-mon[23158]: pgmap v3843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:58 smithi002 ceph-mon[17167]: pgmap v3843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:05:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:05:59 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6860.005998 secs 2022-03-21T16:05:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:05:59 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6860.005998 secs 2022-03-21T16:05:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:05:59 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6860.005998 secs 2022-03-21T16:06:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:00 smithi079 ceph-mon[23158]: pgmap v3844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:00 smithi002 ceph-mon[17167]: pgmap v3844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:00 smithi016 ceph-mon[22010]: pgmap v3844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:02 smithi079 ceph-mon[23158]: pgmap v3845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:02 smithi002 ceph-mon[17167]: pgmap v3845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:02.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:02 smithi016 ceph-mon[22010]: pgmap v3845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:04 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6865.006132 secs 2022-03-21T16:06:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:04 smithi079 ceph-mon[23158]: pgmap v3846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:04.511 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:04 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6865.006132 secs 2022-03-21T16:06:04.511 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:04 smithi016 ceph-mon[22010]: pgmap v3846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:04.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:04 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6865.006132 secs 2022-03-21T16:06:04.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:04 smithi002 ceph-mon[17167]: pgmap v3846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:06:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:06:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:06:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:06:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:06:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:06:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:06:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:06:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:06:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:06 smithi002 ceph-mon[17167]: pgmap v3847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:06:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:06 smithi016 ceph-mon[22010]: pgmap v3847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:06:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:06 smithi079 ceph-mon[23158]: pgmap v3847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:06:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:08 smithi002 ceph-mon[17167]: pgmap v3848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6870.006224 secs 2022-03-21T16:06:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:08 smithi016 ceph-mon[22010]: pgmap v3848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6870.006224 secs 2022-03-21T16:06:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:08 smithi079 ceph-mon[23158]: pgmap v3848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6870.006224 secs 2022-03-21T16:06:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:10 smithi002 ceph-mon[17167]: pgmap v3849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:10 smithi016 ceph-mon[22010]: pgmap v3849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:10 smithi079 ceph-mon[23158]: pgmap v3849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:12 smithi002 ceph-mon[17167]: pgmap v3850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:12 smithi016 ceph-mon[22010]: pgmap v3850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:12 smithi079 ceph-mon[23158]: pgmap v3850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6875.006344 secs 2022-03-21T16:06:13.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6875.006344 secs 2022-03-21T16:06:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6875.006344 secs 2022-03-21T16:06:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:14 smithi002 ceph-mon[17167]: pgmap v3851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:14 smithi016 ceph-mon[22010]: pgmap v3851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:14 smithi079 ceph-mon[23158]: pgmap v3851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:16 smithi002 ceph-mon[17167]: pgmap v3852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:16 smithi016 ceph-mon[22010]: pgmap v3852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:16 smithi079 ceph-mon[23158]: pgmap v3852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:18 smithi002 ceph-mon[17167]: pgmap v3853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6880.006409 secs 2022-03-21T16:06:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:18 smithi016 ceph-mon[22010]: pgmap v3853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6880.006409 secs 2022-03-21T16:06:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:18 smithi079 ceph-mon[23158]: pgmap v3853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6880.006409 secs 2022-03-21T16:06:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:20 smithi002 ceph-mon[17167]: pgmap v3854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:20 smithi016 ceph-mon[22010]: pgmap v3854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:20 smithi079 ceph-mon[23158]: pgmap v3854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:21.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:06:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:06:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:06:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:06:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:06:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:06:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:22 smithi016 ceph-mon[22010]: pgmap v3855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:22 smithi079 ceph-mon[23158]: pgmap v3855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:22 smithi002 ceph-mon[17167]: pgmap v3855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6885.006560 secs 2022-03-21T16:06:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6885.006560 secs 2022-03-21T16:06:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6885.006560 secs 2022-03-21T16:06:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:24 smithi016 ceph-mon[22010]: pgmap v3856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:24 smithi079 ceph-mon[23158]: pgmap v3856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:24 smithi002 ceph-mon[17167]: pgmap v3856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:26 smithi016 ceph-mon[22010]: pgmap v3857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:26 smithi079 ceph-mon[23158]: pgmap v3857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:26 smithi002 ceph-mon[17167]: pgmap v3857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:28 smithi016 ceph-mon[22010]: pgmap v3858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6890.006652 secs 2022-03-21T16:06:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:28 smithi079 ceph-mon[23158]: pgmap v3858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6890.006652 secs 2022-03-21T16:06:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:28 smithi002 ceph-mon[17167]: pgmap v3858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6890.006652 secs 2022-03-21T16:06:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:30 smithi016 ceph-mon[22010]: pgmap v3859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:30 smithi079 ceph-mon[23158]: pgmap v3859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:30 smithi002 ceph-mon[17167]: pgmap v3859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:32 smithi016 ceph-mon[22010]: pgmap v3860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:32 smithi079 ceph-mon[23158]: pgmap v3860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:32 smithi002 ceph-mon[17167]: pgmap v3860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6895.006772 secs 2022-03-21T16:06:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6895.006772 secs 2022-03-21T16:06:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6895.006772 secs 2022-03-21T16:06:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:34 smithi016 ceph-mon[22010]: pgmap v3861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:34 smithi079 ceph-mon[23158]: pgmap v3861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:34 smithi002 ceph-mon[17167]: pgmap v3861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:36 smithi016 ceph-mon[22010]: pgmap v3862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:36 smithi079 ceph-mon[23158]: pgmap v3862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:36 smithi002 ceph-mon[17167]: pgmap v3862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:38 smithi016 ceph-mon[22010]: pgmap v3863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6900.006888 secs 2022-03-21T16:06:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:38 smithi079 ceph-mon[23158]: pgmap v3863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6900.006888 secs 2022-03-21T16:06:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:38 smithi002 ceph-mon[17167]: pgmap v3863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6900.006888 secs 2022-03-21T16:06:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:40 smithi016 ceph-mon[22010]: pgmap v3864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:40 smithi079 ceph-mon[23158]: pgmap v3864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:40 smithi002 ceph-mon[17167]: pgmap v3864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:42 smithi079 ceph-mon[23158]: pgmap v3865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:42 smithi002 ceph-mon[17167]: pgmap v3865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:42 smithi016 ceph-mon[22010]: pgmap v3865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6905.007009 secs 2022-03-21T16:06:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6905.007009 secs 2022-03-21T16:06:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6905.007009 secs 2022-03-21T16:06:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:44 smithi079 ceph-mon[23158]: pgmap v3866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:44 smithi002 ceph-mon[17167]: pgmap v3866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:44 smithi016 ceph-mon[22010]: pgmap v3866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:46 smithi079 ceph-mon[23158]: pgmap v3867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:46 smithi002 ceph-mon[17167]: pgmap v3867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:46 smithi016 ceph-mon[22010]: pgmap v3867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:48 smithi079 ceph-mon[23158]: pgmap v3868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6910.007131 secs 2022-03-21T16:06:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:48 smithi002 ceph-mon[17167]: pgmap v3868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6910.007131 secs 2022-03-21T16:06:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:48 smithi016 ceph-mon[22010]: pgmap v3868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6910.007131 secs 2022-03-21T16:06:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:50 smithi079 ceph-mon[23158]: pgmap v3869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:50 smithi002 ceph-mon[17167]: pgmap v3869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:50 smithi016 ceph-mon[22010]: pgmap v3869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:52 smithi079 ceph-mon[23158]: pgmap v3870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:52 smithi002 ceph-mon[17167]: pgmap v3870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:52.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:52 smithi016 ceph-mon[22010]: pgmap v3870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6915.007266 secs 2022-03-21T16:06:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6915.007266 secs 2022-03-21T16:06:53.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6915.007266 secs 2022-03-21T16:06:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:54 smithi079 ceph-mon[23158]: pgmap v3871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:54 smithi002 ceph-mon[17167]: pgmap v3871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:54 smithi016 ceph-mon[22010]: pgmap v3871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:56 smithi079 ceph-mon[23158]: pgmap v3872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:56 smithi002 ceph-mon[17167]: pgmap v3872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:56.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:56 smithi016 ceph-mon[22010]: pgmap v3872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:06:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:58 smithi079 ceph-mon[23158]: pgmap v3873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:06:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6920.007397 secs 2022-03-21T16:06:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:58 smithi002 ceph-mon[17167]: pgmap v3873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:06:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6920.007397 secs 2022-03-21T16:06:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:58 smithi016 ceph-mon[22010]: pgmap v3873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:06:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:06:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6920.007397 secs 2022-03-21T16:07:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:00 smithi079 ceph-mon[23158]: pgmap v3874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:00 smithi002 ceph-mon[17167]: pgmap v3874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:00 smithi016 ceph-mon[22010]: pgmap v3874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:02 smithi079 ceph-mon[23158]: pgmap v3875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:02 smithi002 ceph-mon[17167]: pgmap v3875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:02 smithi016 ceph-mon[22010]: pgmap v3875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6925.007512 secs 2022-03-21T16:07:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6925.007512 secs 2022-03-21T16:07:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6925.007512 secs 2022-03-21T16:07:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:04 smithi079 ceph-mon[23158]: pgmap v3876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:04 smithi002 ceph-mon[17167]: pgmap v3876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:04 smithi016 ceph-mon[22010]: pgmap v3876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:07:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:07:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:07:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:07:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:07:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:07:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:07:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:07:05.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:07:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:06 smithi002 ceph-mon[17167]: pgmap v3877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:07:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:06 smithi016 ceph-mon[22010]: pgmap v3877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:07:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:06 smithi079 ceph-mon[23158]: pgmap v3877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:07:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:08 smithi002 ceph-mon[17167]: pgmap v3878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6930.007629 secs 2022-03-21T16:07:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:08 smithi016 ceph-mon[22010]: pgmap v3878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6930.007629 secs 2022-03-21T16:07:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:08 smithi079 ceph-mon[23158]: pgmap v3878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6930.007629 secs 2022-03-21T16:07:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:10 smithi002 ceph-mon[17167]: pgmap v3879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:10 smithi016 ceph-mon[22010]: pgmap v3879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:10 smithi079 ceph-mon[23158]: pgmap v3879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:12 smithi002 ceph-mon[17167]: pgmap v3880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:12 smithi016 ceph-mon[22010]: pgmap v3880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:12 smithi079 ceph-mon[23158]: pgmap v3880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6935.007745 secs 2022-03-21T16:07:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6935.007745 secs 2022-03-21T16:07:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6935.007745 secs 2022-03-21T16:07:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:14 smithi002 ceph-mon[17167]: pgmap v3881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:14 smithi016 ceph-mon[22010]: pgmap v3881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:14 smithi079 ceph-mon[23158]: pgmap v3881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:16.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:16 smithi002 ceph-mon[17167]: pgmap v3882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:16 smithi016 ceph-mon[22010]: pgmap v3882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:16 smithi079 ceph-mon[23158]: pgmap v3882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:18 smithi002 ceph-mon[17167]: pgmap v3883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6940.007868 secs 2022-03-21T16:07:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:18 smithi016 ceph-mon[22010]: pgmap v3883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6940.007868 secs 2022-03-21T16:07:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:18 smithi079 ceph-mon[23158]: pgmap v3883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6940.007868 secs 2022-03-21T16:07:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:20 smithi002 ceph-mon[17167]: pgmap v3884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:20 smithi016 ceph-mon[22010]: pgmap v3884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:20 smithi079 ceph-mon[23158]: pgmap v3884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:07:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:07:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:07:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:07:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:07:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:07:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:22 smithi002 ceph-mon[17167]: pgmap v3885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:22.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:22 smithi016 ceph-mon[22010]: pgmap v3885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:22 smithi079 ceph-mon[23158]: pgmap v3885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6945.007992 secs 2022-03-21T16:07:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6945.007992 secs 2022-03-21T16:07:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6945.007992 secs 2022-03-21T16:07:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:24 smithi002 ceph-mon[17167]: pgmap v3886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:24.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:24 smithi016 ceph-mon[22010]: pgmap v3886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:24 smithi079 ceph-mon[23158]: pgmap v3886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:26 smithi002 ceph-mon[17167]: pgmap v3887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:26 smithi016 ceph-mon[22010]: pgmap v3887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:26 smithi079 ceph-mon[23158]: pgmap v3887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:28 smithi002 ceph-mon[17167]: pgmap v3888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6950.008109 secs 2022-03-21T16:07:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:28 smithi016 ceph-mon[22010]: pgmap v3888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6950.008109 secs 2022-03-21T16:07:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:28 smithi079 ceph-mon[23158]: pgmap v3888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6950.008109 secs 2022-03-21T16:07:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:30 smithi002 ceph-mon[17167]: pgmap v3889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:30 smithi016 ceph-mon[22010]: pgmap v3889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:30 smithi079 ceph-mon[23158]: pgmap v3889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:32 smithi002 ceph-mon[17167]: pgmap v3890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:32 smithi016 ceph-mon[22010]: pgmap v3890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:32 smithi079 ceph-mon[23158]: pgmap v3890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6955.008254 secs 2022-03-21T16:07:33.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6955.008254 secs 2022-03-21T16:07:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6955.008254 secs 2022-03-21T16:07:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:34 smithi002 ceph-mon[17167]: pgmap v3891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:34.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:34 smithi016 ceph-mon[22010]: pgmap v3891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:34 smithi079 ceph-mon[23158]: pgmap v3891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:36 smithi002 ceph-mon[17167]: pgmap v3892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:36 smithi016 ceph-mon[22010]: pgmap v3892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:36 smithi079 ceph-mon[23158]: pgmap v3892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:38 smithi002 ceph-mon[17167]: pgmap v3893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6960.008376 secs 2022-03-21T16:07:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:38 smithi016 ceph-mon[22010]: pgmap v3893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6960.008376 secs 2022-03-21T16:07:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:38 smithi079 ceph-mon[23158]: pgmap v3893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6960.008376 secs 2022-03-21T16:07:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:40 smithi002 ceph-mon[17167]: pgmap v3894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:40.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:40 smithi016 ceph-mon[22010]: pgmap v3894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:40 smithi079 ceph-mon[23158]: pgmap v3894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:42 smithi002 ceph-mon[17167]: pgmap v3895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:42 smithi016 ceph-mon[22010]: pgmap v3895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:42 smithi079 ceph-mon[23158]: pgmap v3895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6965.008438 secs 2022-03-21T16:07:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6965.008438 secs 2022-03-21T16:07:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6965.008438 secs 2022-03-21T16:07:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:44 smithi002 ceph-mon[17167]: pgmap v3896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:44 smithi016 ceph-mon[22010]: pgmap v3896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:44 smithi079 ceph-mon[23158]: pgmap v3896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:46 smithi002 ceph-mon[17167]: pgmap v3897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:46 smithi016 ceph-mon[22010]: pgmap v3897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:46 smithi079 ceph-mon[23158]: pgmap v3897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:48 smithi002 ceph-mon[17167]: pgmap v3898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6970.008577 secs 2022-03-21T16:07:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:48 smithi016 ceph-mon[22010]: pgmap v3898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6970.008577 secs 2022-03-21T16:07:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:48 smithi079 ceph-mon[23158]: pgmap v3898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6970.008577 secs 2022-03-21T16:07:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:50 smithi002 ceph-mon[17167]: pgmap v3899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:50 smithi016 ceph-mon[22010]: pgmap v3899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:50 smithi079 ceph-mon[23158]: pgmap v3899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:52 smithi002 ceph-mon[17167]: pgmap v3900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:52 smithi016 ceph-mon[22010]: pgmap v3900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:52 smithi079 ceph-mon[23158]: pgmap v3900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6975.008734 secs 2022-03-21T16:07:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6975.008734 secs 2022-03-21T16:07:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6975.008734 secs 2022-03-21T16:07:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:54 smithi002 ceph-mon[17167]: pgmap v3901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:54.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:54 smithi016 ceph-mon[22010]: pgmap v3901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:54 smithi079 ceph-mon[23158]: pgmap v3901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:56 smithi002 ceph-mon[17167]: pgmap v3902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:56 smithi016 ceph-mon[22010]: pgmap v3902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:56 smithi079 ceph-mon[23158]: pgmap v3902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:07:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:58 smithi002 ceph-mon[17167]: pgmap v3903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:07:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6980.008867 secs 2022-03-21T16:07:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:58 smithi016 ceph-mon[22010]: pgmap v3903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:07:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6980.008867 secs 2022-03-21T16:07:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:58 smithi079 ceph-mon[23158]: pgmap v3903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:07:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:07:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6980.008867 secs 2022-03-21T16:08:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:00 smithi002 ceph-mon[17167]: pgmap v3904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:00 smithi016 ceph-mon[22010]: pgmap v3904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:00 smithi079 ceph-mon[23158]: pgmap v3904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:02 smithi002 ceph-mon[17167]: pgmap v3905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:02 smithi016 ceph-mon[22010]: pgmap v3905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:02 smithi079 ceph-mon[23158]: pgmap v3905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6985.009015 secs 2022-03-21T16:08:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6985.009015 secs 2022-03-21T16:08:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6985.009015 secs 2022-03-21T16:08:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:04 smithi016 ceph-mon[22010]: pgmap v3906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:04 smithi079 ceph-mon[23158]: pgmap v3906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:04 smithi002 ceph-mon[17167]: pgmap v3906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:06.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:06 smithi079 ceph-mon[23158]: pgmap v3907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:06.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:08:06.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:08:06.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:08:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:06 smithi016 ceph-mon[22010]: pgmap v3907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:08:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:08:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:08:07.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:06 smithi002 ceph-mon[17167]: pgmap v3907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:07.061 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:08:07.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:08:07.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:08:08.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:08 smithi002 ceph-mon[17167]: pgmap v3908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:08.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6990.009155 secs 2022-03-21T16:08:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:08 smithi016 ceph-mon[22010]: pgmap v3908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6990.009155 secs 2022-03-21T16:08:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:08 smithi079 ceph-mon[23158]: pgmap v3908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6990.009155 secs 2022-03-21T16:08:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:10 smithi016 ceph-mon[22010]: pgmap v3909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:10 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T16:08:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:10 smithi079 ceph-mon[23158]: pgmap v3909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:10 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T16:08:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:10.728 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:10 smithi002 ceph-mon[17167]: pgmap v3909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:10.728 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:10 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T16:08:10.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:11 smithi079 ceph-mon[23158]: pgmap v3910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:11 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T16:08:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:11 smithi002 ceph-mon[17167]: pgmap v3910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:12.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:11 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T16:08:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:11 smithi016 ceph-mon[22010]: pgmap v3910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:11 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T16:08:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:08:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 6995.009257 secs 2022-03-21T16:08:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:13 smithi079 ceph-mon[23158]: pgmap v3911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:14.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 6995.009257 secs 2022-03-21T16:08:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:13 smithi002 ceph-mon[17167]: pgmap v3911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 6995.009257 secs 2022-03-21T16:08:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:13 smithi016 ceph-mon[22010]: pgmap v3911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:15 smithi079 ceph-mon[23158]: pgmap v3912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:15 smithi002 ceph-mon[17167]: pgmap v3912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:15 smithi016 ceph-mon[22010]: pgmap v3912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:18 smithi016 ceph-mon[22010]: pgmap v3913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:18 smithi079 ceph-mon[23158]: pgmap v3913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:18 smithi002 ceph-mon[17167]: pgmap v3913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:19 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7000.009388 secs 2022-03-21T16:08:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:19 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7000.009388 secs 2022-03-21T16:08:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:19 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7000.009388 secs 2022-03-21T16:08:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:20 smithi016 ceph-mon[22010]: pgmap v3914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:20 smithi079 ceph-mon[23158]: pgmap v3914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:20 smithi002 ceph-mon[17167]: pgmap v3914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:08:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:08:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:08:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:08:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:08:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:22 smithi016 ceph-mon[22010]: pgmap v3915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:22 smithi079 ceph-mon[23158]: pgmap v3915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:22 smithi002 ceph-mon[17167]: pgmap v3915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7005.009505 secs 2022-03-21T16:08:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:24 smithi016 ceph-mon[22010]: pgmap v3916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7005.009505 secs 2022-03-21T16:08:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:24 smithi079 ceph-mon[23158]: pgmap v3916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:24 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7005.009505 secs 2022-03-21T16:08:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:24 smithi002 ceph-mon[17167]: pgmap v3916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:26 smithi016 ceph-mon[22010]: pgmap v3917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:26 smithi079 ceph-mon[23158]: pgmap v3917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:26.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:26 smithi002 ceph-mon[17167]: pgmap v3917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:28 smithi016 ceph-mon[22010]: pgmap v3918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:28 smithi079 ceph-mon[23158]: pgmap v3918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:28 smithi002 ceph-mon[17167]: pgmap v3918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:29 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7010.009645 secs 2022-03-21T16:08:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:29 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7010.009645 secs 2022-03-21T16:08:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:29 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7010.009645 secs 2022-03-21T16:08:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:30 smithi079 ceph-mon[23158]: pgmap v3919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:30 smithi002 ceph-mon[17167]: pgmap v3919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:30 smithi016 ceph-mon[22010]: pgmap v3919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:32 smithi079 ceph-mon[23158]: pgmap v3920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:32 smithi002 ceph-mon[17167]: pgmap v3920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:32.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:32 smithi016 ceph-mon[22010]: pgmap v3920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:34 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7015.009788 secs 2022-03-21T16:08:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:34 smithi079 ceph-mon[23158]: pgmap v3921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:34 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7015.009788 secs 2022-03-21T16:08:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:34 smithi002 ceph-mon[17167]: pgmap v3921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:34 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7015.009788 secs 2022-03-21T16:08:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:34 smithi016 ceph-mon[22010]: pgmap v3921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:36 smithi079 ceph-mon[23158]: pgmap v3922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:36 smithi002 ceph-mon[17167]: pgmap v3922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:36 smithi016 ceph-mon[22010]: pgmap v3922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:38 smithi079 ceph-mon[23158]: pgmap v3923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:38 smithi002 ceph-mon[17167]: pgmap v3923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:38 smithi016 ceph-mon[22010]: pgmap v3923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:39 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7020.009935 secs 2022-03-21T16:08:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:39 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7020.009935 secs 2022-03-21T16:08:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:39 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7020.009935 secs 2022-03-21T16:08:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:40 smithi079 ceph-mon[23158]: pgmap v3924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:40 smithi002 ceph-mon[17167]: pgmap v3924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:40 smithi016 ceph-mon[22010]: pgmap v3924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:42 smithi079 ceph-mon[23158]: pgmap v3925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:42 smithi002 ceph-mon[17167]: pgmap v3925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:42 smithi016 ceph-mon[22010]: pgmap v3925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:44 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7025.010071 secs 2022-03-21T16:08:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:44 smithi079 ceph-mon[23158]: pgmap v3926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:44 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7025.010071 secs 2022-03-21T16:08:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:44 smithi002 ceph-mon[17167]: pgmap v3926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:44 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7025.010071 secs 2022-03-21T16:08:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:44 smithi016 ceph-mon[22010]: pgmap v3926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:46 smithi079 ceph-mon[23158]: pgmap v3927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:46 smithi002 ceph-mon[17167]: pgmap v3927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:46.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:46 smithi016 ceph-mon[22010]: pgmap v3927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:48 smithi079 ceph-mon[23158]: pgmap v3928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:48 smithi002 ceph-mon[17167]: pgmap v3928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:48 smithi016 ceph-mon[22010]: pgmap v3928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:49 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7030.010209 secs 2022-03-21T16:08:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:49 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7030.010209 secs 2022-03-21T16:08:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:49 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7030.010209 secs 2022-03-21T16:08:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:50 smithi079 ceph-mon[23158]: pgmap v3929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:50 smithi002 ceph-mon[17167]: pgmap v3929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:50 smithi016 ceph-mon[22010]: pgmap v3929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:52 smithi079 ceph-mon[23158]: pgmap v3930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:52 smithi002 ceph-mon[17167]: pgmap v3930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:52 smithi016 ceph-mon[22010]: pgmap v3930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:54 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7035.010335 secs 2022-03-21T16:08:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:54 smithi079 ceph-mon[23158]: pgmap v3931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:54 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7035.010335 secs 2022-03-21T16:08:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:54 smithi002 ceph-mon[17167]: pgmap v3931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:54 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7035.010335 secs 2022-03-21T16:08:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:54 smithi016 ceph-mon[22010]: pgmap v3931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:56 smithi002 ceph-mon[17167]: pgmap v3932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:56 smithi016 ceph-mon[22010]: pgmap v3932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:56 smithi079 ceph-mon[23158]: pgmap v3932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:08:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:58 smithi002 ceph-mon[17167]: pgmap v3933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:58 smithi016 ceph-mon[22010]: pgmap v3933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:58 smithi079 ceph-mon[23158]: pgmap v3933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:08:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:08:59 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7040.010479 secs 2022-03-21T16:08:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:08:59 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7040.010479 secs 2022-03-21T16:08:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:08:59 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7040.010479 secs 2022-03-21T16:09:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:00 smithi002 ceph-mon[17167]: pgmap v3934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:00.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:00 smithi016 ceph-mon[22010]: pgmap v3934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:00 smithi079 ceph-mon[23158]: pgmap v3934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:02 smithi002 ceph-mon[17167]: pgmap v3935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:02 smithi016 ceph-mon[22010]: pgmap v3935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:02 smithi079 ceph-mon[23158]: pgmap v3935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:04 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7045.010616 secs 2022-03-21T16:09:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:04 smithi002 ceph-mon[17167]: pgmap v3936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:04 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7045.010616 secs 2022-03-21T16:09:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:04 smithi016 ceph-mon[22010]: pgmap v3936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:04 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7045.010616 secs 2022-03-21T16:09:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:04 smithi079 ceph-mon[23158]: pgmap v3936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:06 smithi002 ceph-mon[17167]: pgmap v3937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:06 smithi016 ceph-mon[22010]: pgmap v3937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:06 smithi079 ceph-mon[23158]: pgmap v3937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:08 smithi002 ceph-mon[17167]: pgmap v3938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:08 smithi016 ceph-mon[22010]: pgmap v3938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:08 smithi079 ceph-mon[23158]: pgmap v3938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:09 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7050.010754 secs 2022-03-21T16:09:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:09 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7050.010754 secs 2022-03-21T16:09:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:09 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7050.010754 secs 2022-03-21T16:09:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:10 smithi002 ceph-mon[17167]: pgmap v3939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:10 smithi016 ceph-mon[22010]: pgmap v3939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:10 smithi079 ceph-mon[23158]: pgmap v3939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:09:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:09:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:09:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:09:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:09:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:09:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:09:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:09:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:09:12.415 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: pgmap v3940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:12.416 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.416 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.416 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.417 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.417 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:09:12.417 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:09:12.417 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: pgmap v3940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:12.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:09:12.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:09:12.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: pgmap v3940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:09:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:09:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:13 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:09:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:13 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:09:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:13 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:09:13.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:13 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:09:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:13 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:09:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:13 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:09:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7055.010817 secs 2022-03-21T16:09:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: pgmap v3941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:14.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:14.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7055.010817 secs 2022-03-21T16:09:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: pgmap v3941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7055.010817 secs 2022-03-21T16:09:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: pgmap v3941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:09:14.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:09:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:15 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T16:09:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:15 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:09:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:15 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:09:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:15 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T16:09:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:15 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:09:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:15 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:09:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:15 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T16:09:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:15 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:09:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:15 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:09:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:16 smithi002 ceph-mon[17167]: pgmap v3942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:16 smithi016 ceph-mon[22010]: pgmap v3942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:16 smithi079 ceph-mon[23158]: pgmap v3942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:18 smithi002 ceph-mon[17167]: pgmap v3943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7060.010964 secs 2022-03-21T16:09:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:18 smithi016 ceph-mon[22010]: pgmap v3943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7060.010964 secs 2022-03-21T16:09:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:18 smithi079 ceph-mon[23158]: pgmap v3943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7060.010964 secs 2022-03-21T16:09:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:20 smithi002 ceph-mon[17167]: pgmap v3944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:20 smithi016 ceph-mon[22010]: pgmap v3944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:20 smithi079 ceph-mon[23158]: pgmap v3944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:09:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:09:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:09:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:09:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:09:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:09:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:22 smithi002 ceph-mon[17167]: pgmap v3945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:22 smithi016 ceph-mon[22010]: pgmap v3945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:22 smithi079 ceph-mon[23158]: pgmap v3945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7065.011097 secs 2022-03-21T16:09:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7065.011097 secs 2022-03-21T16:09:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7065.011097 secs 2022-03-21T16:09:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:24 smithi002 ceph-mon[17167]: pgmap v3946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:24 smithi016 ceph-mon[22010]: pgmap v3946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:24 smithi079 ceph-mon[23158]: pgmap v3946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:26.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:26 smithi002 ceph-mon[17167]: pgmap v3947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:26 smithi016 ceph-mon[22010]: pgmap v3947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:26 smithi079 ceph-mon[23158]: pgmap v3947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:28 smithi002 ceph-mon[17167]: pgmap v3948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7070.011230 secs 2022-03-21T16:09:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:28 smithi016 ceph-mon[22010]: pgmap v3948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7070.011230 secs 2022-03-21T16:09:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:28 smithi079 ceph-mon[23158]: pgmap v3948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7070.011230 secs 2022-03-21T16:09:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:30 smithi002 ceph-mon[17167]: pgmap v3949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:30 smithi016 ceph-mon[22010]: pgmap v3949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:30 smithi079 ceph-mon[23158]: pgmap v3949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:32 smithi002 ceph-mon[17167]: pgmap v3950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:32 smithi016 ceph-mon[22010]: pgmap v3950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:32 smithi079 ceph-mon[23158]: pgmap v3950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:33.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7075.011387 secs 2022-03-21T16:09:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7075.011387 secs 2022-03-21T16:09:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7075.011387 secs 2022-03-21T16:09:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:34 smithi002 ceph-mon[17167]: pgmap v3951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:34 smithi016 ceph-mon[22010]: pgmap v3951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:34 smithi079 ceph-mon[23158]: pgmap v3951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:36 smithi002 ceph-mon[17167]: pgmap v3952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:36 smithi016 ceph-mon[22010]: pgmap v3952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:36 smithi079 ceph-mon[23158]: pgmap v3952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:38 smithi002 ceph-mon[17167]: pgmap v3953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7080.011459 secs 2022-03-21T16:09:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:38 smithi016 ceph-mon[22010]: pgmap v3953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7080.011459 secs 2022-03-21T16:09:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:38 smithi079 ceph-mon[23158]: pgmap v3953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7080.011459 secs 2022-03-21T16:09:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:40 smithi002 ceph-mon[17167]: pgmap v3954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:40 smithi016 ceph-mon[22010]: pgmap v3954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:40 smithi079 ceph-mon[23158]: pgmap v3954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:42 smithi002 ceph-mon[17167]: pgmap v3955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:42.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:42 smithi016 ceph-mon[22010]: pgmap v3955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:42 smithi079 ceph-mon[23158]: pgmap v3955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7085.011617 secs 2022-03-21T16:09:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7085.011617 secs 2022-03-21T16:09:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7085.011617 secs 2022-03-21T16:09:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:44 smithi002 ceph-mon[17167]: pgmap v3956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:44 smithi016 ceph-mon[22010]: pgmap v3956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:44 smithi079 ceph-mon[23158]: pgmap v3956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:46 smithi002 ceph-mon[17167]: pgmap v3957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:46 smithi016 ceph-mon[22010]: pgmap v3957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:46 smithi079 ceph-mon[23158]: pgmap v3957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:48 smithi002 ceph-mon[17167]: pgmap v3958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7090.011725 secs 2022-03-21T16:09:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:48 smithi016 ceph-mon[22010]: pgmap v3958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7090.011725 secs 2022-03-21T16:09:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:48 smithi079 ceph-mon[23158]: pgmap v3958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7090.011725 secs 2022-03-21T16:09:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:50 smithi002 ceph-mon[17167]: pgmap v3959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:50.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:50 smithi016 ceph-mon[22010]: pgmap v3959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:50 smithi079 ceph-mon[23158]: pgmap v3959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:52 smithi002 ceph-mon[17167]: pgmap v3960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:52 smithi016 ceph-mon[22010]: pgmap v3960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:52 smithi079 ceph-mon[23158]: pgmap v3960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7095.011838 secs 2022-03-21T16:09:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7095.011838 secs 2022-03-21T16:09:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7095.011838 secs 2022-03-21T16:09:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:54 smithi016 ceph-mon[22010]: pgmap v3961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:54 smithi079 ceph-mon[23158]: pgmap v3961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:54 smithi002 ceph-mon[17167]: pgmap v3961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:56 smithi016 ceph-mon[22010]: pgmap v3962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:56 smithi079 ceph-mon[23158]: pgmap v3962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:56 smithi002 ceph-mon[17167]: pgmap v3962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:09:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:58 smithi016 ceph-mon[22010]: pgmap v3963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:09:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7100.011959 secs 2022-03-21T16:09:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:58 smithi079 ceph-mon[23158]: pgmap v3963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:09:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7100.011959 secs 2022-03-21T16:09:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:58 smithi002 ceph-mon[17167]: pgmap v3963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:09:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:09:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7100.011959 secs 2022-03-21T16:10:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:00 smithi016 ceph-mon[22010]: pgmap v3964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:00 smithi079 ceph-mon[23158]: pgmap v3964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:10:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:00 smithi002 ceph-mon[17167]: pgmap v3964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:00.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:10:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:02 smithi016 ceph-mon[22010]: pgmap v3965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:02 smithi079 ceph-mon[23158]: pgmap v3965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:02 smithi002 ceph-mon[17167]: pgmap v3965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7105.012062 secs 2022-03-21T16:10:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7105.012062 secs 2022-03-21T16:10:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7105.012062 secs 2022-03-21T16:10:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:04 smithi016 ceph-mon[22010]: pgmap v3966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:04 smithi079 ceph-mon[23158]: pgmap v3966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:04 smithi002 ceph-mon[17167]: pgmap v3966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:06 smithi016 ceph-mon[22010]: pgmap v3967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:06 smithi079 ceph-mon[23158]: pgmap v3967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:06 smithi002 ceph-mon[17167]: pgmap v3967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:08 smithi016 ceph-mon[22010]: pgmap v3968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7110.012190 secs 2022-03-21T16:10:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:08 smithi079 ceph-mon[23158]: pgmap v3968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7110.012190 secs 2022-03-21T16:10:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:08 smithi002 ceph-mon[17167]: pgmap v3968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7110.012190 secs 2022-03-21T16:10:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:10 smithi016 ceph-mon[22010]: pgmap v3969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:10 smithi079 ceph-mon[23158]: pgmap v3969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:10 smithi002 ceph-mon[17167]: pgmap v3969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:12 smithi016 ceph-mon[22010]: pgmap v3970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:12 smithi079 ceph-mon[23158]: pgmap v3970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:12 smithi002 ceph-mon[17167]: pgmap v3970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7115.012284 secs 2022-03-21T16:10:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7115.012284 secs 2022-03-21T16:10:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7115.012284 secs 2022-03-21T16:10:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:14 smithi079 ceph-mon[23158]: pgmap v3971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:10:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:10:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:10:14.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:14 smithi002 ceph-mon[17167]: pgmap v3971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:14.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:10:14.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:10:14.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:10:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:14 smithi016 ceph-mon[22010]: pgmap v3971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:10:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:10:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:10:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:15 smithi016 ceph-mon[22010]: pgmap v3972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:15 smithi079 ceph-mon[23158]: pgmap v3972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:10:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:15 smithi002 ceph-mon[17167]: pgmap v3972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:10:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:18 smithi079 ceph-mon[23158]: pgmap v3973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:18 smithi002 ceph-mon[17167]: pgmap v3973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:18 smithi016 ceph-mon[22010]: pgmap v3973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:19 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7120.012428 secs 2022-03-21T16:10:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:19 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7120.012428 secs 2022-03-21T16:10:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:19 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7120.012428 secs 2022-03-21T16:10:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:20 smithi079 ceph-mon[23158]: pgmap v3974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:20 smithi002 ceph-mon[17167]: pgmap v3974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:20 smithi016 ceph-mon[22010]: pgmap v3974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:10:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:10:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:10:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:10:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:10:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:10:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:22 smithi079 ceph-mon[23158]: pgmap v3975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:22 smithi002 ceph-mon[17167]: pgmap v3975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:22 smithi016 ceph-mon[22010]: pgmap v3975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7125.012576 secs 2022-03-21T16:10:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:24 smithi079 ceph-mon[23158]: pgmap v3976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:24 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7125.012576 secs 2022-03-21T16:10:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:24 smithi002 ceph-mon[17167]: pgmap v3976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7125.012576 secs 2022-03-21T16:10:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:24 smithi016 ceph-mon[22010]: pgmap v3976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:26 smithi079 ceph-mon[23158]: pgmap v3977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:26.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:26 smithi002 ceph-mon[17167]: pgmap v3977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:26 smithi016 ceph-mon[22010]: pgmap v3977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:10:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:28 smithi079 ceph-mon[23158]: pgmap v3978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:28 smithi002 ceph-mon[17167]: pgmap v3978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:28 smithi016 ceph-mon[22010]: pgmap v3978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:29 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7130.012638 secs 2022-03-21T16:10:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:29 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7130.012638 secs 2022-03-21T16:10:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:29 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7130.012638 secs 2022-03-21T16:10:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:30 smithi079 ceph-mon[23158]: pgmap v3979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:30 smithi002 ceph-mon[17167]: pgmap v3979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:30 smithi016 ceph-mon[22010]: pgmap v3979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:32 smithi079 ceph-mon[23158]: pgmap v3980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:32 smithi002 ceph-mon[17167]: pgmap v3980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:32 smithi016 ceph-mon[22010]: pgmap v3980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:34 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7135.012750 secs 2022-03-21T16:10:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:34 smithi079 ceph-mon[23158]: pgmap v3981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:34 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7135.012750 secs 2022-03-21T16:10:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:34 smithi002 ceph-mon[17167]: pgmap v3981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:34 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7135.012750 secs 2022-03-21T16:10:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:34 smithi016 ceph-mon[22010]: pgmap v3981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:36 smithi002 ceph-mon[17167]: pgmap v3982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:36 smithi016 ceph-mon[22010]: pgmap v3982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:36 smithi079 ceph-mon[23158]: pgmap v3982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:38 smithi002 ceph-mon[17167]: pgmap v3983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:38 smithi016 ceph-mon[22010]: pgmap v3983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:38 smithi079 ceph-mon[23158]: pgmap v3983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:39 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7140.012883 secs 2022-03-21T16:10:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:39 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7140.012883 secs 2022-03-21T16:10:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:39 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7140.012883 secs 2022-03-21T16:10:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:40 smithi002 ceph-mon[17167]: pgmap v3984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:40 smithi016 ceph-mon[22010]: pgmap v3984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:40 smithi079 ceph-mon[23158]: pgmap v3984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:42 smithi002 ceph-mon[17167]: pgmap v3985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:42 smithi016 ceph-mon[22010]: pgmap v3985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:42 smithi079 ceph-mon[23158]: pgmap v3985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:44 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7145.012947 secs 2022-03-21T16:10:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:44 smithi002 ceph-mon[17167]: pgmap v3986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:44 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7145.012947 secs 2022-03-21T16:10:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:44 smithi016 ceph-mon[22010]: pgmap v3986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:44 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7145.012947 secs 2022-03-21T16:10:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:44 smithi079 ceph-mon[23158]: pgmap v3986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:46 smithi002 ceph-mon[17167]: pgmap v3987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:46 smithi016 ceph-mon[22010]: pgmap v3987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:46 smithi079 ceph-mon[23158]: pgmap v3987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:48 smithi002 ceph-mon[17167]: pgmap v3988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:48 smithi016 ceph-mon[22010]: pgmap v3988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:48 smithi079 ceph-mon[23158]: pgmap v3988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:49 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7150.013012 secs 2022-03-21T16:10:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:49 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7150.013012 secs 2022-03-21T16:10:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:49 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7150.013012 secs 2022-03-21T16:10:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:50 smithi002 ceph-mon[17167]: pgmap v3989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:50 smithi016 ceph-mon[22010]: pgmap v3989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:50 smithi079 ceph-mon[23158]: pgmap v3989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:52 smithi002 ceph-mon[17167]: pgmap v3990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:52 smithi016 ceph-mon[22010]: pgmap v3990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:52 smithi079 ceph-mon[23158]: pgmap v3990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:54 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7155.013113 secs 2022-03-21T16:10:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:54 smithi002 ceph-mon[17167]: pgmap v3991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:54 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7155.013113 secs 2022-03-21T16:10:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:54 smithi016 ceph-mon[22010]: pgmap v3991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:54 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7155.013113 secs 2022-03-21T16:10:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:54 smithi079 ceph-mon[23158]: pgmap v3991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:56 smithi002 ceph-mon[17167]: pgmap v3992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:56 smithi016 ceph-mon[22010]: pgmap v3992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:56 smithi079 ceph-mon[23158]: pgmap v3992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:10:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:58 smithi002 ceph-mon[17167]: pgmap v3993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:58 smithi016 ceph-mon[22010]: pgmap v3993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:58 smithi079 ceph-mon[23158]: pgmap v3993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:10:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:10:59 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7160.013260 secs 2022-03-21T16:10:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:10:59 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7160.013260 secs 2022-03-21T16:10:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:10:59 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7160.013260 secs 2022-03-21T16:11:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:00 smithi002 ceph-mon[17167]: pgmap v3994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:00 smithi016 ceph-mon[22010]: pgmap v3994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:00 smithi079 ceph-mon[23158]: pgmap v3994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:02 smithi002 ceph-mon[17167]: pgmap v3995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:02 smithi016 ceph-mon[22010]: pgmap v3995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:02 smithi079 ceph-mon[23158]: pgmap v3995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:04 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7165.013362 secs 2022-03-21T16:11:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:04 smithi002 ceph-mon[17167]: pgmap v3996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:04 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7165.013362 secs 2022-03-21T16:11:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:04 smithi016 ceph-mon[22010]: pgmap v3996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:04 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7165.013362 secs 2022-03-21T16:11:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:04 smithi079 ceph-mon[23158]: pgmap v3996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:06 smithi002 ceph-mon[17167]: pgmap v3997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:06 smithi016 ceph-mon[22010]: pgmap v3997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:06 smithi079 ceph-mon[23158]: pgmap v3997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:08 smithi002 ceph-mon[17167]: pgmap v3998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7170.013443 secs 2022-03-21T16:11:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:08 smithi016 ceph-mon[22010]: pgmap v3998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7170.013443 secs 2022-03-21T16:11:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:08 smithi079 ceph-mon[23158]: pgmap v3998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7170.013443 secs 2022-03-21T16:11:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:10 smithi002 ceph-mon[17167]: pgmap v3999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:10 smithi016 ceph-mon[22010]: pgmap v3999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:10 smithi079 ceph-mon[23158]: pgmap v3999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:12 smithi002 ceph-mon[17167]: pgmap v4000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:12 smithi016 ceph-mon[22010]: pgmap v4000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:12 smithi079 ceph-mon[23158]: pgmap v4000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7175.013583 secs 2022-03-21T16:11:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7175.013583 secs 2022-03-21T16:11:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7175.013583 secs 2022-03-21T16:11:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:14 smithi002 ceph-mon[17167]: pgmap v4001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:14 smithi016 ceph-mon[22010]: pgmap v4001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:14 smithi079 ceph-mon[23158]: pgmap v4001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:11:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:11:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:11:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:11:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:11:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:11:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:11:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:11:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:11:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:16 smithi002 ceph-mon[17167]: pgmap v4002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:17.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:11:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:16 smithi016 ceph-mon[22010]: pgmap v4002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:11:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:16 smithi079 ceph-mon[23158]: pgmap v4002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:11:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:18 smithi002 ceph-mon[17167]: pgmap v4003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7180.013704 secs 2022-03-21T16:11:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:18 smithi016 ceph-mon[22010]: pgmap v4003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7180.013704 secs 2022-03-21T16:11:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:18 smithi079 ceph-mon[23158]: pgmap v4003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7180.013704 secs 2022-03-21T16:11:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:20 smithi002 ceph-mon[17167]: pgmap v4004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:20 smithi016 ceph-mon[22010]: pgmap v4004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:20 smithi079 ceph-mon[23158]: pgmap v4004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:11:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:11:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:11:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:11:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:11:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:11:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:22 smithi002 ceph-mon[17167]: pgmap v4005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:22 smithi016 ceph-mon[22010]: pgmap v4005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:22 smithi079 ceph-mon[23158]: pgmap v4005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7185.013822 secs 2022-03-21T16:11:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7185.013822 secs 2022-03-21T16:11:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7185.013822 secs 2022-03-21T16:11:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:24 smithi002 ceph-mon[17167]: pgmap v4006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:24 smithi016 ceph-mon[22010]: pgmap v4006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:24 smithi079 ceph-mon[23158]: pgmap v4006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:26 smithi002 ceph-mon[17167]: pgmap v4007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:26 smithi016 ceph-mon[22010]: pgmap v4007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:26 smithi079 ceph-mon[23158]: pgmap v4007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:28 smithi002 ceph-mon[17167]: pgmap v4008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7190.013957 secs 2022-03-21T16:11:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:28 smithi016 ceph-mon[22010]: pgmap v4008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7190.013957 secs 2022-03-21T16:11:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:28 smithi079 ceph-mon[23158]: pgmap v4008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7190.013957 secs 2022-03-21T16:11:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:30 smithi002 ceph-mon[17167]: pgmap v4009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:30 smithi016 ceph-mon[22010]: pgmap v4009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:30 smithi079 ceph-mon[23158]: pgmap v4009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:32 smithi002 ceph-mon[17167]: pgmap v4010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:32 smithi016 ceph-mon[22010]: pgmap v4010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:32 smithi079 ceph-mon[23158]: pgmap v4010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7195.014092 secs 2022-03-21T16:11:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7195.014092 secs 2022-03-21T16:11:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7195.014092 secs 2022-03-21T16:11:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:34 smithi002 ceph-mon[17167]: pgmap v4011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:34 smithi016 ceph-mon[22010]: pgmap v4011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:34 smithi079 ceph-mon[23158]: pgmap v4011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:36 smithi002 ceph-mon[17167]: pgmap v4012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:36 smithi016 ceph-mon[22010]: pgmap v4012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:36 smithi079 ceph-mon[23158]: pgmap v4012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:37 smithi002 ceph-mon[17167]: pgmap v4013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:37 smithi016 ceph-mon[22010]: pgmap v4013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:37 smithi079 ceph-mon[23158]: pgmap v4013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7200.014205 secs 2022-03-21T16:11:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7200.014205 secs 2022-03-21T16:11:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7200.014205 secs 2022-03-21T16:11:40.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:39 smithi002 ceph-mon[17167]: pgmap v4014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:39 smithi016 ceph-mon[22010]: pgmap v4014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:39 smithi079 ceph-mon[23158]: pgmap v4014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:42 smithi079 ceph-mon[23158]: pgmap v4015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:42 smithi002 ceph-mon[17167]: pgmap v4015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:42 smithi016 ceph-mon[22010]: pgmap v4015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:44 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7205.014342 secs 2022-03-21T16:11:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:44 smithi079 ceph-mon[23158]: pgmap v4016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:44 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7205.014342 secs 2022-03-21T16:11:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:44 smithi002 ceph-mon[17167]: pgmap v4016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:44 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7205.014342 secs 2022-03-21T16:11:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:44 smithi016 ceph-mon[22010]: pgmap v4016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:46 smithi079 ceph-mon[23158]: pgmap v4017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:46 smithi002 ceph-mon[17167]: pgmap v4017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:46 smithi016 ceph-mon[22010]: pgmap v4017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:48 smithi079 ceph-mon[23158]: pgmap v4018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:48 smithi002 ceph-mon[17167]: pgmap v4018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:48 smithi016 ceph-mon[22010]: pgmap v4018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:49 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7210.014459 secs 2022-03-21T16:11:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:49 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7210.014459 secs 2022-03-21T16:11:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:49 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7210.014459 secs 2022-03-21T16:11:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:50 smithi002 ceph-mon[17167]: pgmap v4019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:50 smithi016 ceph-mon[22010]: pgmap v4019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:50 smithi079 ceph-mon[23158]: pgmap v4019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:52 smithi002 ceph-mon[17167]: pgmap v4020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:52 smithi016 ceph-mon[22010]: pgmap v4020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:52 smithi079 ceph-mon[23158]: pgmap v4020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:54 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7215.014581 secs 2022-03-21T16:11:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:54 smithi002 ceph-mon[17167]: pgmap v4021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:54 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7215.014581 secs 2022-03-21T16:11:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:54 smithi016 ceph-mon[22010]: pgmap v4021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:54 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7215.014581 secs 2022-03-21T16:11:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:54 smithi079 ceph-mon[23158]: pgmap v4021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:56 smithi002 ceph-mon[17167]: pgmap v4022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:56 smithi016 ceph-mon[22010]: pgmap v4022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:56 smithi079 ceph-mon[23158]: pgmap v4022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:11:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:58 smithi002 ceph-mon[17167]: pgmap v4023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:58 smithi016 ceph-mon[22010]: pgmap v4023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:58 smithi079 ceph-mon[23158]: pgmap v4023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:11:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:11:59 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7220.014658 secs 2022-03-21T16:11:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:11:59 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7220.014658 secs 2022-03-21T16:11:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:11:59 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7220.014658 secs 2022-03-21T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:00 smithi002 ceph-mon[17167]: pgmap v4024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:00 smithi016 ceph-mon[22010]: pgmap v4024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:00 smithi079 ceph-mon[23158]: pgmap v4024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:02 smithi002 ceph-mon[17167]: pgmap v4025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:02 smithi016 ceph-mon[22010]: pgmap v4025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:02 smithi079 ceph-mon[23158]: pgmap v4025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:04.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:04 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7225.014738 secs 2022-03-21T16:12:04.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:04 smithi002 ceph-mon[17167]: pgmap v4026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:04 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7225.014738 secs 2022-03-21T16:12:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:04 smithi016 ceph-mon[22010]: pgmap v4026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:04 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7225.014738 secs 2022-03-21T16:12:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:04 smithi079 ceph-mon[23158]: pgmap v4026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:06 smithi002 ceph-mon[17167]: pgmap v4027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:06 smithi016 ceph-mon[22010]: pgmap v4027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:06 smithi079 ceph-mon[23158]: pgmap v4027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:08 smithi002 ceph-mon[17167]: pgmap v4028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:08 smithi016 ceph-mon[22010]: pgmap v4028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:08 smithi079 ceph-mon[23158]: pgmap v4028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:09 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7230.014822 secs 2022-03-21T16:12:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:09 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7230.014822 secs 2022-03-21T16:12:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:09 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7230.014822 secs 2022-03-21T16:12:10.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:10 smithi002 ceph-mon[17167]: pgmap v4029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:10 smithi016 ceph-mon[22010]: pgmap v4029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:10 smithi079 ceph-mon[23158]: pgmap v4029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:12 smithi002 ceph-mon[17167]: pgmap v4030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:12 smithi016 ceph-mon[22010]: pgmap v4030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:12 smithi079 ceph-mon[23158]: pgmap v4030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:14 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7235.014906 secs 2022-03-21T16:12:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:14 smithi002 ceph-mon[17167]: pgmap v4031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:14 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7235.014906 secs 2022-03-21T16:12:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:14 smithi016 ceph-mon[22010]: pgmap v4031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:14 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7235.014906 secs 2022-03-21T16:12:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:14 smithi079 ceph-mon[23158]: pgmap v4031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:16 smithi002 ceph-mon[17167]: pgmap v4032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:12:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:12:16.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:16 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:12:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:16 smithi016 ceph-mon[22010]: pgmap v4032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:12:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:12:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:16 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:12:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:16 smithi079 ceph-mon[23158]: pgmap v4032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:12:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:12:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:16 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:12:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:18 smithi002 ceph-mon[17167]: pgmap v4033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:18 smithi016 ceph-mon[22010]: pgmap v4033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:18 smithi079 ceph-mon[23158]: pgmap v4033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:19.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:19 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7240.015025 secs 2022-03-21T16:12:19.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:19.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:19 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7240.015025 secs 2022-03-21T16:12:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:19 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7240.015025 secs 2022-03-21T16:12:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:20 smithi002 ceph-mon[17167]: pgmap v4034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:20 smithi016 ceph-mon[22010]: pgmap v4034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:20 smithi079 ceph-mon[23158]: pgmap v4034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:12:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:12:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:12:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:12:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:12:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:12:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:12:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:22 smithi002 ceph-mon[17167]: pgmap v4035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:22 smithi016 ceph-mon[22010]: pgmap v4035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:22 smithi079 ceph-mon[23158]: pgmap v4035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7245.015144 secs 2022-03-21T16:12:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7245.015144 secs 2022-03-21T16:12:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7245.015144 secs 2022-03-21T16:12:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:24 smithi002 ceph-mon[17167]: pgmap v4036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:24 smithi016 ceph-mon[22010]: pgmap v4036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:24 smithi079 ceph-mon[23158]: pgmap v4036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:26 smithi002 ceph-mon[17167]: pgmap v4037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:26 smithi016 ceph-mon[22010]: pgmap v4037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:26 smithi079 ceph-mon[23158]: pgmap v4037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:28 smithi002 ceph-mon[17167]: pgmap v4038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7250.015253 secs 2022-03-21T16:12:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:28 smithi016 ceph-mon[22010]: pgmap v4038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7250.015253 secs 2022-03-21T16:12:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:28 smithi079 ceph-mon[23158]: pgmap v4038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7250.015253 secs 2022-03-21T16:12:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:30 smithi002 ceph-mon[17167]: pgmap v4039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:30 smithi016 ceph-mon[22010]: pgmap v4039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:30 smithi079 ceph-mon[23158]: pgmap v4039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:32 smithi002 ceph-mon[17167]: pgmap v4040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:32 smithi016 ceph-mon[22010]: pgmap v4040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:32 smithi079 ceph-mon[23158]: pgmap v4040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7255.015364 secs 2022-03-21T16:12:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7255.015364 secs 2022-03-21T16:12:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7255.015364 secs 2022-03-21T16:12:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:34 smithi002 ceph-mon[17167]: pgmap v4041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:34 smithi016 ceph-mon[22010]: pgmap v4041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:34 smithi079 ceph-mon[23158]: pgmap v4041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:36 smithi002 ceph-mon[17167]: pgmap v4042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:36 smithi016 ceph-mon[22010]: pgmap v4042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:36 smithi079 ceph-mon[23158]: pgmap v4042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:38 smithi002 ceph-mon[17167]: pgmap v4043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7260.015472 secs 2022-03-21T16:12:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:38 smithi016 ceph-mon[22010]: pgmap v4043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7260.015472 secs 2022-03-21T16:12:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:38 smithi079 ceph-mon[23158]: pgmap v4043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7260.015472 secs 2022-03-21T16:12:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:40 smithi002 ceph-mon[17167]: pgmap v4044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:41.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:40 smithi079 ceph-mon[23158]: pgmap v4044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:41.324 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:40 smithi016 ceph-mon[22010]: pgmap v4044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:42 smithi002 ceph-mon[17167]: pgmap v4045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:42 smithi016 ceph-mon[22010]: pgmap v4045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:42 smithi079 ceph-mon[23158]: pgmap v4045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:43.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7265.015596 secs 2022-03-21T16:12:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7265.015596 secs 2022-03-21T16:12:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7265.015596 secs 2022-03-21T16:12:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:44 smithi002 ceph-mon[17167]: pgmap v4046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:44 smithi016 ceph-mon[22010]: pgmap v4046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:44 smithi079 ceph-mon[23158]: pgmap v4046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:46 smithi002 ceph-mon[17167]: pgmap v4047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:46 smithi016 ceph-mon[22010]: pgmap v4047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:46 smithi079 ceph-mon[23158]: pgmap v4047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:48 smithi002 ceph-mon[17167]: pgmap v4048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7270.015720 secs 2022-03-21T16:12:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:48 smithi016 ceph-mon[22010]: pgmap v4048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7270.015720 secs 2022-03-21T16:12:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:48 smithi079 ceph-mon[23158]: pgmap v4048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7270.015720 secs 2022-03-21T16:12:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:50 smithi002 ceph-mon[17167]: pgmap v4049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:50 smithi016 ceph-mon[22010]: pgmap v4049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:50 smithi079 ceph-mon[23158]: pgmap v4049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:52 smithi002 ceph-mon[17167]: pgmap v4050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:52 smithi016 ceph-mon[22010]: pgmap v4050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:52 smithi079 ceph-mon[23158]: pgmap v4050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:53.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7275.015839 secs 2022-03-21T16:12:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7275.015839 secs 2022-03-21T16:12:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7275.015839 secs 2022-03-21T16:12:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:54 smithi016 ceph-mon[22010]: pgmap v4051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:54 smithi079 ceph-mon[23158]: pgmap v4051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:54 smithi002 ceph-mon[17167]: pgmap v4051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:56 smithi016 ceph-mon[22010]: pgmap v4052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:56 smithi079 ceph-mon[23158]: pgmap v4052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:56 smithi002 ceph-mon[17167]: pgmap v4052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:12:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:58 smithi016 ceph-mon[22010]: pgmap v4053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:12:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7280.015959 secs 2022-03-21T16:12:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:58 smithi079 ceph-mon[23158]: pgmap v4053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:12:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7280.015959 secs 2022-03-21T16:12:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:58 smithi002 ceph-mon[17167]: pgmap v4053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:12:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:12:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7280.015959 secs 2022-03-21T16:13:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:00 smithi016 ceph-mon[22010]: pgmap v4054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:00 smithi079 ceph-mon[23158]: pgmap v4054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:00 smithi002 ceph-mon[17167]: pgmap v4054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:02 smithi016 ceph-mon[22010]: pgmap v4055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:02 smithi079 ceph-mon[23158]: pgmap v4055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:02 smithi002 ceph-mon[17167]: pgmap v4055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7285.016098 secs 2022-03-21T16:13:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7285.016098 secs 2022-03-21T16:13:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7285.016098 secs 2022-03-21T16:13:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:04 smithi016 ceph-mon[22010]: pgmap v4056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:04 smithi079 ceph-mon[23158]: pgmap v4056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:04 smithi002 ceph-mon[17167]: pgmap v4056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:06 smithi016 ceph-mon[22010]: pgmap v4057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:06 smithi079 ceph-mon[23158]: pgmap v4057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:06 smithi002 ceph-mon[17167]: pgmap v4057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:08 smithi016 ceph-mon[22010]: pgmap v4058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7290.016211 secs 2022-03-21T16:13:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:08 smithi079 ceph-mon[23158]: pgmap v4058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7290.016211 secs 2022-03-21T16:13:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:08 smithi002 ceph-mon[17167]: pgmap v4058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:09.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7290.016211 secs 2022-03-21T16:13:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:10 smithi016 ceph-mon[22010]: pgmap v4059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:10 smithi079 ceph-mon[23158]: pgmap v4059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:10 smithi002 ceph-mon[17167]: pgmap v4059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:12 smithi016 ceph-mon[22010]: pgmap v4060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:12 smithi079 ceph-mon[23158]: pgmap v4060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:12 smithi002 ceph-mon[17167]: pgmap v4060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7295.016324 secs 2022-03-21T16:13:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7295.016324 secs 2022-03-21T16:13:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7295.016324 secs 2022-03-21T16:13:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:14 smithi079 ceph-mon[23158]: pgmap v4061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:14 smithi002 ceph-mon[17167]: pgmap v4061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:14 smithi016 ceph-mon[22010]: pgmap v4061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:16 smithi079 ceph-mon[23158]: pgmap v4062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:16 smithi002 ceph-mon[17167]: pgmap v4062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:16 smithi016 ceph-mon[22010]: pgmap v4062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:18 smithi079 ceph-mon[23158]: pgmap v4063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7300.016438 secs 2022-03-21T16:13:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:18 smithi002 ceph-mon[17167]: pgmap v4063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7300.016438 secs 2022-03-21T16:13:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:18 smithi016 ceph-mon[22010]: pgmap v4063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7300.016438 secs 2022-03-21T16:13:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:20 smithi002 ceph-mon[17167]: pgmap v4064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:13:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:13:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:13:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:20 smithi079 ceph-mon[23158]: pgmap v4064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:13:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:13:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:13:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:20 smithi016 ceph-mon[22010]: pgmap v4064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:13:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:13:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:13:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:13:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:13:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:13:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:13:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:13:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:13:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:13:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:13:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:13:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:22 smithi079 ceph-mon[23158]: pgmap v4065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:22 smithi002 ceph-mon[17167]: pgmap v4065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:22 smithi016 ceph-mon[22010]: pgmap v4065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7305.016574 secs 2022-03-21T16:13:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7305.016574 secs 2022-03-21T16:13:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7305.016574 secs 2022-03-21T16:13:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:24 smithi079 ceph-mon[23158]: pgmap v4066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:24 smithi002 ceph-mon[17167]: pgmap v4066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:24 smithi016 ceph-mon[22010]: pgmap v4066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:26 smithi079 ceph-mon[23158]: pgmap v4067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:26 smithi002 ceph-mon[17167]: pgmap v4067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:26 smithi016 ceph-mon[22010]: pgmap v4067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:28 smithi079 ceph-mon[23158]: pgmap v4068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7310.016692 secs 2022-03-21T16:13:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:28 smithi002 ceph-mon[17167]: pgmap v4068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7310.016692 secs 2022-03-21T16:13:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:28 smithi016 ceph-mon[22010]: pgmap v4068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7310.016692 secs 2022-03-21T16:13:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:30 smithi079 ceph-mon[23158]: pgmap v4069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:30 smithi002 ceph-mon[17167]: pgmap v4069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:30 smithi016 ceph-mon[22010]: pgmap v4069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:32 smithi079 ceph-mon[23158]: pgmap v4070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:32 smithi002 ceph-mon[17167]: pgmap v4070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:32 smithi016 ceph-mon[22010]: pgmap v4070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7315.016786 secs 2022-03-21T16:13:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7315.016786 secs 2022-03-21T16:13:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7315.016786 secs 2022-03-21T16:13:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:34 smithi079 ceph-mon[23158]: pgmap v4071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:34 smithi002 ceph-mon[17167]: pgmap v4071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:34 smithi016 ceph-mon[22010]: pgmap v4071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:36 smithi079 ceph-mon[23158]: pgmap v4072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:36 smithi002 ceph-mon[17167]: pgmap v4072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:36 smithi016 ceph-mon[22010]: pgmap v4072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:38 smithi079 ceph-mon[23158]: pgmap v4073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7320.016895 secs 2022-03-21T16:13:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:38 smithi002 ceph-mon[17167]: pgmap v4073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7320.016895 secs 2022-03-21T16:13:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:38 smithi016 ceph-mon[22010]: pgmap v4073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7320.016895 secs 2022-03-21T16:13:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:40 smithi002 ceph-mon[17167]: pgmap v4074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:40 smithi016 ceph-mon[22010]: pgmap v4074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:40 smithi079 ceph-mon[23158]: pgmap v4074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:42 smithi002 ceph-mon[17167]: pgmap v4075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:42 smithi016 ceph-mon[22010]: pgmap v4075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:42 smithi079 ceph-mon[23158]: pgmap v4075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7325.016971 secs 2022-03-21T16:13:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7325.016971 secs 2022-03-21T16:13:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7325.016971 secs 2022-03-21T16:13:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:44 smithi002 ceph-mon[17167]: pgmap v4076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:44 smithi016 ceph-mon[22010]: pgmap v4076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:44 smithi079 ceph-mon[23158]: pgmap v4076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:46 smithi002 ceph-mon[17167]: pgmap v4077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:46 smithi016 ceph-mon[22010]: pgmap v4077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:46 smithi079 ceph-mon[23158]: pgmap v4077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:48 smithi002 ceph-mon[17167]: pgmap v4078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7330.017062 secs 2022-03-21T16:13:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:48 smithi016 ceph-mon[22010]: pgmap v4078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7330.017062 secs 2022-03-21T16:13:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:48 smithi079 ceph-mon[23158]: pgmap v4078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7330.017062 secs 2022-03-21T16:13:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:50 smithi002 ceph-mon[17167]: pgmap v4079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:50 smithi016 ceph-mon[22010]: pgmap v4079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:50 smithi079 ceph-mon[23158]: pgmap v4079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:52 smithi002 ceph-mon[17167]: pgmap v4080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:52 smithi016 ceph-mon[22010]: pgmap v4080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:52 smithi079 ceph-mon[23158]: pgmap v4080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7335.017205 secs 2022-03-21T16:13:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7335.017205 secs 2022-03-21T16:13:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7335.017205 secs 2022-03-21T16:13:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:54 smithi002 ceph-mon[17167]: pgmap v4081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:54 smithi016 ceph-mon[22010]: pgmap v4081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:54 smithi079 ceph-mon[23158]: pgmap v4081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:56 smithi002 ceph-mon[17167]: pgmap v4082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:56 smithi016 ceph-mon[22010]: pgmap v4082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:56 smithi079 ceph-mon[23158]: pgmap v4082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:13:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:58 smithi002 ceph-mon[17167]: pgmap v4083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:13:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7340.017353 secs 2022-03-21T16:13:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:58 smithi016 ceph-mon[22010]: pgmap v4083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:13:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7340.017353 secs 2022-03-21T16:13:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:58 smithi079 ceph-mon[23158]: pgmap v4083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:13:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:13:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7340.017353 secs 2022-03-21T16:14:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:00 smithi002 ceph-mon[17167]: pgmap v4084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:00 smithi016 ceph-mon[22010]: pgmap v4084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:00 smithi079 ceph-mon[23158]: pgmap v4084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:02 smithi002 ceph-mon[17167]: pgmap v4085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:02 smithi016 ceph-mon[22010]: pgmap v4085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:02 smithi079 ceph-mon[23158]: pgmap v4085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:04.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7345.017466 secs 2022-03-21T16:14:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7345.017466 secs 2022-03-21T16:14:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7345.017466 secs 2022-03-21T16:14:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:04 smithi002 ceph-mon[17167]: pgmap v4086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:04 smithi016 ceph-mon[22010]: pgmap v4086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:04 smithi079 ceph-mon[23158]: pgmap v4086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:07.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:06 smithi002 ceph-mon[17167]: pgmap v4087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:06 smithi016 ceph-mon[22010]: pgmap v4087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:06 smithi079 ceph-mon[23158]: pgmap v4087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:08 smithi002 ceph-mon[17167]: pgmap v4088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7350.017569 secs 2022-03-21T16:14:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:08 smithi016 ceph-mon[22010]: pgmap v4088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7350.017569 secs 2022-03-21T16:14:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:08 smithi079 ceph-mon[23158]: pgmap v4088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7350.017569 secs 2022-03-21T16:14:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:10 smithi002 ceph-mon[17167]: pgmap v4089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:10 smithi016 ceph-mon[22010]: pgmap v4089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:10 smithi079 ceph-mon[23158]: pgmap v4089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:12 smithi002 ceph-mon[17167]: pgmap v4090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:12 smithi016 ceph-mon[22010]: pgmap v4090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:12 smithi079 ceph-mon[23158]: pgmap v4090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:14.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7355.017683 secs 2022-03-21T16:14:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7355.017683 secs 2022-03-21T16:14:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7355.017683 secs 2022-03-21T16:14:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:14 smithi002 ceph-mon[17167]: pgmap v4091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:14 smithi016 ceph-mon[22010]: pgmap v4091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:14 smithi079 ceph-mon[23158]: pgmap v4091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:16 smithi002 ceph-mon[17167]: pgmap v4092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:16 smithi016 ceph-mon[22010]: pgmap v4092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:16 smithi079 ceph-mon[23158]: pgmap v4092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:18 smithi002 ceph-mon[17167]: pgmap v4093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7360.017804 secs 2022-03-21T16:14:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:18 smithi016 ceph-mon[22010]: pgmap v4093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7360.017804 secs 2022-03-21T16:14:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:18 smithi079 ceph-mon[23158]: pgmap v4093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7360.017804 secs 2022-03-21T16:14:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:20 smithi002 ceph-mon[17167]: pgmap v4094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:20 smithi016 ceph-mon[22010]: pgmap v4094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:20 smithi079 ceph-mon[23158]: pgmap v4094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:14:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:14:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:14:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:14:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:14:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:14:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:14:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:14:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:14:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:14:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:14:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:14:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:14:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:14:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:22 smithi002 ceph-mon[17167]: pgmap v4095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:22 smithi016 ceph-mon[22010]: pgmap v4095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:22 smithi079 ceph-mon[23158]: pgmap v4095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:14:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7365.017919 secs 2022-03-21T16:14:24.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:23 smithi002 ceph-mon[17167]: pgmap v4096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7365.017919 secs 2022-03-21T16:14:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:23 smithi016 ceph-mon[22010]: pgmap v4096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7365.017919 secs 2022-03-21T16:14:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:23 smithi079 ceph-mon[23158]: pgmap v4096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:25 smithi079 ceph-mon[23158]: pgmap v4097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:25 smithi002 ceph-mon[17167]: pgmap v4097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:25 smithi016 ceph-mon[22010]: pgmap v4097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:28 smithi002 ceph-mon[17167]: pgmap v4098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:28 smithi016 ceph-mon[22010]: pgmap v4098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:28 smithi079 ceph-mon[23158]: pgmap v4098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:29 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7370.018056 secs 2022-03-21T16:14:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:29 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7370.018056 secs 2022-03-21T16:14:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:29 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7370.018056 secs 2022-03-21T16:14:30.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:30 smithi002 ceph-mon[17167]: pgmap v4099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:30 smithi016 ceph-mon[22010]: pgmap v4099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:30 smithi079 ceph-mon[23158]: pgmap v4099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:32.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:32 smithi002 ceph-mon[17167]: pgmap v4100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:32 smithi016 ceph-mon[22010]: pgmap v4100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:32 smithi079 ceph-mon[23158]: pgmap v4100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:34 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7375.018200 secs 2022-03-21T16:14:34.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:34 smithi002 ceph-mon[17167]: pgmap v4101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:34 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7375.018200 secs 2022-03-21T16:14:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:34 smithi016 ceph-mon[22010]: pgmap v4101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:34 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7375.018200 secs 2022-03-21T16:14:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:34 smithi079 ceph-mon[23158]: pgmap v4101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:36.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:36 smithi002 ceph-mon[17167]: pgmap v4102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:36 smithi016 ceph-mon[22010]: pgmap v4102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:36 smithi079 ceph-mon[23158]: pgmap v4102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:38 smithi002 ceph-mon[17167]: pgmap v4103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:38 smithi016 ceph-mon[22010]: pgmap v4103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:38 smithi079 ceph-mon[23158]: pgmap v4103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:39 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7380.018320 secs 2022-03-21T16:14:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:39 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7380.018320 secs 2022-03-21T16:14:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:39 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7380.018320 secs 2022-03-21T16:14:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:40 smithi002 ceph-mon[17167]: pgmap v4104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:40 smithi016 ceph-mon[22010]: pgmap v4104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:40 smithi079 ceph-mon[23158]: pgmap v4104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:42.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:42 smithi002 ceph-mon[17167]: pgmap v4105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:42 smithi016 ceph-mon[22010]: pgmap v4105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:42 smithi079 ceph-mon[23158]: pgmap v4105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:44.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:44 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7385.018447 secs 2022-03-21T16:14:44.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:44 smithi002 ceph-mon[17167]: pgmap v4106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:44 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7385.018447 secs 2022-03-21T16:14:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:44 smithi016 ceph-mon[22010]: pgmap v4106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:44 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7385.018447 secs 2022-03-21T16:14:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:44 smithi079 ceph-mon[23158]: pgmap v4106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:46.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:46 smithi002 ceph-mon[17167]: pgmap v4107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:46 smithi016 ceph-mon[22010]: pgmap v4107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:46 smithi079 ceph-mon[23158]: pgmap v4107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:48 smithi002 ceph-mon[17167]: pgmap v4108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:48 smithi016 ceph-mon[22010]: pgmap v4108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:48 smithi079 ceph-mon[23158]: pgmap v4108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:49 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7390.018610 secs 2022-03-21T16:14:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:49 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7390.018610 secs 2022-03-21T16:14:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:49 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7390.018610 secs 2022-03-21T16:14:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:50 smithi002 ceph-mon[17167]: pgmap v4109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:50 smithi016 ceph-mon[22010]: pgmap v4109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:50 smithi079 ceph-mon[23158]: pgmap v4109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:52.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:52 smithi002 ceph-mon[17167]: pgmap v4110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:52 smithi016 ceph-mon[22010]: pgmap v4110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:52 smithi079 ceph-mon[23158]: pgmap v4110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7395.018704 secs 2022-03-21T16:14:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7395.018704 secs 2022-03-21T16:14:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7395.018704 secs 2022-03-21T16:14:54.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:54 smithi002 ceph-mon[17167]: pgmap v4111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:54 smithi016 ceph-mon[22010]: pgmap v4111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:54 smithi079 ceph-mon[23158]: pgmap v4111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:56 smithi002 ceph-mon[17167]: pgmap v4112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:56 smithi016 ceph-mon[22010]: pgmap v4112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:56 smithi079 ceph-mon[23158]: pgmap v4112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:14:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:58 smithi002 ceph-mon[17167]: pgmap v4113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:14:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7400.018815 secs 2022-03-21T16:14:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:58 smithi016 ceph-mon[22010]: pgmap v4113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:14:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7400.018815 secs 2022-03-21T16:14:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:58 smithi079 ceph-mon[23158]: pgmap v4113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:14:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:14:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7400.018815 secs 2022-03-21T16:15:00.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:00 smithi002 ceph-mon[17167]: pgmap v4114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:00 smithi016 ceph-mon[22010]: pgmap v4114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:00 smithi079 ceph-mon[23158]: pgmap v4114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:02.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:02 smithi002 ceph-mon[17167]: pgmap v4115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:02 smithi016 ceph-mon[22010]: pgmap v4115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:02 smithi079 ceph-mon[23158]: pgmap v4115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7405.018931 secs 2022-03-21T16:15:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7405.018931 secs 2022-03-21T16:15:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7405.018931 secs 2022-03-21T16:15:04.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:04 smithi002 ceph-mon[17167]: pgmap v4116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:04 smithi016 ceph-mon[22010]: pgmap v4116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:04 smithi079 ceph-mon[23158]: pgmap v4116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:06 smithi002 ceph-mon[17167]: pgmap v4117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:06 smithi016 ceph-mon[22010]: pgmap v4117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:06 smithi079 ceph-mon[23158]: pgmap v4117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:08 smithi016 ceph-mon[22010]: pgmap v4118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7410.019052 secs 2022-03-21T16:15:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:08 smithi079 ceph-mon[23158]: pgmap v4118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7410.019052 secs 2022-03-21T16:15:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:08 smithi002 ceph-mon[17167]: pgmap v4118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7410.019052 secs 2022-03-21T16:15:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:10 smithi016 ceph-mon[22010]: pgmap v4119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:10 smithi079 ceph-mon[23158]: pgmap v4119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:10 smithi002 ceph-mon[17167]: pgmap v4119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:12 smithi016 ceph-mon[22010]: pgmap v4120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:12 smithi079 ceph-mon[23158]: pgmap v4120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:12 smithi002 ceph-mon[17167]: pgmap v4120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7415.019148 secs 2022-03-21T16:15:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7415.019148 secs 2022-03-21T16:15:13.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7415.019148 secs 2022-03-21T16:15:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:14 smithi016 ceph-mon[22010]: pgmap v4121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:14 smithi079 ceph-mon[23158]: pgmap v4121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:14 smithi002 ceph-mon[17167]: pgmap v4121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:16 smithi016 ceph-mon[22010]: pgmap v4122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:16 smithi079 ceph-mon[23158]: pgmap v4122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:16 smithi002 ceph-mon[17167]: pgmap v4122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:18 smithi016 ceph-mon[22010]: pgmap v4123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7420.019269 secs 2022-03-21T16:15:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:18 smithi079 ceph-mon[23158]: pgmap v4123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7420.019269 secs 2022-03-21T16:15:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:18 smithi002 ceph-mon[17167]: pgmap v4123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7420.019269 secs 2022-03-21T16:15:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:20 smithi016 ceph-mon[22010]: pgmap v4124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:20 smithi079 ceph-mon[23158]: pgmap v4124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:20 smithi002 ceph-mon[17167]: pgmap v4124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:15:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:15:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:15:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:15:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:15:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:15:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:22 smithi016 ceph-mon[22010]: pgmap v4125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:22 smithi079 ceph-mon[23158]: pgmap v4125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:22.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:22 smithi002 ceph-mon[17167]: pgmap v4125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7425.019393 secs 2022-03-21T16:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:15:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:15:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:15:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7425.019393 secs 2022-03-21T16:15:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:15:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:15:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:15:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7425.019393 secs 2022-03-21T16:15:23.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:15:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:15:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:15:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:24 smithi002 ceph-mon[17167]: pgmap v4126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:15:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:24 smithi016 ceph-mon[22010]: pgmap v4126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:25.007 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:15:25.008 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:24 smithi079 ceph-mon[23158]: pgmap v4126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:25.008 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:15:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:26 smithi002 ceph-mon[17167]: pgmap v4127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:26 smithi016 ceph-mon[22010]: pgmap v4127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:26 smithi079 ceph-mon[23158]: pgmap v4127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:28 smithi016 ceph-mon[22010]: pgmap v4128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7430.019483 secs 2022-03-21T16:15:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:28 smithi079 ceph-mon[23158]: pgmap v4128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7430.019483 secs 2022-03-21T16:15:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:28 smithi002 ceph-mon[17167]: pgmap v4128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7430.019483 secs 2022-03-21T16:15:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:30 smithi016 ceph-mon[22010]: pgmap v4129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:30 smithi079 ceph-mon[23158]: pgmap v4129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:30 smithi002 ceph-mon[17167]: pgmap v4129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:32 smithi016 ceph-mon[22010]: pgmap v4130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:32 smithi079 ceph-mon[23158]: pgmap v4130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:32 smithi002 ceph-mon[17167]: pgmap v4130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7435.019597 secs 2022-03-21T16:15:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7435.019597 secs 2022-03-21T16:15:34.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7435.019597 secs 2022-03-21T16:15:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:34 smithi016 ceph-mon[22010]: pgmap v4131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:34 smithi079 ceph-mon[23158]: pgmap v4131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:34 smithi002 ceph-mon[17167]: pgmap v4131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:36 smithi016 ceph-mon[22010]: pgmap v4132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:36 smithi079 ceph-mon[23158]: pgmap v4132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:36 smithi002 ceph-mon[17167]: pgmap v4132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:38 smithi016 ceph-mon[22010]: pgmap v4133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7440.019709 secs 2022-03-21T16:15:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:38 smithi079 ceph-mon[23158]: pgmap v4133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7440.019709 secs 2022-03-21T16:15:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:38 smithi002 ceph-mon[17167]: pgmap v4133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7440.019709 secs 2022-03-21T16:15:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:40 smithi016 ceph-mon[22010]: pgmap v4134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:40 smithi079 ceph-mon[23158]: pgmap v4134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:40 smithi002 ceph-mon[17167]: pgmap v4134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:42 smithi016 ceph-mon[22010]: pgmap v4135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:42 smithi079 ceph-mon[23158]: pgmap v4135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:42 smithi002 ceph-mon[17167]: pgmap v4135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7445.019773 secs 2022-03-21T16:15:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7445.019773 secs 2022-03-21T16:15:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7445.019773 secs 2022-03-21T16:15:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:44 smithi016 ceph-mon[22010]: pgmap v4136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:44 smithi079 ceph-mon[23158]: pgmap v4136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:44 smithi002 ceph-mon[17167]: pgmap v4136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:46 smithi016 ceph-mon[22010]: pgmap v4137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:46 smithi079 ceph-mon[23158]: pgmap v4137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:46 smithi002 ceph-mon[17167]: pgmap v4137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:48 smithi016 ceph-mon[22010]: pgmap v4138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7450.019839 secs 2022-03-21T16:15:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:48 smithi079 ceph-mon[23158]: pgmap v4138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7450.019839 secs 2022-03-21T16:15:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:48 smithi002 ceph-mon[17167]: pgmap v4138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7450.019839 secs 2022-03-21T16:15:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:50 smithi079 ceph-mon[23158]: pgmap v4139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:50 smithi002 ceph-mon[17167]: pgmap v4139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:50 smithi016 ceph-mon[22010]: pgmap v4139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:52 smithi079 ceph-mon[23158]: pgmap v4140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:52 smithi002 ceph-mon[17167]: pgmap v4140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:52 smithi016 ceph-mon[22010]: pgmap v4140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7455.019906 secs 2022-03-21T16:15:54.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7455.019906 secs 2022-03-21T16:15:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7455.019906 secs 2022-03-21T16:15:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:54 smithi079 ceph-mon[23158]: pgmap v4141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:54 smithi002 ceph-mon[17167]: pgmap v4141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:54 smithi016 ceph-mon[22010]: pgmap v4141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:56 smithi079 ceph-mon[23158]: pgmap v4142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:56 smithi002 ceph-mon[17167]: pgmap v4142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:56 smithi016 ceph-mon[22010]: pgmap v4142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:15:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:58 smithi079 ceph-mon[23158]: pgmap v4143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:15:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7460.020034 secs 2022-03-21T16:15:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:58 smithi002 ceph-mon[17167]: pgmap v4143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:15:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7460.020034 secs 2022-03-21T16:15:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:58 smithi016 ceph-mon[22010]: pgmap v4143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:15:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:15:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7460.020034 secs 2022-03-21T16:16:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:00 smithi079 ceph-mon[23158]: pgmap v4144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:01.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:00 smithi002 ceph-mon[17167]: pgmap v4144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:00 smithi016 ceph-mon[22010]: pgmap v4144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:02 smithi079 ceph-mon[23158]: pgmap v4145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7465.020110 secs 2022-03-21T16:16:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:02 smithi002 ceph-mon[17167]: pgmap v4145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7465.020110 secs 2022-03-21T16:16:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:02 smithi016 ceph-mon[22010]: pgmap v4145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7465.020110 secs 2022-03-21T16:16:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:04 smithi079 ceph-mon[23158]: pgmap v4146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:04 smithi002 ceph-mon[17167]: pgmap v4146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:04 smithi016 ceph-mon[22010]: pgmap v4146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:06 smithi079 ceph-mon[23158]: pgmap v4147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:06 smithi002 ceph-mon[17167]: pgmap v4147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:06 smithi016 ceph-mon[22010]: pgmap v4147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7470.020234 secs 2022-03-21T16:16:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7470.020234 secs 2022-03-21T16:16:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7470.020234 secs 2022-03-21T16:16:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:08 smithi079 ceph-mon[23158]: pgmap v4148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:08 smithi002 ceph-mon[17167]: pgmap v4148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:08 smithi016 ceph-mon[22010]: pgmap v4148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:10 smithi079 ceph-mon[23158]: pgmap v4149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:10 smithi002 ceph-mon[17167]: pgmap v4149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:10 smithi016 ceph-mon[22010]: pgmap v4149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:12 smithi079 ceph-mon[23158]: pgmap v4150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7475.020333 secs 2022-03-21T16:16:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:12 smithi002 ceph-mon[17167]: pgmap v4150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7475.020333 secs 2022-03-21T16:16:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:12 smithi016 ceph-mon[22010]: pgmap v4150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7475.020333 secs 2022-03-21T16:16:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:14 smithi079 ceph-mon[23158]: pgmap v4151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:14 smithi002 ceph-mon[17167]: pgmap v4151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:14 smithi016 ceph-mon[22010]: pgmap v4151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:16 smithi079 ceph-mon[23158]: pgmap v4152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:16 smithi002 ceph-mon[17167]: pgmap v4152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:16 smithi016 ceph-mon[22010]: pgmap v4152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7480.020469 secs 2022-03-21T16:16:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7480.020469 secs 2022-03-21T16:16:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7480.020469 secs 2022-03-21T16:16:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:18 smithi002 ceph-mon[17167]: pgmap v4153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:18 smithi016 ceph-mon[22010]: pgmap v4153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:18 smithi079 ceph-mon[23158]: pgmap v4153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:20 smithi002 ceph-mon[17167]: pgmap v4154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:20 smithi016 ceph-mon[22010]: pgmap v4154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:20 smithi079 ceph-mon[23158]: pgmap v4154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:16:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:16:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:16:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:16:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:16:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:16:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:22 smithi002 ceph-mon[17167]: pgmap v4155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7485.020624 secs 2022-03-21T16:16:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:22 smithi016 ceph-mon[22010]: pgmap v4155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7485.020624 secs 2022-03-21T16:16:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:22 smithi079 ceph-mon[23158]: pgmap v4155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7485.020624 secs 2022-03-21T16:16:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:16:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:16:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:16:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:16:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:16:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:16:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:16:24.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:16:24.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:16:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:24 smithi002 ceph-mon[17167]: pgmap v4156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:16:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:24 smithi016 ceph-mon[22010]: pgmap v4156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:16:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:24 smithi079 ceph-mon[23158]: pgmap v4156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:16:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:26 smithi002 ceph-mon[17167]: pgmap v4157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:26 smithi016 ceph-mon[22010]: pgmap v4157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:26 smithi079 ceph-mon[23158]: pgmap v4157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7490.020743 secs 2022-03-21T16:16:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7490.020743 secs 2022-03-21T16:16:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7490.020743 secs 2022-03-21T16:16:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:28 smithi002 ceph-mon[17167]: pgmap v4158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:28 smithi016 ceph-mon[22010]: pgmap v4158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:28 smithi079 ceph-mon[23158]: pgmap v4158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:30 smithi002 ceph-mon[17167]: pgmap v4159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:30 smithi016 ceph-mon[22010]: pgmap v4159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:30 smithi079 ceph-mon[23158]: pgmap v4159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:32 smithi002 ceph-mon[17167]: pgmap v4160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7495.020827 secs 2022-03-21T16:16:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:32 smithi016 ceph-mon[22010]: pgmap v4160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7495.020827 secs 2022-03-21T16:16:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:32 smithi079 ceph-mon[23158]: pgmap v4160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7495.020827 secs 2022-03-21T16:16:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:34 smithi002 ceph-mon[17167]: pgmap v4161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:34 smithi016 ceph-mon[22010]: pgmap v4161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:34 smithi079 ceph-mon[23158]: pgmap v4161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:36 smithi002 ceph-mon[17167]: pgmap v4162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:36 smithi016 ceph-mon[22010]: pgmap v4162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:36 smithi079 ceph-mon[23158]: pgmap v4162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7500.020946 secs 2022-03-21T16:16:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7500.020946 secs 2022-03-21T16:16:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7500.020946 secs 2022-03-21T16:16:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:38 smithi002 ceph-mon[17167]: pgmap v4163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:38 smithi016 ceph-mon[22010]: pgmap v4163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:38 smithi079 ceph-mon[23158]: pgmap v4163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:40 smithi002 ceph-mon[17167]: pgmap v4164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:40 smithi016 ceph-mon[22010]: pgmap v4164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:40 smithi079 ceph-mon[23158]: pgmap v4164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:42 smithi002 ceph-mon[17167]: pgmap v4165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7505.021063 secs 2022-03-21T16:16:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:42 smithi016 ceph-mon[22010]: pgmap v4165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7505.021063 secs 2022-03-21T16:16:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:42 smithi079 ceph-mon[23158]: pgmap v4165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7505.021063 secs 2022-03-21T16:16:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:44 smithi002 ceph-mon[17167]: pgmap v4166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:44 smithi016 ceph-mon[22010]: pgmap v4166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:44 smithi079 ceph-mon[23158]: pgmap v4166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:46 smithi002 ceph-mon[17167]: pgmap v4167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:46 smithi016 ceph-mon[22010]: pgmap v4167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:46 smithi079 ceph-mon[23158]: pgmap v4167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7510.021176 secs 2022-03-21T16:16:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7510.021176 secs 2022-03-21T16:16:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7510.021176 secs 2022-03-21T16:16:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:48 smithi002 ceph-mon[17167]: pgmap v4168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:48 smithi016 ceph-mon[22010]: pgmap v4168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:48 smithi079 ceph-mon[23158]: pgmap v4168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:50 smithi002 ceph-mon[17167]: pgmap v4169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:50 smithi016 ceph-mon[22010]: pgmap v4169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:50 smithi079 ceph-mon[23158]: pgmap v4169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:52 smithi002 ceph-mon[17167]: pgmap v4170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7515.021298 secs 2022-03-21T16:16:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:52 smithi016 ceph-mon[22010]: pgmap v4170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7515.021298 secs 2022-03-21T16:16:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:52 smithi079 ceph-mon[23158]: pgmap v4170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7515.021298 secs 2022-03-21T16:16:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:54 smithi002 ceph-mon[17167]: pgmap v4171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:54 smithi016 ceph-mon[22010]: pgmap v4171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:54 smithi079 ceph-mon[23158]: pgmap v4171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:56 smithi002 ceph-mon[17167]: pgmap v4172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:56 smithi016 ceph-mon[22010]: pgmap v4172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:56 smithi079 ceph-mon[23158]: pgmap v4172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:16:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7520.021430 secs 2022-03-21T16:16:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7520.021430 secs 2022-03-21T16:16:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7520.021430 secs 2022-03-21T16:16:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:16:58 smithi002 ceph-mon[17167]: pgmap v4173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:16:58 smithi016 ceph-mon[22010]: pgmap v4173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:16:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:16:58 smithi079 ceph-mon[23158]: pgmap v4173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:00 smithi002 ceph-mon[17167]: pgmap v4174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:00 smithi016 ceph-mon[22010]: pgmap v4174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:00 smithi079 ceph-mon[23158]: pgmap v4174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:02 smithi002 ceph-mon[17167]: pgmap v4175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7525.021564 secs 2022-03-21T16:17:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:02 smithi016 ceph-mon[22010]: pgmap v4175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7525.021564 secs 2022-03-21T16:17:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:02 smithi079 ceph-mon[23158]: pgmap v4175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7525.021564 secs 2022-03-21T16:17:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:04 smithi002 ceph-mon[17167]: pgmap v4176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:04 smithi016 ceph-mon[22010]: pgmap v4176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:04 smithi079 ceph-mon[23158]: pgmap v4176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:06 smithi002 ceph-mon[17167]: pgmap v4177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:06 smithi016 ceph-mon[22010]: pgmap v4177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:06 smithi079 ceph-mon[23158]: pgmap v4177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7530.021673 secs 2022-03-21T16:17:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7530.021673 secs 2022-03-21T16:17:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7530.021673 secs 2022-03-21T16:17:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:08 smithi002 ceph-mon[17167]: pgmap v4178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:08 smithi016 ceph-mon[22010]: pgmap v4178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:08 smithi079 ceph-mon[23158]: pgmap v4178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:10.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:09 smithi002 ceph-mon[17167]: pgmap v4179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:09 smithi016 ceph-mon[22010]: pgmap v4179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:09 smithi079 ceph-mon[23158]: pgmap v4179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:12.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:12 smithi002 ceph-mon[17167]: pgmap v4180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:12 smithi016 ceph-mon[22010]: pgmap v4180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:12 smithi079 ceph-mon[23158]: pgmap v4180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:13.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7535.021748 secs 2022-03-21T16:17:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7535.021748 secs 2022-03-21T16:17:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7535.021748 secs 2022-03-21T16:17:14.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:14 smithi002 ceph-mon[17167]: pgmap v4181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:14 smithi016 ceph-mon[22010]: pgmap v4181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:14 smithi079 ceph-mon[23158]: pgmap v4181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:16.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:16 smithi002 ceph-mon[17167]: pgmap v4182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:16 smithi016 ceph-mon[22010]: pgmap v4182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:16 smithi079 ceph-mon[23158]: pgmap v4182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:18 smithi002 ceph-mon[17167]: pgmap v4183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:18.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7540.021866 secs 2022-03-21T16:17:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:18 smithi016 ceph-mon[22010]: pgmap v4183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7540.021866 secs 2022-03-21T16:17:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:18 smithi079 ceph-mon[23158]: pgmap v4183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7540.021866 secs 2022-03-21T16:17:20.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:20 smithi002 ceph-mon[17167]: pgmap v4184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:20 smithi016 ceph-mon[22010]: pgmap v4184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:20 smithi079 ceph-mon[23158]: pgmap v4184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:17:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:17:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:17:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:17:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:17:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:17:22.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:22 smithi002 ceph-mon[17167]: pgmap v4185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7545.021992 secs 2022-03-21T16:17:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:22 smithi016 ceph-mon[22010]: pgmap v4185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7545.021992 secs 2022-03-21T16:17:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:22 smithi079 ceph-mon[23158]: pgmap v4185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7545.021992 secs 2022-03-21T16:17:24.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:24 smithi002 ceph-mon[17167]: pgmap v4186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:24 smithi016 ceph-mon[22010]: pgmap v4186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:24 smithi079 ceph-mon[23158]: pgmap v4186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:17:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:17:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:17:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:17:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:17:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:17:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:17:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:17:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:17:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:17:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:17:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:17:26.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:26 smithi002 ceph-mon[17167]: pgmap v4187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:26 smithi016 ceph-mon[22010]: pgmap v4187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:26 smithi079 ceph-mon[23158]: pgmap v4187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7550.022133 secs 2022-03-21T16:17:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7550.022133 secs 2022-03-21T16:17:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7550.022133 secs 2022-03-21T16:17:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:28 smithi016 ceph-mon[22010]: pgmap v4188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:28 smithi079 ceph-mon[23158]: pgmap v4188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:28 smithi002 ceph-mon[17167]: pgmap v4188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:30 smithi016 ceph-mon[22010]: pgmap v4189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:30 smithi079 ceph-mon[23158]: pgmap v4189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:30 smithi002 ceph-mon[17167]: pgmap v4189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:32 smithi016 ceph-mon[22010]: pgmap v4190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7555.022219 secs 2022-03-21T16:17:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:32 smithi079 ceph-mon[23158]: pgmap v4190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7555.022219 secs 2022-03-21T16:17:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:32 smithi002 ceph-mon[17167]: pgmap v4190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:32.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7555.022219 secs 2022-03-21T16:17:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:34 smithi016 ceph-mon[22010]: pgmap v4191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:34 smithi079 ceph-mon[23158]: pgmap v4191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:34 smithi002 ceph-mon[17167]: pgmap v4191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:36 smithi016 ceph-mon[22010]: pgmap v4192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:36 smithi079 ceph-mon[23158]: pgmap v4192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:36 smithi002 ceph-mon[17167]: pgmap v4192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7560.022370 secs 2022-03-21T16:17:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7560.022370 secs 2022-03-21T16:17:37.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7560.022370 secs 2022-03-21T16:17:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:38 smithi016 ceph-mon[22010]: pgmap v4193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:38 smithi079 ceph-mon[23158]: pgmap v4193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:38 smithi002 ceph-mon[17167]: pgmap v4193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:40 smithi016 ceph-mon[22010]: pgmap v4194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:40 smithi079 ceph-mon[23158]: pgmap v4194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:40 smithi002 ceph-mon[17167]: pgmap v4194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:42 smithi016 ceph-mon[22010]: pgmap v4195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7565.022434 secs 2022-03-21T16:17:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:42 smithi079 ceph-mon[23158]: pgmap v4195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7565.022434 secs 2022-03-21T16:17:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:42 smithi002 ceph-mon[17167]: pgmap v4195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7565.022434 secs 2022-03-21T16:17:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:44 smithi016 ceph-mon[22010]: pgmap v4196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:44 smithi079 ceph-mon[23158]: pgmap v4196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:44 smithi002 ceph-mon[17167]: pgmap v4196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:46 smithi016 ceph-mon[22010]: pgmap v4197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:46 smithi079 ceph-mon[23158]: pgmap v4197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:46 smithi002 ceph-mon[17167]: pgmap v4197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7570.022565 secs 2022-03-21T16:17:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7570.022565 secs 2022-03-21T16:17:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7570.022565 secs 2022-03-21T16:17:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:48 smithi079 ceph-mon[23158]: pgmap v4198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:48 smithi002 ceph-mon[17167]: pgmap v4198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:48 smithi016 ceph-mon[22010]: pgmap v4198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:50 smithi079 ceph-mon[23158]: pgmap v4199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:50 smithi002 ceph-mon[17167]: pgmap v4199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:50 smithi016 ceph-mon[22010]: pgmap v4199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:52 smithi079 ceph-mon[23158]: pgmap v4200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7575.022677 secs 2022-03-21T16:17:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:52 smithi002 ceph-mon[17167]: pgmap v4200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7575.022677 secs 2022-03-21T16:17:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:52 smithi016 ceph-mon[22010]: pgmap v4200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7575.022677 secs 2022-03-21T16:17:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:54 smithi079 ceph-mon[23158]: pgmap v4201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:54 smithi002 ceph-mon[17167]: pgmap v4201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:54 smithi016 ceph-mon[22010]: pgmap v4201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:56 smithi079 ceph-mon[23158]: pgmap v4202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:56 smithi002 ceph-mon[17167]: pgmap v4202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:56 smithi016 ceph-mon[22010]: pgmap v4202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:17:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7580.022792 secs 2022-03-21T16:17:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7580.022792 secs 2022-03-21T16:17:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7580.022792 secs 2022-03-21T16:17:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:17:58 smithi079 ceph-mon[23158]: pgmap v4203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:17:58 smithi002 ceph-mon[17167]: pgmap v4203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:17:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:17:58 smithi016 ceph-mon[22010]: pgmap v4203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:00 smithi079 ceph-mon[23158]: pgmap v4204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:00 smithi002 ceph-mon[17167]: pgmap v4204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:00 smithi016 ceph-mon[22010]: pgmap v4204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:02 smithi079 ceph-mon[23158]: pgmap v4205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7585.022904 secs 2022-03-21T16:18:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:02 smithi002 ceph-mon[17167]: pgmap v4205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7585.022904 secs 2022-03-21T16:18:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:02 smithi016 ceph-mon[22010]: pgmap v4205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7585.022904 secs 2022-03-21T16:18:04.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:04 smithi079 ceph-mon[23158]: pgmap v4206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:04 smithi002 ceph-mon[17167]: pgmap v4206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:04 smithi016 ceph-mon[22010]: pgmap v4206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:06 smithi079 ceph-mon[23158]: pgmap v4207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:06 smithi002 ceph-mon[17167]: pgmap v4207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:06 smithi016 ceph-mon[22010]: pgmap v4207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7590.022997 secs 2022-03-21T16:18:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7590.022997 secs 2022-03-21T16:18:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7590.022997 secs 2022-03-21T16:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:08 smithi079 ceph-mon[23158]: pgmap v4208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:08 smithi002 ceph-mon[17167]: pgmap v4208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:08 smithi016 ceph-mon[22010]: pgmap v4208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:10 smithi079 ceph-mon[23158]: pgmap v4209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:10 smithi002 ceph-mon[17167]: pgmap v4209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:10 smithi016 ceph-mon[22010]: pgmap v4209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:12 smithi079 ceph-mon[23158]: pgmap v4210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7595.023108 secs 2022-03-21T16:18:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:12 smithi002 ceph-mon[17167]: pgmap v4210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7595.023108 secs 2022-03-21T16:18:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:12 smithi016 ceph-mon[22010]: pgmap v4210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7595.023108 secs 2022-03-21T16:18:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:14 smithi079 ceph-mon[23158]: pgmap v4211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:14 smithi002 ceph-mon[17167]: pgmap v4211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:14 smithi016 ceph-mon[22010]: pgmap v4211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:16 smithi002 ceph-mon[17167]: pgmap v4212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:16 smithi016 ceph-mon[22010]: pgmap v4212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:16 smithi079 ceph-mon[23158]: pgmap v4212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7600.023259 secs 2022-03-21T16:18:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7600.023259 secs 2022-03-21T16:18:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7600.023259 secs 2022-03-21T16:18:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:18 smithi002 ceph-mon[17167]: pgmap v4213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:18 smithi016 ceph-mon[22010]: pgmap v4213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:18 smithi079 ceph-mon[23158]: pgmap v4213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:20 smithi002 ceph-mon[17167]: pgmap v4214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:20 smithi016 ceph-mon[22010]: pgmap v4214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:20 smithi079 ceph-mon[23158]: pgmap v4214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:18:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:18:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:18:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:18:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:18:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:18:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:22 smithi002 ceph-mon[17167]: pgmap v4215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7605.023325 secs 2022-03-21T16:18:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:22 smithi016 ceph-mon[22010]: pgmap v4215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7605.023325 secs 2022-03-21T16:18:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:22 smithi079 ceph-mon[23158]: pgmap v4215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7605.023325 secs 2022-03-21T16:18:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:24 smithi002 ceph-mon[17167]: pgmap v4216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:24 smithi016 ceph-mon[22010]: pgmap v4216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:24 smithi079 ceph-mon[23158]: pgmap v4216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:18:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:18:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:18:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:18:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:18:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:18:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:18:25.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:18:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:18:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:26 smithi002 ceph-mon[17167]: pgmap v4217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:18:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:26 smithi016 ceph-mon[22010]: pgmap v4217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:18:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:26 smithi079 ceph-mon[23158]: pgmap v4217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:18:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7610.023443 secs 2022-03-21T16:18:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7610.023443 secs 2022-03-21T16:18:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7610.023443 secs 2022-03-21T16:18:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:28 smithi002 ceph-mon[17167]: pgmap v4218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:28 smithi016 ceph-mon[22010]: pgmap v4218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:28 smithi079 ceph-mon[23158]: pgmap v4218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:30.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:30 smithi002 ceph-mon[17167]: pgmap v4219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:30 smithi016 ceph-mon[22010]: pgmap v4219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:30 smithi079 ceph-mon[23158]: pgmap v4219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:32 smithi002 ceph-mon[17167]: pgmap v4220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:32.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7615.023577 secs 2022-03-21T16:18:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:32 smithi016 ceph-mon[22010]: pgmap v4220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7615.023577 secs 2022-03-21T16:18:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:32 smithi079 ceph-mon[23158]: pgmap v4220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7615.023577 secs 2022-03-21T16:18:34.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:34 smithi002 ceph-mon[17167]: pgmap v4221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:34 smithi016 ceph-mon[22010]: pgmap v4221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:34 smithi079 ceph-mon[23158]: pgmap v4221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:36.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:36 smithi002 ceph-mon[17167]: pgmap v4222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:36 smithi016 ceph-mon[22010]: pgmap v4222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:36 smithi079 ceph-mon[23158]: pgmap v4222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:37.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7620.023652 secs 2022-03-21T16:18:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7620.023652 secs 2022-03-21T16:18:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7620.023652 secs 2022-03-21T16:18:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:38 smithi002 ceph-mon[17167]: pgmap v4223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:38 smithi016 ceph-mon[22010]: pgmap v4223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:38 smithi079 ceph-mon[23158]: pgmap v4223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:40 smithi002 ceph-mon[17167]: pgmap v4224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:41.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:40 smithi079 ceph-mon[23158]: pgmap v4224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:41.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:40 smithi016 ceph-mon[22010]: pgmap v4224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:42 smithi002 ceph-mon[17167]: pgmap v4225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7625.023771 secs 2022-03-21T16:18:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:42 smithi016 ceph-mon[22010]: pgmap v4225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7625.023771 secs 2022-03-21T16:18:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:42 smithi079 ceph-mon[23158]: pgmap v4225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7625.023771 secs 2022-03-21T16:18:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:44 smithi002 ceph-mon[17167]: pgmap v4226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:44 smithi016 ceph-mon[22010]: pgmap v4226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:44 smithi079 ceph-mon[23158]: pgmap v4226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:46 smithi002 ceph-mon[17167]: pgmap v4227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:46 smithi016 ceph-mon[22010]: pgmap v4227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:46 smithi079 ceph-mon[23158]: pgmap v4227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7630.023906 secs 2022-03-21T16:18:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7630.023906 secs 2022-03-21T16:18:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7630.023906 secs 2022-03-21T16:18:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:48 smithi002 ceph-mon[17167]: pgmap v4228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:48 smithi016 ceph-mon[22010]: pgmap v4228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:48 smithi079 ceph-mon[23158]: pgmap v4228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:50 smithi002 ceph-mon[17167]: pgmap v4229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:50 smithi016 ceph-mon[22010]: pgmap v4229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:50 smithi079 ceph-mon[23158]: pgmap v4229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:52 smithi002 ceph-mon[17167]: pgmap v4230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7635.024015 secs 2022-03-21T16:18:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:52 smithi016 ceph-mon[22010]: pgmap v4230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7635.024015 secs 2022-03-21T16:18:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:52 smithi079 ceph-mon[23158]: pgmap v4230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:18:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7635.024015 secs 2022-03-21T16:18:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:54 smithi002 ceph-mon[17167]: pgmap v4231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:54 smithi016 ceph-mon[22010]: pgmap v4231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:54 smithi079 ceph-mon[23158]: pgmap v4231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:56 smithi002 ceph-mon[17167]: pgmap v4232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:56 smithi016 ceph-mon[22010]: pgmap v4232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:56 smithi079 ceph-mon[23158]: pgmap v4232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:18:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7640.024136 secs 2022-03-21T16:18:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7640.024136 secs 2022-03-21T16:18:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7640.024136 secs 2022-03-21T16:18:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:18:58 smithi002 ceph-mon[17167]: pgmap v4233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:18:58 smithi016 ceph-mon[22010]: pgmap v4233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:18:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:18:58 smithi079 ceph-mon[23158]: pgmap v4233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:00 smithi002 ceph-mon[17167]: pgmap v4234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:00 smithi016 ceph-mon[22010]: pgmap v4234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:00 smithi079 ceph-mon[23158]: pgmap v4234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:02 smithi002 ceph-mon[17167]: pgmap v4235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:02.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7645.024270 secs 2022-03-21T16:19:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:02 smithi016 ceph-mon[22010]: pgmap v4235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7645.024270 secs 2022-03-21T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:02 smithi079 ceph-mon[23158]: pgmap v4235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7645.024270 secs 2022-03-21T16:19:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:04 smithi002 ceph-mon[17167]: pgmap v4236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:04 smithi016 ceph-mon[22010]: pgmap v4236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:04 smithi079 ceph-mon[23158]: pgmap v4236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:06 smithi002 ceph-mon[17167]: pgmap v4237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:06 smithi016 ceph-mon[22010]: pgmap v4237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:06 smithi079 ceph-mon[23158]: pgmap v4237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7650.024394 secs 2022-03-21T16:19:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7650.024394 secs 2022-03-21T16:19:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7650.024394 secs 2022-03-21T16:19:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:08 smithi002 ceph-mon[17167]: pgmap v4238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:08 smithi016 ceph-mon[22010]: pgmap v4238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:08 smithi079 ceph-mon[23158]: pgmap v4238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:10 smithi002 ceph-mon[17167]: pgmap v4239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:10 smithi016 ceph-mon[22010]: pgmap v4239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:10 smithi079 ceph-mon[23158]: pgmap v4239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:12 smithi002 ceph-mon[17167]: pgmap v4240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7655.024531 secs 2022-03-21T16:19:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:12 smithi016 ceph-mon[22010]: pgmap v4240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7655.024531 secs 2022-03-21T16:19:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:12 smithi079 ceph-mon[23158]: pgmap v4240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7655.024531 secs 2022-03-21T16:19:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:14 smithi002 ceph-mon[17167]: pgmap v4241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:14 smithi016 ceph-mon[22010]: pgmap v4241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:14 smithi079 ceph-mon[23158]: pgmap v4241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:16 smithi002 ceph-mon[17167]: pgmap v4242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:16 smithi016 ceph-mon[22010]: pgmap v4242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:16 smithi079 ceph-mon[23158]: pgmap v4242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7660.024651 secs 2022-03-21T16:19:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7660.024651 secs 2022-03-21T16:19:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7660.024651 secs 2022-03-21T16:19:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:18 smithi016 ceph-mon[22010]: pgmap v4243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:18 smithi079 ceph-mon[23158]: pgmap v4243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:18 smithi002 ceph-mon[17167]: pgmap v4243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:20 smithi016 ceph-mon[22010]: pgmap v4244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:20 smithi079 ceph-mon[23158]: pgmap v4244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:20 smithi002 ceph-mon[17167]: pgmap v4244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:19:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:19:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:19:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:19:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:19:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:19:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:22 smithi016 ceph-mon[22010]: pgmap v4245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7665.024769 secs 2022-03-21T16:19:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:22 smithi079 ceph-mon[23158]: pgmap v4245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7665.024769 secs 2022-03-21T16:19:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:22 smithi002 ceph-mon[17167]: pgmap v4245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7665.024769 secs 2022-03-21T16:19:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:24 smithi016 ceph-mon[22010]: pgmap v4246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:24 smithi079 ceph-mon[23158]: pgmap v4246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:24 smithi002 ceph-mon[17167]: pgmap v4246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:26 smithi016 ceph-mon[22010]: pgmap v4247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:19:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:19:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:19:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:26 smithi079 ceph-mon[23158]: pgmap v4247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:19:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:19:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:19:27.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:26 smithi002 ceph-mon[17167]: pgmap v4247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:27.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:19:27.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:19:27.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:19:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:19:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:19:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:19:27.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:19:27.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:19:27.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7670.024898 secs 2022-03-21T16:19:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:19:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:19:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:19:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:19:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:19:27.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7670.024898 secs 2022-03-21T16:19:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:19:28.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:19:28.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:19:28.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:19:28.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:19:28.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:19:28.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7670.024898 secs 2022-03-21T16:19:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:28 smithi016 ceph-mon[22010]: pgmap v4248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:28 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:19:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:28 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:19:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:28 smithi079 ceph-mon[23158]: pgmap v4248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:28 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:19:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:28 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:19:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:28 smithi002 ceph-mon[17167]: pgmap v4248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:29.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:28 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:19:29.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:28 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:19:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:30 smithi016 ceph-mon[22010]: pgmap v4249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:30 smithi079 ceph-mon[23158]: pgmap v4249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:30 smithi002 ceph-mon[17167]: pgmap v4249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:32 smithi016 ceph-mon[22010]: pgmap v4250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7675.024953 secs 2022-03-21T16:19:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:32 smithi079 ceph-mon[23158]: pgmap v4250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7675.024953 secs 2022-03-21T16:19:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:32 smithi002 ceph-mon[17167]: pgmap v4250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7675.024953 secs 2022-03-21T16:19:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:34 smithi016 ceph-mon[22010]: pgmap v4251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:34 smithi079 ceph-mon[23158]: pgmap v4251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:34 smithi002 ceph-mon[17167]: pgmap v4251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:36 smithi016 ceph-mon[22010]: pgmap v4252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:36 smithi079 ceph-mon[23158]: pgmap v4252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:36 smithi002 ceph-mon[17167]: pgmap v4252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7680.025071 secs 2022-03-21T16:19:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7680.025071 secs 2022-03-21T16:19:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7680.025071 secs 2022-03-21T16:19:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:38 smithi079 ceph-mon[23158]: pgmap v4253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:38 smithi002 ceph-mon[17167]: pgmap v4253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:38 smithi016 ceph-mon[22010]: pgmap v4253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:40 smithi079 ceph-mon[23158]: pgmap v4254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:40 smithi002 ceph-mon[17167]: pgmap v4254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:40 smithi016 ceph-mon[22010]: pgmap v4254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:42 smithi079 ceph-mon[23158]: pgmap v4255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7685.025207 secs 2022-03-21T16:19:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:42 smithi002 ceph-mon[17167]: pgmap v4255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7685.025207 secs 2022-03-21T16:19:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:42 smithi016 ceph-mon[22010]: pgmap v4255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7685.025207 secs 2022-03-21T16:19:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:44 smithi079 ceph-mon[23158]: pgmap v4256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:44 smithi002 ceph-mon[17167]: pgmap v4256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:44 smithi016 ceph-mon[22010]: pgmap v4256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:46 smithi079 ceph-mon[23158]: pgmap v4257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:46 smithi002 ceph-mon[17167]: pgmap v4257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:46 smithi016 ceph-mon[22010]: pgmap v4257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7690.025339 secs 2022-03-21T16:19:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7690.025339 secs 2022-03-21T16:19:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7690.025339 secs 2022-03-21T16:19:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:48 smithi079 ceph-mon[23158]: pgmap v4258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:48 smithi002 ceph-mon[17167]: pgmap v4258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:48 smithi016 ceph-mon[22010]: pgmap v4258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:50 smithi079 ceph-mon[23158]: pgmap v4259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:50 smithi002 ceph-mon[17167]: pgmap v4259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:50 smithi016 ceph-mon[22010]: pgmap v4259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:52 smithi079 ceph-mon[23158]: pgmap v4260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7695.025473 secs 2022-03-21T16:19:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:52 smithi002 ceph-mon[17167]: pgmap v4260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7695.025473 secs 2022-03-21T16:19:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:52 smithi016 ceph-mon[22010]: pgmap v4260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7695.025473 secs 2022-03-21T16:19:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:54 smithi079 ceph-mon[23158]: pgmap v4261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:54 smithi002 ceph-mon[17167]: pgmap v4261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:54 smithi016 ceph-mon[22010]: pgmap v4261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:56 smithi079 ceph-mon[23158]: pgmap v4262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:56 smithi002 ceph-mon[17167]: pgmap v4262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:56 smithi016 ceph-mon[22010]: pgmap v4262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:19:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7700.025628 secs 2022-03-21T16:19:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7700.025628 secs 2022-03-21T16:19:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7700.025628 secs 2022-03-21T16:19:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:19:58 smithi079 ceph-mon[23158]: pgmap v4263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:19:58 smithi002 ceph-mon[17167]: pgmap v4263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:19:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:19:58 smithi016 ceph-mon[22010]: pgmap v4263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:00 smithi079 ceph-mon[23158]: pgmap v4264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:20:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:00 smithi002 ceph-mon[17167]: pgmap v4264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:20:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:00 smithi016 ceph-mon[22010]: pgmap v4264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:20:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:02 smithi079 ceph-mon[23158]: pgmap v4265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7705.025713 secs 2022-03-21T16:20:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:02 smithi002 ceph-mon[17167]: pgmap v4265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7705.025713 secs 2022-03-21T16:20:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:02 smithi016 ceph-mon[22010]: pgmap v4265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7705.025713 secs 2022-03-21T16:20:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:04 smithi002 ceph-mon[17167]: pgmap v4266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:04 smithi016 ceph-mon[22010]: pgmap v4266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:04 smithi079 ceph-mon[23158]: pgmap v4266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:06 smithi002 ceph-mon[17167]: pgmap v4267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:06 smithi016 ceph-mon[22010]: pgmap v4267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:06 smithi079 ceph-mon[23158]: pgmap v4267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7710.025850 secs 2022-03-21T16:20:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7710.025850 secs 2022-03-21T16:20:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7710.025850 secs 2022-03-21T16:20:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:08 smithi002 ceph-mon[17167]: pgmap v4268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:08 smithi016 ceph-mon[22010]: pgmap v4268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:08 smithi079 ceph-mon[23158]: pgmap v4268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:10 smithi002 ceph-mon[17167]: pgmap v4269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:10 smithi016 ceph-mon[22010]: pgmap v4269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:10 smithi079 ceph-mon[23158]: pgmap v4269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:12 smithi002 ceph-mon[17167]: pgmap v4270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7715.025974 secs 2022-03-21T16:20:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:12 smithi016 ceph-mon[22010]: pgmap v4270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7715.025974 secs 2022-03-21T16:20:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:12 smithi079 ceph-mon[23158]: pgmap v4270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7715.025974 secs 2022-03-21T16:20:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:14 smithi002 ceph-mon[17167]: pgmap v4271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:14 smithi016 ceph-mon[22010]: pgmap v4271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:14 smithi079 ceph-mon[23158]: pgmap v4271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:16 smithi002 ceph-mon[17167]: pgmap v4272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:16 smithi016 ceph-mon[22010]: pgmap v4272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:16 smithi079 ceph-mon[23158]: pgmap v4272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7720.026038 secs 2022-03-21T16:20:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7720.026038 secs 2022-03-21T16:20:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7720.026038 secs 2022-03-21T16:20:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:18 smithi002 ceph-mon[17167]: pgmap v4273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:18 smithi016 ceph-mon[22010]: pgmap v4273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:18 smithi079 ceph-mon[23158]: pgmap v4273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:20 smithi002 ceph-mon[17167]: pgmap v4274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:20 smithi016 ceph-mon[22010]: pgmap v4274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:20 smithi079 ceph-mon[23158]: pgmap v4274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:20:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:20:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:20:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:20:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:20:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:20:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:22 smithi002 ceph-mon[17167]: pgmap v4275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7725.026182 secs 2022-03-21T16:20:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:22 smithi016 ceph-mon[22010]: pgmap v4275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7725.026182 secs 2022-03-21T16:20:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:22 smithi079 ceph-mon[23158]: pgmap v4275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7725.026182 secs 2022-03-21T16:20:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:24 smithi002 ceph-mon[17167]: pgmap v4276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:24 smithi016 ceph-mon[22010]: pgmap v4276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:24 smithi079 ceph-mon[23158]: pgmap v4276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:27.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:26 smithi002 ceph-mon[17167]: pgmap v4277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:27.121 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:26 smithi079 ceph-mon[23158]: pgmap v4277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:27.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:26 smithi016 ceph-mon[22010]: pgmap v4277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:20:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:20:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:20:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7730.026293 secs 2022-03-21T16:20:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:20:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:20:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:20:28.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7730.026293 secs 2022-03-21T16:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:20:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:20:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7730.026293 secs 2022-03-21T16:20:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:28 smithi002 ceph-mon[17167]: pgmap v4278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:20:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:28 smithi016 ceph-mon[22010]: pgmap v4278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:20:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:28 smithi079 ceph-mon[23158]: pgmap v4278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:20:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:30 smithi002 ceph-mon[17167]: pgmap v4279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:30 smithi016 ceph-mon[22010]: pgmap v4279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:30 smithi079 ceph-mon[23158]: pgmap v4279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:32 smithi002 ceph-mon[17167]: pgmap v4280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7735.026438 secs 2022-03-21T16:20:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:32 smithi016 ceph-mon[22010]: pgmap v4280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7735.026438 secs 2022-03-21T16:20:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:32 smithi079 ceph-mon[23158]: pgmap v4280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7735.026438 secs 2022-03-21T16:20:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:34 smithi002 ceph-mon[17167]: pgmap v4281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:34 smithi016 ceph-mon[22010]: pgmap v4281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:34 smithi079 ceph-mon[23158]: pgmap v4281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:36 smithi002 ceph-mon[17167]: pgmap v4282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:36 smithi016 ceph-mon[22010]: pgmap v4282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:36 smithi079 ceph-mon[23158]: pgmap v4282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7740.026571 secs 2022-03-21T16:20:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7740.026571 secs 2022-03-21T16:20:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7740.026571 secs 2022-03-21T16:20:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:38 smithi002 ceph-mon[17167]: pgmap v4283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:38 smithi016 ceph-mon[22010]: pgmap v4283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:38 smithi079 ceph-mon[23158]: pgmap v4283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:40 smithi002 ceph-mon[17167]: pgmap v4284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:40 smithi016 ceph-mon[22010]: pgmap v4284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:40 smithi079 ceph-mon[23158]: pgmap v4284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:42 smithi002 ceph-mon[17167]: pgmap v4285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7745.026622 secs 2022-03-21T16:20:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:42 smithi016 ceph-mon[22010]: pgmap v4285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7745.026622 secs 2022-03-21T16:20:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:42 smithi079 ceph-mon[23158]: pgmap v4285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7745.026622 secs 2022-03-21T16:20:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:44 smithi002 ceph-mon[17167]: pgmap v4286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:44 smithi016 ceph-mon[22010]: pgmap v4286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:44 smithi079 ceph-mon[23158]: pgmap v4286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:46 smithi002 ceph-mon[17167]: pgmap v4287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:46 smithi016 ceph-mon[22010]: pgmap v4287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:46 smithi079 ceph-mon[23158]: pgmap v4287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7750.026713 secs 2022-03-21T16:20:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7750.026713 secs 2022-03-21T16:20:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7750.026713 secs 2022-03-21T16:20:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:48 smithi002 ceph-mon[17167]: pgmap v4288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:48 smithi016 ceph-mon[22010]: pgmap v4288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:48 smithi079 ceph-mon[23158]: pgmap v4288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:50 smithi002 ceph-mon[17167]: pgmap v4289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:50 smithi016 ceph-mon[22010]: pgmap v4289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:50 smithi079 ceph-mon[23158]: pgmap v4289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:52 smithi002 ceph-mon[17167]: pgmap v4290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7755.026831 secs 2022-03-21T16:20:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:52 smithi016 ceph-mon[22010]: pgmap v4290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7755.026831 secs 2022-03-21T16:20:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:52 smithi079 ceph-mon[23158]: pgmap v4290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7755.026831 secs 2022-03-21T16:20:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:54 smithi002 ceph-mon[17167]: pgmap v4291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:54 smithi016 ceph-mon[22010]: pgmap v4291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:54 smithi079 ceph-mon[23158]: pgmap v4291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:56 smithi002 ceph-mon[17167]: pgmap v4292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:56 smithi016 ceph-mon[22010]: pgmap v4292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:56 smithi079 ceph-mon[23158]: pgmap v4292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:20:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7760.026960 secs 2022-03-21T16:20:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7760.026960 secs 2022-03-21T16:20:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7760.026960 secs 2022-03-21T16:20:59.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:20:58 smithi002 ceph-mon[17167]: pgmap v4293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:20:58 smithi016 ceph-mon[22010]: pgmap v4293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:20:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:20:58 smithi079 ceph-mon[23158]: pgmap v4293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:00 smithi002 ceph-mon[17167]: pgmap v4294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:00 smithi016 ceph-mon[22010]: pgmap v4294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:00 smithi079 ceph-mon[23158]: pgmap v4294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:02 smithi016 ceph-mon[22010]: pgmap v4295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7765.027114 secs 2022-03-21T16:21:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:02 smithi079 ceph-mon[23158]: pgmap v4295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7765.027114 secs 2022-03-21T16:21:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:02 smithi002 ceph-mon[17167]: pgmap v4295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7765.027114 secs 2022-03-21T16:21:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:04 smithi016 ceph-mon[22010]: pgmap v4296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:04 smithi079 ceph-mon[23158]: pgmap v4296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:04 smithi002 ceph-mon[17167]: pgmap v4296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:06 smithi016 ceph-mon[22010]: pgmap v4297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:06 smithi079 ceph-mon[23158]: pgmap v4297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:06 smithi002 ceph-mon[17167]: pgmap v4297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7770.027236 secs 2022-03-21T16:21:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7770.027236 secs 2022-03-21T16:21:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7770.027236 secs 2022-03-21T16:21:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:08 smithi016 ceph-mon[22010]: pgmap v4298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:08 smithi079 ceph-mon[23158]: pgmap v4298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:08 smithi002 ceph-mon[17167]: pgmap v4298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:10 smithi016 ceph-mon[22010]: pgmap v4299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:10 smithi079 ceph-mon[23158]: pgmap v4299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:10 smithi002 ceph-mon[17167]: pgmap v4299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:12 smithi016 ceph-mon[22010]: pgmap v4300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7775.027348 secs 2022-03-21T16:21:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:12 smithi079 ceph-mon[23158]: pgmap v4300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7775.027348 secs 2022-03-21T16:21:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:12 smithi002 ceph-mon[17167]: pgmap v4300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7775.027348 secs 2022-03-21T16:21:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:14 smithi016 ceph-mon[22010]: pgmap v4301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:14 smithi079 ceph-mon[23158]: pgmap v4301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:14 smithi002 ceph-mon[17167]: pgmap v4301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:16 smithi016 ceph-mon[22010]: pgmap v4302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:16 smithi079 ceph-mon[23158]: pgmap v4302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:16 smithi002 ceph-mon[17167]: pgmap v4302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7780.027417 secs 2022-03-21T16:21:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7780.027417 secs 2022-03-21T16:21:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7780.027417 secs 2022-03-21T16:21:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:18 smithi016 ceph-mon[22010]: pgmap v4303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:18 smithi079 ceph-mon[23158]: pgmap v4303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:18 smithi002 ceph-mon[17167]: pgmap v4303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:19 smithi016 ceph-mon[22010]: pgmap v4304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:19 smithi079 ceph-mon[23158]: pgmap v4304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:19 smithi002 ceph-mon[17167]: pgmap v4304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:22 smithi002 ceph-mon[17167]: pgmap v4305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:21:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:21:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:22 smithi016 ceph-mon[22010]: pgmap v4305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:21:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:21:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:22 smithi079 ceph-mon[23158]: pgmap v4305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:21:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:21:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7785.027540 secs 2022-03-21T16:21:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7785.027540 secs 2022-03-21T16:21:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7785.027540 secs 2022-03-21T16:21:24.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:24 smithi002 ceph-mon[17167]: pgmap v4306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:24 smithi016 ceph-mon[22010]: pgmap v4306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:24 smithi079 ceph-mon[23158]: pgmap v4306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:26.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:26 smithi002 ceph-mon[17167]: pgmap v4307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:26 smithi016 ceph-mon[22010]: pgmap v4307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:26 smithi079 ceph-mon[23158]: pgmap v4307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7790.027656 secs 2022-03-21T16:21:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7790.027656 secs 2022-03-21T16:21:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7790.027656 secs 2022-03-21T16:21:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:28 smithi079 ceph-mon[23158]: pgmap v4308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:21:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:21:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:21:28.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:28 smithi002 ceph-mon[17167]: pgmap v4308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:28.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:21:28.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:21:28.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:21:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:28 smithi016 ceph-mon[22010]: pgmap v4308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:21:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:21:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:21:30.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:21:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:21:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:21:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:30 smithi002 ceph-mon[17167]: pgmap v4309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:30 smithi016 ceph-mon[22010]: pgmap v4309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:30 smithi079 ceph-mon[23158]: pgmap v4309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:32 smithi002 ceph-mon[17167]: pgmap v4310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7795.027740 secs 2022-03-21T16:21:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:32 smithi016 ceph-mon[22010]: pgmap v4310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7795.027740 secs 2022-03-21T16:21:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:32 smithi079 ceph-mon[23158]: pgmap v4310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7795.027740 secs 2022-03-21T16:21:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:34 smithi002 ceph-mon[17167]: pgmap v4311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:34 smithi016 ceph-mon[22010]: pgmap v4311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:34 smithi079 ceph-mon[23158]: pgmap v4311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:36 smithi002 ceph-mon[17167]: pgmap v4312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:36 smithi016 ceph-mon[22010]: pgmap v4312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:36 smithi079 ceph-mon[23158]: pgmap v4312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7800.027871 secs 2022-03-21T16:21:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7800.027871 secs 2022-03-21T16:21:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7800.027871 secs 2022-03-21T16:21:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:38 smithi002 ceph-mon[17167]: pgmap v4313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:38 smithi016 ceph-mon[22010]: pgmap v4313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:38 smithi079 ceph-mon[23158]: pgmap v4313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:41.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:40 smithi002 ceph-mon[17167]: pgmap v4314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:40 smithi016 ceph-mon[22010]: pgmap v4314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:40 smithi079 ceph-mon[23158]: pgmap v4314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:42 smithi002 ceph-mon[17167]: pgmap v4315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7805.027998 secs 2022-03-21T16:21:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:42 smithi016 ceph-mon[22010]: pgmap v4315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7805.027998 secs 2022-03-21T16:21:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:42 smithi079 ceph-mon[23158]: pgmap v4315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7805.027998 secs 2022-03-21T16:21:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:44 smithi002 ceph-mon[17167]: pgmap v4316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:44 smithi016 ceph-mon[22010]: pgmap v4316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:44 smithi079 ceph-mon[23158]: pgmap v4316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:46 smithi002 ceph-mon[17167]: pgmap v4317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:46 smithi016 ceph-mon[22010]: pgmap v4317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:46 smithi079 ceph-mon[23158]: pgmap v4317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7810.028113 secs 2022-03-21T16:21:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7810.028113 secs 2022-03-21T16:21:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7810.028113 secs 2022-03-21T16:21:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:48 smithi002 ceph-mon[17167]: pgmap v4318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:48 smithi016 ceph-mon[22010]: pgmap v4318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:48 smithi079 ceph-mon[23158]: pgmap v4318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:50 smithi016 ceph-mon[22010]: pgmap v4319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:50 smithi079 ceph-mon[23158]: pgmap v4319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:50 smithi002 ceph-mon[17167]: pgmap v4319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:52 smithi016 ceph-mon[22010]: pgmap v4320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7815.028241 secs 2022-03-21T16:21:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:52 smithi079 ceph-mon[23158]: pgmap v4320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7815.028241 secs 2022-03-21T16:21:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:52 smithi002 ceph-mon[17167]: pgmap v4320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7815.028241 secs 2022-03-21T16:21:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:54 smithi016 ceph-mon[22010]: pgmap v4321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:54 smithi079 ceph-mon[23158]: pgmap v4321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:54 smithi002 ceph-mon[17167]: pgmap v4321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:56 smithi016 ceph-mon[22010]: pgmap v4322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:56 smithi079 ceph-mon[23158]: pgmap v4322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:56 smithi002 ceph-mon[17167]: pgmap v4322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:21:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7820.028306 secs 2022-03-21T16:21:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7820.028306 secs 2022-03-21T16:21:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7820.028306 secs 2022-03-21T16:21:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:21:58 smithi016 ceph-mon[22010]: pgmap v4323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:21:58 smithi079 ceph-mon[23158]: pgmap v4323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:21:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:21:58 smithi002 ceph-mon[17167]: pgmap v4323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:00 smithi016 ceph-mon[22010]: pgmap v4324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:00 smithi079 ceph-mon[23158]: pgmap v4324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:00 smithi002 ceph-mon[17167]: pgmap v4324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:02 smithi016 ceph-mon[22010]: pgmap v4325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7825.028423 secs 2022-03-21T16:22:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:02 smithi079 ceph-mon[23158]: pgmap v4325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7825.028423 secs 2022-03-21T16:22:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:02 smithi002 ceph-mon[17167]: pgmap v4325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7825.028423 secs 2022-03-21T16:22:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:04 smithi016 ceph-mon[22010]: pgmap v4326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:04 smithi079 ceph-mon[23158]: pgmap v4326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:04 smithi002 ceph-mon[17167]: pgmap v4326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:06 smithi016 ceph-mon[22010]: pgmap v4327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:06 smithi079 ceph-mon[23158]: pgmap v4327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:06 smithi002 ceph-mon[17167]: pgmap v4327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7830.028491 secs 2022-03-21T16:22:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7830.028491 secs 2022-03-21T16:22:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7830.028491 secs 2022-03-21T16:22:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:08 smithi016 ceph-mon[22010]: pgmap v4328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:08 smithi079 ceph-mon[23158]: pgmap v4328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:08 smithi002 ceph-mon[17167]: pgmap v4328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:10 smithi016 ceph-mon[22010]: pgmap v4329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:10 smithi079 ceph-mon[23158]: pgmap v4329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:10 smithi002 ceph-mon[17167]: pgmap v4329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:12 smithi079 ceph-mon[23158]: pgmap v4330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7835.028590 secs 2022-03-21T16:22:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:12 smithi002 ceph-mon[17167]: pgmap v4330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7835.028590 secs 2022-03-21T16:22:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:12 smithi016 ceph-mon[22010]: pgmap v4330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7835.028590 secs 2022-03-21T16:22:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:13 smithi079 ceph-mon[23158]: pgmap v4331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:13 smithi002 ceph-mon[17167]: pgmap v4331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:13 smithi016 ceph-mon[22010]: pgmap v4331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:16.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:16 smithi002 ceph-mon[17167]: pgmap v4332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:16 smithi016 ceph-mon[22010]: pgmap v4332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:16 smithi079 ceph-mon[23158]: pgmap v4332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7840.028682 secs 2022-03-21T16:22:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7840.028682 secs 2022-03-21T16:22:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7840.028682 secs 2022-03-21T16:22:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:18 smithi079 ceph-mon[23158]: pgmap v4333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:18 smithi002 ceph-mon[17167]: pgmap v4333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:18 smithi016 ceph-mon[22010]: pgmap v4333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:20 smithi079 ceph-mon[23158]: pgmap v4334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:20 smithi002 ceph-mon[17167]: pgmap v4334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:20 smithi016 ceph-mon[22010]: pgmap v4334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:22:22.083 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:22:22.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:22:22.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:22:22.088 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:22:22.088 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:22:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:22 smithi079 ceph-mon[23158]: pgmap v4335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7845.028794 secs 2022-03-21T16:22:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:22 smithi002 ceph-mon[17167]: pgmap v4335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7845.028794 secs 2022-03-21T16:22:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:22 smithi016 ceph-mon[22010]: pgmap v4335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7845.028794 secs 2022-03-21T16:22:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:24 smithi079 ceph-mon[23158]: pgmap v4336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:24 smithi002 ceph-mon[17167]: pgmap v4336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:24 smithi016 ceph-mon[22010]: pgmap v4336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:26 smithi079 ceph-mon[23158]: pgmap v4337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:26 smithi002 ceph-mon[17167]: pgmap v4337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:26.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:26 smithi016 ceph-mon[22010]: pgmap v4337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7850.028916 secs 2022-03-21T16:22:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7850.028916 secs 2022-03-21T16:22:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7850.028916 secs 2022-03-21T16:22:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:28 smithi079 ceph-mon[23158]: pgmap v4338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:28 smithi002 ceph-mon[17167]: pgmap v4338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:28.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:28 smithi016 ceph-mon[22010]: pgmap v4338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:22:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:22:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:29 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:22:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:22:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:22:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:29 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:22:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:22:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:22:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:29 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:22:30.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:30 smithi002 ceph-mon[17167]: pgmap v4339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:30 smithi016 ceph-mon[22010]: pgmap v4339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:30 smithi079 ceph-mon[23158]: pgmap v4339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:32 smithi079 ceph-mon[23158]: pgmap v4340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7855.029040 secs 2022-03-21T16:22:32.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:32 smithi002 ceph-mon[17167]: pgmap v4340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:32.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.742 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7855.029040 secs 2022-03-21T16:22:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:32 smithi016 ceph-mon[22010]: pgmap v4340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7855.029040 secs 2022-03-21T16:22:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:22:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:34 smithi016 ceph-mon[22010]: pgmap v4341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:34 smithi079 ceph-mon[23158]: pgmap v4341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:34 smithi002 ceph-mon[17167]: pgmap v4341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:36 smithi016 ceph-mon[22010]: pgmap v4342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:36 smithi079 ceph-mon[23158]: pgmap v4342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:36 smithi002 ceph-mon[17167]: pgmap v4342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7860.029160 secs 2022-03-21T16:22:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7860.029160 secs 2022-03-21T16:22:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7860.029160 secs 2022-03-21T16:22:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:38 smithi016 ceph-mon[22010]: pgmap v4343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:38 smithi079 ceph-mon[23158]: pgmap v4343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:38 smithi002 ceph-mon[17167]: pgmap v4343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:40 smithi016 ceph-mon[22010]: pgmap v4344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:40 smithi079 ceph-mon[23158]: pgmap v4344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:40 smithi002 ceph-mon[17167]: pgmap v4344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:42 smithi016 ceph-mon[22010]: pgmap v4345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7865.029287 secs 2022-03-21T16:22:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:42 smithi079 ceph-mon[23158]: pgmap v4345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7865.029287 secs 2022-03-21T16:22:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:42 smithi002 ceph-mon[17167]: pgmap v4345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7865.029287 secs 2022-03-21T16:22:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:44 smithi016 ceph-mon[22010]: pgmap v4346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:44 smithi079 ceph-mon[23158]: pgmap v4346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:44 smithi002 ceph-mon[17167]: pgmap v4346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:46 smithi016 ceph-mon[22010]: pgmap v4347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:46 smithi079 ceph-mon[23158]: pgmap v4347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:46 smithi002 ceph-mon[17167]: pgmap v4347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7870.029428 secs 2022-03-21T16:22:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7870.029428 secs 2022-03-21T16:22:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7870.029428 secs 2022-03-21T16:22:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:48 smithi016 ceph-mon[22010]: pgmap v4348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:48 smithi079 ceph-mon[23158]: pgmap v4348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:48 smithi002 ceph-mon[17167]: pgmap v4348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:50 smithi016 ceph-mon[22010]: pgmap v4349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:50 smithi079 ceph-mon[23158]: pgmap v4349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:50 smithi002 ceph-mon[17167]: pgmap v4349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:52 smithi079 ceph-mon[23158]: pgmap v4350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7875.029556 secs 2022-03-21T16:22:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:52 smithi002 ceph-mon[17167]: pgmap v4350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7875.029556 secs 2022-03-21T16:22:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:52 smithi016 ceph-mon[22010]: pgmap v4350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7875.029556 secs 2022-03-21T16:22:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:54 smithi079 ceph-mon[23158]: pgmap v4351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:54 smithi002 ceph-mon[17167]: pgmap v4351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:54 smithi016 ceph-mon[22010]: pgmap v4351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:56 smithi079 ceph-mon[23158]: pgmap v4352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:56 smithi002 ceph-mon[17167]: pgmap v4352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:56 smithi016 ceph-mon[22010]: pgmap v4352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:22:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7880.029678 secs 2022-03-21T16:22:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7880.029678 secs 2022-03-21T16:22:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7880.029678 secs 2022-03-21T16:22:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:22:58 smithi079 ceph-mon[23158]: pgmap v4353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:22:58 smithi002 ceph-mon[17167]: pgmap v4353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:22:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:22:58 smithi016 ceph-mon[22010]: pgmap v4353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:00 smithi079 ceph-mon[23158]: pgmap v4354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:00 smithi002 ceph-mon[17167]: pgmap v4354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:00 smithi016 ceph-mon[22010]: pgmap v4354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:02 smithi079 ceph-mon[23158]: pgmap v4355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7885.029802 secs 2022-03-21T16:23:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:02 smithi002 ceph-mon[17167]: pgmap v4355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7885.029802 secs 2022-03-21T16:23:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:02 smithi016 ceph-mon[22010]: pgmap v4355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7885.029802 secs 2022-03-21T16:23:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:04 smithi079 ceph-mon[23158]: pgmap v4356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:04 smithi002 ceph-mon[17167]: pgmap v4356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:04 smithi016 ceph-mon[22010]: pgmap v4356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:06 smithi079 ceph-mon[23158]: pgmap v4357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:06 smithi002 ceph-mon[17167]: pgmap v4357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:06 smithi016 ceph-mon[22010]: pgmap v4357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7890.029866 secs 2022-03-21T16:23:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7890.029866 secs 2022-03-21T16:23:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7890.029866 secs 2022-03-21T16:23:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:08 smithi079 ceph-mon[23158]: pgmap v4358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:08 smithi002 ceph-mon[17167]: pgmap v4358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:08 smithi016 ceph-mon[22010]: pgmap v4358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:10 smithi079 ceph-mon[23158]: pgmap v4359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:10 smithi002 ceph-mon[17167]: pgmap v4359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:10 smithi016 ceph-mon[22010]: pgmap v4359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:12 smithi079 ceph-mon[23158]: pgmap v4360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7895.029979 secs 2022-03-21T16:23:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:12 smithi002 ceph-mon[17167]: pgmap v4360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7895.029979 secs 2022-03-21T16:23:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:12 smithi016 ceph-mon[22010]: pgmap v4360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7895.029979 secs 2022-03-21T16:23:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:14 smithi079 ceph-mon[23158]: pgmap v4361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:14 smithi002 ceph-mon[17167]: pgmap v4361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:14 smithi016 ceph-mon[22010]: pgmap v4361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:16 smithi079 ceph-mon[23158]: pgmap v4362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:16 smithi002 ceph-mon[17167]: pgmap v4362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:16 smithi016 ceph-mon[22010]: pgmap v4362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7900.030118 secs 2022-03-21T16:23:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7900.030118 secs 2022-03-21T16:23:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7900.030118 secs 2022-03-21T16:23:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:18 smithi002 ceph-mon[17167]: pgmap v4363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:18 smithi016 ceph-mon[22010]: pgmap v4363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:18 smithi079 ceph-mon[23158]: pgmap v4363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:20 smithi002 ceph-mon[17167]: pgmap v4364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:20 smithi016 ceph-mon[22010]: pgmap v4364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:20 smithi079 ceph-mon[23158]: pgmap v4364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:23:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:23:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:23:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:23:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:23:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:23:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:22 smithi002 ceph-mon[17167]: pgmap v4365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7905.030254 secs 2022-03-21T16:23:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:22 smithi016 ceph-mon[22010]: pgmap v4365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7905.030254 secs 2022-03-21T16:23:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:22 smithi079 ceph-mon[23158]: pgmap v4365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7905.030254 secs 2022-03-21T16:23:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:24 smithi002 ceph-mon[17167]: pgmap v4366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:24 smithi016 ceph-mon[22010]: pgmap v4366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:24 smithi079 ceph-mon[23158]: pgmap v4366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:26 smithi002 ceph-mon[17167]: pgmap v4367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:26 smithi016 ceph-mon[22010]: pgmap v4367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:26 smithi079 ceph-mon[23158]: pgmap v4367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7910.030393 secs 2022-03-21T16:23:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7910.030393 secs 2022-03-21T16:23:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7910.030393 secs 2022-03-21T16:23:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:28 smithi002 ceph-mon[17167]: pgmap v4368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:28 smithi016 ceph-mon[22010]: pgmap v4368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:28 smithi079 ceph-mon[23158]: pgmap v4368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:30 smithi002 ceph-mon[17167]: pgmap v4369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:30 smithi016 ceph-mon[22010]: pgmap v4369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:30 smithi079 ceph-mon[23158]: pgmap v4369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:32 smithi002 ceph-mon[17167]: pgmap v4370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7915.030566 secs 2022-03-21T16:23:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:32 smithi016 ceph-mon[22010]: pgmap v4370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7915.030566 secs 2022-03-21T16:23:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:32 smithi079 ceph-mon[23158]: pgmap v4370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7915.030566 secs 2022-03-21T16:23:33.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:23:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:23:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:23:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:23:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:23:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:23:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:23:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:23:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:23:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:34 smithi002 ceph-mon[17167]: pgmap v4371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:23:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:34 smithi016 ceph-mon[22010]: pgmap v4371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:23:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:34 smithi079 ceph-mon[23158]: pgmap v4371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:23:37.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:36 smithi002 ceph-mon[17167]: pgmap v4372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:37.208 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:36 smithi016 ceph-mon[22010]: pgmap v4372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:36 smithi079 ceph-mon[23158]: pgmap v4372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7920.030707 secs 2022-03-21T16:23:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7920.030707 secs 2022-03-21T16:23:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7920.030707 secs 2022-03-21T16:23:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:38 smithi002 ceph-mon[17167]: pgmap v4373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:38 smithi016 ceph-mon[22010]: pgmap v4373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:38 smithi079 ceph-mon[23158]: pgmap v4373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:40 smithi002 ceph-mon[17167]: pgmap v4374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:40 smithi016 ceph-mon[22010]: pgmap v4374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:40 smithi079 ceph-mon[23158]: pgmap v4374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:42 smithi002 ceph-mon[17167]: pgmap v4375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7925.030776 secs 2022-03-21T16:23:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:42 smithi016 ceph-mon[22010]: pgmap v4375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7925.030776 secs 2022-03-21T16:23:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:42 smithi079 ceph-mon[23158]: pgmap v4375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7925.030776 secs 2022-03-21T16:23:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:44 smithi002 ceph-mon[17167]: pgmap v4376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:44 smithi016 ceph-mon[22010]: pgmap v4376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:44 smithi079 ceph-mon[23158]: pgmap v4376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:46 smithi002 ceph-mon[17167]: pgmap v4377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:46 smithi016 ceph-mon[22010]: pgmap v4377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:46 smithi079 ceph-mon[23158]: pgmap v4377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7930.030886 secs 2022-03-21T16:23:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7930.030886 secs 2022-03-21T16:23:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7930.030886 secs 2022-03-21T16:23:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:48 smithi002 ceph-mon[17167]: pgmap v4378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:48 smithi016 ceph-mon[22010]: pgmap v4378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:48 smithi079 ceph-mon[23158]: pgmap v4378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:50 smithi002 ceph-mon[17167]: pgmap v4379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:50 smithi016 ceph-mon[22010]: pgmap v4379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:50 smithi079 ceph-mon[23158]: pgmap v4379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:52 smithi002 ceph-mon[17167]: pgmap v4380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7935.030980 secs 2022-03-21T16:23:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:52 smithi016 ceph-mon[22010]: pgmap v4380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7935.030980 secs 2022-03-21T16:23:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:52 smithi079 ceph-mon[23158]: pgmap v4380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:23:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7935.030980 secs 2022-03-21T16:23:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:54 smithi002 ceph-mon[17167]: pgmap v4381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:54 smithi016 ceph-mon[22010]: pgmap v4381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:54 smithi079 ceph-mon[23158]: pgmap v4381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:23:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:56 smithi002 ceph-mon[17167]: pgmap v4382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:56 smithi016 ceph-mon[22010]: pgmap v4382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:56 smithi079 ceph-mon[23158]: pgmap v4382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:23:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7940.031098 secs 2022-03-21T16:23:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7940.031098 secs 2022-03-21T16:23:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7940.031098 secs 2022-03-21T16:23:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:23:58 smithi002 ceph-mon[17167]: pgmap v4383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:23:58 smithi016 ceph-mon[22010]: pgmap v4383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:23:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:23:58 smithi079 ceph-mon[23158]: pgmap v4383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:00 smithi002 ceph-mon[17167]: pgmap v4384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:00 smithi016 ceph-mon[22010]: pgmap v4384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:00 smithi079 ceph-mon[23158]: pgmap v4384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:02 smithi002 ceph-mon[17167]: pgmap v4385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7945.031235 secs 2022-03-21T16:24:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:02 smithi016 ceph-mon[22010]: pgmap v4385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7945.031235 secs 2022-03-21T16:24:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:02 smithi079 ceph-mon[23158]: pgmap v4385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7945.031235 secs 2022-03-21T16:24:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:04 smithi002 ceph-mon[17167]: pgmap v4386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:04 smithi016 ceph-mon[22010]: pgmap v4386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:04 smithi079 ceph-mon[23158]: pgmap v4386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:06 smithi002 ceph-mon[17167]: pgmap v4387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:06 smithi016 ceph-mon[22010]: pgmap v4387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:06 smithi079 ceph-mon[23158]: pgmap v4387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7950.031372 secs 2022-03-21T16:24:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7950.031372 secs 2022-03-21T16:24:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7950.031372 secs 2022-03-21T16:24:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:08 smithi002 ceph-mon[17167]: pgmap v4388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:08 smithi016 ceph-mon[22010]: pgmap v4388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:08 smithi079 ceph-mon[23158]: pgmap v4388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:10 smithi002 ceph-mon[17167]: pgmap v4389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:10 smithi016 ceph-mon[22010]: pgmap v4389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:10 smithi079 ceph-mon[23158]: pgmap v4389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:12 smithi002 ceph-mon[17167]: pgmap v4390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7955.031495 secs 2022-03-21T16:24:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:12 smithi016 ceph-mon[22010]: pgmap v4390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7955.031495 secs 2022-03-21T16:24:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:12 smithi079 ceph-mon[23158]: pgmap v4390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7955.031495 secs 2022-03-21T16:24:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:14 smithi002 ceph-mon[17167]: pgmap v4391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:14 smithi016 ceph-mon[22010]: pgmap v4391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:14 smithi079 ceph-mon[23158]: pgmap v4391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:17.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:16 smithi002 ceph-mon[17167]: pgmap v4392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:17.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:16 smithi016 ceph-mon[22010]: pgmap v4392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:16 smithi079 ceph-mon[23158]: pgmap v4392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7960.031595 secs 2022-03-21T16:24:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7960.031595 secs 2022-03-21T16:24:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7960.031595 secs 2022-03-21T16:24:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:18 smithi016 ceph-mon[22010]: pgmap v4393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:18 smithi079 ceph-mon[23158]: pgmap v4393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:18 smithi002 ceph-mon[17167]: pgmap v4393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:20 smithi016 ceph-mon[22010]: pgmap v4394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:20 smithi079 ceph-mon[23158]: pgmap v4394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:20 smithi002 ceph-mon[17167]: pgmap v4394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:24:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:24:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:24:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:24:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:24:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:24:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:22 smithi016 ceph-mon[22010]: pgmap v4395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7965.031719 secs 2022-03-21T16:24:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:22 smithi079 ceph-mon[23158]: pgmap v4395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7965.031719 secs 2022-03-21T16:24:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:22 smithi002 ceph-mon[17167]: pgmap v4395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7965.031719 secs 2022-03-21T16:24:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:24 smithi016 ceph-mon[22010]: pgmap v4396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:24 smithi079 ceph-mon[23158]: pgmap v4396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:24 smithi002 ceph-mon[17167]: pgmap v4396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:26 smithi016 ceph-mon[22010]: pgmap v4397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:26 smithi079 ceph-mon[23158]: pgmap v4397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:26 smithi002 ceph-mon[17167]: pgmap v4397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7970.031829 secs 2022-03-21T16:24:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7970.031829 secs 2022-03-21T16:24:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7970.031829 secs 2022-03-21T16:24:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:28 smithi016 ceph-mon[22010]: pgmap v4398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:28 smithi079 ceph-mon[23158]: pgmap v4398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:28 smithi002 ceph-mon[17167]: pgmap v4398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:30 smithi016 ceph-mon[22010]: pgmap v4399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:30 smithi079 ceph-mon[23158]: pgmap v4399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:30 smithi002 ceph-mon[17167]: pgmap v4399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:32 smithi016 ceph-mon[22010]: pgmap v4400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7975.031975 secs 2022-03-21T16:24:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:32 smithi079 ceph-mon[23158]: pgmap v4400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7975.031975 secs 2022-03-21T16:24:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:32 smithi002 ceph-mon[17167]: pgmap v4400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7975.031975 secs 2022-03-21T16:24:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: pgmap v4401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:35.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: pgmap v4401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:35.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:24:35.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:24:35.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:24:35.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:35.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: pgmap v4401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:24:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:24:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:24:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:35.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:35 smithi002 ceph-mon[17167]: pgmap v4402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:36.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:35 smithi079 ceph-mon[23158]: pgmap v4402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:35 smithi016 ceph-mon[22010]: pgmap v4402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:24:37.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7980.032090 secs 2022-03-21T16:24:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7980.032090 secs 2022-03-21T16:24:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7980.032090 secs 2022-03-21T16:24:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:38 smithi002 ceph-mon[17167]: pgmap v4403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:38 smithi016 ceph-mon[22010]: pgmap v4403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:38 smithi079 ceph-mon[23158]: pgmap v4403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:40.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:40 smithi002 ceph-mon[17167]: pgmap v4404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:40 smithi016 ceph-mon[22010]: pgmap v4404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:40 smithi079 ceph-mon[23158]: pgmap v4404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:42 smithi002 ceph-mon[17167]: pgmap v4405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7985.032190 secs 2022-03-21T16:24:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:42 smithi016 ceph-mon[22010]: pgmap v4405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7985.032190 secs 2022-03-21T16:24:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:42 smithi079 ceph-mon[23158]: pgmap v4405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7985.032190 secs 2022-03-21T16:24:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:44 smithi002 ceph-mon[17167]: pgmap v4406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:44 smithi016 ceph-mon[22010]: pgmap v4406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:44 smithi079 ceph-mon[23158]: pgmap v4406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:46 smithi002 ceph-mon[17167]: pgmap v4407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:46 smithi016 ceph-mon[22010]: pgmap v4407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:46 smithi079 ceph-mon[23158]: pgmap v4407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:47.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7990.032324 secs 2022-03-21T16:24:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7990.032324 secs 2022-03-21T16:24:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7990.032324 secs 2022-03-21T16:24:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:48 smithi002 ceph-mon[17167]: pgmap v4408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:48 smithi016 ceph-mon[22010]: pgmap v4408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:48 smithi079 ceph-mon[23158]: pgmap v4408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:50 smithi002 ceph-mon[17167]: pgmap v4409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:50 smithi016 ceph-mon[22010]: pgmap v4409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:50 smithi079 ceph-mon[23158]: pgmap v4409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:52 smithi002 ceph-mon[17167]: pgmap v4410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 7995.032441 secs 2022-03-21T16:24:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:52 smithi016 ceph-mon[22010]: pgmap v4410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 7995.032441 secs 2022-03-21T16:24:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:52 smithi079 ceph-mon[23158]: pgmap v4410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:24:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 7995.032441 secs 2022-03-21T16:24:54.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:54 smithi002 ceph-mon[17167]: pgmap v4411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:54 smithi016 ceph-mon[22010]: pgmap v4411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:54 smithi079 ceph-mon[23158]: pgmap v4411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:24:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:56 smithi002 ceph-mon[17167]: pgmap v4412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:56 smithi016 ceph-mon[22010]: pgmap v4412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:56 smithi079 ceph-mon[23158]: pgmap v4412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:24:57.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8000.032588 secs 2022-03-21T16:24:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8000.032588 secs 2022-03-21T16:24:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8000.032588 secs 2022-03-21T16:24:58.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:24:58 smithi002 ceph-mon[17167]: pgmap v4413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:24:58 smithi016 ceph-mon[22010]: pgmap v4413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:24:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:24:58 smithi079 ceph-mon[23158]: pgmap v4413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:00.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:00 smithi002 ceph-mon[17167]: pgmap v4414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:00 smithi016 ceph-mon[22010]: pgmap v4414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:00 smithi079 ceph-mon[23158]: pgmap v4414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:02 smithi002 ceph-mon[17167]: pgmap v4415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8005.032713 secs 2022-03-21T16:25:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:02 smithi016 ceph-mon[22010]: pgmap v4415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8005.032713 secs 2022-03-21T16:25:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:02 smithi079 ceph-mon[23158]: pgmap v4415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8005.032713 secs 2022-03-21T16:25:04.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:04 smithi002 ceph-mon[17167]: pgmap v4416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:04 smithi016 ceph-mon[22010]: pgmap v4416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:04 smithi079 ceph-mon[23158]: pgmap v4416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:06 smithi002 ceph-mon[17167]: pgmap v4417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:06 smithi016 ceph-mon[22010]: pgmap v4417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:06 smithi079 ceph-mon[23158]: pgmap v4417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:07.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8010.032828 secs 2022-03-21T16:25:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8010.032828 secs 2022-03-21T16:25:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8010.032828 secs 2022-03-21T16:25:08.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:08 smithi002 ceph-mon[17167]: pgmap v4418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:08 smithi016 ceph-mon[22010]: pgmap v4418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:08 smithi079 ceph-mon[23158]: pgmap v4418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:10 smithi002 ceph-mon[17167]: pgmap v4419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:10 smithi016 ceph-mon[22010]: pgmap v4419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:10 smithi079 ceph-mon[23158]: pgmap v4419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:12 smithi002 ceph-mon[17167]: pgmap v4420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:12.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8015.032945 secs 2022-03-21T16:25:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:12 smithi016 ceph-mon[22010]: pgmap v4420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:12.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8015.032945 secs 2022-03-21T16:25:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:12 smithi079 ceph-mon[23158]: pgmap v4420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8015.032945 secs 2022-03-21T16:25:14.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:14 smithi002 ceph-mon[17167]: pgmap v4421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:14 smithi016 ceph-mon[22010]: pgmap v4421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:14 smithi079 ceph-mon[23158]: pgmap v4421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:16 smithi002 ceph-mon[17167]: pgmap v4422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:16 smithi016 ceph-mon[22010]: pgmap v4422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:16 smithi079 ceph-mon[23158]: pgmap v4422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8020.033070 secs 2022-03-21T16:25:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8020.033070 secs 2022-03-21T16:25:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8020.033070 secs 2022-03-21T16:25:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:18 smithi002 ceph-mon[17167]: pgmap v4423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:18 smithi016 ceph-mon[22010]: pgmap v4423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:18 smithi079 ceph-mon[23158]: pgmap v4423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:20 smithi002 ceph-mon[17167]: pgmap v4424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:20 smithi016 ceph-mon[22010]: pgmap v4424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:20 smithi079 ceph-mon[23158]: pgmap v4424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:25:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:25:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:25:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:25:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:25:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:25:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:22 smithi002 ceph-mon[17167]: pgmap v4425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8025.033198 secs 2022-03-21T16:25:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:22 smithi016 ceph-mon[22010]: pgmap v4425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8025.033198 secs 2022-03-21T16:25:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:22 smithi079 ceph-mon[23158]: pgmap v4425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8025.033198 secs 2022-03-21T16:25:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:24 smithi002 ceph-mon[17167]: pgmap v4426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:24 smithi016 ceph-mon[22010]: pgmap v4426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:24 smithi079 ceph-mon[23158]: pgmap v4426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:26 smithi002 ceph-mon[17167]: pgmap v4427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:26 smithi016 ceph-mon[22010]: pgmap v4427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:26 smithi079 ceph-mon[23158]: pgmap v4427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8030.033322 secs 2022-03-21T16:25:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8030.033322 secs 2022-03-21T16:25:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8030.033322 secs 2022-03-21T16:25:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:28 smithi002 ceph-mon[17167]: pgmap v4428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:28 smithi016 ceph-mon[22010]: pgmap v4428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:28 smithi079 ceph-mon[23158]: pgmap v4428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:30 smithi016 ceph-mon[22010]: pgmap v4429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:30 smithi079 ceph-mon[23158]: pgmap v4429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:30 smithi002 ceph-mon[17167]: pgmap v4429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:32 smithi016 ceph-mon[22010]: pgmap v4430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8035.033395 secs 2022-03-21T16:25:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:32 smithi079 ceph-mon[23158]: pgmap v4430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8035.033395 secs 2022-03-21T16:25:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:32 smithi002 ceph-mon[17167]: pgmap v4430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8035.033395 secs 2022-03-21T16:25:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:34 smithi016 ceph-mon[22010]: pgmap v4431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:34 smithi079 ceph-mon[23158]: pgmap v4431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:34 smithi002 ceph-mon[17167]: pgmap v4431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:36 smithi016 ceph-mon[22010]: pgmap v4432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:25:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:25:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:25:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:36 smithi079 ceph-mon[23158]: pgmap v4432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:25:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:25:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:25:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:36 smithi002 ceph-mon[17167]: pgmap v4432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:25:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:25:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:25:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:25:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8040.033518 secs 2022-03-21T16:25:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:25:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8040.033518 secs 2022-03-21T16:25:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:25:38.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8040.033518 secs 2022-03-21T16:25:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:38 smithi016 ceph-mon[22010]: pgmap v4433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:38 smithi079 ceph-mon[23158]: pgmap v4433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:38 smithi002 ceph-mon[17167]: pgmap v4433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:40 smithi016 ceph-mon[22010]: pgmap v4434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:40 smithi079 ceph-mon[23158]: pgmap v4434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:40 smithi002 ceph-mon[17167]: pgmap v4434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:42 smithi016 ceph-mon[22010]: pgmap v4435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8045.033641 secs 2022-03-21T16:25:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:42 smithi079 ceph-mon[23158]: pgmap v4435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8045.033641 secs 2022-03-21T16:25:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:42 smithi002 ceph-mon[17167]: pgmap v4435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8045.033641 secs 2022-03-21T16:25:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:44 smithi016 ceph-mon[22010]: pgmap v4436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:44 smithi079 ceph-mon[23158]: pgmap v4436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:44 smithi002 ceph-mon[17167]: pgmap v4436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:46 smithi016 ceph-mon[22010]: pgmap v4437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:46 smithi079 ceph-mon[23158]: pgmap v4437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:46 smithi002 ceph-mon[17167]: pgmap v4437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8050.033782 secs 2022-03-21T16:25:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8050.033782 secs 2022-03-21T16:25:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8050.033782 secs 2022-03-21T16:25:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:48 smithi016 ceph-mon[22010]: pgmap v4438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:48 smithi079 ceph-mon[23158]: pgmap v4438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:48 smithi002 ceph-mon[17167]: pgmap v4438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:50 smithi079 ceph-mon[23158]: pgmap v4439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:50 smithi002 ceph-mon[17167]: pgmap v4439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:50 smithi016 ceph-mon[22010]: pgmap v4439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:52 smithi079 ceph-mon[23158]: pgmap v4440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8055.033934 secs 2022-03-21T16:25:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:52 smithi002 ceph-mon[17167]: pgmap v4440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8055.033934 secs 2022-03-21T16:25:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:52 smithi016 ceph-mon[22010]: pgmap v4440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:25:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8055.033934 secs 2022-03-21T16:25:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:54 smithi079 ceph-mon[23158]: pgmap v4441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:54 smithi002 ceph-mon[17167]: pgmap v4441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:54 smithi016 ceph-mon[22010]: pgmap v4441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:25:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:56 smithi079 ceph-mon[23158]: pgmap v4442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:56 smithi002 ceph-mon[17167]: pgmap v4442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:56 smithi016 ceph-mon[22010]: pgmap v4442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:25:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8060.034053 secs 2022-03-21T16:25:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8060.034053 secs 2022-03-21T16:25:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8060.034053 secs 2022-03-21T16:25:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:25:58 smithi079 ceph-mon[23158]: pgmap v4443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:25:58 smithi002 ceph-mon[17167]: pgmap v4443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:25:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:25:58 smithi016 ceph-mon[22010]: pgmap v4443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:00 smithi079 ceph-mon[23158]: pgmap v4444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:00 smithi002 ceph-mon[17167]: pgmap v4444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:00 smithi016 ceph-mon[22010]: pgmap v4444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:02 smithi079 ceph-mon[23158]: pgmap v4445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8065.034144 secs 2022-03-21T16:26:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:02 smithi002 ceph-mon[17167]: pgmap v4445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8065.034144 secs 2022-03-21T16:26:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:02 smithi016 ceph-mon[22010]: pgmap v4445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8065.034144 secs 2022-03-21T16:26:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:04 smithi079 ceph-mon[23158]: pgmap v4446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T16:26:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:04 smithi002 ceph-mon[17167]: pgmap v4446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T16:26:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:04 smithi016 ceph-mon[22010]: pgmap v4446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 4 op/s 2022-03-21T16:26:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:06 smithi079 ceph-mon[23158]: pgmap v4447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:06 smithi002 ceph-mon[17167]: pgmap v4447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:06 smithi016 ceph-mon[22010]: pgmap v4447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8070.034269 secs 2022-03-21T16:26:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8070.034269 secs 2022-03-21T16:26:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8070.034269 secs 2022-03-21T16:26:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:08 smithi079 ceph-mon[23158]: pgmap v4448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:08 smithi002 ceph-mon[17167]: pgmap v4448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:08 smithi016 ceph-mon[22010]: pgmap v4448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:10 smithi079 ceph-mon[23158]: pgmap v4449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:10 smithi002 ceph-mon[17167]: pgmap v4449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:10 smithi016 ceph-mon[22010]: pgmap v4449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:12 smithi079 ceph-mon[23158]: pgmap v4450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8075.034388 secs 2022-03-21T16:26:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:12 smithi002 ceph-mon[17167]: pgmap v4450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8075.034388 secs 2022-03-21T16:26:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:12 smithi016 ceph-mon[22010]: pgmap v4450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8075.034388 secs 2022-03-21T16:26:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:14 smithi002 ceph-mon[17167]: pgmap v4451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:14 smithi016 ceph-mon[22010]: pgmap v4451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:14 smithi079 ceph-mon[23158]: pgmap v4451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:16 smithi002 ceph-mon[17167]: pgmap v4452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:16 smithi016 ceph-mon[22010]: pgmap v4452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:16 smithi079 ceph-mon[23158]: pgmap v4452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:18.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8080.034457 secs 2022-03-21T16:26:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8080.034457 secs 2022-03-21T16:26:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8080.034457 secs 2022-03-21T16:26:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:18 smithi002 ceph-mon[17167]: pgmap v4453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:18 smithi016 ceph-mon[22010]: pgmap v4453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:18 smithi079 ceph-mon[23158]: pgmap v4453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:20 smithi002 ceph-mon[17167]: pgmap v4454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:20 smithi016 ceph-mon[22010]: pgmap v4454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:20 smithi079 ceph-mon[23158]: pgmap v4454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:26:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:26:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:26:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:26:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:26:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:26:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:22 smithi002 ceph-mon[17167]: pgmap v4455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8085.034585 secs 2022-03-21T16:26:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:22 smithi016 ceph-mon[22010]: pgmap v4455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8085.034585 secs 2022-03-21T16:26:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:22 smithi079 ceph-mon[23158]: pgmap v4455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8085.034585 secs 2022-03-21T16:26:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:24 smithi002 ceph-mon[17167]: pgmap v4456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:24 smithi016 ceph-mon[22010]: pgmap v4456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:24 smithi079 ceph-mon[23158]: pgmap v4456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:26 smithi002 ceph-mon[17167]: pgmap v4457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:26 smithi016 ceph-mon[22010]: pgmap v4457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:26 smithi079 ceph-mon[23158]: pgmap v4457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8090.034654 secs 2022-03-21T16:26:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8090.034654 secs 2022-03-21T16:26:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8090.034654 secs 2022-03-21T16:26:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:28 smithi002 ceph-mon[17167]: pgmap v4458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:28 smithi016 ceph-mon[22010]: pgmap v4458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:28 smithi079 ceph-mon[23158]: pgmap v4458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:31.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:30 smithi002 ceph-mon[17167]: pgmap v4459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:30 smithi016 ceph-mon[22010]: pgmap v4459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:30 smithi079 ceph-mon[23158]: pgmap v4459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:32 smithi002 ceph-mon[17167]: pgmap v4460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8095.034771 secs 2022-03-21T16:26:33.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:32 smithi016 ceph-mon[22010]: pgmap v4460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:33.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8095.034771 secs 2022-03-21T16:26:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:32 smithi079 ceph-mon[23158]: pgmap v4460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8095.034771 secs 2022-03-21T16:26:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:34 smithi002 ceph-mon[17167]: pgmap v4461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:26:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:34 smithi016 ceph-mon[22010]: pgmap v4461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:26:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:34 smithi079 ceph-mon[23158]: pgmap v4461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:26:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:36 smithi079 ceph-mon[23158]: pgmap v4462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:26:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:26:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:26:37.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:36 smithi016 ceph-mon[22010]: pgmap v4462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:37.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:26:37.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:26:37.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:26:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:36 smithi002 ceph-mon[17167]: pgmap v4462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:26:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:26:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:26:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:26:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8100.034866 secs 2022-03-21T16:26:38.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:26:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8100.034866 secs 2022-03-21T16:26:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:26:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8100.034866 secs 2022-03-21T16:26:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:26:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:38 smithi002 ceph-mon[17167]: pgmap v4463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:38 smithi016 ceph-mon[22010]: pgmap v4463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:38 smithi079 ceph-mon[23158]: pgmap v4463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:40 smithi002 ceph-mon[17167]: pgmap v4464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:40 smithi016 ceph-mon[22010]: pgmap v4464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:40 smithi079 ceph-mon[23158]: pgmap v4464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:42 smithi002 ceph-mon[17167]: pgmap v4465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8105.034985 secs 2022-03-21T16:26:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:42 smithi016 ceph-mon[22010]: pgmap v4465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8105.034985 secs 2022-03-21T16:26:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:42 smithi079 ceph-mon[23158]: pgmap v4465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8105.034985 secs 2022-03-21T16:26:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:44 smithi002 ceph-mon[17167]: pgmap v4466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:44 smithi016 ceph-mon[22010]: pgmap v4466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:44 smithi079 ceph-mon[23158]: pgmap v4466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:46 smithi002 ceph-mon[17167]: pgmap v4467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:46 smithi016 ceph-mon[22010]: pgmap v4467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:46 smithi079 ceph-mon[23158]: pgmap v4467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8110.035108 secs 2022-03-21T16:26:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8110.035108 secs 2022-03-21T16:26:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8110.035108 secs 2022-03-21T16:26:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:48 smithi002 ceph-mon[17167]: pgmap v4468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:48 smithi016 ceph-mon[22010]: pgmap v4468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:48 smithi079 ceph-mon[23158]: pgmap v4468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:50 smithi002 ceph-mon[17167]: pgmap v4469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:50 smithi016 ceph-mon[22010]: pgmap v4469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:50 smithi079 ceph-mon[23158]: pgmap v4469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:52 smithi002 ceph-mon[17167]: pgmap v4470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8115.035172 secs 2022-03-21T16:26:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:52 smithi016 ceph-mon[22010]: pgmap v4470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8115.035172 secs 2022-03-21T16:26:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:52 smithi079 ceph-mon[23158]: pgmap v4470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:26:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8115.035172 secs 2022-03-21T16:26:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:54 smithi002 ceph-mon[17167]: pgmap v4471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:54 smithi016 ceph-mon[22010]: pgmap v4471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:54 smithi079 ceph-mon[23158]: pgmap v4471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:26:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:56 smithi002 ceph-mon[17167]: pgmap v4472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:56 smithi016 ceph-mon[22010]: pgmap v4472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:56 smithi079 ceph-mon[23158]: pgmap v4472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:26:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8120.035294 secs 2022-03-21T16:26:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8120.035294 secs 2022-03-21T16:26:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8120.035294 secs 2022-03-21T16:26:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:26:58 smithi002 ceph-mon[17167]: pgmap v4473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:26:58 smithi016 ceph-mon[22010]: pgmap v4473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:26:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:26:58 smithi079 ceph-mon[23158]: pgmap v4473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:00 smithi002 ceph-mon[17167]: pgmap v4474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:00 smithi016 ceph-mon[22010]: pgmap v4474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:00 smithi079 ceph-mon[23158]: pgmap v4474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:02 smithi002 ceph-mon[17167]: pgmap v4475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8125.035413 secs 2022-03-21T16:27:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:02 smithi016 ceph-mon[22010]: pgmap v4475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8125.035413 secs 2022-03-21T16:27:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:02 smithi079 ceph-mon[23158]: pgmap v4475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8125.035413 secs 2022-03-21T16:27:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:04 smithi002 ceph-mon[17167]: pgmap v4476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:04 smithi016 ceph-mon[22010]: pgmap v4476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:04 smithi079 ceph-mon[23158]: pgmap v4476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:07.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:06 smithi002 ceph-mon[17167]: pgmap v4477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:06 smithi016 ceph-mon[22010]: pgmap v4477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:06 smithi079 ceph-mon[23158]: pgmap v4477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8130.035533 secs 2022-03-21T16:27:08.257 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8130.035533 secs 2022-03-21T16:27:08.258 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8130.035533 secs 2022-03-21T16:27:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:08 smithi002 ceph-mon[17167]: pgmap v4478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:08 smithi016 ceph-mon[22010]: pgmap v4478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:08 smithi079 ceph-mon[23158]: pgmap v4478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:10 smithi002 ceph-mon[17167]: pgmap v4479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:10 smithi016 ceph-mon[22010]: pgmap v4479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:10 smithi079 ceph-mon[23158]: pgmap v4479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:12 smithi002 ceph-mon[17167]: pgmap v4480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8135.035645 secs 2022-03-21T16:27:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:12 smithi016 ceph-mon[22010]: pgmap v4480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8135.035645 secs 2022-03-21T16:27:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:12 smithi079 ceph-mon[23158]: pgmap v4480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8135.035645 secs 2022-03-21T16:27:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:14 smithi002 ceph-mon[17167]: pgmap v4481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:14 smithi016 ceph-mon[22010]: pgmap v4481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:14 smithi079 ceph-mon[23158]: pgmap v4481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:16 smithi002 ceph-mon[17167]: pgmap v4482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:16 smithi016 ceph-mon[22010]: pgmap v4482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:16 smithi079 ceph-mon[23158]: pgmap v4482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8140.035743 secs 2022-03-21T16:27:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8140.035743 secs 2022-03-21T16:27:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8140.035743 secs 2022-03-21T16:27:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:18 smithi016 ceph-mon[22010]: pgmap v4483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:18 smithi079 ceph-mon[23158]: pgmap v4483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:18 smithi002 ceph-mon[17167]: pgmap v4483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:20 smithi016 ceph-mon[22010]: pgmap v4484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:20 smithi079 ceph-mon[23158]: pgmap v4484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:20 smithi002 ceph-mon[17167]: pgmap v4484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:27:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:27:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:27:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:27:22.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:27:22.371 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:27:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:22 smithi016 ceph-mon[22010]: pgmap v4485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8145.035867 secs 2022-03-21T16:27:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:22 smithi079 ceph-mon[23158]: pgmap v4485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8145.035867 secs 2022-03-21T16:27:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:22 smithi002 ceph-mon[17167]: pgmap v4485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8145.035867 secs 2022-03-21T16:27:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:24 smithi016 ceph-mon[22010]: pgmap v4486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:24 smithi079 ceph-mon[23158]: pgmap v4486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:24 smithi002 ceph-mon[17167]: pgmap v4486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:26 smithi016 ceph-mon[22010]: pgmap v4487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:26 smithi079 ceph-mon[23158]: pgmap v4487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:26 smithi002 ceph-mon[17167]: pgmap v4487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8150.035981 secs 2022-03-21T16:27:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8150.035981 secs 2022-03-21T16:27:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8150.035981 secs 2022-03-21T16:27:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:28 smithi016 ceph-mon[22010]: pgmap v4488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:28 smithi079 ceph-mon[23158]: pgmap v4488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:28 smithi002 ceph-mon[17167]: pgmap v4488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:30 smithi016 ceph-mon[22010]: pgmap v4489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:30 smithi079 ceph-mon[23158]: pgmap v4489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:30 smithi002 ceph-mon[17167]: pgmap v4489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:32 smithi016 ceph-mon[22010]: pgmap v4490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8155.036094 secs 2022-03-21T16:27:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:32 smithi079 ceph-mon[23158]: pgmap v4490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8155.036094 secs 2022-03-21T16:27:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:32 smithi002 ceph-mon[17167]: pgmap v4490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8155.036094 secs 2022-03-21T16:27:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:34 smithi016 ceph-mon[22010]: pgmap v4491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:34 smithi079 ceph-mon[23158]: pgmap v4491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:34 smithi002 ceph-mon[17167]: pgmap v4491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:36 smithi016 ceph-mon[22010]: pgmap v4492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:36 smithi079 ceph-mon[23158]: pgmap v4492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:36 smithi002 ceph-mon[17167]: pgmap v4492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8160.036213 secs 2022-03-21T16:27:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:27:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:27:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:27:38.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8160.036213 secs 2022-03-21T16:27:38.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:27:38.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:27:38.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:27:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8160.036213 secs 2022-03-21T16:27:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:27:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:27:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:27:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:38 smithi079 ceph-mon[23158]: pgmap v4493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:27:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:38 smithi002 ceph-mon[17167]: pgmap v4493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:27:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:38 smithi016 ceph-mon[22010]: pgmap v4493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:27:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:40 smithi079 ceph-mon[23158]: pgmap v4494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:40 smithi002 ceph-mon[17167]: pgmap v4494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:40 smithi016 ceph-mon[22010]: pgmap v4494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:42 smithi079 ceph-mon[23158]: pgmap v4495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8165.036334 secs 2022-03-21T16:27:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:42 smithi002 ceph-mon[17167]: pgmap v4495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8165.036334 secs 2022-03-21T16:27:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:42 smithi016 ceph-mon[22010]: pgmap v4495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8165.036334 secs 2022-03-21T16:27:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:44 smithi079 ceph-mon[23158]: pgmap v4496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:44 smithi002 ceph-mon[17167]: pgmap v4496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:44 smithi016 ceph-mon[22010]: pgmap v4496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:46 smithi079 ceph-mon[23158]: pgmap v4497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:46 smithi002 ceph-mon[17167]: pgmap v4497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:46 smithi016 ceph-mon[22010]: pgmap v4497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8170.036446 secs 2022-03-21T16:27:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8170.036446 secs 2022-03-21T16:27:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8170.036446 secs 2022-03-21T16:27:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:48 smithi079 ceph-mon[23158]: pgmap v4498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:48 smithi002 ceph-mon[17167]: pgmap v4498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:48 smithi016 ceph-mon[22010]: pgmap v4498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:50 smithi079 ceph-mon[23158]: pgmap v4499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:50 smithi002 ceph-mon[17167]: pgmap v4499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:50 smithi016 ceph-mon[22010]: pgmap v4499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:52 smithi079 ceph-mon[23158]: pgmap v4500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8175.036591 secs 2022-03-21T16:27:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:52 smithi002 ceph-mon[17167]: pgmap v4500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8175.036591 secs 2022-03-21T16:27:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:52 smithi016 ceph-mon[22010]: pgmap v4500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:27:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8175.036591 secs 2022-03-21T16:27:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:54 smithi079 ceph-mon[23158]: pgmap v4501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:54 smithi002 ceph-mon[17167]: pgmap v4501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:54 smithi016 ceph-mon[22010]: pgmap v4501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:27:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:56 smithi079 ceph-mon[23158]: pgmap v4502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:56 smithi002 ceph-mon[17167]: pgmap v4502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:57.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:56 smithi016 ceph-mon[22010]: pgmap v4502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:27:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8180.036643 secs 2022-03-21T16:27:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8180.036643 secs 2022-03-21T16:27:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8180.036643 secs 2022-03-21T16:27:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:27:58 smithi079 ceph-mon[23158]: pgmap v4503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:27:58 smithi002 ceph-mon[17167]: pgmap v4503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:27:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:27:58 smithi016 ceph-mon[22010]: pgmap v4503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:00 smithi079 ceph-mon[23158]: pgmap v4504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:00 smithi002 ceph-mon[17167]: pgmap v4504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:00 smithi016 ceph-mon[22010]: pgmap v4504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:02 smithi079 ceph-mon[23158]: pgmap v4505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8185.036740 secs 2022-03-21T16:28:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:02 smithi002 ceph-mon[17167]: pgmap v4505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8185.036740 secs 2022-03-21T16:28:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:02 smithi016 ceph-mon[22010]: pgmap v4505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8185.036740 secs 2022-03-21T16:28:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:04 smithi002 ceph-mon[17167]: pgmap v4506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:04 smithi016 ceph-mon[22010]: pgmap v4506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:04 smithi079 ceph-mon[23158]: pgmap v4506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:06 smithi002 ceph-mon[17167]: pgmap v4507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:06 smithi016 ceph-mon[22010]: pgmap v4507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:06 smithi079 ceph-mon[23158]: pgmap v4507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8190.036857 secs 2022-03-21T16:28:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8190.036857 secs 2022-03-21T16:28:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8190.036857 secs 2022-03-21T16:28:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:08 smithi002 ceph-mon[17167]: pgmap v4508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:08 smithi016 ceph-mon[22010]: pgmap v4508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:08 smithi079 ceph-mon[23158]: pgmap v4508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:10 smithi002 ceph-mon[17167]: pgmap v4509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:10 smithi016 ceph-mon[22010]: pgmap v4509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:10 smithi079 ceph-mon[23158]: pgmap v4509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:12.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:12 smithi002 ceph-mon[17167]: pgmap v4510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:12.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8195.036976 secs 2022-03-21T16:28:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:12 smithi016 ceph-mon[22010]: pgmap v4510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8195.036976 secs 2022-03-21T16:28:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:12 smithi079 ceph-mon[23158]: pgmap v4510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8195.036976 secs 2022-03-21T16:28:14.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:14 smithi002 ceph-mon[17167]: pgmap v4511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:14 smithi016 ceph-mon[22010]: pgmap v4511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:14 smithi079 ceph-mon[23158]: pgmap v4511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:16.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:16 smithi002 ceph-mon[17167]: pgmap v4512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:16 smithi016 ceph-mon[22010]: pgmap v4512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:16 smithi079 ceph-mon[23158]: pgmap v4512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8200.037114 secs 2022-03-21T16:28:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8200.037114 secs 2022-03-21T16:28:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8200.037114 secs 2022-03-21T16:28:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:18 smithi002 ceph-mon[17167]: pgmap v4513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:18 smithi016 ceph-mon[22010]: pgmap v4513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:18 smithi079 ceph-mon[23158]: pgmap v4513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:20.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:20 smithi002 ceph-mon[17167]: pgmap v4514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:20 smithi016 ceph-mon[22010]: pgmap v4514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:20 smithi079 ceph-mon[23158]: pgmap v4514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:21.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:28:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:28:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:28:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:28:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:28:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:28:22.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:22 smithi002 ceph-mon[17167]: pgmap v4515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:22.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8205.037265 secs 2022-03-21T16:28:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:22 smithi016 ceph-mon[22010]: pgmap v4515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8205.037265 secs 2022-03-21T16:28:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:22 smithi079 ceph-mon[23158]: pgmap v4515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8205.037265 secs 2022-03-21T16:28:24.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:24 smithi002 ceph-mon[17167]: pgmap v4516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:24 smithi016 ceph-mon[22010]: pgmap v4516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:24 smithi079 ceph-mon[23158]: pgmap v4516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:26.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:26 smithi002 ceph-mon[17167]: pgmap v4517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:26 smithi016 ceph-mon[22010]: pgmap v4517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:26 smithi079 ceph-mon[23158]: pgmap v4517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:27.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8210.037381 secs 2022-03-21T16:28:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8210.037381 secs 2022-03-21T16:28:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8210.037381 secs 2022-03-21T16:28:28.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:28 smithi002 ceph-mon[17167]: pgmap v4518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:28 smithi016 ceph-mon[22010]: pgmap v4518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:28 smithi079 ceph-mon[23158]: pgmap v4518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:30 smithi016 ceph-mon[22010]: pgmap v4519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:30 smithi079 ceph-mon[23158]: pgmap v4519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:31.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:30 smithi002 ceph-mon[17167]: pgmap v4519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:32 smithi016 ceph-mon[22010]: pgmap v4520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8215.037510 secs 2022-03-21T16:28:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:32 smithi079 ceph-mon[23158]: pgmap v4520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8215.037510 secs 2022-03-21T16:28:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:32 smithi002 ceph-mon[17167]: pgmap v4520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8215.037510 secs 2022-03-21T16:28:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:34 smithi016 ceph-mon[22010]: pgmap v4521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:34 smithi079 ceph-mon[23158]: pgmap v4521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:34 smithi002 ceph-mon[17167]: pgmap v4521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:36 smithi016 ceph-mon[22010]: pgmap v4522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:36 smithi079 ceph-mon[23158]: pgmap v4522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:36 smithi002 ceph-mon[17167]: pgmap v4522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8220.037607 secs 2022-03-21T16:28:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8220.037607 secs 2022-03-21T16:28:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8220.037607 secs 2022-03-21T16:28:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:38 smithi016 ceph-mon[22010]: pgmap v4523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:28:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:28:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:28:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:38 smithi079 ceph-mon[23158]: pgmap v4523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:28:38.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:38 smithi002 ceph-mon[17167]: pgmap v4523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:28:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:28:38.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:28:40.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:40 smithi002 ceph-mon[17167]: pgmap v4524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:28:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:40 smithi016 ceph-mon[22010]: pgmap v4524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:28:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:40 smithi079 ceph-mon[23158]: pgmap v4524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:28:42.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:42 smithi002 ceph-mon[17167]: pgmap v4525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:42.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8225.037707 secs 2022-03-21T16:28:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:42 smithi016 ceph-mon[22010]: pgmap v4525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8225.037707 secs 2022-03-21T16:28:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:42 smithi079 ceph-mon[23158]: pgmap v4525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8225.037707 secs 2022-03-21T16:28:44.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:44 smithi002 ceph-mon[17167]: pgmap v4526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:44 smithi016 ceph-mon[22010]: pgmap v4526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:44 smithi079 ceph-mon[23158]: pgmap v4526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:46 smithi002 ceph-mon[17167]: pgmap v4527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:46 smithi016 ceph-mon[22010]: pgmap v4527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:46 smithi079 ceph-mon[23158]: pgmap v4527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8230.037808 secs 2022-03-21T16:28:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8230.037808 secs 2022-03-21T16:28:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8230.037808 secs 2022-03-21T16:28:48.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:48 smithi002 ceph-mon[17167]: pgmap v4528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:48 smithi016 ceph-mon[22010]: pgmap v4528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:48 smithi079 ceph-mon[23158]: pgmap v4528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:50.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:50 smithi002 ceph-mon[17167]: pgmap v4529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:50 smithi016 ceph-mon[22010]: pgmap v4529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:50 smithi079 ceph-mon[23158]: pgmap v4529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:52.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:52 smithi002 ceph-mon[17167]: pgmap v4530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:52.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8235.037921 secs 2022-03-21T16:28:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:52 smithi016 ceph-mon[22010]: pgmap v4530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8235.037921 secs 2022-03-21T16:28:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:52 smithi079 ceph-mon[23158]: pgmap v4530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:28:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8235.037921 secs 2022-03-21T16:28:54.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:54 smithi002 ceph-mon[17167]: pgmap v4531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:54 smithi016 ceph-mon[22010]: pgmap v4531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:54 smithi079 ceph-mon[23158]: pgmap v4531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:28:56.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:56 smithi002 ceph-mon[17167]: pgmap v4532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:56 smithi016 ceph-mon[22010]: pgmap v4532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:56 smithi079 ceph-mon[23158]: pgmap v4532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:28:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8240.038010 secs 2022-03-21T16:28:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8240.038010 secs 2022-03-21T16:28:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8240.038010 secs 2022-03-21T16:28:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:28:58 smithi016 ceph-mon[22010]: pgmap v4533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:28:58 smithi079 ceph-mon[23158]: pgmap v4533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:28:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:28:58 smithi002 ceph-mon[17167]: pgmap v4533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:00 smithi016 ceph-mon[22010]: pgmap v4534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:00 smithi079 ceph-mon[23158]: pgmap v4534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:00 smithi002 ceph-mon[17167]: pgmap v4534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:02 smithi016 ceph-mon[22010]: pgmap v4535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8245.038124 secs 2022-03-21T16:29:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:02 smithi079 ceph-mon[23158]: pgmap v4535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8245.038124 secs 2022-03-21T16:29:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:02 smithi002 ceph-mon[17167]: pgmap v4535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8245.038124 secs 2022-03-21T16:29:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:04 smithi016 ceph-mon[22010]: pgmap v4536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:04 smithi079 ceph-mon[23158]: pgmap v4536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:04 smithi002 ceph-mon[17167]: pgmap v4536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:06 smithi016 ceph-mon[22010]: pgmap v4537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:06 smithi079 ceph-mon[23158]: pgmap v4537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:06 smithi002 ceph-mon[17167]: pgmap v4537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8250.038234 secs 2022-03-21T16:29:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8250.038234 secs 2022-03-21T16:29:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8250.038234 secs 2022-03-21T16:29:08.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:08 smithi016 ceph-mon[22010]: pgmap v4538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:08 smithi079 ceph-mon[23158]: pgmap v4538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:09.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:08 smithi002 ceph-mon[17167]: pgmap v4538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:10 smithi016 ceph-mon[22010]: pgmap v4539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:10 smithi079 ceph-mon[23158]: pgmap v4539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:10 smithi002 ceph-mon[17167]: pgmap v4539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:12 smithi016 ceph-mon[22010]: pgmap v4540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8255.038369 secs 2022-03-21T16:29:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:12 smithi079 ceph-mon[23158]: pgmap v4540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8255.038369 secs 2022-03-21T16:29:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:12 smithi002 ceph-mon[17167]: pgmap v4540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8255.038369 secs 2022-03-21T16:29:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:14 smithi016 ceph-mon[22010]: pgmap v4541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:14 smithi079 ceph-mon[23158]: pgmap v4541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:14 smithi002 ceph-mon[17167]: pgmap v4541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:16 smithi016 ceph-mon[22010]: pgmap v4542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:16 smithi079 ceph-mon[23158]: pgmap v4542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:16 smithi002 ceph-mon[17167]: pgmap v4542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8260.038432 secs 2022-03-21T16:29:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8260.038432 secs 2022-03-21T16:29:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8260.038432 secs 2022-03-21T16:29:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:18 smithi016 ceph-mon[22010]: pgmap v4543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:18 smithi079 ceph-mon[23158]: pgmap v4543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:18 smithi002 ceph-mon[17167]: pgmap v4543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:20 smithi079 ceph-mon[23158]: pgmap v4544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:20 smithi002 ceph-mon[17167]: pgmap v4544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:20 smithi016 ceph-mon[22010]: pgmap v4544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:29:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:29:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:29:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:29:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:29:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:29:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:22 smithi079 ceph-mon[23158]: pgmap v4545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8265.038567 secs 2022-03-21T16:29:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:22 smithi002 ceph-mon[17167]: pgmap v4545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8265.038567 secs 2022-03-21T16:29:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:22 smithi016 ceph-mon[22010]: pgmap v4545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8265.038567 secs 2022-03-21T16:29:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:24 smithi079 ceph-mon[23158]: pgmap v4546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:24 smithi002 ceph-mon[17167]: pgmap v4546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:24 smithi016 ceph-mon[22010]: pgmap v4546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:26 smithi079 ceph-mon[23158]: pgmap v4547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:26 smithi002 ceph-mon[17167]: pgmap v4547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:26 smithi016 ceph-mon[22010]: pgmap v4547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8270.038685 secs 2022-03-21T16:29:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8270.038685 secs 2022-03-21T16:29:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8270.038685 secs 2022-03-21T16:29:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:28 smithi079 ceph-mon[23158]: pgmap v4548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:28 smithi002 ceph-mon[17167]: pgmap v4548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:28 smithi016 ceph-mon[22010]: pgmap v4548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:30 smithi079 ceph-mon[23158]: pgmap v4549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:30 smithi002 ceph-mon[17167]: pgmap v4549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:30 smithi016 ceph-mon[22010]: pgmap v4549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:32 smithi079 ceph-mon[23158]: pgmap v4550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8275.038805 secs 2022-03-21T16:29:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:32 smithi002 ceph-mon[17167]: pgmap v4550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8275.038805 secs 2022-03-21T16:29:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:32 smithi016 ceph-mon[22010]: pgmap v4550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8275.038805 secs 2022-03-21T16:29:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:34 smithi079 ceph-mon[23158]: pgmap v4551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:34 smithi002 ceph-mon[17167]: pgmap v4551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:34 smithi016 ceph-mon[22010]: pgmap v4551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:36 smithi079 ceph-mon[23158]: pgmap v4552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:36 smithi002 ceph-mon[17167]: pgmap v4552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:36 smithi016 ceph-mon[22010]: pgmap v4552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8280.038873 secs 2022-03-21T16:29:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8280.038873 secs 2022-03-21T16:29:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8280.038873 secs 2022-03-21T16:29:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:38 smithi079 ceph-mon[23158]: pgmap v4553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:38 smithi002 ceph-mon[17167]: pgmap v4553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:38 smithi016 ceph-mon[22010]: pgmap v4553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:29:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:29:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:29:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:39 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:29:40.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:29:40.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:29:40.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:39 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:29:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:29:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:29:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:39 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: pgmap v4554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:41.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:29:41.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:29:41.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:29:41.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:29:41.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:29:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: pgmap v4554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:29:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:29:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:29:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:29:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:29:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: pgmap v4554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:29:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:29:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:29:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:29:42.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:41 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:29:42.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:41 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:29:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:41 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:29:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:41 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:29:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:41 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:29:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:41 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:29:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:42 smithi002 ceph-mon[17167]: pgmap v4555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8285.038994 secs 2022-03-21T16:29:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:42 smithi016 ceph-mon[22010]: pgmap v4555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8285.038994 secs 2022-03-21T16:29:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:42 smithi079 ceph-mon[23158]: pgmap v4555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8285.038994 secs 2022-03-21T16:29:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:44 smithi002 ceph-mon[17167]: pgmap v4556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:44 smithi016 ceph-mon[22010]: pgmap v4556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:44 smithi079 ceph-mon[23158]: pgmap v4556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:29:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:46 smithi002 ceph-mon[17167]: pgmap v4557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:46 smithi016 ceph-mon[22010]: pgmap v4557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:46 smithi079 ceph-mon[23158]: pgmap v4557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8290.039090 secs 2022-03-21T16:29:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8290.039090 secs 2022-03-21T16:29:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8290.039090 secs 2022-03-21T16:29:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:48 smithi002 ceph-mon[17167]: pgmap v4558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:48 smithi016 ceph-mon[22010]: pgmap v4558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:48 smithi079 ceph-mon[23158]: pgmap v4558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:50 smithi002 ceph-mon[17167]: pgmap v4559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:50 smithi016 ceph-mon[22010]: pgmap v4559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:50 smithi079 ceph-mon[23158]: pgmap v4559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:52 smithi002 ceph-mon[17167]: pgmap v4560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8295.039208 secs 2022-03-21T16:29:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:52 smithi016 ceph-mon[22010]: pgmap v4560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8295.039208 secs 2022-03-21T16:29:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:52 smithi079 ceph-mon[23158]: pgmap v4560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:29:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8295.039208 secs 2022-03-21T16:29:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:54 smithi002 ceph-mon[17167]: pgmap v4561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:54 smithi016 ceph-mon[22010]: pgmap v4561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:54 smithi079 ceph-mon[23158]: pgmap v4561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:56 smithi002 ceph-mon[17167]: pgmap v4562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:56 smithi016 ceph-mon[22010]: pgmap v4562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:56 smithi079 ceph-mon[23158]: pgmap v4562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:29:58.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8300.039330 secs 2022-03-21T16:29:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8300.039330 secs 2022-03-21T16:29:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8300.039330 secs 2022-03-21T16:29:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:29:58 smithi002 ceph-mon[17167]: pgmap v4563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:29:58 smithi016 ceph-mon[22010]: pgmap v4563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:29:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:29:58 smithi079 ceph-mon[23158]: pgmap v4563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:00 smithi002 ceph-mon[17167]: pgmap v4564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:30:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:00 smithi016 ceph-mon[22010]: pgmap v4564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:30:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:00 smithi079 ceph-mon[23158]: pgmap v4564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:30:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:02 smithi002 ceph-mon[17167]: pgmap v4565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8305.039431 secs 2022-03-21T16:30:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:02 smithi016 ceph-mon[22010]: pgmap v4565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8305.039431 secs 2022-03-21T16:30:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:02 smithi079 ceph-mon[23158]: pgmap v4565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8305.039431 secs 2022-03-21T16:30:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:04 smithi002 ceph-mon[17167]: pgmap v4566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:04 smithi016 ceph-mon[22010]: pgmap v4566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:04 smithi079 ceph-mon[23158]: pgmap v4566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:06 smithi002 ceph-mon[17167]: pgmap v4567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:06 smithi016 ceph-mon[22010]: pgmap v4567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:06 smithi079 ceph-mon[23158]: pgmap v4567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8310.039567 secs 2022-03-21T16:30:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8310.039567 secs 2022-03-21T16:30:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8310.039567 secs 2022-03-21T16:30:09.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:08 smithi002 ceph-mon[17167]: pgmap v4568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:08 smithi016 ceph-mon[22010]: pgmap v4568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:08 smithi079 ceph-mon[23158]: pgmap v4568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:10 smithi002 ceph-mon[17167]: pgmap v4569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:10 smithi016 ceph-mon[22010]: pgmap v4569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:10 smithi079 ceph-mon[23158]: pgmap v4569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:12 smithi002 ceph-mon[17167]: pgmap v4570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8315.039692 secs 2022-03-21T16:30:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:12 smithi016 ceph-mon[22010]: pgmap v4570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8315.039692 secs 2022-03-21T16:30:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:12 smithi079 ceph-mon[23158]: pgmap v4570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8315.039692 secs 2022-03-21T16:30:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:14 smithi002 ceph-mon[17167]: pgmap v4571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:14 smithi016 ceph-mon[22010]: pgmap v4571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:14 smithi079 ceph-mon[23158]: pgmap v4571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:16 smithi002 ceph-mon[17167]: pgmap v4572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:16 smithi016 ceph-mon[22010]: pgmap v4572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:16 smithi079 ceph-mon[23158]: pgmap v4572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8320.039834 secs 2022-03-21T16:30:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8320.039834 secs 2022-03-21T16:30:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8320.039834 secs 2022-03-21T16:30:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:18 smithi002 ceph-mon[17167]: pgmap v4573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:18 smithi016 ceph-mon[22010]: pgmap v4573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:18 smithi079 ceph-mon[23158]: pgmap v4573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:20 smithi002 ceph-mon[17167]: pgmap v4574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:20 smithi016 ceph-mon[22010]: pgmap v4574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:20 smithi079 ceph-mon[23158]: pgmap v4574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:30:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:30:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:30:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:30:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:30:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:30:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:22 smithi002 ceph-mon[17167]: pgmap v4575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8325.039957 secs 2022-03-21T16:30:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:22 smithi016 ceph-mon[22010]: pgmap v4575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8325.039957 secs 2022-03-21T16:30:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:22 smithi079 ceph-mon[23158]: pgmap v4575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8325.039957 secs 2022-03-21T16:30:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:24 smithi002 ceph-mon[17167]: pgmap v4576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:24 smithi016 ceph-mon[22010]: pgmap v4576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:24 smithi079 ceph-mon[23158]: pgmap v4576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:26 smithi002 ceph-mon[17167]: pgmap v4577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:26 smithi016 ceph-mon[22010]: pgmap v4577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:26 smithi079 ceph-mon[23158]: pgmap v4577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8330.040094 secs 2022-03-21T16:30:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8330.040094 secs 2022-03-21T16:30:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8330.040094 secs 2022-03-21T16:30:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:28 smithi002 ceph-mon[17167]: pgmap v4578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:28 smithi016 ceph-mon[22010]: pgmap v4578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:28 smithi079 ceph-mon[23158]: pgmap v4578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:30 smithi002 ceph-mon[17167]: pgmap v4579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:30 smithi016 ceph-mon[22010]: pgmap v4579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:30 smithi079 ceph-mon[23158]: pgmap v4579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:32 smithi002 ceph-mon[17167]: pgmap v4580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8335.040210 secs 2022-03-21T16:30:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:32 smithi016 ceph-mon[22010]: pgmap v4580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8335.040210 secs 2022-03-21T16:30:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:32 smithi079 ceph-mon[23158]: pgmap v4580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8335.040210 secs 2022-03-21T16:30:35.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:34 smithi002 ceph-mon[17167]: pgmap v4581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:34 smithi016 ceph-mon[22010]: pgmap v4581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:34 smithi079 ceph-mon[23158]: pgmap v4581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:37.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:36 smithi002 ceph-mon[17167]: pgmap v4582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:36 smithi016 ceph-mon[22010]: pgmap v4582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:36 smithi079 ceph-mon[23158]: pgmap v4582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8340.040347 secs 2022-03-21T16:30:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8340.040347 secs 2022-03-21T16:30:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8340.040347 secs 2022-03-21T16:30:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:38 smithi002 ceph-mon[17167]: pgmap v4583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:38 smithi016 ceph-mon[22010]: pgmap v4583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:38 smithi079 ceph-mon[23158]: pgmap v4583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:41.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:40 smithi002 ceph-mon[17167]: pgmap v4584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:41.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:30:41.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:30:41.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:40 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:30:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:40 smithi016 ceph-mon[22010]: pgmap v4584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:41.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:30:41.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:30:41.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:40 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:30:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:40 smithi079 ceph-mon[23158]: pgmap v4584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:40 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:30:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:30:42.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:30:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:30:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:42 smithi079 ceph-mon[23158]: pgmap v4585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8345.040467 secs 2022-03-21T16:30:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:42 smithi002 ceph-mon[17167]: pgmap v4585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8345.040467 secs 2022-03-21T16:30:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:42 smithi016 ceph-mon[22010]: pgmap v4585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8345.040467 secs 2022-03-21T16:30:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:44 smithi079 ceph-mon[23158]: pgmap v4586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:44 smithi002 ceph-mon[17167]: pgmap v4586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:44 smithi016 ceph-mon[22010]: pgmap v4586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:46 smithi079 ceph-mon[23158]: pgmap v4587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:46 smithi002 ceph-mon[17167]: pgmap v4587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:46 smithi016 ceph-mon[22010]: pgmap v4587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8350.040567 secs 2022-03-21T16:30:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8350.040567 secs 2022-03-21T16:30:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8350.040567 secs 2022-03-21T16:30:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:48 smithi079 ceph-mon[23158]: pgmap v4588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:48 smithi002 ceph-mon[17167]: pgmap v4588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:48 smithi016 ceph-mon[22010]: pgmap v4588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:50 smithi002 ceph-mon[17167]: pgmap v4589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:50 smithi016 ceph-mon[22010]: pgmap v4589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:50 smithi079 ceph-mon[23158]: pgmap v4589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:52 smithi002 ceph-mon[17167]: pgmap v4590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8355.040653 secs 2022-03-21T16:30:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:52 smithi016 ceph-mon[22010]: pgmap v4590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8355.040653 secs 2022-03-21T16:30:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:52 smithi079 ceph-mon[23158]: pgmap v4590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:30:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8355.040653 secs 2022-03-21T16:30:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:54 smithi002 ceph-mon[17167]: pgmap v4591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:54 smithi016 ceph-mon[22010]: pgmap v4591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:54 smithi079 ceph-mon[23158]: pgmap v4591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:30:57.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:56 smithi002 ceph-mon[17167]: pgmap v4592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:57.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:56 smithi016 ceph-mon[22010]: pgmap v4592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:56 smithi079 ceph-mon[23158]: pgmap v4592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:30:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8360.040718 secs 2022-03-21T16:30:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8360.040718 secs 2022-03-21T16:30:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8360.040718 secs 2022-03-21T16:30:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:30:58 smithi002 ceph-mon[17167]: pgmap v4593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:30:58 smithi016 ceph-mon[22010]: pgmap v4593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:30:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:30:58 smithi079 ceph-mon[23158]: pgmap v4593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:00 smithi002 ceph-mon[17167]: pgmap v4594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:00 smithi016 ceph-mon[22010]: pgmap v4594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:00 smithi079 ceph-mon[23158]: pgmap v4594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:02 smithi002 ceph-mon[17167]: pgmap v4595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8365.040835 secs 2022-03-21T16:31:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:02 smithi016 ceph-mon[22010]: pgmap v4595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8365.040835 secs 2022-03-21T16:31:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:02 smithi079 ceph-mon[23158]: pgmap v4595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8365.040835 secs 2022-03-21T16:31:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:05 smithi002 ceph-mon[17167]: pgmap v4596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:05 smithi016 ceph-mon[22010]: pgmap v4596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:05 smithi079 ceph-mon[23158]: pgmap v4596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:07 smithi002 ceph-mon[17167]: pgmap v4597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:07 smithi016 ceph-mon[22010]: pgmap v4597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:07 smithi079 ceph-mon[23158]: pgmap v4597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8370.040950 secs 2022-03-21T16:31:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8370.040950 secs 2022-03-21T16:31:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8370.040950 secs 2022-03-21T16:31:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:09 smithi002 ceph-mon[17167]: pgmap v4598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:09 smithi016 ceph-mon[22010]: pgmap v4598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:09 smithi079 ceph-mon[23158]: pgmap v4598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:11 smithi002 ceph-mon[17167]: pgmap v4599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:11 smithi016 ceph-mon[22010]: pgmap v4599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:11 smithi079 ceph-mon[23158]: pgmap v4599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:13 smithi002 ceph-mon[17167]: pgmap v4600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8375.041099 secs 2022-03-21T16:31:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:13 smithi016 ceph-mon[22010]: pgmap v4600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8375.041099 secs 2022-03-21T16:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:13 smithi079 ceph-mon[23158]: pgmap v4600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8375.041099 secs 2022-03-21T16:31:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:15 smithi002 ceph-mon[17167]: pgmap v4601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:15 smithi016 ceph-mon[22010]: pgmap v4601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:15 smithi079 ceph-mon[23158]: pgmap v4601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:17 smithi002 ceph-mon[17167]: pgmap v4602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:17 smithi016 ceph-mon[22010]: pgmap v4602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:17 smithi079 ceph-mon[23158]: pgmap v4602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8380.041232 secs 2022-03-21T16:31:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8380.041232 secs 2022-03-21T16:31:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8380.041232 secs 2022-03-21T16:31:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:19 smithi002 ceph-mon[17167]: pgmap v4603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:19 smithi016 ceph-mon[22010]: pgmap v4603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:19 smithi079 ceph-mon[23158]: pgmap v4603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:20.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:20 smithi002 ceph-mon[17167]: pgmap v4604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:20 smithi016 ceph-mon[22010]: pgmap v4604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:20 smithi079 ceph-mon[23158]: pgmap v4604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:31:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:31:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:31:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:31:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:31:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:31:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:22 smithi016 ceph-mon[22010]: pgmap v4605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8385.041349 secs 2022-03-21T16:31:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:22 smithi079 ceph-mon[23158]: pgmap v4605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8385.041349 secs 2022-03-21T16:31:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:22 smithi002 ceph-mon[17167]: pgmap v4605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8385.041349 secs 2022-03-21T16:31:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:24 smithi016 ceph-mon[22010]: pgmap v4606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:24 smithi079 ceph-mon[23158]: pgmap v4606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:25.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:24 smithi002 ceph-mon[17167]: pgmap v4606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:26 smithi016 ceph-mon[22010]: pgmap v4607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:26 smithi079 ceph-mon[23158]: pgmap v4607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:26 smithi002 ceph-mon[17167]: pgmap v4607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8390.041469 secs 2022-03-21T16:31:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8390.041469 secs 2022-03-21T16:31:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8390.041469 secs 2022-03-21T16:31:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:28 smithi016 ceph-mon[22010]: pgmap v4608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:28 smithi079 ceph-mon[23158]: pgmap v4608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:28 smithi002 ceph-mon[17167]: pgmap v4608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:30 smithi079 ceph-mon[23158]: pgmap v4609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:30 smithi002 ceph-mon[17167]: pgmap v4609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:30 smithi016 ceph-mon[22010]: pgmap v4609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:32 smithi079 ceph-mon[23158]: pgmap v4610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8395.041584 secs 2022-03-21T16:31:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:32 smithi002 ceph-mon[17167]: pgmap v4610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8395.041584 secs 2022-03-21T16:31:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:32 smithi016 ceph-mon[22010]: pgmap v4610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8395.041584 secs 2022-03-21T16:31:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:34 smithi079 ceph-mon[23158]: pgmap v4611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:34 smithi002 ceph-mon[17167]: pgmap v4611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:34 smithi016 ceph-mon[22010]: pgmap v4611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:36 smithi079 ceph-mon[23158]: pgmap v4612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:36 smithi002 ceph-mon[17167]: pgmap v4612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:36 smithi016 ceph-mon[22010]: pgmap v4612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8400.041700 secs 2022-03-21T16:31:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8400.041700 secs 2022-03-21T16:31:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8400.041700 secs 2022-03-21T16:31:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:38 smithi079 ceph-mon[23158]: pgmap v4613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:38 smithi002 ceph-mon[17167]: pgmap v4613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:38 smithi016 ceph-mon[22010]: pgmap v4613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:40 smithi079 ceph-mon[23158]: pgmap v4614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:40.991 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:40 smithi016 ceph-mon[22010]: pgmap v4614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:41.024 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:40 smithi002 ceph-mon[17167]: pgmap v4614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:31:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:31:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:31:42.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:31:42.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:31:42.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:31:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:31:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:31:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:31:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:42 smithi002 ceph-mon[17167]: pgmap v4615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:31:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8405.041800 secs 2022-03-21T16:31:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:42 smithi016 ceph-mon[22010]: pgmap v4615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:31:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8405.041800 secs 2022-03-21T16:31:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:42 smithi079 ceph-mon[23158]: pgmap v4615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:31:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8405.041800 secs 2022-03-21T16:31:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:44 smithi002 ceph-mon[17167]: pgmap v4616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:44 smithi016 ceph-mon[22010]: pgmap v4616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:44 smithi079 ceph-mon[23158]: pgmap v4616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:46 smithi002 ceph-mon[17167]: pgmap v4617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:46 smithi016 ceph-mon[22010]: pgmap v4617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:46 smithi079 ceph-mon[23158]: pgmap v4617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8410.041923 secs 2022-03-21T16:31:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8410.041923 secs 2022-03-21T16:31:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8410.041923 secs 2022-03-21T16:31:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:48 smithi002 ceph-mon[17167]: pgmap v4618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:48 smithi016 ceph-mon[22010]: pgmap v4618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:48 smithi079 ceph-mon[23158]: pgmap v4618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:50 smithi002 ceph-mon[17167]: pgmap v4619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:50 smithi016 ceph-mon[22010]: pgmap v4619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:50 smithi079 ceph-mon[23158]: pgmap v4619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:52 smithi002 ceph-mon[17167]: pgmap v4620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8415.042060 secs 2022-03-21T16:31:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:52 smithi016 ceph-mon[22010]: pgmap v4620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8415.042060 secs 2022-03-21T16:31:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:52 smithi079 ceph-mon[23158]: pgmap v4620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:31:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8415.042060 secs 2022-03-21T16:31:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:54 smithi002 ceph-mon[17167]: pgmap v4621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:55.284 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:54 smithi079 ceph-mon[23158]: pgmap v4621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:55.285 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:54 smithi016 ceph-mon[22010]: pgmap v4621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:31:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:56 smithi002 ceph-mon[17167]: pgmap v4622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:56 smithi016 ceph-mon[22010]: pgmap v4622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:56 smithi079 ceph-mon[23158]: pgmap v4622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:31:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8420.042180 secs 2022-03-21T16:31:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8420.042180 secs 2022-03-21T16:31:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8420.042180 secs 2022-03-21T16:31:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:31:58 smithi002 ceph-mon[17167]: pgmap v4623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:31:58 smithi016 ceph-mon[22010]: pgmap v4623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:31:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:31:58 smithi079 ceph-mon[23158]: pgmap v4623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:00 smithi002 ceph-mon[17167]: pgmap v4624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:00 smithi016 ceph-mon[22010]: pgmap v4624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:00 smithi079 ceph-mon[23158]: pgmap v4624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:02 smithi002 ceph-mon[17167]: pgmap v4625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8425.042303 secs 2022-03-21T16:32:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:02 smithi016 ceph-mon[22010]: pgmap v4625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8425.042303 secs 2022-03-21T16:32:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:02 smithi079 ceph-mon[23158]: pgmap v4625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8425.042303 secs 2022-03-21T16:32:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:04 smithi002 ceph-mon[17167]: pgmap v4626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:04 smithi016 ceph-mon[22010]: pgmap v4626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:04 smithi079 ceph-mon[23158]: pgmap v4626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:06 smithi002 ceph-mon[17167]: pgmap v4627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:06 smithi016 ceph-mon[22010]: pgmap v4627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:06 smithi079 ceph-mon[23158]: pgmap v4627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8430.042428 secs 2022-03-21T16:32:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8430.042428 secs 2022-03-21T16:32:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8430.042428 secs 2022-03-21T16:32:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:08 smithi016 ceph-mon[22010]: pgmap v4628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:08 smithi079 ceph-mon[23158]: pgmap v4628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:08 smithi002 ceph-mon[17167]: pgmap v4628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:10 smithi016 ceph-mon[22010]: pgmap v4629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:10 smithi079 ceph-mon[23158]: pgmap v4629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:10 smithi002 ceph-mon[17167]: pgmap v4629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:12 smithi016 ceph-mon[22010]: pgmap v4630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8435.042570 secs 2022-03-21T16:32:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:12 smithi079 ceph-mon[23158]: pgmap v4630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8435.042570 secs 2022-03-21T16:32:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:12 smithi002 ceph-mon[17167]: pgmap v4630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8435.042570 secs 2022-03-21T16:32:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:14 smithi016 ceph-mon[22010]: pgmap v4631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:14 smithi079 ceph-mon[23158]: pgmap v4631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:14 smithi002 ceph-mon[17167]: pgmap v4631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:16 smithi016 ceph-mon[22010]: pgmap v4632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:16 smithi079 ceph-mon[23158]: pgmap v4632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:16 smithi002 ceph-mon[17167]: pgmap v4632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8440.042701 secs 2022-03-21T16:32:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8440.042701 secs 2022-03-21T16:32:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8440.042701 secs 2022-03-21T16:32:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:18 smithi016 ceph-mon[22010]: pgmap v4633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:18 smithi079 ceph-mon[23158]: pgmap v4633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:18 smithi002 ceph-mon[17167]: pgmap v4633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:20 smithi016 ceph-mon[22010]: pgmap v4634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:20 smithi079 ceph-mon[23158]: pgmap v4634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:20 smithi002 ceph-mon[17167]: pgmap v4634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:32:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:32:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:32:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:32:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:32:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:32:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:22 smithi079 ceph-mon[23158]: pgmap v4635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8445.042820 secs 2022-03-21T16:32:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:22 smithi002 ceph-mon[17167]: pgmap v4635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8445.042820 secs 2022-03-21T16:32:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:22 smithi016 ceph-mon[22010]: pgmap v4635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8445.042820 secs 2022-03-21T16:32:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:24 smithi079 ceph-mon[23158]: pgmap v4636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:24 smithi002 ceph-mon[17167]: pgmap v4636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:24 smithi016 ceph-mon[22010]: pgmap v4636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:26 smithi079 ceph-mon[23158]: pgmap v4637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:26 smithi002 ceph-mon[17167]: pgmap v4637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:26 smithi016 ceph-mon[22010]: pgmap v4637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8450.042913 secs 2022-03-21T16:32:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8450.042913 secs 2022-03-21T16:32:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8450.042913 secs 2022-03-21T16:32:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:28 smithi079 ceph-mon[23158]: pgmap v4638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:28 smithi002 ceph-mon[17167]: pgmap v4638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:28 smithi016 ceph-mon[22010]: pgmap v4638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:30 smithi079 ceph-mon[23158]: pgmap v4639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:30 smithi002 ceph-mon[17167]: pgmap v4639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:30 smithi016 ceph-mon[22010]: pgmap v4639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:32 smithi079 ceph-mon[23158]: pgmap v4640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8455.043034 secs 2022-03-21T16:32:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:32 smithi002 ceph-mon[17167]: pgmap v4640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8455.043034 secs 2022-03-21T16:32:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:32 smithi016 ceph-mon[22010]: pgmap v4640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8455.043034 secs 2022-03-21T16:32:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:34 smithi079 ceph-mon[23158]: pgmap v4641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:34 smithi002 ceph-mon[17167]: pgmap v4641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:34 smithi016 ceph-mon[22010]: pgmap v4641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:36 smithi079 ceph-mon[23158]: pgmap v4642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:36 smithi002 ceph-mon[17167]: pgmap v4642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:36 smithi016 ceph-mon[22010]: pgmap v4642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8460.043188 secs 2022-03-21T16:32:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8460.043188 secs 2022-03-21T16:32:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8460.043188 secs 2022-03-21T16:32:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:38 smithi079 ceph-mon[23158]: pgmap v4643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:38 smithi002 ceph-mon[17167]: pgmap v4643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:38 smithi016 ceph-mon[22010]: pgmap v4643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:40 smithi079 ceph-mon[23158]: pgmap v4644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:40 smithi002 ceph-mon[17167]: pgmap v4644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:40 smithi016 ceph-mon[22010]: pgmap v4644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:42 smithi079 ceph-mon[23158]: pgmap v4645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:32:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:32:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:42 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:32:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8465.043253 secs 2022-03-21T16:32:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:42 smithi002 ceph-mon[17167]: pgmap v4645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:32:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:32:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:42 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:32:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8465.043253 secs 2022-03-21T16:32:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:42 smithi016 ceph-mon[22010]: pgmap v4645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:32:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:32:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:42 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:32:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8465.043253 secs 2022-03-21T16:32:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:44 smithi079 ceph-mon[23158]: pgmap v4646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:44 smithi002 ceph-mon[17167]: pgmap v4646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:45.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:45.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:44 smithi016 ceph-mon[22010]: pgmap v4646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:46 smithi002 ceph-mon[17167]: pgmap v4647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:46.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:46 smithi016 ceph-mon[22010]: pgmap v4647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:46 smithi079 ceph-mon[23158]: pgmap v4647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:32:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8470.043353 secs 2022-03-21T16:32:48.150 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8470.043353 secs 2022-03-21T16:32:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8470.043353 secs 2022-03-21T16:32:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:48 smithi079 ceph-mon[23158]: pgmap v4648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:49.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:48 smithi002 ceph-mon[17167]: pgmap v4648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:48 smithi016 ceph-mon[22010]: pgmap v4648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:50 smithi079 ceph-mon[23158]: pgmap v4649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:50 smithi002 ceph-mon[17167]: pgmap v4649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:50 smithi016 ceph-mon[22010]: pgmap v4649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:52 smithi079 ceph-mon[23158]: pgmap v4650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8475.043475 secs 2022-03-21T16:32:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:52 smithi002 ceph-mon[17167]: pgmap v4650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:53.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8475.043475 secs 2022-03-21T16:32:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:52 smithi016 ceph-mon[22010]: pgmap v4650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:32:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8475.043475 secs 2022-03-21T16:32:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:54 smithi079 ceph-mon[23158]: pgmap v4651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:54 smithi002 ceph-mon[17167]: pgmap v4651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:54 smithi016 ceph-mon[22010]: pgmap v4651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:32:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:56 smithi079 ceph-mon[23158]: pgmap v4652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:56 smithi002 ceph-mon[17167]: pgmap v4652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:56 smithi016 ceph-mon[22010]: pgmap v4652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:32:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8480.043607 secs 2022-03-21T16:32:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8480.043607 secs 2022-03-21T16:32:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8480.043607 secs 2022-03-21T16:32:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:32:58 smithi079 ceph-mon[23158]: pgmap v4653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:32:58 smithi002 ceph-mon[17167]: pgmap v4653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:32:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:32:58 smithi016 ceph-mon[22010]: pgmap v4653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:00 smithi079 ceph-mon[23158]: pgmap v4654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:00 smithi002 ceph-mon[17167]: pgmap v4654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:00 smithi016 ceph-mon[22010]: pgmap v4654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:02 smithi079 ceph-mon[23158]: pgmap v4655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8485.043728 secs 2022-03-21T16:33:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:02 smithi002 ceph-mon[17167]: pgmap v4655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8485.043728 secs 2022-03-21T16:33:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:02 smithi016 ceph-mon[22010]: pgmap v4655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8485.043728 secs 2022-03-21T16:33:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:04 smithi079 ceph-mon[23158]: pgmap v4656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:04 smithi002 ceph-mon[17167]: pgmap v4656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:04 smithi016 ceph-mon[22010]: pgmap v4656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:06 smithi079 ceph-mon[23158]: pgmap v4657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:06 smithi002 ceph-mon[17167]: pgmap v4657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:06 smithi016 ceph-mon[22010]: pgmap v4657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8490.043800 secs 2022-03-21T16:33:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8490.043800 secs 2022-03-21T16:33:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8490.043800 secs 2022-03-21T16:33:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:08 smithi002 ceph-mon[17167]: pgmap v4658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:08 smithi016 ceph-mon[22010]: pgmap v4658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:08 smithi079 ceph-mon[23158]: pgmap v4658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:10 smithi002 ceph-mon[17167]: pgmap v4659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:10 smithi016 ceph-mon[22010]: pgmap v4659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:10 smithi079 ceph-mon[23158]: pgmap v4659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:12 smithi002 ceph-mon[17167]: pgmap v4660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8495.043925 secs 2022-03-21T16:33:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:12 smithi016 ceph-mon[22010]: pgmap v4660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8495.043925 secs 2022-03-21T16:33:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:12 smithi079 ceph-mon[23158]: pgmap v4660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8495.043925 secs 2022-03-21T16:33:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:14 smithi002 ceph-mon[17167]: pgmap v4661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:14 smithi016 ceph-mon[22010]: pgmap v4661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:14 smithi079 ceph-mon[23158]: pgmap v4661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:16 smithi002 ceph-mon[17167]: pgmap v4662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:16 smithi016 ceph-mon[22010]: pgmap v4662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:16 smithi079 ceph-mon[23158]: pgmap v4662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8500.044053 secs 2022-03-21T16:33:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8500.044053 secs 2022-03-21T16:33:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8500.044053 secs 2022-03-21T16:33:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:18 smithi002 ceph-mon[17167]: pgmap v4663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:18 smithi016 ceph-mon[22010]: pgmap v4663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:18 smithi079 ceph-mon[23158]: pgmap v4663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:21.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:20 smithi002 ceph-mon[17167]: pgmap v4664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:20 smithi016 ceph-mon[22010]: pgmap v4664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:20 smithi079 ceph-mon[23158]: pgmap v4664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:33:22.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:33:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:33:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:33:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:33:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:33:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:22 smithi002 ceph-mon[17167]: pgmap v4665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8505.044170 secs 2022-03-21T16:33:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:22 smithi016 ceph-mon[22010]: pgmap v4665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8505.044170 secs 2022-03-21T16:33:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:22 smithi079 ceph-mon[23158]: pgmap v4665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8505.044170 secs 2022-03-21T16:33:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:24 smithi002 ceph-mon[17167]: pgmap v4666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:24 smithi016 ceph-mon[22010]: pgmap v4666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:24 smithi079 ceph-mon[23158]: pgmap v4666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:26 smithi002 ceph-mon[17167]: pgmap v4667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:26 smithi016 ceph-mon[22010]: pgmap v4667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:26 smithi079 ceph-mon[23158]: pgmap v4667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8510.044279 secs 2022-03-21T16:33:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8510.044279 secs 2022-03-21T16:33:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8510.044279 secs 2022-03-21T16:33:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:28 smithi002 ceph-mon[17167]: pgmap v4668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:28 smithi016 ceph-mon[22010]: pgmap v4668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:28 smithi079 ceph-mon[23158]: pgmap v4668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:30 smithi002 ceph-mon[17167]: pgmap v4669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:30 smithi016 ceph-mon[22010]: pgmap v4669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:30 smithi079 ceph-mon[23158]: pgmap v4669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:32 smithi002 ceph-mon[17167]: pgmap v4670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8515.044416 secs 2022-03-21T16:33:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:32 smithi016 ceph-mon[22010]: pgmap v4670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8515.044416 secs 2022-03-21T16:33:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:32 smithi079 ceph-mon[23158]: pgmap v4670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8515.044416 secs 2022-03-21T16:33:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:34 smithi002 ceph-mon[17167]: pgmap v4671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:34 smithi016 ceph-mon[22010]: pgmap v4671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:34 smithi079 ceph-mon[23158]: pgmap v4671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:36 smithi002 ceph-mon[17167]: pgmap v4672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:36 smithi016 ceph-mon[22010]: pgmap v4672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:36 smithi079 ceph-mon[23158]: pgmap v4672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8520.044539 secs 2022-03-21T16:33:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8520.044539 secs 2022-03-21T16:33:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8520.044539 secs 2022-03-21T16:33:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:38 smithi002 ceph-mon[17167]: pgmap v4673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:38 smithi016 ceph-mon[22010]: pgmap v4673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:38 smithi079 ceph-mon[23158]: pgmap v4673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:41.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:40 smithi002 ceph-mon[17167]: pgmap v4674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:40 smithi016 ceph-mon[22010]: pgmap v4674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:40 smithi079 ceph-mon[23158]: pgmap v4674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:42 smithi002 ceph-mon[17167]: pgmap v4675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8525.044654 secs 2022-03-21T16:33:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:42 smithi016 ceph-mon[22010]: pgmap v4675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8525.044654 secs 2022-03-21T16:33:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:42 smithi079 ceph-mon[23158]: pgmap v4675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8525.044654 secs 2022-03-21T16:33:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:44 smithi002 ceph-mon[17167]: pgmap v4676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:44 smithi016 ceph-mon[22010]: pgmap v4676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:44 smithi079 ceph-mon[23158]: pgmap v4676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:46 smithi002 ceph-mon[17167]: pgmap v4677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:33:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:33:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:33:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:46 smithi016 ceph-mon[22010]: pgmap v4677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:33:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:33:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:33:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:46 smithi079 ceph-mon[23158]: pgmap v4677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:33:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:33:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:33:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:48 smithi002 ceph-mon[17167]: pgmap v4678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8530.044764 secs 2022-03-21T16:33:48.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:48 smithi016 ceph-mon[22010]: pgmap v4678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8530.044764 secs 2022-03-21T16:33:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:48 smithi079 ceph-mon[23158]: pgmap v4678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8530.044764 secs 2022-03-21T16:33:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:33:50.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:50 smithi002 ceph-mon[17167]: pgmap v4679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:50 smithi016 ceph-mon[22010]: pgmap v4679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:50 smithi079 ceph-mon[23158]: pgmap v4679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:52 smithi079 ceph-mon[23158]: pgmap v4680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8535.044880 secs 2022-03-21T16:33:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:52 smithi002 ceph-mon[17167]: pgmap v4680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8535.044880 secs 2022-03-21T16:33:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:52 smithi016 ceph-mon[22010]: pgmap v4680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:33:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8535.044880 secs 2022-03-21T16:33:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:54 smithi079 ceph-mon[23158]: pgmap v4681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:54 smithi002 ceph-mon[17167]: pgmap v4681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:54 smithi016 ceph-mon[22010]: pgmap v4681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:33:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:56 smithi079 ceph-mon[23158]: pgmap v4682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:56 smithi002 ceph-mon[17167]: pgmap v4682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:56 smithi016 ceph-mon[22010]: pgmap v4682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:33:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8540.044979 secs 2022-03-21T16:33:58.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8540.044979 secs 2022-03-21T16:33:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8540.044979 secs 2022-03-21T16:33:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:33:58 smithi079 ceph-mon[23158]: pgmap v4683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:33:58 smithi002 ceph-mon[17167]: pgmap v4683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:33:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:33:58 smithi016 ceph-mon[22010]: pgmap v4683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:00 smithi079 ceph-mon[23158]: pgmap v4684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:00 smithi002 ceph-mon[17167]: pgmap v4684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:00 smithi016 ceph-mon[22010]: pgmap v4684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:02 smithi079 ceph-mon[23158]: pgmap v4685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8545.045092 secs 2022-03-21T16:34:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:02 smithi002 ceph-mon[17167]: pgmap v4685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8545.045092 secs 2022-03-21T16:34:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:02 smithi016 ceph-mon[22010]: pgmap v4685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8545.045092 secs 2022-03-21T16:34:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:04 smithi002 ceph-mon[17167]: pgmap v4686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:04 smithi016 ceph-mon[22010]: pgmap v4686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:04 smithi079 ceph-mon[23158]: pgmap v4686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:06 smithi002 ceph-mon[17167]: pgmap v4687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:06 smithi016 ceph-mon[22010]: pgmap v4687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:06 smithi079 ceph-mon[23158]: pgmap v4687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8550.045206 secs 2022-03-21T16:34:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8550.045206 secs 2022-03-21T16:34:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8550.045206 secs 2022-03-21T16:34:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:08 smithi002 ceph-mon[17167]: pgmap v4688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:08 smithi016 ceph-mon[22010]: pgmap v4688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:08 smithi079 ceph-mon[23158]: pgmap v4688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:10 smithi002 ceph-mon[17167]: pgmap v4689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:10 smithi016 ceph-mon[22010]: pgmap v4689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:10 smithi079 ceph-mon[23158]: pgmap v4689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:12 smithi002 ceph-mon[17167]: pgmap v4690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8555.045323 secs 2022-03-21T16:34:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:12 smithi016 ceph-mon[22010]: pgmap v4690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8555.045323 secs 2022-03-21T16:34:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:12 smithi079 ceph-mon[23158]: pgmap v4690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8555.045323 secs 2022-03-21T16:34:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:14 smithi002 ceph-mon[17167]: pgmap v4691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:14 smithi016 ceph-mon[22010]: pgmap v4691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:14 smithi079 ceph-mon[23158]: pgmap v4691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:16 smithi002 ceph-mon[17167]: pgmap v4692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:16 smithi016 ceph-mon[22010]: pgmap v4692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:16 smithi079 ceph-mon[23158]: pgmap v4692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8560.045427 secs 2022-03-21T16:34:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8560.045427 secs 2022-03-21T16:34:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8560.045427 secs 2022-03-21T16:34:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:18 smithi002 ceph-mon[17167]: pgmap v4693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:18 smithi016 ceph-mon[22010]: pgmap v4693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:18 smithi079 ceph-mon[23158]: pgmap v4693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:20 smithi002 ceph-mon[17167]: pgmap v4694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:20 smithi016 ceph-mon[22010]: pgmap v4694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:20 smithi079 ceph-mon[23158]: pgmap v4694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:34:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:34:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:34:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:34:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:34:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:34:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:22 smithi002 ceph-mon[17167]: pgmap v4695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8565.045561 secs 2022-03-21T16:34:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:22 smithi016 ceph-mon[22010]: pgmap v4695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8565.045561 secs 2022-03-21T16:34:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:22 smithi079 ceph-mon[23158]: pgmap v4695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8565.045561 secs 2022-03-21T16:34:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:24 smithi002 ceph-mon[17167]: pgmap v4696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:24 smithi016 ceph-mon[22010]: pgmap v4696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:24 smithi079 ceph-mon[23158]: pgmap v4696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:26 smithi002 ceph-mon[17167]: pgmap v4697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:26 smithi016 ceph-mon[22010]: pgmap v4697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:26 smithi079 ceph-mon[23158]: pgmap v4697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8570.045650 secs 2022-03-21T16:34:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8570.045650 secs 2022-03-21T16:34:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8570.045650 secs 2022-03-21T16:34:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:28 smithi002 ceph-mon[17167]: pgmap v4698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:28 smithi016 ceph-mon[22010]: pgmap v4698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:28 smithi079 ceph-mon[23158]: pgmap v4698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:30 smithi002 ceph-mon[17167]: pgmap v4699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:30 smithi016 ceph-mon[22010]: pgmap v4699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:30 smithi079 ceph-mon[23158]: pgmap v4699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:32 smithi002 ceph-mon[17167]: pgmap v4700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8575.045734 secs 2022-03-21T16:34:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:32 smithi016 ceph-mon[22010]: pgmap v4700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8575.045734 secs 2022-03-21T16:34:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:32 smithi079 ceph-mon[23158]: pgmap v4700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8575.045734 secs 2022-03-21T16:34:35.123 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:34 smithi002 ceph-mon[17167]: pgmap v4701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:34 smithi016 ceph-mon[22010]: pgmap v4701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:34 smithi079 ceph-mon[23158]: pgmap v4701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:36 smithi002 ceph-mon[17167]: pgmap v4702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:36 smithi016 ceph-mon[22010]: pgmap v4702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:36 smithi079 ceph-mon[23158]: pgmap v4702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8580.045858 secs 2022-03-21T16:34:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8580.045858 secs 2022-03-21T16:34:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8580.045858 secs 2022-03-21T16:34:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:38 smithi002 ceph-mon[17167]: pgmap v4703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:38 smithi016 ceph-mon[22010]: pgmap v4703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:38 smithi079 ceph-mon[23158]: pgmap v4703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:40 smithi002 ceph-mon[17167]: pgmap v4704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:41.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:40 smithi016 ceph-mon[22010]: pgmap v4704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:40 smithi079 ceph-mon[23158]: pgmap v4704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:42 smithi002 ceph-mon[17167]: pgmap v4705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8585.045926 secs 2022-03-21T16:34:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:42 smithi016 ceph-mon[22010]: pgmap v4705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8585.045926 secs 2022-03-21T16:34:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:42 smithi079 ceph-mon[23158]: pgmap v4705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8585.045926 secs 2022-03-21T16:34:45.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:44 smithi002 ceph-mon[17167]: pgmap v4706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:44 smithi016 ceph-mon[22010]: pgmap v4706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:44 smithi079 ceph-mon[23158]: pgmap v4706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:34:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:46 smithi002 ceph-mon[17167]: pgmap v4707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:46 smithi016 ceph-mon[22010]: pgmap v4707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:46 smithi079 ceph-mon[23158]: pgmap v4707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:34:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8590.046047 secs 2022-03-21T16:34:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8590.046047 secs 2022-03-21T16:34:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8590.046047 secs 2022-03-21T16:34:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: pgmap v4708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:34:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:34:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:34:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:48 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: pgmap v4708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:49.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:34:49.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:34:49.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:34:49.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:48 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: pgmap v4708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:34:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:34:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:34:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:48 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:34:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:51 smithi002 ceph-mon[17167]: pgmap v4709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:51 smithi016 ceph-mon[22010]: pgmap v4709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:51 smithi079 ceph-mon[23158]: pgmap v4709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:53 smithi002 ceph-mon[17167]: pgmap v4710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8595.046161 secs 2022-03-21T16:34:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:53 smithi016 ceph-mon[22010]: pgmap v4710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8595.046161 secs 2022-03-21T16:34:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:53 smithi079 ceph-mon[23158]: pgmap v4710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:34:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8595.046161 secs 2022-03-21T16:34:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:55 smithi002 ceph-mon[17167]: pgmap v4711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:34:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:55 smithi016 ceph-mon[22010]: pgmap v4711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:34:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:55 smithi079 ceph-mon[23158]: pgmap v4711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:34:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:56 smithi016 ceph-mon[22010]: pgmap v4712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:56 smithi079 ceph-mon[23158]: pgmap v4712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:56.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:56 smithi002 ceph-mon[17167]: pgmap v4712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:34:58.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8600.046285 secs 2022-03-21T16:34:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8600.046285 secs 2022-03-21T16:34:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8600.046285 secs 2022-03-21T16:34:59.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:34:58 smithi002 ceph-mon[17167]: pgmap v4713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:34:58 smithi016 ceph-mon[22010]: pgmap v4713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:34:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:34:58 smithi079 ceph-mon[23158]: pgmap v4713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:00 smithi002 ceph-mon[17167]: pgmap v4714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:00 smithi016 ceph-mon[22010]: pgmap v4714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:00 smithi079 ceph-mon[23158]: pgmap v4714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:02 smithi002 ceph-mon[17167]: pgmap v4715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8605.046424 secs 2022-03-21T16:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:02 smithi016 ceph-mon[22010]: pgmap v4715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8605.046424 secs 2022-03-21T16:35:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:02 smithi079 ceph-mon[23158]: pgmap v4715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8605.046424 secs 2022-03-21T16:35:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:04 smithi002 ceph-mon[17167]: pgmap v4716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:04 smithi016 ceph-mon[22010]: pgmap v4716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:04 smithi079 ceph-mon[23158]: pgmap v4716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:06 smithi002 ceph-mon[17167]: pgmap v4717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:06 smithi016 ceph-mon[22010]: pgmap v4717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:06 smithi079 ceph-mon[23158]: pgmap v4717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8610.046569 secs 2022-03-21T16:35:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8610.046569 secs 2022-03-21T16:35:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8610.046569 secs 2022-03-21T16:35:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:08 smithi002 ceph-mon[17167]: pgmap v4718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:08 smithi016 ceph-mon[22010]: pgmap v4718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:08 smithi079 ceph-mon[23158]: pgmap v4718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:10 smithi002 ceph-mon[17167]: pgmap v4719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:10 smithi016 ceph-mon[22010]: pgmap v4719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:11.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:10 smithi079 ceph-mon[23158]: pgmap v4719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:12 smithi002 ceph-mon[17167]: pgmap v4720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8615.046686 secs 2022-03-21T16:35:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:12 smithi016 ceph-mon[22010]: pgmap v4720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8615.046686 secs 2022-03-21T16:35:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:12 smithi079 ceph-mon[23158]: pgmap v4720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8615.046686 secs 2022-03-21T16:35:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:14 smithi002 ceph-mon[17167]: pgmap v4721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:14 smithi016 ceph-mon[22010]: pgmap v4721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:14 smithi079 ceph-mon[23158]: pgmap v4721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:16 smithi002 ceph-mon[17167]: pgmap v4722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:16 smithi016 ceph-mon[22010]: pgmap v4722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:16 smithi079 ceph-mon[23158]: pgmap v4722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8620.046789 secs 2022-03-21T16:35:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8620.046789 secs 2022-03-21T16:35:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8620.046789 secs 2022-03-21T16:35:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:18 smithi002 ceph-mon[17167]: pgmap v4723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:18 smithi016 ceph-mon[22010]: pgmap v4723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:18 smithi079 ceph-mon[23158]: pgmap v4723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:20 smithi002 ceph-mon[17167]: pgmap v4724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:20 smithi016 ceph-mon[22010]: pgmap v4724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:20 smithi079 ceph-mon[23158]: pgmap v4724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:35:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:35:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:35:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:35:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:35:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:35:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:22 smithi002 ceph-mon[17167]: pgmap v4725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8625.046887 secs 2022-03-21T16:35:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:22 smithi016 ceph-mon[22010]: pgmap v4725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8625.046887 secs 2022-03-21T16:35:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:22 smithi079 ceph-mon[23158]: pgmap v4725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8625.046887 secs 2022-03-21T16:35:25.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:24 smithi002 ceph-mon[17167]: pgmap v4726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:24 smithi016 ceph-mon[22010]: pgmap v4726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:24 smithi079 ceph-mon[23158]: pgmap v4726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:26 smithi002 ceph-mon[17167]: pgmap v4727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:27.256 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:26 smithi079 ceph-mon[23158]: pgmap v4727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:27.257 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:26 smithi016 ceph-mon[22010]: pgmap v4727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8630.047001 secs 2022-03-21T16:35:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8630.047001 secs 2022-03-21T16:35:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8630.047001 secs 2022-03-21T16:35:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:28 smithi002 ceph-mon[17167]: pgmap v4728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:28 smithi016 ceph-mon[22010]: pgmap v4728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:28 smithi079 ceph-mon[23158]: pgmap v4728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:30 smithi002 ceph-mon[17167]: pgmap v4729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:30 smithi016 ceph-mon[22010]: pgmap v4729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:30 smithi079 ceph-mon[23158]: pgmap v4729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:32 smithi002 ceph-mon[17167]: pgmap v4730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8635.047151 secs 2022-03-21T16:35:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:32 smithi016 ceph-mon[22010]: pgmap v4730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8635.047151 secs 2022-03-21T16:35:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:32 smithi079 ceph-mon[23158]: pgmap v4730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8635.047151 secs 2022-03-21T16:35:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:34 smithi002 ceph-mon[17167]: pgmap v4731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:34 smithi016 ceph-mon[22010]: pgmap v4731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:34 smithi079 ceph-mon[23158]: pgmap v4731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:36 smithi002 ceph-mon[17167]: pgmap v4732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:36 smithi016 ceph-mon[22010]: pgmap v4732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:36 smithi079 ceph-mon[23158]: pgmap v4732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8640.047304 secs 2022-03-21T16:35:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8640.047304 secs 2022-03-21T16:35:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8640.047304 secs 2022-03-21T16:35:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:38 smithi002 ceph-mon[17167]: pgmap v4733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:38 smithi016 ceph-mon[22010]: pgmap v4733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:38 smithi079 ceph-mon[23158]: pgmap v4733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:40 smithi002 ceph-mon[17167]: pgmap v4734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:40 smithi016 ceph-mon[22010]: pgmap v4734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:40 smithi079 ceph-mon[23158]: pgmap v4734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:42 smithi002 ceph-mon[17167]: pgmap v4735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8645.047410 secs 2022-03-21T16:35:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:42 smithi016 ceph-mon[22010]: pgmap v4735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8645.047410 secs 2022-03-21T16:35:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:42 smithi079 ceph-mon[23158]: pgmap v4735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8645.047410 secs 2022-03-21T16:35:45.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:44 smithi002 ceph-mon[17167]: pgmap v4736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:44 smithi016 ceph-mon[22010]: pgmap v4736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:44 smithi079 ceph-mon[23158]: pgmap v4736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:47.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:46 smithi002 ceph-mon[17167]: pgmap v4737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:46 smithi016 ceph-mon[22010]: pgmap v4737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:46 smithi079 ceph-mon[23158]: pgmap v4737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:48.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8650.047528 secs 2022-03-21T16:35:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8650.047528 secs 2022-03-21T16:35:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8650.047528 secs 2022-03-21T16:35:49.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:48 smithi002 ceph-mon[17167]: pgmap v4738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:48 smithi016 ceph-mon[22010]: pgmap v4738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:48 smithi079 ceph-mon[23158]: pgmap v4738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:50.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:35:50.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:35:50.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:49 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:35:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:35:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:35:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:49 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:35:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:35:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:35:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:49 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:35:51.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:50 smithi002 ceph-mon[17167]: pgmap v4739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:51.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:35:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:50 smithi016 ceph-mon[22010]: pgmap v4739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:35:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:50 smithi079 ceph-mon[23158]: pgmap v4739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:35:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:52 smithi002 ceph-mon[17167]: pgmap v4740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8655.047645 secs 2022-03-21T16:35:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:52 smithi016 ceph-mon[22010]: pgmap v4740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8655.047645 secs 2022-03-21T16:35:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:52 smithi079 ceph-mon[23158]: pgmap v4740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:35:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8655.047645 secs 2022-03-21T16:35:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:54 smithi002 ceph-mon[17167]: pgmap v4741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:54 smithi016 ceph-mon[22010]: pgmap v4741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:54 smithi079 ceph-mon[23158]: pgmap v4741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:35:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:56 smithi002 ceph-mon[17167]: pgmap v4742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:56 smithi016 ceph-mon[22010]: pgmap v4742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:56 smithi079 ceph-mon[23158]: pgmap v4742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:35:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8660.047758 secs 2022-03-21T16:35:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8660.047758 secs 2022-03-21T16:35:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8660.047758 secs 2022-03-21T16:35:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:35:58 smithi016 ceph-mon[22010]: pgmap v4743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:35:58 smithi079 ceph-mon[23158]: pgmap v4743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:35:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:35:58 smithi002 ceph-mon[17167]: pgmap v4743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:00 smithi016 ceph-mon[22010]: pgmap v4744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:00 smithi079 ceph-mon[23158]: pgmap v4744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:00 smithi002 ceph-mon[17167]: pgmap v4744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:02 smithi016 ceph-mon[22010]: pgmap v4745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8665.047879 secs 2022-03-21T16:36:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:02 smithi079 ceph-mon[23158]: pgmap v4745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8665.047879 secs 2022-03-21T16:36:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:02 smithi002 ceph-mon[17167]: pgmap v4745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8665.047879 secs 2022-03-21T16:36:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:04 smithi016 ceph-mon[22010]: pgmap v4746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:04 smithi079 ceph-mon[23158]: pgmap v4746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:04 smithi002 ceph-mon[17167]: pgmap v4746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:06 smithi016 ceph-mon[22010]: pgmap v4747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:06 smithi079 ceph-mon[23158]: pgmap v4747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:06 smithi002 ceph-mon[17167]: pgmap v4747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8670.047967 secs 2022-03-21T16:36:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8670.047967 secs 2022-03-21T16:36:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8670.047967 secs 2022-03-21T16:36:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:08 smithi016 ceph-mon[22010]: pgmap v4748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:08 smithi079 ceph-mon[23158]: pgmap v4748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:08 smithi002 ceph-mon[17167]: pgmap v4748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:10 smithi016 ceph-mon[22010]: pgmap v4749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:10 smithi079 ceph-mon[23158]: pgmap v4749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:10 smithi002 ceph-mon[17167]: pgmap v4749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:12 smithi016 ceph-mon[22010]: pgmap v4750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8675.048092 secs 2022-03-21T16:36:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:12 smithi079 ceph-mon[23158]: pgmap v4750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8675.048092 secs 2022-03-21T16:36:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:12 smithi002 ceph-mon[17167]: pgmap v4750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8675.048092 secs 2022-03-21T16:36:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:14 smithi016 ceph-mon[22010]: pgmap v4751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:14 smithi079 ceph-mon[23158]: pgmap v4751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:14 smithi002 ceph-mon[17167]: pgmap v4751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:16 smithi016 ceph-mon[22010]: pgmap v4752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:16 smithi079 ceph-mon[23158]: pgmap v4752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:16 smithi002 ceph-mon[17167]: pgmap v4752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8680.048213 secs 2022-03-21T16:36:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8680.048213 secs 2022-03-21T16:36:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8680.048213 secs 2022-03-21T16:36:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:18 smithi079 ceph-mon[23158]: pgmap v4753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:18 smithi002 ceph-mon[17167]: pgmap v4753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:18 smithi016 ceph-mon[22010]: pgmap v4753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:20 smithi079 ceph-mon[23158]: pgmap v4754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:20 smithi002 ceph-mon[17167]: pgmap v4754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:20 smithi016 ceph-mon[22010]: pgmap v4754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:36:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:36:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:36:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:36:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:36:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:36:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:22 smithi079 ceph-mon[23158]: pgmap v4755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8685.048332 secs 2022-03-21T16:36:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:22 smithi002 ceph-mon[17167]: pgmap v4755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8685.048332 secs 2022-03-21T16:36:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:22 smithi016 ceph-mon[22010]: pgmap v4755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8685.048332 secs 2022-03-21T16:36:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:24 smithi079 ceph-mon[23158]: pgmap v4756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:24 smithi002 ceph-mon[17167]: pgmap v4756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:24 smithi016 ceph-mon[22010]: pgmap v4756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:26 smithi079 ceph-mon[23158]: pgmap v4757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:26 smithi002 ceph-mon[17167]: pgmap v4757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:26 smithi016 ceph-mon[22010]: pgmap v4757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8690.048451 secs 2022-03-21T16:36:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8690.048451 secs 2022-03-21T16:36:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8690.048451 secs 2022-03-21T16:36:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:28 smithi079 ceph-mon[23158]: pgmap v4758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:28 smithi002 ceph-mon[17167]: pgmap v4758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:28 smithi016 ceph-mon[22010]: pgmap v4758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:30 smithi079 ceph-mon[23158]: pgmap v4759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:30 smithi002 ceph-mon[17167]: pgmap v4759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:30 smithi016 ceph-mon[22010]: pgmap v4759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:32 smithi079 ceph-mon[23158]: pgmap v4760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8695.048552 secs 2022-03-21T16:36:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:32 smithi002 ceph-mon[17167]: pgmap v4760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8695.048552 secs 2022-03-21T16:36:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:32 smithi016 ceph-mon[22010]: pgmap v4760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8695.048552 secs 2022-03-21T16:36:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:34 smithi079 ceph-mon[23158]: pgmap v4761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:34 smithi002 ceph-mon[17167]: pgmap v4761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:34 smithi016 ceph-mon[22010]: pgmap v4761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:36 smithi079 ceph-mon[23158]: pgmap v4762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:36 smithi002 ceph-mon[17167]: pgmap v4762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:36 smithi016 ceph-mon[22010]: pgmap v4762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8700.048663 secs 2022-03-21T16:36:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8700.048663 secs 2022-03-21T16:36:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8700.048663 secs 2022-03-21T16:36:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:38 smithi079 ceph-mon[23158]: pgmap v4763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:38 smithi002 ceph-mon[17167]: pgmap v4763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:38 smithi016 ceph-mon[22010]: pgmap v4763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:40 smithi079 ceph-mon[23158]: pgmap v4764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:40 smithi002 ceph-mon[17167]: pgmap v4764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:40 smithi016 ceph-mon[22010]: pgmap v4764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:42 smithi079 ceph-mon[23158]: pgmap v4765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8705.048777 secs 2022-03-21T16:36:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:42 smithi002 ceph-mon[17167]: pgmap v4765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8705.048777 secs 2022-03-21T16:36:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:42 smithi016 ceph-mon[22010]: pgmap v4765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8705.048777 secs 2022-03-21T16:36:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:44 smithi002 ceph-mon[17167]: pgmap v4766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:44 smithi016 ceph-mon[22010]: pgmap v4766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:44 smithi079 ceph-mon[23158]: pgmap v4766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:46 smithi002 ceph-mon[17167]: pgmap v4767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:46 smithi016 ceph-mon[22010]: pgmap v4767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:46 smithi079 ceph-mon[23158]: pgmap v4767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8710.048900 secs 2022-03-21T16:36:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8710.048900 secs 2022-03-21T16:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8710.048900 secs 2022-03-21T16:36:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:48 smithi002 ceph-mon[17167]: pgmap v4768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:48 smithi016 ceph-mon[22010]: pgmap v4768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:48 smithi079 ceph-mon[23158]: pgmap v4768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:50 smithi002 ceph-mon[17167]: pgmap v4769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:36:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:36:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:50 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:36:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:50 smithi016 ceph-mon[22010]: pgmap v4769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:36:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:36:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:50 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:36:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:50 smithi079 ceph-mon[23158]: pgmap v4769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:36:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:36:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:50 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:36:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:36:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:36:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:36:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:53 smithi002 ceph-mon[17167]: pgmap v4770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8715.049038 secs 2022-03-21T16:36:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:53 smithi016 ceph-mon[22010]: pgmap v4770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8715.049038 secs 2022-03-21T16:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:53 smithi079 ceph-mon[23158]: pgmap v4770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:36:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8715.049038 secs 2022-03-21T16:36:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:55 smithi002 ceph-mon[17167]: pgmap v4771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:55 smithi016 ceph-mon[22010]: pgmap v4771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:55 smithi079 ceph-mon[23158]: pgmap v4771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:36:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:57 smithi002 ceph-mon[17167]: pgmap v4772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:57 smithi016 ceph-mon[22010]: pgmap v4772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:57 smithi079 ceph-mon[23158]: pgmap v4772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:36:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8720.049165 secs 2022-03-21T16:36:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8720.049165 secs 2022-03-21T16:36:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8720.049165 secs 2022-03-21T16:36:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:36:59 smithi002 ceph-mon[17167]: pgmap v4773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:36:59 smithi016 ceph-mon[22010]: pgmap v4773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:36:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:36:59 smithi079 ceph-mon[23158]: pgmap v4773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:00.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:00 smithi002 ceph-mon[17167]: pgmap v4774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:00 smithi016 ceph-mon[22010]: pgmap v4774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:00 smithi079 ceph-mon[23158]: pgmap v4774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:02 smithi002 ceph-mon[17167]: pgmap v4775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8725.049284 secs 2022-03-21T16:37:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:02 smithi016 ceph-mon[22010]: pgmap v4775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8725.049284 secs 2022-03-21T16:37:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:02 smithi079 ceph-mon[23158]: pgmap v4775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8725.049284 secs 2022-03-21T16:37:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:04 smithi002 ceph-mon[17167]: pgmap v4776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:04 smithi016 ceph-mon[22010]: pgmap v4776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:04 smithi079 ceph-mon[23158]: pgmap v4776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:06 smithi002 ceph-mon[17167]: pgmap v4777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:06 smithi016 ceph-mon[22010]: pgmap v4777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:06 smithi079 ceph-mon[23158]: pgmap v4777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8730.049434 secs 2022-03-21T16:37:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8730.049434 secs 2022-03-21T16:37:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8730.049434 secs 2022-03-21T16:37:09.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:08 smithi002 ceph-mon[17167]: pgmap v4778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:08 smithi016 ceph-mon[22010]: pgmap v4778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:08 smithi079 ceph-mon[23158]: pgmap v4778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:11.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:10 smithi002 ceph-mon[17167]: pgmap v4779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:10 smithi016 ceph-mon[22010]: pgmap v4779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:10 smithi079 ceph-mon[23158]: pgmap v4779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:12 smithi002 ceph-mon[17167]: pgmap v4780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8735.049570 secs 2022-03-21T16:37:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:12 smithi016 ceph-mon[22010]: pgmap v4780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8735.049570 secs 2022-03-21T16:37:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:12 smithi079 ceph-mon[23158]: pgmap v4780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8735.049570 secs 2022-03-21T16:37:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:14 smithi002 ceph-mon[17167]: pgmap v4781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:14 smithi016 ceph-mon[22010]: pgmap v4781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:14 smithi079 ceph-mon[23158]: pgmap v4781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:17.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:16 smithi002 ceph-mon[17167]: pgmap v4782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:16 smithi016 ceph-mon[22010]: pgmap v4782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:16 smithi079 ceph-mon[23158]: pgmap v4782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8740.049689 secs 2022-03-21T16:37:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8740.049689 secs 2022-03-21T16:37:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8740.049689 secs 2022-03-21T16:37:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:18 smithi002 ceph-mon[17167]: pgmap v4783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:18 smithi016 ceph-mon[22010]: pgmap v4783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:18 smithi079 ceph-mon[23158]: pgmap v4783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:20 smithi002 ceph-mon[17167]: pgmap v4784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:20 smithi016 ceph-mon[22010]: pgmap v4784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:20 smithi079 ceph-mon[23158]: pgmap v4784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:37:22.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:37:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:37:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:37:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:37:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:37:23.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:22 smithi002 ceph-mon[17167]: pgmap v4785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:23.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8745.049824 secs 2022-03-21T16:37:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:22 smithi016 ceph-mon[22010]: pgmap v4785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8745.049824 secs 2022-03-21T16:37:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:22 smithi079 ceph-mon[23158]: pgmap v4785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8745.049824 secs 2022-03-21T16:37:25.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:24 smithi002 ceph-mon[17167]: pgmap v4786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:24 smithi016 ceph-mon[22010]: pgmap v4786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:24 smithi079 ceph-mon[23158]: pgmap v4786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:27.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:26 smithi002 ceph-mon[17167]: pgmap v4787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:26 smithi016 ceph-mon[22010]: pgmap v4787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:26 smithi079 ceph-mon[23158]: pgmap v4787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:28.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8750.049962 secs 2022-03-21T16:37:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8750.049962 secs 2022-03-21T16:37:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8750.049962 secs 2022-03-21T16:37:29.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:28 smithi002 ceph-mon[17167]: pgmap v4788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:28 smithi016 ceph-mon[22010]: pgmap v4788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:28 smithi079 ceph-mon[23158]: pgmap v4788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:31.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:30 smithi002 ceph-mon[17167]: pgmap v4789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:30 smithi016 ceph-mon[22010]: pgmap v4789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:30 smithi079 ceph-mon[23158]: pgmap v4789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:33.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:32 smithi002 ceph-mon[17167]: pgmap v4790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:33.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8755.050086 secs 2022-03-21T16:37:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:32 smithi016 ceph-mon[22010]: pgmap v4790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8755.050086 secs 2022-03-21T16:37:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:32 smithi079 ceph-mon[23158]: pgmap v4790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8755.050086 secs 2022-03-21T16:37:35.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:34 smithi002 ceph-mon[17167]: pgmap v4791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:34 smithi016 ceph-mon[22010]: pgmap v4791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:34 smithi079 ceph-mon[23158]: pgmap v4791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:37.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:36 smithi002 ceph-mon[17167]: pgmap v4792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:36 smithi016 ceph-mon[22010]: pgmap v4792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:36 smithi079 ceph-mon[23158]: pgmap v4792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:38.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8760.050221 secs 2022-03-21T16:37:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8760.050221 secs 2022-03-21T16:37:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8760.050221 secs 2022-03-21T16:37:39.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:38 smithi002 ceph-mon[17167]: pgmap v4793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:38 smithi016 ceph-mon[22010]: pgmap v4793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:38 smithi079 ceph-mon[23158]: pgmap v4793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:41.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:40 smithi002 ceph-mon[17167]: pgmap v4794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:40 smithi016 ceph-mon[22010]: pgmap v4794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:40 smithi079 ceph-mon[23158]: pgmap v4794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:43.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:42 smithi002 ceph-mon[17167]: pgmap v4795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8765.050334 secs 2022-03-21T16:37:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:42 smithi016 ceph-mon[22010]: pgmap v4795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8765.050334 secs 2022-03-21T16:37:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:42 smithi079 ceph-mon[23158]: pgmap v4795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8765.050334 secs 2022-03-21T16:37:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:44 smithi016 ceph-mon[22010]: pgmap v4796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:44 smithi079 ceph-mon[23158]: pgmap v4796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:44 smithi002 ceph-mon[17167]: pgmap v4796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:46 smithi016 ceph-mon[22010]: pgmap v4797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:46 smithi079 ceph-mon[23158]: pgmap v4797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:46 smithi002 ceph-mon[17167]: pgmap v4797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8770.050452 secs 2022-03-21T16:37:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8770.050452 secs 2022-03-21T16:37:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8770.050452 secs 2022-03-21T16:37:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:48 smithi016 ceph-mon[22010]: pgmap v4798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:48 smithi079 ceph-mon[23158]: pgmap v4798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:48 smithi002 ceph-mon[17167]: pgmap v4798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:51.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:50 smithi016 ceph-mon[22010]: pgmap v4799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:51.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:50 smithi079 ceph-mon[23158]: pgmap v4799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:51.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:50 smithi002 ceph-mon[17167]: pgmap v4799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:37:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:37:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:51 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:37:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:37:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:37:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:51 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:37:52.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:37:52.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:37:52.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:51 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:37:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:52 smithi079 ceph-mon[23158]: pgmap v4800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:37:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8775.050575 secs 2022-03-21T16:37:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:52 smithi002 ceph-mon[17167]: pgmap v4800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:37:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8775.050575 secs 2022-03-21T16:37:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:52 smithi016 ceph-mon[22010]: pgmap v4800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:37:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:37:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8775.050575 secs 2022-03-21T16:37:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:54 smithi002 ceph-mon[17167]: pgmap v4801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:54 smithi016 ceph-mon[22010]: pgmap v4801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:54 smithi079 ceph-mon[23158]: pgmap v4801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:37:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:56 smithi002 ceph-mon[17167]: pgmap v4802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:56 smithi016 ceph-mon[22010]: pgmap v4802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:56 smithi079 ceph-mon[23158]: pgmap v4802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:37:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8780.050718 secs 2022-03-21T16:37:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8780.050718 secs 2022-03-21T16:37:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8780.050718 secs 2022-03-21T16:37:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:37:58 smithi002 ceph-mon[17167]: pgmap v4803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:37:58 smithi016 ceph-mon[22010]: pgmap v4803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:37:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:37:58 smithi079 ceph-mon[23158]: pgmap v4803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:00 smithi002 ceph-mon[17167]: pgmap v4804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:00 smithi016 ceph-mon[22010]: pgmap v4804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:00 smithi079 ceph-mon[23158]: pgmap v4804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:02 smithi002 ceph-mon[17167]: pgmap v4805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8785.050850 secs 2022-03-21T16:38:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:02 smithi016 ceph-mon[22010]: pgmap v4805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8785.050850 secs 2022-03-21T16:38:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:02 smithi079 ceph-mon[23158]: pgmap v4805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8785.050850 secs 2022-03-21T16:38:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:04 smithi002 ceph-mon[17167]: pgmap v4806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:04 smithi016 ceph-mon[22010]: pgmap v4806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:04 smithi079 ceph-mon[23158]: pgmap v4806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:07 smithi002 ceph-mon[17167]: pgmap v4807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:07 smithi016 ceph-mon[22010]: pgmap v4807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:07 smithi079 ceph-mon[23158]: pgmap v4807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8790.050969 secs 2022-03-21T16:38:08.510 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8790.050969 secs 2022-03-21T16:38:08.511 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8790.050969 secs 2022-03-21T16:38:09.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:09 smithi002 ceph-mon[17167]: pgmap v4808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:09 smithi016 ceph-mon[22010]: pgmap v4808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:09 smithi079 ceph-mon[23158]: pgmap v4808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:11 smithi002 ceph-mon[17167]: pgmap v4809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:11 smithi016 ceph-mon[22010]: pgmap v4809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:11 smithi079 ceph-mon[23158]: pgmap v4809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:13 smithi002 ceph-mon[17167]: pgmap v4810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8795.051083 secs 2022-03-21T16:38:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:13 smithi016 ceph-mon[22010]: pgmap v4810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8795.051083 secs 2022-03-21T16:38:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:13 smithi079 ceph-mon[23158]: pgmap v4810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8795.051083 secs 2022-03-21T16:38:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:15 smithi002 ceph-mon[17167]: pgmap v4811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:15 smithi016 ceph-mon[22010]: pgmap v4811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:15 smithi079 ceph-mon[23158]: pgmap v4811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:17 smithi002 ceph-mon[17167]: pgmap v4812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:17 smithi016 ceph-mon[22010]: pgmap v4812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:17 smithi079 ceph-mon[23158]: pgmap v4812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8800.051230 secs 2022-03-21T16:38:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8800.051230 secs 2022-03-21T16:38:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8800.051230 secs 2022-03-21T16:38:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:19 smithi002 ceph-mon[17167]: pgmap v4813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:19 smithi016 ceph-mon[22010]: pgmap v4813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:19 smithi079 ceph-mon[23158]: pgmap v4813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:21 smithi002 ceph-mon[17167]: pgmap v4814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:21 smithi016 ceph-mon[22010]: pgmap v4814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:21 smithi079 ceph-mon[23158]: pgmap v4814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:38:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:38:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:38:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:38:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:38:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:38:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:23 smithi002 ceph-mon[17167]: pgmap v4815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8805.051355 secs 2022-03-21T16:38:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:23 smithi016 ceph-mon[22010]: pgmap v4815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8805.051355 secs 2022-03-21T16:38:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:23 smithi079 ceph-mon[23158]: pgmap v4815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8805.051355 secs 2022-03-21T16:38:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:25 smithi002 ceph-mon[17167]: pgmap v4816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:25 smithi016 ceph-mon[22010]: pgmap v4816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:25 smithi079 ceph-mon[23158]: pgmap v4816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:27 smithi002 ceph-mon[17167]: pgmap v4817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:27 smithi016 ceph-mon[22010]: pgmap v4817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:27 smithi079 ceph-mon[23158]: pgmap v4817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8810.051479 secs 2022-03-21T16:38:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8810.051479 secs 2022-03-21T16:38:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8810.051479 secs 2022-03-21T16:38:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:29 smithi002 ceph-mon[17167]: pgmap v4818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:29 smithi016 ceph-mon[22010]: pgmap v4818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:29 smithi079 ceph-mon[23158]: pgmap v4818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:31 smithi002 ceph-mon[17167]: pgmap v4819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:31 smithi016 ceph-mon[22010]: pgmap v4819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:31 smithi079 ceph-mon[23158]: pgmap v4819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:33 smithi002 ceph-mon[17167]: pgmap v4820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8815.051606 secs 2022-03-21T16:38:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:33 smithi016 ceph-mon[22010]: pgmap v4820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8815.051606 secs 2022-03-21T16:38:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:33 smithi079 ceph-mon[23158]: pgmap v4820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8815.051606 secs 2022-03-21T16:38:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:35 smithi002 ceph-mon[17167]: pgmap v4821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:35 smithi016 ceph-mon[22010]: pgmap v4821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:35 smithi079 ceph-mon[23158]: pgmap v4821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:37 smithi002 ceph-mon[17167]: pgmap v4822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:37 smithi016 ceph-mon[22010]: pgmap v4822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:37 smithi079 ceph-mon[23158]: pgmap v4822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8820.051685 secs 2022-03-21T16:38:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8820.051685 secs 2022-03-21T16:38:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8820.051685 secs 2022-03-21T16:38:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:39 smithi002 ceph-mon[17167]: pgmap v4823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:39 smithi016 ceph-mon[22010]: pgmap v4823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:39 smithi079 ceph-mon[23158]: pgmap v4823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:41 smithi002 ceph-mon[17167]: pgmap v4824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:41 smithi016 ceph-mon[22010]: pgmap v4824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:41 smithi079 ceph-mon[23158]: pgmap v4824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:43 smithi002 ceph-mon[17167]: pgmap v4825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8825.051812 secs 2022-03-21T16:38:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:43 smithi016 ceph-mon[22010]: pgmap v4825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8825.051812 secs 2022-03-21T16:38:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:43 smithi079 ceph-mon[23158]: pgmap v4825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8825.051812 secs 2022-03-21T16:38:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:45 smithi002 ceph-mon[17167]: pgmap v4826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:45 smithi016 ceph-mon[22010]: pgmap v4826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:45 smithi079 ceph-mon[23158]: pgmap v4826: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:47 smithi002 ceph-mon[17167]: pgmap v4827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:47 smithi016 ceph-mon[22010]: pgmap v4827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:47 smithi079 ceph-mon[23158]: pgmap v4827: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8830.051926 secs 2022-03-21T16:38:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8830.051926 secs 2022-03-21T16:38:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8830.051926 secs 2022-03-21T16:38:49.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:49 smithi002 ceph-mon[17167]: pgmap v4828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:49 smithi016 ceph-mon[22010]: pgmap v4828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:49 smithi079 ceph-mon[23158]: pgmap v4828: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:51 smithi002 ceph-mon[17167]: pgmap v4829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:51 smithi016 ceph-mon[22010]: pgmap v4829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:51 smithi079 ceph-mon[23158]: pgmap v4829: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:53.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:52 smithi002 ceph-mon[17167]: pgmap v4830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:38:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:38:53.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:52 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:38:53.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8835.052021 secs 2022-03-21T16:38:53.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:52 smithi016 ceph-mon[22010]: pgmap v4830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:53.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:38:53.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:38:53.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:52 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:38:53.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8835.052021 secs 2022-03-21T16:38:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:52 smithi079 ceph-mon[23158]: pgmap v4830: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:38:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:38:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:38:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:52 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:38:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8835.052021 secs 2022-03-21T16:38:55.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:54 smithi002 ceph-mon[17167]: pgmap v4831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:54 smithi016 ceph-mon[22010]: pgmap v4831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:54 smithi079 ceph-mon[23158]: pgmap v4831: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:38:56.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:56 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T16:38:56.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:56.494 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:56 smithi002 ceph-mon[17167]: pgmap v4832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:56 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T16:38:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:56 smithi016 ceph-mon[22010]: pgmap v4832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:56 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T16:38:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:56 smithi079 ceph-mon[23158]: pgmap v4832: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:38:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:57 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T16:38:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8840.052115 secs 2022-03-21T16:38:58.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:57 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T16:38:58.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:58.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:58.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8840.052115 secs 2022-03-21T16:38:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:57 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T16:38:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:38:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8840.052115 secs 2022-03-21T16:38:59.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:38:58 smithi002 ceph-mon[17167]: pgmap v4833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:38:58 smithi016 ceph-mon[22010]: pgmap v4833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:38:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:38:58 smithi079 ceph-mon[23158]: pgmap v4833: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:01.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:00 smithi002 ceph-mon[17167]: pgmap v4834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:00 smithi016 ceph-mon[22010]: pgmap v4834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:00 smithi079 ceph-mon[23158]: pgmap v4834: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:02 smithi002 ceph-mon[17167]: pgmap v4835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:03.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8845.052239 secs 2022-03-21T16:39:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:02 smithi016 ceph-mon[22010]: pgmap v4835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8845.052239 secs 2022-03-21T16:39:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:02 smithi079 ceph-mon[23158]: pgmap v4835: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8845.052239 secs 2022-03-21T16:39:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:04 smithi002 ceph-mon[17167]: pgmap v4836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:04 smithi016 ceph-mon[22010]: pgmap v4836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:04 smithi079 ceph-mon[23158]: pgmap v4836: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:07.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:06 smithi002 ceph-mon[17167]: pgmap v4837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:06 smithi016 ceph-mon[22010]: pgmap v4837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:06 smithi079 ceph-mon[23158]: pgmap v4837: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8850.052352 secs 2022-03-21T16:39:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8850.052352 secs 2022-03-21T16:39:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8850.052352 secs 2022-03-21T16:39:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:08 smithi002 ceph-mon[17167]: pgmap v4838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:08 smithi016 ceph-mon[22010]: pgmap v4838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:08 smithi079 ceph-mon[23158]: pgmap v4838: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:10 smithi002 ceph-mon[17167]: pgmap v4839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:10 smithi016 ceph-mon[22010]: pgmap v4839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:10 smithi079 ceph-mon[23158]: pgmap v4839: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:13.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:12 smithi002 ceph-mon[17167]: pgmap v4840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:13.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8855.052465 secs 2022-03-21T16:39:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:12 smithi016 ceph-mon[22010]: pgmap v4840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8855.052465 secs 2022-03-21T16:39:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:12 smithi079 ceph-mon[23158]: pgmap v4840: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8855.052465 secs 2022-03-21T16:39:15.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:14 smithi002 ceph-mon[17167]: pgmap v4841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:14 smithi016 ceph-mon[22010]: pgmap v4841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:14 smithi079 ceph-mon[23158]: pgmap v4841: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:17.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:16 smithi002 ceph-mon[17167]: pgmap v4842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:16 smithi016 ceph-mon[22010]: pgmap v4842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:16 smithi079 ceph-mon[23158]: pgmap v4842: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:18.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8860.052590 secs 2022-03-21T16:39:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8860.052590 secs 2022-03-21T16:39:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8860.052590 secs 2022-03-21T16:39:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:18 smithi002 ceph-mon[17167]: pgmap v4843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:18 smithi016 ceph-mon[22010]: pgmap v4843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:18 smithi079 ceph-mon[23158]: pgmap v4843: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:21.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:20 smithi002 ceph-mon[17167]: pgmap v4844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:20 smithi016 ceph-mon[22010]: pgmap v4844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:20 smithi079 ceph-mon[23158]: pgmap v4844: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:22.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:39:22.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:39:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:39:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:39:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:39:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:39:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:22 smithi016 ceph-mon[22010]: pgmap v4845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8865.052704 secs 2022-03-21T16:39:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:22 smithi079 ceph-mon[23158]: pgmap v4845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8865.052704 secs 2022-03-21T16:39:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:22 smithi002 ceph-mon[17167]: pgmap v4845: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8865.052704 secs 2022-03-21T16:39:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:24 smithi016 ceph-mon[22010]: pgmap v4846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:24 smithi079 ceph-mon[23158]: pgmap v4846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:24 smithi002 ceph-mon[17167]: pgmap v4846: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:26 smithi016 ceph-mon[22010]: pgmap v4847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:26 smithi079 ceph-mon[23158]: pgmap v4847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:26 smithi002 ceph-mon[17167]: pgmap v4847: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8870.052780 secs 2022-03-21T16:39:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8870.052780 secs 2022-03-21T16:39:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8870.052780 secs 2022-03-21T16:39:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:28 smithi016 ceph-mon[22010]: pgmap v4848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:28 smithi079 ceph-mon[23158]: pgmap v4848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:28 smithi002 ceph-mon[17167]: pgmap v4848: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:30 smithi016 ceph-mon[22010]: pgmap v4849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:30 smithi079 ceph-mon[23158]: pgmap v4849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:30 smithi002 ceph-mon[17167]: pgmap v4849: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:32 smithi016 ceph-mon[22010]: pgmap v4850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8875.052904 secs 2022-03-21T16:39:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:32 smithi079 ceph-mon[23158]: pgmap v4850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8875.052904 secs 2022-03-21T16:39:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:32 smithi002 ceph-mon[17167]: pgmap v4850: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8875.052904 secs 2022-03-21T16:39:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:34 smithi016 ceph-mon[22010]: pgmap v4851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:34 smithi079 ceph-mon[23158]: pgmap v4851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:35.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:34 smithi002 ceph-mon[17167]: pgmap v4851: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:36 smithi016 ceph-mon[22010]: pgmap v4852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:36 smithi079 ceph-mon[23158]: pgmap v4852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:36 smithi002 ceph-mon[17167]: pgmap v4852: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8880.053025 secs 2022-03-21T16:39:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8880.053025 secs 2022-03-21T16:39:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8880.053025 secs 2022-03-21T16:39:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:38 smithi016 ceph-mon[22010]: pgmap v4853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:38 smithi079 ceph-mon[23158]: pgmap v4853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:38 smithi002 ceph-mon[17167]: pgmap v4853: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:40 smithi016 ceph-mon[22010]: pgmap v4854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:40 smithi079 ceph-mon[23158]: pgmap v4854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:40 smithi002 ceph-mon[17167]: pgmap v4854: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:42 smithi016 ceph-mon[22010]: pgmap v4855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8885.053095 secs 2022-03-21T16:39:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:42 smithi079 ceph-mon[23158]: pgmap v4855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8885.053095 secs 2022-03-21T16:39:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:42 smithi002 ceph-mon[17167]: pgmap v4855: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8885.053095 secs 2022-03-21T16:39:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:44 smithi016 ceph-mon[22010]: pgmap v4856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:44 smithi079 ceph-mon[23158]: pgmap v4856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:44 smithi002 ceph-mon[17167]: pgmap v4856: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:46 smithi079 ceph-mon[23158]: pgmap v4857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:46 smithi002 ceph-mon[17167]: pgmap v4857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:46 smithi016 ceph-mon[22010]: pgmap v4857: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8890.053231 secs 2022-03-21T16:39:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8890.053231 secs 2022-03-21T16:39:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8890.053231 secs 2022-03-21T16:39:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:48 smithi079 ceph-mon[23158]: pgmap v4858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:48 smithi002 ceph-mon[17167]: pgmap v4858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:48 smithi016 ceph-mon[22010]: pgmap v4858: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:50 smithi079 ceph-mon[23158]: pgmap v4859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:50 smithi002 ceph-mon[17167]: pgmap v4859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:50 smithi016 ceph-mon[22010]: pgmap v4859: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:52 smithi079 ceph-mon[23158]: pgmap v4860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8895.053350 secs 2022-03-21T16:39:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:52 smithi002 ceph-mon[17167]: pgmap v4860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8895.053350 secs 2022-03-21T16:39:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:52 smithi016 ceph-mon[22010]: pgmap v4860: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:39:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8895.053350 secs 2022-03-21T16:39:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:54 smithi079 ceph-mon[23158]: pgmap v4861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:54 smithi002 ceph-mon[17167]: pgmap v4861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:54 smithi016 ceph-mon[22010]: pgmap v4861: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:39:57.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:56 smithi002 ceph-mon[17167]: pgmap v4862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:39:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:39:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:56 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:39:57.138 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:56 smithi079 ceph-mon[23158]: pgmap v4862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:57.138 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:39:57.138 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:39:57.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:56 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:39:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:56 smithi016 ceph-mon[22010]: pgmap v4862: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:39:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:39:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:39:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:56 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:39:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8900.053468 secs 2022-03-21T16:39:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:57 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8900.053468 secs 2022-03-21T16:39:58.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:57 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8900.053468 secs 2022-03-21T16:39:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:57 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:39:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:58 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:39:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:58 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:39:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:58 smithi079 ceph-mon[23158]: pgmap v4863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:58 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:39:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:39:58 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:39:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:58 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:39:59.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:58 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:39:59.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:58 smithi002 ceph-mon[17167]: pgmap v4863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:59.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:58 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:39:59.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:39:58 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:39:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:58 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:39:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:58 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:39:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:58 smithi016 ceph-mon[22010]: pgmap v4863: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:39:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:58 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:39:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:39:58 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:40:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: pgmap v4864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T16:40:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:40:01.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:40:01.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:40:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: pgmap v4864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T16:40:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:40:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: pgmap v4864: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T16:40:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:40:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:40:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:40:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:40:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:40:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:02 smithi002 ceph-mon[17167]: pgmap v4865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:03.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8905.053586 secs 2022-03-21T16:40:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:02 smithi016 ceph-mon[22010]: pgmap v4865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8905.053586 secs 2022-03-21T16:40:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:02 smithi079 ceph-mon[23158]: pgmap v4865: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8905.053586 secs 2022-03-21T16:40:05.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:04 smithi002 ceph-mon[17167]: pgmap v4866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:04 smithi016 ceph-mon[22010]: pgmap v4866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:04 smithi079 ceph-mon[23158]: pgmap v4866: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:07.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:06 smithi002 ceph-mon[17167]: pgmap v4867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:06 smithi016 ceph-mon[22010]: pgmap v4867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:06 smithi079 ceph-mon[23158]: pgmap v4867: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:08.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8910.053666 secs 2022-03-21T16:40:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8910.053666 secs 2022-03-21T16:40:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8910.053666 secs 2022-03-21T16:40:09.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:08 smithi002 ceph-mon[17167]: pgmap v4868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:08 smithi016 ceph-mon[22010]: pgmap v4868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:08 smithi079 ceph-mon[23158]: pgmap v4868: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:11.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:10 smithi002 ceph-mon[17167]: pgmap v4869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:10 smithi016 ceph-mon[22010]: pgmap v4869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:10 smithi079 ceph-mon[23158]: pgmap v4869: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:12 smithi016 ceph-mon[22010]: pgmap v4870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8915.053787 secs 2022-03-21T16:40:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:12 smithi079 ceph-mon[23158]: pgmap v4870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8915.053787 secs 2022-03-21T16:40:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:12 smithi002 ceph-mon[17167]: pgmap v4870: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8915.053787 secs 2022-03-21T16:40:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:14 smithi016 ceph-mon[22010]: pgmap v4871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:14 smithi079 ceph-mon[23158]: pgmap v4871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:14 smithi002 ceph-mon[17167]: pgmap v4871: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:16 smithi016 ceph-mon[22010]: pgmap v4872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:16 smithi079 ceph-mon[23158]: pgmap v4872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:16 smithi002 ceph-mon[17167]: pgmap v4872: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8920.053906 secs 2022-03-21T16:40:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8920.053906 secs 2022-03-21T16:40:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8920.053906 secs 2022-03-21T16:40:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:18 smithi016 ceph-mon[22010]: pgmap v4873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:18 smithi079 ceph-mon[23158]: pgmap v4873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:18 smithi002 ceph-mon[17167]: pgmap v4873: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:20 smithi016 ceph-mon[22010]: pgmap v4874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:20 smithi079 ceph-mon[23158]: pgmap v4874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:20 smithi002 ceph-mon[17167]: pgmap v4874: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:40:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:40:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:40:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:40:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:40:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:40:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:22 smithi016 ceph-mon[22010]: pgmap v4875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8925.054030 secs 2022-03-21T16:40:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:22 smithi079 ceph-mon[23158]: pgmap v4875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8925.054030 secs 2022-03-21T16:40:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:22 smithi002 ceph-mon[17167]: pgmap v4875: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8925.054030 secs 2022-03-21T16:40:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:24 smithi016 ceph-mon[22010]: pgmap v4876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:24 smithi079 ceph-mon[23158]: pgmap v4876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:24 smithi002 ceph-mon[17167]: pgmap v4876: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:26 smithi016 ceph-mon[22010]: pgmap v4877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:26 smithi079 ceph-mon[23158]: pgmap v4877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:26 smithi002 ceph-mon[17167]: pgmap v4877: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8930.054146 secs 2022-03-21T16:40:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8930.054146 secs 2022-03-21T16:40:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8930.054146 secs 2022-03-21T16:40:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:28 smithi016 ceph-mon[22010]: pgmap v4878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:28 smithi079 ceph-mon[23158]: pgmap v4878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:28 smithi002 ceph-mon[17167]: pgmap v4878: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:30 smithi016 ceph-mon[22010]: pgmap v4879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:30 smithi079 ceph-mon[23158]: pgmap v4879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:30 smithi002 ceph-mon[17167]: pgmap v4879: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:32 smithi079 ceph-mon[23158]: pgmap v4880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8935.054290 secs 2022-03-21T16:40:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:32 smithi002 ceph-mon[17167]: pgmap v4880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8935.054290 secs 2022-03-21T16:40:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:32 smithi016 ceph-mon[22010]: pgmap v4880: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8935.054290 secs 2022-03-21T16:40:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:34 smithi079 ceph-mon[23158]: pgmap v4881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:34 smithi002 ceph-mon[17167]: pgmap v4881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:34 smithi016 ceph-mon[22010]: pgmap v4881: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:36 smithi079 ceph-mon[23158]: pgmap v4882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:36 smithi002 ceph-mon[17167]: pgmap v4882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:36 smithi016 ceph-mon[22010]: pgmap v4882: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8940.054412 secs 2022-03-21T16:40:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8940.054412 secs 2022-03-21T16:40:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8940.054412 secs 2022-03-21T16:40:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:38 smithi079 ceph-mon[23158]: pgmap v4883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:38 smithi002 ceph-mon[17167]: pgmap v4883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:38 smithi016 ceph-mon[22010]: pgmap v4883: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:40 smithi079 ceph-mon[23158]: pgmap v4884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:40 smithi002 ceph-mon[17167]: pgmap v4884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:40 smithi016 ceph-mon[22010]: pgmap v4884: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:42 smithi079 ceph-mon[23158]: pgmap v4885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8945.054534 secs 2022-03-21T16:40:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:42 smithi002 ceph-mon[17167]: pgmap v4885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8945.054534 secs 2022-03-21T16:40:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:42 smithi016 ceph-mon[22010]: pgmap v4885: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8945.054534 secs 2022-03-21T16:40:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:44 smithi016 ceph-mon[22010]: pgmap v4886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:44 smithi079 ceph-mon[23158]: pgmap v4886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:44 smithi002 ceph-mon[17167]: pgmap v4886: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:46 smithi079 ceph-mon[23158]: pgmap v4887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:46 smithi002 ceph-mon[17167]: pgmap v4887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:46 smithi016 ceph-mon[22010]: pgmap v4887: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8950.054646 secs 2022-03-21T16:40:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8950.054646 secs 2022-03-21T16:40:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8950.054646 secs 2022-03-21T16:40:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:48 smithi079 ceph-mon[23158]: pgmap v4888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:48 smithi002 ceph-mon[17167]: pgmap v4888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:48 smithi016 ceph-mon[22010]: pgmap v4888: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:50 smithi079 ceph-mon[23158]: pgmap v4889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:50 smithi002 ceph-mon[17167]: pgmap v4889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:50 smithi016 ceph-mon[22010]: pgmap v4889: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:52 smithi079 ceph-mon[23158]: pgmap v4890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:53.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8955.054797 secs 2022-03-21T16:40:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:52 smithi002 ceph-mon[17167]: pgmap v4890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8955.054797 secs 2022-03-21T16:40:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:52 smithi016 ceph-mon[22010]: pgmap v4890: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:40:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8955.054797 secs 2022-03-21T16:40:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:54 smithi079 ceph-mon[23158]: pgmap v4891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:54 smithi002 ceph-mon[17167]: pgmap v4891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:54 smithi016 ceph-mon[22010]: pgmap v4891: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:40:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:56 smithi079 ceph-mon[23158]: pgmap v4892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:56 smithi002 ceph-mon[17167]: pgmap v4892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:56 smithi016 ceph-mon[22010]: pgmap v4892: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:40:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:57 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8960.054917 secs 2022-03-21T16:40:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:57 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8960.054917 secs 2022-03-21T16:40:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:57 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8960.054917 secs 2022-03-21T16:40:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:58 smithi002 ceph-mon[17167]: pgmap v4893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:58 smithi016 ceph-mon[22010]: pgmap v4893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:40:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:58 smithi079 ceph-mon[23158]: pgmap v4893: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:41:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:41:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:40:59 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:41:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:41:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:41:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:40:59 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:41:00.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:41:00.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:41:00.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:40:59 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:41:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:00 smithi002 ceph-mon[17167]: pgmap v4894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:00 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:41:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:00 smithi016 ceph-mon[22010]: pgmap v4894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:00 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:41:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:00 smithi079 ceph-mon[23158]: pgmap v4894: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:00 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:41:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:02 smithi002 ceph-mon[17167]: pgmap v4895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8965.055065 secs 2022-03-21T16:41:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:02 smithi016 ceph-mon[22010]: pgmap v4895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8965.055065 secs 2022-03-21T16:41:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:02 smithi079 ceph-mon[23158]: pgmap v4895: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8965.055065 secs 2022-03-21T16:41:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:04 smithi002 ceph-mon[17167]: pgmap v4896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:04 smithi016 ceph-mon[22010]: pgmap v4896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:04 smithi079 ceph-mon[23158]: pgmap v4896: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:06 smithi002 ceph-mon[17167]: pgmap v4897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:06 smithi016 ceph-mon[22010]: pgmap v4897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:06 smithi079 ceph-mon[23158]: pgmap v4897: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8970.055185 secs 2022-03-21T16:41:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8970.055185 secs 2022-03-21T16:41:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8970.055185 secs 2022-03-21T16:41:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:09 smithi002 ceph-mon[17167]: pgmap v4898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:09 smithi016 ceph-mon[22010]: pgmap v4898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:09 smithi079 ceph-mon[23158]: pgmap v4898: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:11 smithi002 ceph-mon[17167]: pgmap v4899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:11 smithi016 ceph-mon[22010]: pgmap v4899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:11 smithi079 ceph-mon[23158]: pgmap v4899: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:13 smithi002 ceph-mon[17167]: pgmap v4900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8975.055332 secs 2022-03-21T16:41:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:13 smithi016 ceph-mon[22010]: pgmap v4900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8975.055332 secs 2022-03-21T16:41:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:13 smithi079 ceph-mon[23158]: pgmap v4900: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8975.055332 secs 2022-03-21T16:41:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:15 smithi002 ceph-mon[17167]: pgmap v4901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:15 smithi016 ceph-mon[22010]: pgmap v4901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:15 smithi079 ceph-mon[23158]: pgmap v4901: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:17 smithi002 ceph-mon[17167]: pgmap v4902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:17 smithi016 ceph-mon[22010]: pgmap v4902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:17 smithi079 ceph-mon[23158]: pgmap v4902: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8980.055454 secs 2022-03-21T16:41:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8980.055454 secs 2022-03-21T16:41:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8980.055454 secs 2022-03-21T16:41:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:19 smithi002 ceph-mon[17167]: pgmap v4903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:19 smithi016 ceph-mon[22010]: pgmap v4903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:19 smithi079 ceph-mon[23158]: pgmap v4903: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:21 smithi002 ceph-mon[17167]: pgmap v4904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:21 smithi016 ceph-mon[22010]: pgmap v4904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:21 smithi079 ceph-mon[23158]: pgmap v4904: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:41:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:41:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:41:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:41:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:41:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:41:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:23 smithi002 ceph-mon[17167]: pgmap v4905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8985.055555 secs 2022-03-21T16:41:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:23 smithi016 ceph-mon[22010]: pgmap v4905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8985.055555 secs 2022-03-21T16:41:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:23 smithi079 ceph-mon[23158]: pgmap v4905: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8985.055555 secs 2022-03-21T16:41:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:25 smithi002 ceph-mon[17167]: pgmap v4906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:25 smithi016 ceph-mon[22010]: pgmap v4906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:25 smithi079 ceph-mon[23158]: pgmap v4906: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:27 smithi002 ceph-mon[17167]: pgmap v4907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:27 smithi016 ceph-mon[22010]: pgmap v4907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:27 smithi079 ceph-mon[23158]: pgmap v4907: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8990.055686 secs 2022-03-21T16:41:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8990.055686 secs 2022-03-21T16:41:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8990.055686 secs 2022-03-21T16:41:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:29 smithi002 ceph-mon[17167]: pgmap v4908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:29 smithi016 ceph-mon[22010]: pgmap v4908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:29 smithi079 ceph-mon[23158]: pgmap v4908: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:31 smithi002 ceph-mon[17167]: pgmap v4909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:31 smithi016 ceph-mon[22010]: pgmap v4909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:31 smithi079 ceph-mon[23158]: pgmap v4909: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:33 smithi002 ceph-mon[17167]: pgmap v4910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 8995.055780 secs 2022-03-21T16:41:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:33 smithi016 ceph-mon[22010]: pgmap v4910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 8995.055780 secs 2022-03-21T16:41:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:33 smithi079 ceph-mon[23158]: pgmap v4910: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 8995.055780 secs 2022-03-21T16:41:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:35 smithi002 ceph-mon[17167]: pgmap v4911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:35 smithi016 ceph-mon[22010]: pgmap v4911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:35 smithi079 ceph-mon[23158]: pgmap v4911: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:37 smithi002 ceph-mon[17167]: pgmap v4912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:37 smithi016 ceph-mon[22010]: pgmap v4912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:37 smithi079 ceph-mon[23158]: pgmap v4912: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9000.055903 secs 2022-03-21T16:41:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9000.055903 secs 2022-03-21T16:41:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9000.055903 secs 2022-03-21T16:41:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:39 smithi002 ceph-mon[17167]: pgmap v4913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:39 smithi016 ceph-mon[22010]: pgmap v4913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:39 smithi079 ceph-mon[23158]: pgmap v4913: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:41 smithi002 ceph-mon[17167]: pgmap v4914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:41 smithi016 ceph-mon[22010]: pgmap v4914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:41 smithi079 ceph-mon[23158]: pgmap v4914: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:43 smithi002 ceph-mon[17167]: pgmap v4915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9005.055991 secs 2022-03-21T16:41:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:43 smithi016 ceph-mon[22010]: pgmap v4915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9005.055991 secs 2022-03-21T16:41:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:43 smithi079 ceph-mon[23158]: pgmap v4915: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9005.055991 secs 2022-03-21T16:41:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:45 smithi002 ceph-mon[17167]: pgmap v4916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:45 smithi016 ceph-mon[22010]: pgmap v4916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:45 smithi079 ceph-mon[23158]: pgmap v4916: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:47 smithi002 ceph-mon[17167]: pgmap v4917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:47 smithi016 ceph-mon[22010]: pgmap v4917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:47 smithi079 ceph-mon[23158]: pgmap v4917: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9010.056128 secs 2022-03-21T16:41:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9010.056128 secs 2022-03-21T16:41:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9010.056128 secs 2022-03-21T16:41:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:49 smithi002 ceph-mon[17167]: pgmap v4918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:49 smithi016 ceph-mon[22010]: pgmap v4918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:49 smithi079 ceph-mon[23158]: pgmap v4918: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:51 smithi002 ceph-mon[17167]: pgmap v4919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:51 smithi016 ceph-mon[22010]: pgmap v4919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:51 smithi079 ceph-mon[23158]: pgmap v4919: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:53 smithi002 ceph-mon[17167]: pgmap v4920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9015.056276 secs 2022-03-21T16:41:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:53 smithi016 ceph-mon[22010]: pgmap v4920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9015.056276 secs 2022-03-21T16:41:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:53 smithi079 ceph-mon[23158]: pgmap v4920: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:41:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9015.056276 secs 2022-03-21T16:41:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:55 smithi002 ceph-mon[17167]: pgmap v4921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:55 smithi016 ceph-mon[22010]: pgmap v4921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:55 smithi079 ceph-mon[23158]: pgmap v4921: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:41:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:57 smithi002 ceph-mon[17167]: pgmap v4922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:57 smithi016 ceph-mon[22010]: pgmap v4922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:57 smithi079 ceph-mon[23158]: pgmap v4922: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:41:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9020.056405 secs 2022-03-21T16:41:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9020.056405 secs 2022-03-21T16:41:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9020.056405 secs 2022-03-21T16:41:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:41:59 smithi016 ceph-mon[22010]: pgmap v4923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:41:59 smithi079 ceph-mon[23158]: pgmap v4923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:41:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:41:59 smithi002 ceph-mon[17167]: pgmap v4923: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:01 smithi016 ceph-mon[22010]: pgmap v4924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:42:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:42:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:01 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:42:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:01 smithi079 ceph-mon[23158]: pgmap v4924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:42:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:42:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:01 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:42:01.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:01 smithi002 ceph-mon[17167]: pgmap v4924: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:01.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:42:01.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:42:01.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:01 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:42:02.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:02 smithi002 ceph-mon[17167]: pgmap v4925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:02.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:42:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:02 smithi016 ceph-mon[22010]: pgmap v4925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:42:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:02 smithi079 ceph-mon[23158]: pgmap v4925: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:42:03.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9025.056559 secs 2022-03-21T16:42:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9025.056559 secs 2022-03-21T16:42:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9025.056559 secs 2022-03-21T16:42:04.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:04 smithi002 ceph-mon[17167]: pgmap v4926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:04 smithi016 ceph-mon[22010]: pgmap v4926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:04 smithi079 ceph-mon[23158]: pgmap v4926: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:06.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:06 smithi002 ceph-mon[17167]: pgmap v4927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:06 smithi016 ceph-mon[22010]: pgmap v4927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:06 smithi079 ceph-mon[23158]: pgmap v4927: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9030.056663 secs 2022-03-21T16:42:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9030.056663 secs 2022-03-21T16:42:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9030.056663 secs 2022-03-21T16:42:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:08 smithi016 ceph-mon[22010]: pgmap v4928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:08 smithi079 ceph-mon[23158]: pgmap v4928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:08 smithi002 ceph-mon[17167]: pgmap v4928: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:10 smithi016 ceph-mon[22010]: pgmap v4929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:10 smithi079 ceph-mon[23158]: pgmap v4929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:10 smithi002 ceph-mon[17167]: pgmap v4929: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:12 smithi016 ceph-mon[22010]: pgmap v4930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9035.056780 secs 2022-03-21T16:42:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:12 smithi079 ceph-mon[23158]: pgmap v4930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9035.056780 secs 2022-03-21T16:42:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:12 smithi002 ceph-mon[17167]: pgmap v4930: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9035.056780 secs 2022-03-21T16:42:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:14 smithi016 ceph-mon[22010]: pgmap v4931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:14 smithi079 ceph-mon[23158]: pgmap v4931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:14 smithi002 ceph-mon[17167]: pgmap v4931: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:16 smithi016 ceph-mon[22010]: pgmap v4932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:16 smithi079 ceph-mon[23158]: pgmap v4932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:16 smithi002 ceph-mon[17167]: pgmap v4932: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9040.056900 secs 2022-03-21T16:42:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9040.056900 secs 2022-03-21T16:42:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9040.056900 secs 2022-03-21T16:42:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:18 smithi079 ceph-mon[23158]: pgmap v4933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:18 smithi002 ceph-mon[17167]: pgmap v4933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:18 smithi016 ceph-mon[22010]: pgmap v4933: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:20 smithi079 ceph-mon[23158]: pgmap v4934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:21.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:20 smithi002 ceph-mon[17167]: pgmap v4934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:20 smithi016 ceph-mon[22010]: pgmap v4934: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:42:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:42:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:42:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:42:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:42:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:42:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:22 smithi079 ceph-mon[23158]: pgmap v4935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9045.057015 secs 2022-03-21T16:42:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:22 smithi002 ceph-mon[17167]: pgmap v4935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9045.057015 secs 2022-03-21T16:42:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:22 smithi016 ceph-mon[22010]: pgmap v4935: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9045.057015 secs 2022-03-21T16:42:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:24 smithi079 ceph-mon[23158]: pgmap v4936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:24 smithi002 ceph-mon[17167]: pgmap v4936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:24 smithi016 ceph-mon[22010]: pgmap v4936: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:26 smithi079 ceph-mon[23158]: pgmap v4937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:26 smithi002 ceph-mon[17167]: pgmap v4937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:26 smithi016 ceph-mon[22010]: pgmap v4937: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9050.057131 secs 2022-03-21T16:42:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9050.057131 secs 2022-03-21T16:42:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9050.057131 secs 2022-03-21T16:42:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:28 smithi079 ceph-mon[23158]: pgmap v4938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:28 smithi002 ceph-mon[17167]: pgmap v4938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:28 smithi016 ceph-mon[22010]: pgmap v4938: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:30 smithi079 ceph-mon[23158]: pgmap v4939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:30 smithi002 ceph-mon[17167]: pgmap v4939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:30 smithi016 ceph-mon[22010]: pgmap v4939: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:32 smithi079 ceph-mon[23158]: pgmap v4940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9055.057251 secs 2022-03-21T16:42:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:32 smithi002 ceph-mon[17167]: pgmap v4940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9055.057251 secs 2022-03-21T16:42:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:32 smithi016 ceph-mon[22010]: pgmap v4940: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9055.057251 secs 2022-03-21T16:42:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:34 smithi079 ceph-mon[23158]: pgmap v4941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:42:35.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:34 smithi002 ceph-mon[17167]: pgmap v4941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:42:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:34 smithi016 ceph-mon[22010]: pgmap v4941: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4 op/s 2022-03-21T16:42:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:36 smithi079 ceph-mon[23158]: pgmap v4942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:36 smithi002 ceph-mon[17167]: pgmap v4942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:36 smithi016 ceph-mon[22010]: pgmap v4942: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:42:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9060.057405 secs 2022-03-21T16:42:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9060.057405 secs 2022-03-21T16:42:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9060.057405 secs 2022-03-21T16:42:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:38 smithi079 ceph-mon[23158]: pgmap v4943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:38 smithi002 ceph-mon[17167]: pgmap v4943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:38 smithi016 ceph-mon[22010]: pgmap v4943: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:40 smithi079 ceph-mon[23158]: pgmap v4944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:40 smithi002 ceph-mon[17167]: pgmap v4944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:40 smithi016 ceph-mon[22010]: pgmap v4944: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:42 smithi079 ceph-mon[23158]: pgmap v4945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9065.057530 secs 2022-03-21T16:42:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:42 smithi002 ceph-mon[17167]: pgmap v4945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9065.057530 secs 2022-03-21T16:42:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:42 smithi016 ceph-mon[22010]: pgmap v4945: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9065.057530 secs 2022-03-21T16:42:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:44 smithi002 ceph-mon[17167]: pgmap v4946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:44 smithi016 ceph-mon[22010]: pgmap v4946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:44 smithi079 ceph-mon[23158]: pgmap v4946: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:47.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:46 smithi002 ceph-mon[17167]: pgmap v4947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:46 smithi016 ceph-mon[22010]: pgmap v4947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:46 smithi079 ceph-mon[23158]: pgmap v4947: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:48.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:47 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9070.057646 secs 2022-03-21T16:42:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:47 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9070.057646 secs 2022-03-21T16:42:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:47 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9070.057646 secs 2022-03-21T16:42:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:48 smithi002 ceph-mon[17167]: pgmap v4948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:48 smithi016 ceph-mon[22010]: pgmap v4948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:48 smithi079 ceph-mon[23158]: pgmap v4948: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:50 smithi002 ceph-mon[17167]: pgmap v4949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:50 smithi016 ceph-mon[22010]: pgmap v4949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:50 smithi079 ceph-mon[23158]: pgmap v4949: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:52 smithi002 ceph-mon[17167]: pgmap v4950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:52 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9075.057739 secs 2022-03-21T16:42:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:52 smithi016 ceph-mon[22010]: pgmap v4950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:52 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9075.057739 secs 2022-03-21T16:42:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:52 smithi079 ceph-mon[23158]: pgmap v4950: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:42:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:52 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9075.057739 secs 2022-03-21T16:42:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:54 smithi002 ceph-mon[17167]: pgmap v4951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:54 smithi016 ceph-mon[22010]: pgmap v4951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:54 smithi079 ceph-mon[23158]: pgmap v4951: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:42:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:57 smithi002 ceph-mon[17167]: pgmap v4952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:57 smithi016 ceph-mon[22010]: pgmap v4952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:57 smithi079 ceph-mon[23158]: pgmap v4952: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:42:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9080.057868 secs 2022-03-21T16:42:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9080.057868 secs 2022-03-21T16:42:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9080.057868 secs 2022-03-21T16:42:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:42:59 smithi002 ceph-mon[17167]: pgmap v4953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:42:59 smithi016 ceph-mon[22010]: pgmap v4953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:42:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:42:59 smithi079 ceph-mon[23158]: pgmap v4953: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:01 smithi002 ceph-mon[17167]: pgmap v4954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:01 smithi016 ceph-mon[22010]: pgmap v4954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:01 smithi079 ceph-mon[23158]: pgmap v4954: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:02.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:43:02.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:43:02.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:02 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:43:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:43:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:43:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:02 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:43:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:43:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:43:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:02 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:43:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:03 smithi002 ceph-mon[17167]: pgmap v4955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9085.057927 secs 2022-03-21T16:43:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:03 smithi016 ceph-mon[22010]: pgmap v4955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9085.057927 secs 2022-03-21T16:43:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:03 smithi079 ceph-mon[23158]: pgmap v4955: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9085.057927 secs 2022-03-21T16:43:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:05 smithi002 ceph-mon[17167]: pgmap v4956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:05 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:05 smithi016 ceph-mon[22010]: pgmap v4956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:05 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:05 smithi079 ceph-mon[23158]: pgmap v4956: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:05 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:06 smithi002 ceph-mon[17167]: pgmap v4957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:06.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:06 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:06 smithi016 ceph-mon[22010]: pgmap v4957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:06 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:06 smithi079 ceph-mon[23158]: pgmap v4957: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:06 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:43:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9090.058049 secs 2022-03-21T16:43:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9090.058049 secs 2022-03-21T16:43:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9090.058049 secs 2022-03-21T16:43:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:08 smithi016 ceph-mon[22010]: pgmap v4958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:08 smithi079 ceph-mon[23158]: pgmap v4958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:08 smithi002 ceph-mon[17167]: pgmap v4958: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:10 smithi079 ceph-mon[23158]: pgmap v4959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:10 smithi002 ceph-mon[17167]: pgmap v4959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:10 smithi016 ceph-mon[22010]: pgmap v4959: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:12 smithi079 ceph-mon[23158]: pgmap v4960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9095.058172 secs 2022-03-21T16:43:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:12 smithi002 ceph-mon[17167]: pgmap v4960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9095.058172 secs 2022-03-21T16:43:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:12 smithi016 ceph-mon[22010]: pgmap v4960: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9095.058172 secs 2022-03-21T16:43:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:14 smithi079 ceph-mon[23158]: pgmap v4961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:14 smithi002 ceph-mon[17167]: pgmap v4961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:14 smithi016 ceph-mon[22010]: pgmap v4961: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:16 smithi079 ceph-mon[23158]: pgmap v4962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:17.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:16 smithi002 ceph-mon[17167]: pgmap v4962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:16 smithi016 ceph-mon[22010]: pgmap v4962: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9100.058246 secs 2022-03-21T16:43:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9100.058246 secs 2022-03-21T16:43:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9100.058246 secs 2022-03-21T16:43:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:18 smithi079 ceph-mon[23158]: pgmap v4963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:18 smithi002 ceph-mon[17167]: pgmap v4963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:18 smithi016 ceph-mon[22010]: pgmap v4963: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:20 smithi079 ceph-mon[23158]: pgmap v4964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:20 smithi002 ceph-mon[17167]: pgmap v4964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:20 smithi016 ceph-mon[22010]: pgmap v4964: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:43:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:43:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:43:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:43:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:43:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:43:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:22 smithi079 ceph-mon[23158]: pgmap v4965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9105.058360 secs 2022-03-21T16:43:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:22 smithi002 ceph-mon[17167]: pgmap v4965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9105.058360 secs 2022-03-21T16:43:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:22 smithi016 ceph-mon[22010]: pgmap v4965: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9105.058360 secs 2022-03-21T16:43:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:24 smithi016 ceph-mon[22010]: pgmap v4966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:24 smithi079 ceph-mon[23158]: pgmap v4966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:24 smithi002 ceph-mon[17167]: pgmap v4966: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:26 smithi079 ceph-mon[23158]: pgmap v4967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:26 smithi002 ceph-mon[17167]: pgmap v4967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:27.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:26 smithi016 ceph-mon[22010]: pgmap v4967: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9110.058487 secs 2022-03-21T16:43:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:27 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9110.058487 secs 2022-03-21T16:43:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9110.058487 secs 2022-03-21T16:43:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:28 smithi016 ceph-mon[22010]: pgmap v4968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:28 smithi079 ceph-mon[23158]: pgmap v4968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:28 smithi002 ceph-mon[17167]: pgmap v4968: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:30 smithi079 ceph-mon[23158]: pgmap v4969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:30 smithi002 ceph-mon[17167]: pgmap v4969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:30 smithi016 ceph-mon[22010]: pgmap v4969: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:32 smithi079 ceph-mon[23158]: pgmap v4970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:32 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9115.058605 secs 2022-03-21T16:43:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:32 smithi002 ceph-mon[17167]: pgmap v4970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:32 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9115.058605 secs 2022-03-21T16:43:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:32 smithi016 ceph-mon[22010]: pgmap v4970: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:32 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9115.058605 secs 2022-03-21T16:43:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:34 smithi079 ceph-mon[23158]: pgmap v4971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:34 smithi002 ceph-mon[17167]: pgmap v4971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:34 smithi016 ceph-mon[22010]: pgmap v4971: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:36 smithi002 ceph-mon[17167]: pgmap v4972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:36 smithi016 ceph-mon[22010]: pgmap v4972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:36 smithi079 ceph-mon[23158]: pgmap v4972: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:38.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:37 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9120.058717 secs 2022-03-21T16:43:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9120.058717 secs 2022-03-21T16:43:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9120.058717 secs 2022-03-21T16:43:39.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:38 smithi002 ceph-mon[17167]: pgmap v4973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:38 smithi016 ceph-mon[22010]: pgmap v4973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:38 smithi079 ceph-mon[23158]: pgmap v4973: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:40 smithi002 ceph-mon[17167]: pgmap v4974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:40 smithi016 ceph-mon[22010]: pgmap v4974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:40 smithi079 ceph-mon[23158]: pgmap v4974: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:42 smithi002 ceph-mon[17167]: pgmap v4975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:42 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9125.058858 secs 2022-03-21T16:43:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:42 smithi016 ceph-mon[22010]: pgmap v4975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9125.058858 secs 2022-03-21T16:43:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:42 smithi079 ceph-mon[23158]: pgmap v4975: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9125.058858 secs 2022-03-21T16:43:45.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:44 smithi002 ceph-mon[17167]: pgmap v4976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:44 smithi016 ceph-mon[22010]: pgmap v4976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:44 smithi079 ceph-mon[23158]: pgmap v4976: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:46 smithi002 ceph-mon[17167]: pgmap v4977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:46 smithi016 ceph-mon[22010]: pgmap v4977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:46 smithi079 ceph-mon[23158]: pgmap v4977: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:48.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9130.058968 secs 2022-03-21T16:43:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9130.058968 secs 2022-03-21T16:43:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9130.058968 secs 2022-03-21T16:43:49.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:49 smithi002 ceph-mon[17167]: pgmap v4978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:49 smithi016 ceph-mon[22010]: pgmap v4978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:49 smithi079 ceph-mon[23158]: pgmap v4978: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:51 smithi002 ceph-mon[17167]: pgmap v4979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:51 smithi016 ceph-mon[22010]: pgmap v4979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:51 smithi079 ceph-mon[23158]: pgmap v4979: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:53.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:53 smithi002 ceph-mon[17167]: pgmap v4980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9135.059070 secs 2022-03-21T16:43:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:53 smithi016 ceph-mon[22010]: pgmap v4980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9135.059070 secs 2022-03-21T16:43:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:53 smithi079 ceph-mon[23158]: pgmap v4980: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:43:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9135.059070 secs 2022-03-21T16:43:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:55 smithi002 ceph-mon[17167]: pgmap v4981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:55 smithi016 ceph-mon[22010]: pgmap v4981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:55 smithi079 ceph-mon[23158]: pgmap v4981: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:43:57.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:57 smithi002 ceph-mon[17167]: pgmap v4982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:57 smithi016 ceph-mon[22010]: pgmap v4982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:57 smithi079 ceph-mon[23158]: pgmap v4982: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:43:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9140.059208 secs 2022-03-21T16:43:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9140.059208 secs 2022-03-21T16:43:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9140.059208 secs 2022-03-21T16:43:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:43:59 smithi002 ceph-mon[17167]: pgmap v4983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:43:59 smithi016 ceph-mon[22010]: pgmap v4983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:43:59.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:43:59 smithi079 ceph-mon[23158]: pgmap v4983: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:01 smithi002 ceph-mon[17167]: pgmap v4984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:01 smithi016 ceph-mon[22010]: pgmap v4984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:01 smithi079 ceph-mon[23158]: pgmap v4984: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:03 smithi002 ceph-mon[17167]: pgmap v4985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9145.059334 secs 2022-03-21T16:44:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:03 smithi016 ceph-mon[22010]: pgmap v4985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:03.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9145.059334 secs 2022-03-21T16:44:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:03 smithi079 ceph-mon[23158]: pgmap v4985: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9145.059334 secs 2022-03-21T16:44:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:05 smithi002 ceph-mon[17167]: pgmap v4986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:05 smithi016 ceph-mon[22010]: pgmap v4986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:05 smithi079 ceph-mon[23158]: pgmap v4986: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:07.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:07 smithi002 ceph-mon[17167]: pgmap v4987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:44:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:44:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:44:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:07 smithi016 ceph-mon[22010]: pgmap v4987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:44:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:44:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:44:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:07 smithi079 ceph-mon[23158]: pgmap v4987: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:44:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:44:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:44:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:44:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9150.059441 secs 2022-03-21T16:44:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:44:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9150.059441 secs 2022-03-21T16:44:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:44:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9150.059441 secs 2022-03-21T16:44:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:09 smithi002 ceph-mon[17167]: pgmap v4988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:09 smithi016 ceph-mon[22010]: pgmap v4988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:09 smithi079 ceph-mon[23158]: pgmap v4988: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:11 smithi002 ceph-mon[17167]: pgmap v4989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:11 smithi016 ceph-mon[22010]: pgmap v4989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:11 smithi079 ceph-mon[23158]: pgmap v4989: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:13 smithi002 ceph-mon[17167]: pgmap v4990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9155.059579 secs 2022-03-21T16:44:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:13 smithi016 ceph-mon[22010]: pgmap v4990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9155.059579 secs 2022-03-21T16:44:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:13 smithi079 ceph-mon[23158]: pgmap v4990: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9155.059579 secs 2022-03-21T16:44:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:15 smithi002 ceph-mon[17167]: pgmap v4991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:15 smithi016 ceph-mon[22010]: pgmap v4991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:15 smithi079 ceph-mon[23158]: pgmap v4991: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:17 smithi002 ceph-mon[17167]: pgmap v4992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:17 smithi016 ceph-mon[22010]: pgmap v4992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:17 smithi079 ceph-mon[23158]: pgmap v4992: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9160.059679 secs 2022-03-21T16:44:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9160.059679 secs 2022-03-21T16:44:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9160.059679 secs 2022-03-21T16:44:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:19 smithi002 ceph-mon[17167]: pgmap v4993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:19 smithi016 ceph-mon[22010]: pgmap v4993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:19 smithi079 ceph-mon[23158]: pgmap v4993: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:21 smithi002 ceph-mon[17167]: pgmap v4994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:21 smithi016 ceph-mon[22010]: pgmap v4994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:21 smithi079 ceph-mon[23158]: pgmap v4994: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:44:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:44:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:44:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:44:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:44:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:44:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:23 smithi002 ceph-mon[17167]: pgmap v4995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9165.059804 secs 2022-03-21T16:44:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:23 smithi016 ceph-mon[22010]: pgmap v4995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9165.059804 secs 2022-03-21T16:44:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:23 smithi079 ceph-mon[23158]: pgmap v4995: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9165.059804 secs 2022-03-21T16:44:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:25 smithi002 ceph-mon[17167]: pgmap v4996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:25 smithi016 ceph-mon[22010]: pgmap v4996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:25 smithi079 ceph-mon[23158]: pgmap v4996: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:27 smithi016 ceph-mon[22010]: pgmap v4997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:27 smithi079 ceph-mon[23158]: pgmap v4997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:27 smithi002 ceph-mon[17167]: pgmap v4997: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9170.059923 secs 2022-03-21T16:44:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9170.059923 secs 2022-03-21T16:44:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9170.059923 secs 2022-03-21T16:44:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:29 smithi016 ceph-mon[22010]: pgmap v4998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:29 smithi079 ceph-mon[23158]: pgmap v4998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:29 smithi002 ceph-mon[17167]: pgmap v4998: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:31 smithi016 ceph-mon[22010]: pgmap v4999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:31 smithi079 ceph-mon[23158]: pgmap v4999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:31.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:31 smithi002 ceph-mon[17167]: pgmap v4999: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:33 smithi016 ceph-mon[22010]: pgmap v5000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9175.060046 secs 2022-03-21T16:44:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:33 smithi079 ceph-mon[23158]: pgmap v5000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9175.060046 secs 2022-03-21T16:44:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:33 smithi002 ceph-mon[17167]: pgmap v5000: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9175.060046 secs 2022-03-21T16:44:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:35 smithi016 ceph-mon[22010]: pgmap v5001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:35 smithi079 ceph-mon[23158]: pgmap v5001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:35.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:35 smithi002 ceph-mon[17167]: pgmap v5001: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:37 smithi016 ceph-mon[22010]: pgmap v5002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:37 smithi079 ceph-mon[23158]: pgmap v5002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:37 smithi002 ceph-mon[17167]: pgmap v5002: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9180.060177 secs 2022-03-21T16:44:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9180.060177 secs 2022-03-21T16:44:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9180.060177 secs 2022-03-21T16:44:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:39 smithi016 ceph-mon[22010]: pgmap v5003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:39 smithi079 ceph-mon[23158]: pgmap v5003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:39.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:39 smithi002 ceph-mon[17167]: pgmap v5003: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:41 smithi016 ceph-mon[22010]: pgmap v5004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:41 smithi079 ceph-mon[23158]: pgmap v5004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:41 smithi002 ceph-mon[17167]: pgmap v5004: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:43 smithi016 ceph-mon[22010]: pgmap v5005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9185.060300 secs 2022-03-21T16:44:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:43 smithi079 ceph-mon[23158]: pgmap v5005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9185.060300 secs 2022-03-21T16:44:43.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:43 smithi002 ceph-mon[17167]: pgmap v5005: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9185.060300 secs 2022-03-21T16:44:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:45 smithi079 ceph-mon[23158]: pgmap v5006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:45 smithi002 ceph-mon[17167]: pgmap v5006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:45 smithi016 ceph-mon[22010]: pgmap v5006: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:47 smithi079 ceph-mon[23158]: pgmap v5007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:47 smithi002 ceph-mon[17167]: pgmap v5007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:47 smithi016 ceph-mon[22010]: pgmap v5007: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9190.060417 secs 2022-03-21T16:44:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9190.060417 secs 2022-03-21T16:44:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9190.060417 secs 2022-03-21T16:44:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:49 smithi079 ceph-mon[23158]: pgmap v5008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:49 smithi002 ceph-mon[17167]: pgmap v5008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:49 smithi016 ceph-mon[22010]: pgmap v5008: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:51 smithi079 ceph-mon[23158]: pgmap v5009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:51 smithi002 ceph-mon[17167]: pgmap v5009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:51 smithi016 ceph-mon[22010]: pgmap v5009: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:53 smithi079 ceph-mon[23158]: pgmap v5010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9195.060538 secs 2022-03-21T16:44:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:53 smithi002 ceph-mon[17167]: pgmap v5010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9195.060538 secs 2022-03-21T16:44:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:53 smithi016 ceph-mon[22010]: pgmap v5010: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:44:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9195.060538 secs 2022-03-21T16:44:55.418 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:55 smithi079 ceph-mon[23158]: pgmap v5011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:55 smithi002 ceph-mon[17167]: pgmap v5011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:55 smithi016 ceph-mon[22010]: pgmap v5011: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:44:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:57 smithi079 ceph-mon[23158]: pgmap v5012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:57 smithi002 ceph-mon[17167]: pgmap v5012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:57 smithi016 ceph-mon[22010]: pgmap v5012: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:44:58.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9200.060653 secs 2022-03-21T16:44:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9200.060653 secs 2022-03-21T16:44:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9200.060653 secs 2022-03-21T16:44:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:44:59 smithi079 ceph-mon[23158]: pgmap v5013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:44:59 smithi002 ceph-mon[17167]: pgmap v5013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:44:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:44:59 smithi016 ceph-mon[22010]: pgmap v5013: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:01 smithi079 ceph-mon[23158]: pgmap v5014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:01 smithi002 ceph-mon[17167]: pgmap v5014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:01 smithi016 ceph-mon[22010]: pgmap v5014: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:02 smithi079 ceph-mon[23158]: pgmap v5015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:02 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9205.060780 secs 2022-03-21T16:45:03.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:02 smithi002 ceph-mon[17167]: pgmap v5015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:02 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9205.060780 secs 2022-03-21T16:45:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:02 smithi016 ceph-mon[22010]: pgmap v5015: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:02 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9205.060780 secs 2022-03-21T16:45:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:04 smithi079 ceph-mon[23158]: pgmap v5016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:05.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:04 smithi002 ceph-mon[17167]: pgmap v5016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:04 smithi016 ceph-mon[22010]: pgmap v5016: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:07.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:06 smithi002 ceph-mon[17167]: pgmap v5017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:06 smithi079 ceph-mon[23158]: pgmap v5017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:06 smithi016 ceph-mon[22010]: pgmap v5017: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:45:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:45:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:45:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9210.060872 secs 2022-03-21T16:45:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:07 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:45:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:45:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:45:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9210.060872 secs 2022-03-21T16:45:08.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:07 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:45:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:45:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:45:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9210.060872 secs 2022-03-21T16:45:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:07 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:08 smithi079 ceph-mon[23158]: pgmap v5018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:08 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:08 smithi002 ceph-mon[17167]: pgmap v5018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:08 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:08 smithi016 ceph-mon[22010]: pgmap v5018: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:08 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:45:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:10 smithi079 ceph-mon[23158]: pgmap v5019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:11.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:10 smithi002 ceph-mon[17167]: pgmap v5019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:10 smithi016 ceph-mon[22010]: pgmap v5019: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:12 smithi079 ceph-mon[23158]: pgmap v5020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9215.061007 secs 2022-03-21T16:45:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:12 smithi002 ceph-mon[17167]: pgmap v5020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9215.061007 secs 2022-03-21T16:45:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:12 smithi016 ceph-mon[22010]: pgmap v5020: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9215.061007 secs 2022-03-21T16:45:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:14 smithi079 ceph-mon[23158]: pgmap v5021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:14 smithi002 ceph-mon[17167]: pgmap v5021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:14 smithi016 ceph-mon[22010]: pgmap v5021: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:16 smithi002 ceph-mon[17167]: pgmap v5022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:16 smithi016 ceph-mon[22010]: pgmap v5022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:16 smithi079 ceph-mon[23158]: pgmap v5022: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9220.061161 secs 2022-03-21T16:45:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9220.061161 secs 2022-03-21T16:45:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9220.061161 secs 2022-03-21T16:45:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:18 smithi002 ceph-mon[17167]: pgmap v5023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:18 smithi016 ceph-mon[22010]: pgmap v5023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:18 smithi079 ceph-mon[23158]: pgmap v5023: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:20 smithi002 ceph-mon[17167]: pgmap v5024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:20 smithi016 ceph-mon[22010]: pgmap v5024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:20 smithi079 ceph-mon[23158]: pgmap v5024: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:45:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:45:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:45:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:45:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:45:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:45:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:22 smithi002 ceph-mon[17167]: pgmap v5025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:22 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9225.061283 secs 2022-03-21T16:45:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:22 smithi016 ceph-mon[22010]: pgmap v5025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9225.061283 secs 2022-03-21T16:45:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:22 smithi079 ceph-mon[23158]: pgmap v5025: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9225.061283 secs 2022-03-21T16:45:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:24 smithi002 ceph-mon[17167]: pgmap v5026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:24 smithi016 ceph-mon[22010]: pgmap v5026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:24 smithi079 ceph-mon[23158]: pgmap v5026: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:27.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:26 smithi002 ceph-mon[17167]: pgmap v5027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:26 smithi016 ceph-mon[22010]: pgmap v5027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:26 smithi079 ceph-mon[23158]: pgmap v5027: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9230.061344 secs 2022-03-21T16:45:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9230.061344 secs 2022-03-21T16:45:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9230.061344 secs 2022-03-21T16:45:29.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:29 smithi002 ceph-mon[17167]: pgmap v5028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:29 smithi016 ceph-mon[22010]: pgmap v5028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:29 smithi079 ceph-mon[23158]: pgmap v5028: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:31 smithi002 ceph-mon[17167]: pgmap v5029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:31 smithi016 ceph-mon[22010]: pgmap v5029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:31 smithi079 ceph-mon[23158]: pgmap v5029: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:33.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:33 smithi002 ceph-mon[17167]: pgmap v5030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9235.061475 secs 2022-03-21T16:45:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:33 smithi016 ceph-mon[22010]: pgmap v5030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9235.061475 secs 2022-03-21T16:45:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:33 smithi079 ceph-mon[23158]: pgmap v5030: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9235.061475 secs 2022-03-21T16:45:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:35 smithi002 ceph-mon[17167]: pgmap v5031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:35 smithi016 ceph-mon[22010]: pgmap v5031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:35 smithi079 ceph-mon[23158]: pgmap v5031: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:37 smithi002 ceph-mon[17167]: pgmap v5032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:37 smithi016 ceph-mon[22010]: pgmap v5032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:37 smithi079 ceph-mon[23158]: pgmap v5032: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:38.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9240.061629 secs 2022-03-21T16:45:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9240.061629 secs 2022-03-21T16:45:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9240.061629 secs 2022-03-21T16:45:39.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:39 smithi002 ceph-mon[17167]: pgmap v5033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:39 smithi016 ceph-mon[22010]: pgmap v5033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:39 smithi079 ceph-mon[23158]: pgmap v5033: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:41.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:41 smithi002 ceph-mon[17167]: pgmap v5034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:41 smithi016 ceph-mon[22010]: pgmap v5034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:41 smithi079 ceph-mon[23158]: pgmap v5034: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:43.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:43 smithi002 ceph-mon[17167]: pgmap v5035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9245.061711 secs 2022-03-21T16:45:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:43 smithi016 ceph-mon[22010]: pgmap v5035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9245.061711 secs 2022-03-21T16:45:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:43 smithi079 ceph-mon[23158]: pgmap v5035: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9245.061711 secs 2022-03-21T16:45:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:45 smithi002 ceph-mon[17167]: pgmap v5036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:45 smithi016 ceph-mon[22010]: pgmap v5036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:45 smithi079 ceph-mon[23158]: pgmap v5036: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:47 smithi002 ceph-mon[17167]: pgmap v5037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:47 smithi016 ceph-mon[22010]: pgmap v5037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:47 smithi079 ceph-mon[23158]: pgmap v5037: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:48.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9250.061821 secs 2022-03-21T16:45:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9250.061821 secs 2022-03-21T16:45:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9250.061821 secs 2022-03-21T16:45:49.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:49 smithi002 ceph-mon[17167]: pgmap v5038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:49 smithi016 ceph-mon[22010]: pgmap v5038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:49 smithi079 ceph-mon[23158]: pgmap v5038: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:51 smithi002 ceph-mon[17167]: pgmap v5039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:51 smithi016 ceph-mon[22010]: pgmap v5039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:51 smithi079 ceph-mon[23158]: pgmap v5039: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:53 smithi002 ceph-mon[17167]: pgmap v5040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9255.061942 secs 2022-03-21T16:45:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:53 smithi016 ceph-mon[22010]: pgmap v5040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9255.061942 secs 2022-03-21T16:45:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:53 smithi079 ceph-mon[23158]: pgmap v5040: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:45:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9255.061942 secs 2022-03-21T16:45:55.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:55 smithi002 ceph-mon[17167]: pgmap v5041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:55 smithi016 ceph-mon[22010]: pgmap v5041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:55 smithi079 ceph-mon[23158]: pgmap v5041: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:45:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:57 smithi002 ceph-mon[17167]: pgmap v5042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:57 smithi016 ceph-mon[22010]: pgmap v5042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:57 smithi079 ceph-mon[23158]: pgmap v5042: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:45:58.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9260.062064 secs 2022-03-21T16:45:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9260.062064 secs 2022-03-21T16:45:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9260.062064 secs 2022-03-21T16:45:59.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:45:59 smithi002 ceph-mon[17167]: pgmap v5043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:45:59 smithi016 ceph-mon[22010]: pgmap v5043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:45:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:45:59 smithi079 ceph-mon[23158]: pgmap v5043: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:01.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:01 smithi002 ceph-mon[17167]: pgmap v5044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:01 smithi016 ceph-mon[22010]: pgmap v5044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:01 smithi079 ceph-mon[23158]: pgmap v5044: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:03 smithi002 ceph-mon[17167]: pgmap v5045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:46:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9265.062190 secs 2022-03-21T16:46:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:03 smithi016 ceph-mon[22010]: pgmap v5045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:46:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9265.062190 secs 2022-03-21T16:46:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:03 smithi079 ceph-mon[23158]: pgmap v5045: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2022-03-21T16:46:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9265.062190 secs 2022-03-21T16:46:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:05 smithi002 ceph-mon[17167]: pgmap v5046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:05 smithi016 ceph-mon[22010]: pgmap v5046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:05 smithi079 ceph-mon[23158]: pgmap v5046: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:07 smithi002 ceph-mon[17167]: pgmap v5047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:07 smithi016 ceph-mon[22010]: pgmap v5047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:07 smithi079 ceph-mon[23158]: pgmap v5047: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:08.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9270.062325 secs 2022-03-21T16:46:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9270.062325 secs 2022-03-21T16:46:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9270.062325 secs 2022-03-21T16:46:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:09 smithi002 ceph-mon[17167]: pgmap v5048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:46:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:46:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:09 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:46:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:09 smithi016 ceph-mon[22010]: pgmap v5048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:46:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:46:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:09 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:46:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:09 smithi079 ceph-mon[23158]: pgmap v5048: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:46:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:46:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:09 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:46:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:46:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:10 smithi079 ceph-mon[23158]: pgmap v5049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:10.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:46:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:10 smithi002 ceph-mon[17167]: pgmap v5049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:46:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:10 smithi016 ceph-mon[22010]: pgmap v5049: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:13.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:12 smithi002 ceph-mon[17167]: pgmap v5050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:12 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9275.062467 secs 2022-03-21T16:46:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:12 smithi016 ceph-mon[22010]: pgmap v5050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9275.062467 secs 2022-03-21T16:46:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:12 smithi079 ceph-mon[23158]: pgmap v5050: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9275.062467 secs 2022-03-21T16:46:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:14 smithi002 ceph-mon[17167]: pgmap v5051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:14 smithi016 ceph-mon[22010]: pgmap v5051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:14 smithi079 ceph-mon[23158]: pgmap v5051: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 4 op/s 2022-03-21T16:46:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:16 smithi002 ceph-mon[17167]: pgmap v5052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:16 smithi016 ceph-mon[22010]: pgmap v5052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:16 smithi079 ceph-mon[23158]: pgmap v5052: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:18.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:17 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9280.062619 secs 2022-03-21T16:46:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:17 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9280.062619 secs 2022-03-21T16:46:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:17 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9280.062619 secs 2022-03-21T16:46:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:18 smithi002 ceph-mon[17167]: pgmap v5053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:18 smithi016 ceph-mon[22010]: pgmap v5053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:18 smithi079 ceph-mon[23158]: pgmap v5053: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:20 smithi002 ceph-mon[17167]: pgmap v5054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:20 smithi016 ceph-mon[22010]: pgmap v5054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:20 smithi079 ceph-mon[23158]: pgmap v5054: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:22.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:46:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:46:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:46:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:46:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:46:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:46:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:23 smithi002 ceph-mon[17167]: pgmap v5055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9285.062737 secs 2022-03-21T16:46:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:23 smithi016 ceph-mon[22010]: pgmap v5055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9285.062737 secs 2022-03-21T16:46:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:23 smithi079 ceph-mon[23158]: pgmap v5055: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9285.062737 secs 2022-03-21T16:46:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:25 smithi002 ceph-mon[17167]: pgmap v5056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:25 smithi016 ceph-mon[22010]: pgmap v5056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:25 smithi079 ceph-mon[23158]: pgmap v5056: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:27 smithi002 ceph-mon[17167]: pgmap v5057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:27 smithi016 ceph-mon[22010]: pgmap v5057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:27 smithi079 ceph-mon[23158]: pgmap v5057: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:28.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9290.062810 secs 2022-03-21T16:46:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9290.062810 secs 2022-03-21T16:46:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9290.062810 secs 2022-03-21T16:46:29.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:29 smithi002 ceph-mon[17167]: pgmap v5058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:29 smithi016 ceph-mon[22010]: pgmap v5058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:29 smithi079 ceph-mon[23158]: pgmap v5058: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:31.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:31 smithi002 ceph-mon[17167]: pgmap v5059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:31 smithi016 ceph-mon[22010]: pgmap v5059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:31 smithi079 ceph-mon[23158]: pgmap v5059: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:33 smithi002 ceph-mon[17167]: pgmap v5060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9295.062913 secs 2022-03-21T16:46:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:33 smithi016 ceph-mon[22010]: pgmap v5060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:33.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9295.062913 secs 2022-03-21T16:46:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:33 smithi079 ceph-mon[23158]: pgmap v5060: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9295.062913 secs 2022-03-21T16:46:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:35 smithi002 ceph-mon[17167]: pgmap v5061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:35 smithi016 ceph-mon[22010]: pgmap v5061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:35 smithi079 ceph-mon[23158]: pgmap v5061: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:37.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:37 smithi002 ceph-mon[17167]: pgmap v5062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:37 smithi016 ceph-mon[22010]: pgmap v5062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:37 smithi079 ceph-mon[23158]: pgmap v5062: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:38.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9300.063032 secs 2022-03-21T16:46:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9300.063032 secs 2022-03-21T16:46:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9300.063032 secs 2022-03-21T16:46:39.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:39 smithi002 ceph-mon[17167]: pgmap v5063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:39 smithi016 ceph-mon[22010]: pgmap v5063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:39 smithi079 ceph-mon[23158]: pgmap v5063: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:41 smithi002 ceph-mon[17167]: pgmap v5064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:41 smithi016 ceph-mon[22010]: pgmap v5064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:41 smithi079 ceph-mon[23158]: pgmap v5064: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:43 smithi002 ceph-mon[17167]: pgmap v5065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:43.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9305.063169 secs 2022-03-21T16:46:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:43 smithi016 ceph-mon[22010]: pgmap v5065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9305.063169 secs 2022-03-21T16:46:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:43 smithi079 ceph-mon[23158]: pgmap v5065: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9305.063169 secs 2022-03-21T16:46:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:45 smithi002 ceph-mon[17167]: pgmap v5066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:45 smithi016 ceph-mon[22010]: pgmap v5066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:45 smithi079 ceph-mon[23158]: pgmap v5066: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:47 smithi002 ceph-mon[17167]: pgmap v5067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:47 smithi016 ceph-mon[22010]: pgmap v5067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:47 smithi079 ceph-mon[23158]: pgmap v5067: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:48.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9310.063288 secs 2022-03-21T16:46:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9310.063288 secs 2022-03-21T16:46:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9310.063288 secs 2022-03-21T16:46:49.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:49 smithi002 ceph-mon[17167]: pgmap v5068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:49 smithi016 ceph-mon[22010]: pgmap v5068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:49 smithi079 ceph-mon[23158]: pgmap v5068: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:51.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:51 smithi002 ceph-mon[17167]: pgmap v5069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:51 smithi016 ceph-mon[22010]: pgmap v5069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:51 smithi079 ceph-mon[23158]: pgmap v5069: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:53 smithi002 ceph-mon[17167]: pgmap v5070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9315.063406 secs 2022-03-21T16:46:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:53 smithi016 ceph-mon[22010]: pgmap v5070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9315.063406 secs 2022-03-21T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:53 smithi079 ceph-mon[23158]: pgmap v5070: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9315.063406 secs 2022-03-21T16:46:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:55 smithi002 ceph-mon[17167]: pgmap v5071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:55 smithi016 ceph-mon[22010]: pgmap v5071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:55 smithi079 ceph-mon[23158]: pgmap v5071: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:57 smithi002 ceph-mon[17167]: pgmap v5072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:57 smithi016 ceph-mon[22010]: pgmap v5072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:57 smithi079 ceph-mon[23158]: pgmap v5072: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:46:58.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9320.063499 secs 2022-03-21T16:46:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9320.063499 secs 2022-03-21T16:46:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9320.063499 secs 2022-03-21T16:46:59.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:46:59 smithi002 ceph-mon[17167]: pgmap v5073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:46:59 smithi016 ceph-mon[22010]: pgmap v5073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:46:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:46:59 smithi079 ceph-mon[23158]: pgmap v5073: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:01 smithi002 ceph-mon[17167]: pgmap v5074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:01 smithi016 ceph-mon[22010]: pgmap v5074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:01 smithi079 ceph-mon[23158]: pgmap v5074: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:03 smithi002 ceph-mon[17167]: pgmap v5075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9325.063616 secs 2022-03-21T16:47:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:03 smithi016 ceph-mon[22010]: pgmap v5075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9325.063616 secs 2022-03-21T16:47:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:03 smithi079 ceph-mon[23158]: pgmap v5075: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9325.063616 secs 2022-03-21T16:47:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:05 smithi002 ceph-mon[17167]: pgmap v5076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:05 smithi016 ceph-mon[22010]: pgmap v5076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:05 smithi079 ceph-mon[23158]: pgmap v5076: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:07 smithi002 ceph-mon[17167]: pgmap v5077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:07 smithi016 ceph-mon[22010]: pgmap v5077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:07 smithi079 ceph-mon[23158]: pgmap v5077: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9330.063739 secs 2022-03-21T16:47:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9330.063739 secs 2022-03-21T16:47:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9330.063739 secs 2022-03-21T16:47:09.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:09 smithi002 ceph-mon[17167]: pgmap v5078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:09 smithi016 ceph-mon[22010]: pgmap v5078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:09.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:09 smithi079 ceph-mon[23158]: pgmap v5078: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:47:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:47:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:10 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:47:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:47:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:47:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:10 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:47:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:47:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:47:10.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:10 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:47:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:11 smithi002 ceph-mon[17167]: pgmap v5079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:47:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:11 smithi016 ceph-mon[22010]: pgmap v5079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:47:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:11 smithi079 ceph-mon[23158]: pgmap v5079: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:47:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:13 smithi002 ceph-mon[17167]: pgmap v5080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9335.063821 secs 2022-03-21T16:47:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:13 smithi016 ceph-mon[22010]: pgmap v5080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9335.063821 secs 2022-03-21T16:47:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:13 smithi079 ceph-mon[23158]: pgmap v5080: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9335.063821 secs 2022-03-21T16:47:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:15 smithi016 ceph-mon[22010]: pgmap v5081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:15 smithi079 ceph-mon[23158]: pgmap v5081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:15 smithi002 ceph-mon[17167]: pgmap v5081: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:17 smithi016 ceph-mon[22010]: pgmap v5082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:17 smithi079 ceph-mon[23158]: pgmap v5082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:17 smithi002 ceph-mon[17167]: pgmap v5082: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9340.063940 secs 2022-03-21T16:47:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9340.063940 secs 2022-03-21T16:47:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9340.063940 secs 2022-03-21T16:47:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:19 smithi016 ceph-mon[22010]: pgmap v5083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:19 smithi079 ceph-mon[23158]: pgmap v5083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:19 smithi002 ceph-mon[17167]: pgmap v5083: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:21 smithi016 ceph-mon[22010]: pgmap v5084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:21 smithi079 ceph-mon[23158]: pgmap v5084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:21 smithi002 ceph-mon[17167]: pgmap v5084: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:47:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:47:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:47:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:47:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:47:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:23 smithi016 ceph-mon[22010]: pgmap v5085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9345.064079 secs 2022-03-21T16:47:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:23 smithi079 ceph-mon[23158]: pgmap v5085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9345.064079 secs 2022-03-21T16:47:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:23 smithi002 ceph-mon[17167]: pgmap v5085: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9345.064079 secs 2022-03-21T16:47:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:25 smithi016 ceph-mon[22010]: pgmap v5086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:25 smithi079 ceph-mon[23158]: pgmap v5086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:25.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:25 smithi002 ceph-mon[17167]: pgmap v5086: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:27 smithi016 ceph-mon[22010]: pgmap v5087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:27 smithi079 ceph-mon[23158]: pgmap v5087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:27 smithi002 ceph-mon[17167]: pgmap v5087: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9350.064195 secs 2022-03-21T16:47:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9350.064195 secs 2022-03-21T16:47:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9350.064195 secs 2022-03-21T16:47:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:29 smithi016 ceph-mon[22010]: pgmap v5088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:29 smithi079 ceph-mon[23158]: pgmap v5088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:29.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:29 smithi002 ceph-mon[17167]: pgmap v5088: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:31 smithi016 ceph-mon[22010]: pgmap v5089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:31 smithi079 ceph-mon[23158]: pgmap v5089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:31 smithi002 ceph-mon[17167]: pgmap v5089: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:33 smithi016 ceph-mon[22010]: pgmap v5090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9355.064321 secs 2022-03-21T16:47:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:33 smithi079 ceph-mon[23158]: pgmap v5090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9355.064321 secs 2022-03-21T16:47:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:33 smithi002 ceph-mon[17167]: pgmap v5090: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9355.064321 secs 2022-03-21T16:47:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:35 smithi079 ceph-mon[23158]: pgmap v5091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:35 smithi002 ceph-mon[17167]: pgmap v5091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:35 smithi016 ceph-mon[22010]: pgmap v5091: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:37 smithi079 ceph-mon[23158]: pgmap v5092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:37 smithi002 ceph-mon[17167]: pgmap v5092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:37 smithi016 ceph-mon[22010]: pgmap v5092: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9360.064441 secs 2022-03-21T16:47:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9360.064441 secs 2022-03-21T16:47:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9360.064441 secs 2022-03-21T16:47:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:39 smithi079 ceph-mon[23158]: pgmap v5093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:39 smithi002 ceph-mon[17167]: pgmap v5093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:39 smithi016 ceph-mon[22010]: pgmap v5093: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:41 smithi079 ceph-mon[23158]: pgmap v5094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:41 smithi002 ceph-mon[17167]: pgmap v5094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:41 smithi016 ceph-mon[22010]: pgmap v5094: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:43 smithi079 ceph-mon[23158]: pgmap v5095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9365.064571 secs 2022-03-21T16:47:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:43 smithi002 ceph-mon[17167]: pgmap v5095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:43.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9365.064571 secs 2022-03-21T16:47:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:43 smithi016 ceph-mon[22010]: pgmap v5095: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9365.064571 secs 2022-03-21T16:47:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:45 smithi079 ceph-mon[23158]: pgmap v5096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:45 smithi002 ceph-mon[17167]: pgmap v5096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:45 smithi016 ceph-mon[22010]: pgmap v5096: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:47 smithi079 ceph-mon[23158]: pgmap v5097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:47 smithi002 ceph-mon[17167]: pgmap v5097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:47 smithi016 ceph-mon[22010]: pgmap v5097: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9370.064696 secs 2022-03-21T16:47:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9370.064696 secs 2022-03-21T16:47:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9370.064696 secs 2022-03-21T16:47:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:49 smithi079 ceph-mon[23158]: pgmap v5098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:49 smithi002 ceph-mon[17167]: pgmap v5098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:49 smithi016 ceph-mon[22010]: pgmap v5098: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:51 smithi079 ceph-mon[23158]: pgmap v5099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:51 smithi002 ceph-mon[17167]: pgmap v5099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:51 smithi016 ceph-mon[22010]: pgmap v5099: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:53 smithi079 ceph-mon[23158]: pgmap v5100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9375.064814 secs 2022-03-21T16:47:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:53 smithi002 ceph-mon[17167]: pgmap v5100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9375.064814 secs 2022-03-21T16:47:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:53 smithi016 ceph-mon[22010]: pgmap v5100: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9375.064814 secs 2022-03-21T16:47:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:55 smithi079 ceph-mon[23158]: pgmap v5101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:55 smithi002 ceph-mon[17167]: pgmap v5101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:55 smithi016 ceph-mon[22010]: pgmap v5101: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:57 smithi079 ceph-mon[23158]: pgmap v5102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:57 smithi002 ceph-mon[17167]: pgmap v5102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:57 smithi016 ceph-mon[22010]: pgmap v5102: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9380.064907 secs 2022-03-21T16:47:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9380.064907 secs 2022-03-21T16:47:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9380.064907 secs 2022-03-21T16:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:47:59 smithi079 ceph-mon[23158]: pgmap v5103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:47:59 smithi002 ceph-mon[17167]: pgmap v5103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:47:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:47:59 smithi016 ceph-mon[22010]: pgmap v5103: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:01 smithi002 ceph-mon[17167]: pgmap v5104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:01 smithi016 ceph-mon[22010]: pgmap v5104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:01 smithi079 ceph-mon[23158]: pgmap v5104: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:03 smithi002 ceph-mon[17167]: pgmap v5105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9385.065027 secs 2022-03-21T16:48:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:03 smithi016 ceph-mon[22010]: pgmap v5105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9385.065027 secs 2022-03-21T16:48:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:03 smithi079 ceph-mon[23158]: pgmap v5105: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9385.065027 secs 2022-03-21T16:48:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:05 smithi002 ceph-mon[17167]: pgmap v5106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:05 smithi016 ceph-mon[22010]: pgmap v5106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:05 smithi079 ceph-mon[23158]: pgmap v5106: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:07 smithi002 ceph-mon[17167]: pgmap v5107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:07 smithi016 ceph-mon[22010]: pgmap v5107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:07 smithi079 ceph-mon[23158]: pgmap v5107: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9390.065159 secs 2022-03-21T16:48:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9390.065159 secs 2022-03-21T16:48:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9390.065159 secs 2022-03-21T16:48:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:09 smithi002 ceph-mon[17167]: pgmap v5108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:09 smithi016 ceph-mon[22010]: pgmap v5108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:09 smithi079 ceph-mon[23158]: pgmap v5108: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:10.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:10 smithi002 ceph-mon[17167]: pgmap v5109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:10 smithi016 ceph-mon[22010]: pgmap v5109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:10 smithi079 ceph-mon[23158]: pgmap v5109: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:48:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:48:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:48:11.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:11 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:48:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:48:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:48:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:48:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:11 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:48:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:48:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:48:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:48:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:11 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:48:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:13 smithi002 ceph-mon[17167]: pgmap v5110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9395.065279 secs 2022-03-21T16:48:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:13 smithi016 ceph-mon[22010]: pgmap v5110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9395.065279 secs 2022-03-21T16:48:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:13 smithi079 ceph-mon[23158]: pgmap v5110: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9395.065279 secs 2022-03-21T16:48:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:15 smithi002 ceph-mon[17167]: pgmap v5111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:15 smithi016 ceph-mon[22010]: pgmap v5111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:15 smithi079 ceph-mon[23158]: pgmap v5111: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:17 smithi002 ceph-mon[17167]: pgmap v5112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:17 smithi016 ceph-mon[22010]: pgmap v5112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:17 smithi079 ceph-mon[23158]: pgmap v5112: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9400.065381 secs 2022-03-21T16:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9400.065381 secs 2022-03-21T16:48:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9400.065381 secs 2022-03-21T16:48:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:19 smithi002 ceph-mon[17167]: pgmap v5113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:19 smithi016 ceph-mon[22010]: pgmap v5113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:19 smithi079 ceph-mon[23158]: pgmap v5113: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:21 smithi002 ceph-mon[17167]: pgmap v5114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:21 smithi016 ceph-mon[22010]: pgmap v5114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:21 smithi079 ceph-mon[23158]: pgmap v5114: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:48:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:48:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:48:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:48:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:48:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:48:23.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:23 smithi002 ceph-mon[17167]: pgmap v5115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9405.065522 secs 2022-03-21T16:48:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:23 smithi016 ceph-mon[22010]: pgmap v5115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9405.065522 secs 2022-03-21T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:23 smithi079 ceph-mon[23158]: pgmap v5115: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9405.065522 secs 2022-03-21T16:48:25.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:25 smithi002 ceph-mon[17167]: pgmap v5116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:25 smithi016 ceph-mon[22010]: pgmap v5116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:25 smithi079 ceph-mon[23158]: pgmap v5116: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:27 smithi002 ceph-mon[17167]: pgmap v5117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:27 smithi016 ceph-mon[22010]: pgmap v5117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:27 smithi079 ceph-mon[23158]: pgmap v5117: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:28.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9410.065638 secs 2022-03-21T16:48:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9410.065638 secs 2022-03-21T16:48:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9410.065638 secs 2022-03-21T16:48:29.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:29 smithi002 ceph-mon[17167]: pgmap v5118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:29 smithi016 ceph-mon[22010]: pgmap v5118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:29 smithi079 ceph-mon[23158]: pgmap v5118: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:31 smithi002 ceph-mon[17167]: pgmap v5119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:31 smithi016 ceph-mon[22010]: pgmap v5119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:31 smithi079 ceph-mon[23158]: pgmap v5119: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:33 smithi002 ceph-mon[17167]: pgmap v5120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9415.065714 secs 2022-03-21T16:48:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:33 smithi016 ceph-mon[22010]: pgmap v5120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9415.065714 secs 2022-03-21T16:48:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:33 smithi079 ceph-mon[23158]: pgmap v5120: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9415.065714 secs 2022-03-21T16:48:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:35 smithi002 ceph-mon[17167]: pgmap v5121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:35 smithi016 ceph-mon[22010]: pgmap v5121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:35 smithi079 ceph-mon[23158]: pgmap v5121: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:37 smithi002 ceph-mon[17167]: pgmap v5122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:37 smithi016 ceph-mon[22010]: pgmap v5122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:37 smithi079 ceph-mon[23158]: pgmap v5122: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:38.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9420.065839 secs 2022-03-21T16:48:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9420.065839 secs 2022-03-21T16:48:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9420.065839 secs 2022-03-21T16:48:39.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:39 smithi002 ceph-mon[17167]: pgmap v5123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:39 smithi016 ceph-mon[22010]: pgmap v5123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:39 smithi079 ceph-mon[23158]: pgmap v5123: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:41 smithi002 ceph-mon[17167]: pgmap v5124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:41 smithi016 ceph-mon[22010]: pgmap v5124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:41 smithi079 ceph-mon[23158]: pgmap v5124: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:43 smithi002 ceph-mon[17167]: pgmap v5125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9425.065974 secs 2022-03-21T16:48:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:43 smithi016 ceph-mon[22010]: pgmap v5125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9425.065974 secs 2022-03-21T16:48:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:43 smithi079 ceph-mon[23158]: pgmap v5125: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9425.065974 secs 2022-03-21T16:48:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:45 smithi002 ceph-mon[17167]: pgmap v5126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:45 smithi016 ceph-mon[22010]: pgmap v5126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:45 smithi079 ceph-mon[23158]: pgmap v5126: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:47 smithi002 ceph-mon[17167]: pgmap v5127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:47 smithi016 ceph-mon[22010]: pgmap v5127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:47 smithi079 ceph-mon[23158]: pgmap v5127: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:48.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9430.066095 secs 2022-03-21T16:48:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9430.066095 secs 2022-03-21T16:48:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9430.066095 secs 2022-03-21T16:48:49.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:49 smithi002 ceph-mon[17167]: pgmap v5128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:49 smithi016 ceph-mon[22010]: pgmap v5128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:49 smithi079 ceph-mon[23158]: pgmap v5128: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:51 smithi002 ceph-mon[17167]: pgmap v5129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:51 smithi016 ceph-mon[22010]: pgmap v5129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:51 smithi079 ceph-mon[23158]: pgmap v5129: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:53 smithi002 ceph-mon[17167]: pgmap v5130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9435.066193 secs 2022-03-21T16:48:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:53 smithi016 ceph-mon[22010]: pgmap v5130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9435.066193 secs 2022-03-21T16:48:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:53 smithi079 ceph-mon[23158]: pgmap v5130: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9435.066193 secs 2022-03-21T16:48:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:55 smithi016 ceph-mon[22010]: pgmap v5131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:55 smithi079 ceph-mon[23158]: pgmap v5131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:55 smithi002 ceph-mon[17167]: pgmap v5131: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:57 smithi016 ceph-mon[22010]: pgmap v5132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:57 smithi079 ceph-mon[23158]: pgmap v5132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:57 smithi002 ceph-mon[17167]: pgmap v5132: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:48:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9440.066330 secs 2022-03-21T16:48:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9440.066330 secs 2022-03-21T16:48:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9440.066330 secs 2022-03-21T16:48:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:48:59 smithi016 ceph-mon[22010]: pgmap v5133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:48:59 smithi079 ceph-mon[23158]: pgmap v5133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:48:59.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:48:59 smithi002 ceph-mon[17167]: pgmap v5133: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:01 smithi016 ceph-mon[22010]: pgmap v5134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:01 smithi079 ceph-mon[23158]: pgmap v5134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:01 smithi002 ceph-mon[17167]: pgmap v5134: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:03 smithi016 ceph-mon[22010]: pgmap v5135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9445.066445 secs 2022-03-21T16:49:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:03 smithi079 ceph-mon[23158]: pgmap v5135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9445.066445 secs 2022-03-21T16:49:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:03 smithi002 ceph-mon[17167]: pgmap v5135: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9445.066445 secs 2022-03-21T16:49:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:05 smithi016 ceph-mon[22010]: pgmap v5136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:05 smithi079 ceph-mon[23158]: pgmap v5136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:05 smithi002 ceph-mon[17167]: pgmap v5136: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:07 smithi016 ceph-mon[22010]: pgmap v5137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:07 smithi079 ceph-mon[23158]: pgmap v5137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:07 smithi002 ceph-mon[17167]: pgmap v5137: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9450.066587 secs 2022-03-21T16:49:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9450.066587 secs 2022-03-21T16:49:08.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9450.066587 secs 2022-03-21T16:49:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:09 smithi016 ceph-mon[22010]: pgmap v5138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:09 smithi079 ceph-mon[23158]: pgmap v5138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:09.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:09 smithi002 ceph-mon[17167]: pgmap v5138: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:11 smithi016 ceph-mon[22010]: pgmap v5139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:11 smithi079 ceph-mon[23158]: pgmap v5139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:11.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:11 smithi002 ceph-mon[17167]: pgmap v5139: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:49:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:49:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:49:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:49:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:49:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:49:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:49:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:49:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:49:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:49:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:49:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:49:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:13 smithi016 ceph-mon[22010]: pgmap v5140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9455.066670 secs 2022-03-21T16:49:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:13 smithi079 ceph-mon[23158]: pgmap v5140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9455.066670 secs 2022-03-21T16:49:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:13 smithi002 ceph-mon[17167]: pgmap v5140: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9455.066670 secs 2022-03-21T16:49:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:15 smithi079 ceph-mon[23158]: pgmap v5141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:15 smithi002 ceph-mon[17167]: pgmap v5141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:15 smithi016 ceph-mon[22010]: pgmap v5141: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:17 smithi079 ceph-mon[23158]: pgmap v5142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:17.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:17 smithi002 ceph-mon[17167]: pgmap v5142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:17 smithi016 ceph-mon[22010]: pgmap v5142: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9460.066795 secs 2022-03-21T16:49:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9460.066795 secs 2022-03-21T16:49:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9460.066795 secs 2022-03-21T16:49:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:19 smithi079 ceph-mon[23158]: pgmap v5143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:19 smithi002 ceph-mon[17167]: pgmap v5143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:19 smithi016 ceph-mon[22010]: pgmap v5143: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:21 smithi079 ceph-mon[23158]: pgmap v5144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:21 smithi002 ceph-mon[17167]: pgmap v5144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:21 smithi016 ceph-mon[22010]: pgmap v5144: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:49:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:49:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:49:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:49:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:49:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:49:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:23 smithi079 ceph-mon[23158]: pgmap v5145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9465.066883 secs 2022-03-21T16:49:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:23 smithi002 ceph-mon[17167]: pgmap v5145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9465.066883 secs 2022-03-21T16:49:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:23 smithi016 ceph-mon[22010]: pgmap v5145: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9465.066883 secs 2022-03-21T16:49:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:25 smithi079 ceph-mon[23158]: pgmap v5146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:25 smithi002 ceph-mon[17167]: pgmap v5146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:25 smithi016 ceph-mon[22010]: pgmap v5146: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:27 smithi079 ceph-mon[23158]: pgmap v5147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:27 smithi002 ceph-mon[17167]: pgmap v5147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:27 smithi016 ceph-mon[22010]: pgmap v5147: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9470.067002 secs 2022-03-21T16:49:28.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9470.067002 secs 2022-03-21T16:49:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9470.067002 secs 2022-03-21T16:49:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:29 smithi079 ceph-mon[23158]: pgmap v5148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:29 smithi002 ceph-mon[17167]: pgmap v5148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:29 smithi016 ceph-mon[22010]: pgmap v5148: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:31 smithi079 ceph-mon[23158]: pgmap v5149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:31 smithi002 ceph-mon[17167]: pgmap v5149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:31 smithi016 ceph-mon[22010]: pgmap v5149: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:33 smithi079 ceph-mon[23158]: pgmap v5150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9475.067137 secs 2022-03-21T16:49:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:33 smithi002 ceph-mon[17167]: pgmap v5150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9475.067137 secs 2022-03-21T16:49:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:33 smithi016 ceph-mon[22010]: pgmap v5150: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9475.067137 secs 2022-03-21T16:49:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:35 smithi079 ceph-mon[23158]: pgmap v5151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:35 smithi002 ceph-mon[17167]: pgmap v5151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:35 smithi016 ceph-mon[22010]: pgmap v5151: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:37 smithi079 ceph-mon[23158]: pgmap v5152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:37 smithi002 ceph-mon[17167]: pgmap v5152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:37 smithi016 ceph-mon[22010]: pgmap v5152: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9480.067246 secs 2022-03-21T16:49:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9480.067246 secs 2022-03-21T16:49:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9480.067246 secs 2022-03-21T16:49:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:39 smithi002 ceph-mon[17167]: pgmap v5153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:39 smithi016 ceph-mon[22010]: pgmap v5153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:39 smithi079 ceph-mon[23158]: pgmap v5153: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:41 smithi002 ceph-mon[17167]: pgmap v5154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:41 smithi016 ceph-mon[22010]: pgmap v5154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:41 smithi079 ceph-mon[23158]: pgmap v5154: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:43 smithi002 ceph-mon[17167]: pgmap v5155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9485.067361 secs 2022-03-21T16:49:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:43 smithi016 ceph-mon[22010]: pgmap v5155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9485.067361 secs 2022-03-21T16:49:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:43 smithi079 ceph-mon[23158]: pgmap v5155: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9485.067361 secs 2022-03-21T16:49:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:45 smithi002 ceph-mon[17167]: pgmap v5156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:45 smithi016 ceph-mon[22010]: pgmap v5156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:45 smithi079 ceph-mon[23158]: pgmap v5156: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:47 smithi002 ceph-mon[17167]: pgmap v5157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:47 smithi016 ceph-mon[22010]: pgmap v5157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:47 smithi079 ceph-mon[23158]: pgmap v5157: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9490.067475 secs 2022-03-21T16:49:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9490.067475 secs 2022-03-21T16:49:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9490.067475 secs 2022-03-21T16:49:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:49 smithi002 ceph-mon[17167]: pgmap v5158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:49 smithi016 ceph-mon[22010]: pgmap v5158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:49 smithi079 ceph-mon[23158]: pgmap v5158: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:51 smithi002 ceph-mon[17167]: pgmap v5159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:51 smithi016 ceph-mon[22010]: pgmap v5159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:51 smithi079 ceph-mon[23158]: pgmap v5159: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:53 smithi002 ceph-mon[17167]: pgmap v5160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9495.067596 secs 2022-03-21T16:49:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:53 smithi016 ceph-mon[22010]: pgmap v5160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9495.067596 secs 2022-03-21T16:49:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:53 smithi079 ceph-mon[23158]: pgmap v5160: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9495.067596 secs 2022-03-21T16:49:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:55 smithi002 ceph-mon[17167]: pgmap v5161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:55 smithi016 ceph-mon[22010]: pgmap v5161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:55 smithi079 ceph-mon[23158]: pgmap v5161: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:57.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:57 smithi002 ceph-mon[17167]: pgmap v5162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:57 smithi016 ceph-mon[22010]: pgmap v5162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:57 smithi079 ceph-mon[23158]: pgmap v5162: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:49:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9500.067709 secs 2022-03-21T16:49:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9500.067709 secs 2022-03-21T16:49:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9500.067709 secs 2022-03-21T16:49:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:49:59 smithi002 ceph-mon[17167]: pgmap v5163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:49:59 smithi016 ceph-mon[22010]: pgmap v5163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:49:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:49:59 smithi079 ceph-mon[23158]: pgmap v5163: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:00.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:00 smithi002 ceph-mon[17167]: pgmap v5164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:00.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:50:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:00 smithi016 ceph-mon[22010]: pgmap v5164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:50:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:00 smithi079 ceph-mon[23158]: pgmap v5164: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T16:50:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:03 smithi002 ceph-mon[17167]: pgmap v5165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9505.067811 secs 2022-03-21T16:50:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:03 smithi016 ceph-mon[22010]: pgmap v5165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9505.067811 secs 2022-03-21T16:50:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:03 smithi079 ceph-mon[23158]: pgmap v5165: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9505.067811 secs 2022-03-21T16:50:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:05 smithi002 ceph-mon[17167]: pgmap v5166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:05 smithi016 ceph-mon[22010]: pgmap v5166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:05 smithi079 ceph-mon[23158]: pgmap v5166: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:07 smithi002 ceph-mon[17167]: pgmap v5167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:07 smithi016 ceph-mon[22010]: pgmap v5167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:07 smithi079 ceph-mon[23158]: pgmap v5167: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:08.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9510.067932 secs 2022-03-21T16:50:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9510.067932 secs 2022-03-21T16:50:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9510.067932 secs 2022-03-21T16:50:09.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:09 smithi002 ceph-mon[17167]: pgmap v5168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:09 smithi016 ceph-mon[22010]: pgmap v5168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:09 smithi079 ceph-mon[23158]: pgmap v5168: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:11 smithi002 ceph-mon[17167]: pgmap v5169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:11 smithi016 ceph-mon[22010]: pgmap v5169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:11 smithi079 ceph-mon[23158]: pgmap v5169: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:50:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:50:12.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:12 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:50:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:50:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:50:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:12 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:50:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:50:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:50:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:12 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:50:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: pgmap v5170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9515.068066 secs 2022-03-21T16:50:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.371 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:50:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: pgmap v5170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9515.068066 secs 2022-03-21T16:50:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:50:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: pgmap v5170: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9515.068066 secs 2022-03-21T16:50:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T16:50:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:50:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:50:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:50:14.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:50:14.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:50:14.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:50:14.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:14 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:50:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:50:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:50:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:50:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:50:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:50:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:14 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:50:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T16:50:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T16:50:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T16:50:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T16:50:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T16:50:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:14 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T16:50:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:15 smithi002 ceph-mon[17167]: pgmap v5171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:15 smithi016 ceph-mon[22010]: pgmap v5171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:15 smithi079 ceph-mon[23158]: pgmap v5171: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:17 smithi002 ceph-mon[17167]: pgmap v5172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:17 smithi016 ceph-mon[22010]: pgmap v5172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:17 smithi079 ceph-mon[23158]: pgmap v5172: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9520.068158 secs 2022-03-21T16:50:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9520.068158 secs 2022-03-21T16:50:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9520.068158 secs 2022-03-21T16:50:19.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:19 smithi002 ceph-mon[17167]: pgmap v5173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:19 smithi016 ceph-mon[22010]: pgmap v5173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:19 smithi079 ceph-mon[23158]: pgmap v5173: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:21 smithi002 ceph-mon[17167]: pgmap v5174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:21 smithi016 ceph-mon[22010]: pgmap v5174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:21 smithi079 ceph-mon[23158]: pgmap v5174: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:50:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:50:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:50:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:50:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:50:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:50:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:23 smithi016 ceph-mon[22010]: pgmap v5175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9525.068277 secs 2022-03-21T16:50:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:23 smithi079 ceph-mon[23158]: pgmap v5175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9525.068277 secs 2022-03-21T16:50:23.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:23 smithi002 ceph-mon[17167]: pgmap v5175: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9525.068277 secs 2022-03-21T16:50:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:25 smithi016 ceph-mon[22010]: pgmap v5176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:25 smithi079 ceph-mon[23158]: pgmap v5176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:25 smithi002 ceph-mon[17167]: pgmap v5176: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:27 smithi016 ceph-mon[22010]: pgmap v5177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:27 smithi079 ceph-mon[23158]: pgmap v5177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:27 smithi002 ceph-mon[17167]: pgmap v5177: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9530.068379 secs 2022-03-21T16:50:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9530.068379 secs 2022-03-21T16:50:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9530.068379 secs 2022-03-21T16:50:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:29 smithi016 ceph-mon[22010]: pgmap v5178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:29 smithi079 ceph-mon[23158]: pgmap v5178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:29 smithi002 ceph-mon[17167]: pgmap v5178: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:31 smithi016 ceph-mon[22010]: pgmap v5179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:31 smithi079 ceph-mon[23158]: pgmap v5179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:31.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:31 smithi002 ceph-mon[17167]: pgmap v5179: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:33 smithi016 ceph-mon[22010]: pgmap v5180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9535.068525 secs 2022-03-21T16:50:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:33 smithi079 ceph-mon[23158]: pgmap v5180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9535.068525 secs 2022-03-21T16:50:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:33 smithi002 ceph-mon[17167]: pgmap v5180: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9535.068525 secs 2022-03-21T16:50:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:35 smithi016 ceph-mon[22010]: pgmap v5181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:35 smithi079 ceph-mon[23158]: pgmap v5181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:35 smithi002 ceph-mon[17167]: pgmap v5181: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:37 smithi016 ceph-mon[22010]: pgmap v5182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:37 smithi079 ceph-mon[23158]: pgmap v5182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:37.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:37 smithi002 ceph-mon[17167]: pgmap v5182: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9540.068625 secs 2022-03-21T16:50:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9540.068625 secs 2022-03-21T16:50:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9540.068625 secs 2022-03-21T16:50:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:39 smithi016 ceph-mon[22010]: pgmap v5183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:39 smithi079 ceph-mon[23158]: pgmap v5183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:39 smithi002 ceph-mon[17167]: pgmap v5183: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:41 smithi079 ceph-mon[23158]: pgmap v5184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:41 smithi002 ceph-mon[17167]: pgmap v5184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:41 smithi016 ceph-mon[22010]: pgmap v5184: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:43 smithi079 ceph-mon[23158]: pgmap v5185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9545.068757 secs 2022-03-21T16:50:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:43 smithi002 ceph-mon[17167]: pgmap v5185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9545.068757 secs 2022-03-21T16:50:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:43 smithi016 ceph-mon[22010]: pgmap v5185: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9545.068757 secs 2022-03-21T16:50:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:45 smithi079 ceph-mon[23158]: pgmap v5186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:45 smithi002 ceph-mon[17167]: pgmap v5186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:45 smithi016 ceph-mon[22010]: pgmap v5186: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:47 smithi079 ceph-mon[23158]: pgmap v5187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:47.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:47 smithi002 ceph-mon[17167]: pgmap v5187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:47 smithi016 ceph-mon[22010]: pgmap v5187: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9550.068879 secs 2022-03-21T16:50:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9550.068879 secs 2022-03-21T16:50:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9550.068879 secs 2022-03-21T16:50:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:49 smithi079 ceph-mon[23158]: pgmap v5188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:49 smithi002 ceph-mon[17167]: pgmap v5188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:49 smithi016 ceph-mon[22010]: pgmap v5188: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:51 smithi079 ceph-mon[23158]: pgmap v5189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:51 smithi002 ceph-mon[17167]: pgmap v5189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:51 smithi016 ceph-mon[22010]: pgmap v5189: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:50:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:53 smithi079 ceph-mon[23158]: pgmap v5190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9555.069004 secs 2022-03-21T16:50:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:53 smithi002 ceph-mon[17167]: pgmap v5190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9555.069004 secs 2022-03-21T16:50:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:53 smithi016 ceph-mon[22010]: pgmap v5190: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9555.069004 secs 2022-03-21T16:50:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:55 smithi079 ceph-mon[23158]: pgmap v5191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:55 smithi002 ceph-mon[17167]: pgmap v5191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:55 smithi016 ceph-mon[22010]: pgmap v5191: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:57 smithi079 ceph-mon[23158]: pgmap v5192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:57 smithi002 ceph-mon[17167]: pgmap v5192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:57 smithi016 ceph-mon[22010]: pgmap v5192: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:50:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9560.069121 secs 2022-03-21T16:50:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9560.069121 secs 2022-03-21T16:50:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9560.069121 secs 2022-03-21T16:50:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:50:59 smithi079 ceph-mon[23158]: pgmap v5193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:50:59 smithi002 ceph-mon[17167]: pgmap v5193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:50:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:50:59 smithi016 ceph-mon[22010]: pgmap v5193: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:01 smithi079 ceph-mon[23158]: pgmap v5194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:01.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:01 smithi002 ceph-mon[17167]: pgmap v5194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:01 smithi016 ceph-mon[22010]: pgmap v5194: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:03 smithi079 ceph-mon[23158]: pgmap v5195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9565.069247 secs 2022-03-21T16:51:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:03 smithi002 ceph-mon[17167]: pgmap v5195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9565.069247 secs 2022-03-21T16:51:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:03 smithi016 ceph-mon[22010]: pgmap v5195: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9565.069247 secs 2022-03-21T16:51:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:05 smithi079 ceph-mon[23158]: pgmap v5196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:05 smithi002 ceph-mon[17167]: pgmap v5196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:05 smithi016 ceph-mon[22010]: pgmap v5196: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:07 smithi079 ceph-mon[23158]: pgmap v5197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:07 smithi002 ceph-mon[17167]: pgmap v5197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:07 smithi016 ceph-mon[22010]: pgmap v5197: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9570.069368 secs 2022-03-21T16:51:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9570.069368 secs 2022-03-21T16:51:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9570.069368 secs 2022-03-21T16:51:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:09 smithi002 ceph-mon[17167]: pgmap v5198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:09 smithi016 ceph-mon[22010]: pgmap v5198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:09 smithi079 ceph-mon[23158]: pgmap v5198: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:11 smithi002 ceph-mon[17167]: pgmap v5199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:11 smithi016 ceph-mon[22010]: pgmap v5199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:11 smithi079 ceph-mon[23158]: pgmap v5199: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:13 smithi002 ceph-mon[17167]: pgmap v5200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9575.069431 secs 2022-03-21T16:51:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:51:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:51:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:13 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:51:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:13 smithi016 ceph-mon[22010]: pgmap v5200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9575.069431 secs 2022-03-21T16:51:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:51:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:51:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:13 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:51:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:13 smithi079 ceph-mon[23158]: pgmap v5200: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9575.069431 secs 2022-03-21T16:51:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:51:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:51:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:13 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:51:15.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:14 smithi002 ceph-mon[17167]: pgmap v5201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:15.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:51:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:14 smithi016 ceph-mon[22010]: pgmap v5201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:51:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:14 smithi079 ceph-mon[23158]: pgmap v5201: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:51:17.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:16 smithi002 ceph-mon[17167]: pgmap v5202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:16 smithi016 ceph-mon[22010]: pgmap v5202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:16 smithi079 ceph-mon[23158]: pgmap v5202: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:18.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9580.069504 secs 2022-03-21T16:51:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9580.069504 secs 2022-03-21T16:51:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9580.069504 secs 2022-03-21T16:51:19.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:19 smithi002 ceph-mon[17167]: pgmap v5203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:19 smithi016 ceph-mon[22010]: pgmap v5203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:19 smithi079 ceph-mon[23158]: pgmap v5203: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:21 smithi002 ceph-mon[17167]: pgmap v5204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:21 smithi016 ceph-mon[22010]: pgmap v5204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:21 smithi079 ceph-mon[23158]: pgmap v5204: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:51:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:51:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:51:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:51:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:51:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:51:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:23 smithi002 ceph-mon[17167]: pgmap v5205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9585.069644 secs 2022-03-21T16:51:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:23 smithi016 ceph-mon[22010]: pgmap v5205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9585.069644 secs 2022-03-21T16:51:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:23 smithi079 ceph-mon[23158]: pgmap v5205: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9585.069644 secs 2022-03-21T16:51:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:25 smithi002 ceph-mon[17167]: pgmap v5206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:25 smithi016 ceph-mon[22010]: pgmap v5206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:25 smithi079 ceph-mon[23158]: pgmap v5206: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:27 smithi002 ceph-mon[17167]: pgmap v5207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:27 smithi016 ceph-mon[22010]: pgmap v5207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:27 smithi079 ceph-mon[23158]: pgmap v5207: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:28.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9590.069757 secs 2022-03-21T16:51:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9590.069757 secs 2022-03-21T16:51:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9590.069757 secs 2022-03-21T16:51:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:29 smithi016 ceph-mon[22010]: pgmap v5208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:29 smithi079 ceph-mon[23158]: pgmap v5208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:29 smithi002 ceph-mon[17167]: pgmap v5208: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:31 smithi016 ceph-mon[22010]: pgmap v5209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:31 smithi079 ceph-mon[23158]: pgmap v5209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:31 smithi002 ceph-mon[17167]: pgmap v5209: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:33 smithi016 ceph-mon[22010]: pgmap v5210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9595.069862 secs 2022-03-21T16:51:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:33 smithi079 ceph-mon[23158]: pgmap v5210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9595.069862 secs 2022-03-21T16:51:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:33 smithi002 ceph-mon[17167]: pgmap v5210: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9595.069862 secs 2022-03-21T16:51:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:35 smithi016 ceph-mon[22010]: pgmap v5211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:35 smithi079 ceph-mon[23158]: pgmap v5211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:35 smithi002 ceph-mon[17167]: pgmap v5211: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:37 smithi016 ceph-mon[22010]: pgmap v5212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:37 smithi079 ceph-mon[23158]: pgmap v5212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:37 smithi002 ceph-mon[17167]: pgmap v5212: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9600.069983 secs 2022-03-21T16:51:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9600.069983 secs 2022-03-21T16:51:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9600.069983 secs 2022-03-21T16:51:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:39 smithi016 ceph-mon[22010]: pgmap v5213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:39 smithi079 ceph-mon[23158]: pgmap v5213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:39 smithi002 ceph-mon[17167]: pgmap v5213: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:41 smithi016 ceph-mon[22010]: pgmap v5214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:41 smithi079 ceph-mon[23158]: pgmap v5214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:41 smithi002 ceph-mon[17167]: pgmap v5214: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:43 smithi016 ceph-mon[22010]: pgmap v5215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9605.070106 secs 2022-03-21T16:51:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:43 smithi016 ceph-mon[22010]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 9604.955275 seconds ago 2022-03-21T16:51:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:43 smithi079 ceph-mon[23158]: pgmap v5215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9605.070106 secs 2022-03-21T16:51:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:43 smithi079 ceph-mon[23158]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 9604.955275 seconds ago 2022-03-21T16:51:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:43 smithi002 ceph-mon[17167]: pgmap v5215: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9605.070106 secs 2022-03-21T16:51:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:43 smithi002 ceph-mon[17167]: client.14964 isn't responding to mclientcaps(revoke), ino 0x10000000265 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 9604.955275 seconds ago 2022-03-21T16:51:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:45 smithi016 ceph-mon[22010]: pgmap v5216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:45 smithi079 ceph-mon[23158]: pgmap v5216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:45 smithi002 ceph-mon[17167]: pgmap v5216: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:47 smithi016 ceph-mon[22010]: pgmap v5217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:47 smithi079 ceph-mon[23158]: pgmap v5217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:47 smithi002 ceph-mon[17167]: pgmap v5217: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9610.070243 secs 2022-03-21T16:51:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9610.070243 secs 2022-03-21T16:51:48.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9610.070243 secs 2022-03-21T16:51:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:49 smithi016 ceph-mon[22010]: pgmap v5218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:49 smithi079 ceph-mon[23158]: pgmap v5218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:49 smithi002 ceph-mon[17167]: pgmap v5218: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:51 smithi016 ceph-mon[22010]: pgmap v5219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:51 smithi079 ceph-mon[23158]: pgmap v5219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:51 smithi002 ceph-mon[17167]: pgmap v5219: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:53 smithi079 ceph-mon[23158]: pgmap v5220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9615.070388 secs 2022-03-21T16:51:53.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:53 smithi002 ceph-mon[17167]: pgmap v5220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9615.070388 secs 2022-03-21T16:51:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:53 smithi016 ceph-mon[22010]: pgmap v5220: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9615.070388 secs 2022-03-21T16:51:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:55 smithi079 ceph-mon[23158]: pgmap v5221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T16:51:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:55 smithi002 ceph-mon[17167]: pgmap v5221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T16:51:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:55 smithi016 ceph-mon[22010]: pgmap v5221: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 3 op/s 2022-03-21T16:51:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:57 smithi079 ceph-mon[23158]: pgmap v5222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:57.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:57 smithi002 ceph-mon[17167]: pgmap v5222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:57 smithi016 ceph-mon[22010]: pgmap v5222: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:51:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9620.070462 secs 2022-03-21T16:51:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9620.070462 secs 2022-03-21T16:51:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9620.070462 secs 2022-03-21T16:51:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:51:59 smithi079 ceph-mon[23158]: pgmap v5223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:51:59 smithi002 ceph-mon[17167]: pgmap v5223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:51:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:51:59 smithi016 ceph-mon[22010]: pgmap v5223: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:01 smithi079 ceph-mon[23158]: pgmap v5224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:01 smithi002 ceph-mon[17167]: pgmap v5224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:01 smithi016 ceph-mon[22010]: pgmap v5224: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:03 smithi079 ceph-mon[23158]: pgmap v5225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9625.070577 secs 2022-03-21T16:52:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:03 smithi002 ceph-mon[17167]: pgmap v5225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9625.070577 secs 2022-03-21T16:52:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:03 smithi016 ceph-mon[22010]: pgmap v5225: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9625.070577 secs 2022-03-21T16:52:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:05 smithi079 ceph-mon[23158]: pgmap v5226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:05 smithi002 ceph-mon[17167]: pgmap v5226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:05 smithi016 ceph-mon[22010]: pgmap v5226: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:07.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:07 smithi079 ceph-mon[23158]: pgmap v5227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:07.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:07 smithi002 ceph-mon[17167]: pgmap v5227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:07 smithi016 ceph-mon[22010]: pgmap v5227: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9630.070679 secs 2022-03-21T16:52:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9630.070679 secs 2022-03-21T16:52:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9630.070679 secs 2022-03-21T16:52:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:09 smithi079 ceph-mon[23158]: pgmap v5228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:09 smithi002 ceph-mon[17167]: pgmap v5228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:09 smithi016 ceph-mon[22010]: pgmap v5228: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:11 smithi079 ceph-mon[23158]: pgmap v5229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:11 smithi002 ceph-mon[17167]: pgmap v5229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:11 smithi016 ceph-mon[22010]: pgmap v5229: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:13 smithi079 ceph-mon[23158]: pgmap v5230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9635.070775 secs 2022-03-21T16:52:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:13 smithi002 ceph-mon[17167]: pgmap v5230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9635.070775 secs 2022-03-21T16:52:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:13 smithi016 ceph-mon[22010]: pgmap v5230: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9635.070775 secs 2022-03-21T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:52:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:14 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:52:14.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:52:14.610 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:52:14.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:14 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:52:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:52:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:52:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:14 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:52:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:15 smithi079 ceph-mon[23158]: pgmap v5231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:52:15.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:15 smithi002 ceph-mon[17167]: pgmap v5231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:52:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:15 smithi016 ceph-mon[22010]: pgmap v5231: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:52:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:17 smithi079 ceph-mon[23158]: pgmap v5232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:17 smithi002 ceph-mon[17167]: pgmap v5232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:17 smithi016 ceph-mon[22010]: pgmap v5232: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9640.070898 secs 2022-03-21T16:52:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9640.070898 secs 2022-03-21T16:52:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9640.070898 secs 2022-03-21T16:52:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:19 smithi002 ceph-mon[17167]: pgmap v5233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:19 smithi016 ceph-mon[22010]: pgmap v5233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:19 smithi079 ceph-mon[23158]: pgmap v5233: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:21.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:21 smithi002 ceph-mon[17167]: pgmap v5234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:21 smithi016 ceph-mon[22010]: pgmap v5234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:21 smithi079 ceph-mon[23158]: pgmap v5234: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:52:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:52:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:52:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:52:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:52:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:52:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:23 smithi002 ceph-mon[17167]: pgmap v5235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9645.070979 secs 2022-03-21T16:52:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:23 smithi016 ceph-mon[22010]: pgmap v5235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9645.070979 secs 2022-03-21T16:52:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:23 smithi079 ceph-mon[23158]: pgmap v5235: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9645.070979 secs 2022-03-21T16:52:25.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:25 smithi002 ceph-mon[17167]: pgmap v5236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:25 smithi016 ceph-mon[22010]: pgmap v5236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:25 smithi079 ceph-mon[23158]: pgmap v5236: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:27 smithi002 ceph-mon[17167]: pgmap v5237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:27 smithi016 ceph-mon[22010]: pgmap v5237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:27 smithi079 ceph-mon[23158]: pgmap v5237: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9650.071042 secs 2022-03-21T16:52:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9650.071042 secs 2022-03-21T16:52:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9650.071042 secs 2022-03-21T16:52:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:29 smithi002 ceph-mon[17167]: pgmap v5238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:29 smithi016 ceph-mon[22010]: pgmap v5238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:29 smithi079 ceph-mon[23158]: pgmap v5238: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:31 smithi002 ceph-mon[17167]: pgmap v5239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:31 smithi016 ceph-mon[22010]: pgmap v5239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:31 smithi079 ceph-mon[23158]: pgmap v5239: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:33 smithi002 ceph-mon[17167]: pgmap v5240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9655.071158 secs 2022-03-21T16:52:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:33 smithi016 ceph-mon[22010]: pgmap v5240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9655.071158 secs 2022-03-21T16:52:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:33 smithi079 ceph-mon[23158]: pgmap v5240: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9655.071158 secs 2022-03-21T16:52:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:35 smithi002 ceph-mon[17167]: pgmap v5241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:35 smithi016 ceph-mon[22010]: pgmap v5241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:35 smithi079 ceph-mon[23158]: pgmap v5241: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:37 smithi002 ceph-mon[17167]: pgmap v5242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:37 smithi016 ceph-mon[22010]: pgmap v5242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:37 smithi079 ceph-mon[23158]: pgmap v5242: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:38.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9660.071276 secs 2022-03-21T16:52:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9660.071276 secs 2022-03-21T16:52:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9660.071276 secs 2022-03-21T16:52:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:39 smithi002 ceph-mon[17167]: pgmap v5243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:39 smithi016 ceph-mon[22010]: pgmap v5243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:39 smithi079 ceph-mon[23158]: pgmap v5243: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:41 smithi002 ceph-mon[17167]: pgmap v5244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:41 smithi016 ceph-mon[22010]: pgmap v5244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:41 smithi079 ceph-mon[23158]: pgmap v5244: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:43 smithi002 ceph-mon[17167]: pgmap v5245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9665.071379 secs 2022-03-21T16:52:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:43 smithi016 ceph-mon[22010]: pgmap v5245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9665.071379 secs 2022-03-21T16:52:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:43 smithi079 ceph-mon[23158]: pgmap v5245: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9665.071379 secs 2022-03-21T16:52:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:45 smithi002 ceph-mon[17167]: pgmap v5246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:45 smithi016 ceph-mon[22010]: pgmap v5246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:45 smithi079 ceph-mon[23158]: pgmap v5246: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:47 smithi002 ceph-mon[17167]: pgmap v5247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:47 smithi016 ceph-mon[22010]: pgmap v5247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:47 smithi079 ceph-mon[23158]: pgmap v5247: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9670.071498 secs 2022-03-21T16:52:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9670.071498 secs 2022-03-21T16:52:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9670.071498 secs 2022-03-21T16:52:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:49 smithi002 ceph-mon[17167]: pgmap v5248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:49 smithi016 ceph-mon[22010]: pgmap v5248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:49 smithi079 ceph-mon[23158]: pgmap v5248: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:51.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:51 smithi002 ceph-mon[17167]: pgmap v5249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:51 smithi016 ceph-mon[22010]: pgmap v5249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:51 smithi079 ceph-mon[23158]: pgmap v5249: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:53 smithi016 ceph-mon[22010]: pgmap v5250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9675.071612 secs 2022-03-21T16:52:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:53 smithi079 ceph-mon[23158]: pgmap v5250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9675.071612 secs 2022-03-21T16:52:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:53 smithi002 ceph-mon[17167]: pgmap v5250: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9675.071612 secs 2022-03-21T16:52:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:55 smithi016 ceph-mon[22010]: pgmap v5251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:55 smithi079 ceph-mon[23158]: pgmap v5251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:55.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:55 smithi002 ceph-mon[17167]: pgmap v5251: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:57 smithi016 ceph-mon[22010]: pgmap v5252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:57 smithi079 ceph-mon[23158]: pgmap v5252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:57 smithi002 ceph-mon[17167]: pgmap v5252: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:52:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9680.071710 secs 2022-03-21T16:52:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9680.071710 secs 2022-03-21T16:52:58.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9680.071710 secs 2022-03-21T16:52:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:52:59 smithi016 ceph-mon[22010]: pgmap v5253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:52:59 smithi079 ceph-mon[23158]: pgmap v5253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:52:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:52:59 smithi002 ceph-mon[17167]: pgmap v5253: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:01 smithi016 ceph-mon[22010]: pgmap v5254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:01 smithi079 ceph-mon[23158]: pgmap v5254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:01 smithi002 ceph-mon[17167]: pgmap v5254: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:03 smithi016 ceph-mon[22010]: pgmap v5255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9685.071809 secs 2022-03-21T16:53:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:03 smithi079 ceph-mon[23158]: pgmap v5255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9685.071809 secs 2022-03-21T16:53:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:03 smithi002 ceph-mon[17167]: pgmap v5255: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9685.071809 secs 2022-03-21T16:53:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:05 smithi016 ceph-mon[22010]: pgmap v5256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:05 smithi079 ceph-mon[23158]: pgmap v5256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:05 smithi002 ceph-mon[17167]: pgmap v5256: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:07 smithi016 ceph-mon[22010]: pgmap v5257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:07 smithi079 ceph-mon[23158]: pgmap v5257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:07 smithi002 ceph-mon[17167]: pgmap v5257: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9690.071917 secs 2022-03-21T16:53:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9690.071917 secs 2022-03-21T16:53:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9690.071917 secs 2022-03-21T16:53:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:09 smithi016 ceph-mon[22010]: pgmap v5258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:09 smithi079 ceph-mon[23158]: pgmap v5258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:09 smithi002 ceph-mon[17167]: pgmap v5258: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:11 smithi079 ceph-mon[23158]: pgmap v5259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:11 smithi002 ceph-mon[17167]: pgmap v5259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:11 smithi016 ceph-mon[22010]: pgmap v5259: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:13 smithi079 ceph-mon[23158]: pgmap v5260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9695.072014 secs 2022-03-21T16:53:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:13 smithi002 ceph-mon[17167]: pgmap v5260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9695.072014 secs 2022-03-21T16:53:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:13 smithi016 ceph-mon[22010]: pgmap v5260: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9695.072014 secs 2022-03-21T16:53:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:15 smithi079 ceph-mon[23158]: pgmap v5261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:53:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:53:15.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:15 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:53:15.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:15 smithi002 ceph-mon[17167]: pgmap v5261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:15.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:53:15.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:53:15.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:15 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:53:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:15 smithi016 ceph-mon[22010]: pgmap v5261: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:53:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:53:15.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:15 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:53:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:17 smithi079 ceph-mon[23158]: pgmap v5262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:17 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:17 smithi002 ceph-mon[17167]: pgmap v5262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:17 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:17.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:17 smithi016 ceph-mon[22010]: pgmap v5262: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:17.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:17 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:18.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:18 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:18.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9700.072087 secs 2022-03-21T16:53:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:18 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9700.072087 secs 2022-03-21T16:53:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:18 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9700.072087 secs 2022-03-21T16:53:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:19 smithi079 ceph-mon[23158]: pgmap v5263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:19 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:19 smithi002 ceph-mon[17167]: pgmap v5263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:19 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:19 smithi016 ceph-mon[22010]: pgmap v5263: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:19 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:53:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:21 smithi002 ceph-mon[17167]: pgmap v5264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:21 smithi016 ceph-mon[22010]: pgmap v5264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:21 smithi079 ceph-mon[23158]: pgmap v5264: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:53:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:53:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:53:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:53:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:53:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:53:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:23 smithi002 ceph-mon[17167]: pgmap v5265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9705.072212 secs 2022-03-21T16:53:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:23 smithi016 ceph-mon[22010]: pgmap v5265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9705.072212 secs 2022-03-21T16:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:23 smithi079 ceph-mon[23158]: pgmap v5265: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9705.072212 secs 2022-03-21T16:53:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:25 smithi002 ceph-mon[17167]: pgmap v5266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:25 smithi016 ceph-mon[22010]: pgmap v5266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:25 smithi079 ceph-mon[23158]: pgmap v5266: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:27 smithi002 ceph-mon[17167]: pgmap v5267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:27 smithi016 ceph-mon[22010]: pgmap v5267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:27 smithi079 ceph-mon[23158]: pgmap v5267: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9710.072352 secs 2022-03-21T16:53:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9710.072352 secs 2022-03-21T16:53:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9710.072352 secs 2022-03-21T16:53:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:29 smithi002 ceph-mon[17167]: pgmap v5268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:29 smithi016 ceph-mon[22010]: pgmap v5268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:29 smithi079 ceph-mon[23158]: pgmap v5268: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:31 smithi002 ceph-mon[17167]: pgmap v5269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:31 smithi016 ceph-mon[22010]: pgmap v5269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:31 smithi079 ceph-mon[23158]: pgmap v5269: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:33 smithi002 ceph-mon[17167]: pgmap v5270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9715.072503 secs 2022-03-21T16:53:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:33 smithi016 ceph-mon[22010]: pgmap v5270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9715.072503 secs 2022-03-21T16:53:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:33 smithi079 ceph-mon[23158]: pgmap v5270: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9715.072503 secs 2022-03-21T16:53:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:35 smithi002 ceph-mon[17167]: pgmap v5271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:35 smithi016 ceph-mon[22010]: pgmap v5271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:35 smithi079 ceph-mon[23158]: pgmap v5271: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:37 smithi002 ceph-mon[17167]: pgmap v5272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:37 smithi016 ceph-mon[22010]: pgmap v5272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:37 smithi079 ceph-mon[23158]: pgmap v5272: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9720.072600 secs 2022-03-21T16:53:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9720.072600 secs 2022-03-21T16:53:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9720.072600 secs 2022-03-21T16:53:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:39 smithi002 ceph-mon[17167]: pgmap v5273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:39 smithi016 ceph-mon[22010]: pgmap v5273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:39 smithi079 ceph-mon[23158]: pgmap v5273: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:40.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:40 smithi002 ceph-mon[17167]: pgmap v5274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:40 smithi016 ceph-mon[22010]: pgmap v5274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:40 smithi079 ceph-mon[23158]: pgmap v5274: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:43 smithi016 ceph-mon[22010]: pgmap v5275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9725.072723 secs 2022-03-21T16:53:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:43 smithi079 ceph-mon[23158]: pgmap v5275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9725.072723 secs 2022-03-21T16:53:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:43 smithi002 ceph-mon[17167]: pgmap v5275: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9725.072723 secs 2022-03-21T16:53:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:45 smithi016 ceph-mon[22010]: pgmap v5276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:45 smithi079 ceph-mon[23158]: pgmap v5276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:45 smithi002 ceph-mon[17167]: pgmap v5276: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:47 smithi016 ceph-mon[22010]: pgmap v5277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:47 smithi079 ceph-mon[23158]: pgmap v5277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:47 smithi002 ceph-mon[17167]: pgmap v5277: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9730.072840 secs 2022-03-21T16:53:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9730.072840 secs 2022-03-21T16:53:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9730.072840 secs 2022-03-21T16:53:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:49 smithi079 ceph-mon[23158]: pgmap v5278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:49 smithi002 ceph-mon[17167]: pgmap v5278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:49 smithi016 ceph-mon[22010]: pgmap v5278: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:51 smithi079 ceph-mon[23158]: pgmap v5279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:51 smithi002 ceph-mon[17167]: pgmap v5279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:51 smithi016 ceph-mon[22010]: pgmap v5279: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:53 smithi079 ceph-mon[23158]: pgmap v5280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9735.072966 secs 2022-03-21T16:53:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:53 smithi002 ceph-mon[17167]: pgmap v5280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9735.072966 secs 2022-03-21T16:53:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:53 smithi016 ceph-mon[22010]: pgmap v5280: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9735.072966 secs 2022-03-21T16:53:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:55 smithi079 ceph-mon[23158]: pgmap v5281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:55 smithi002 ceph-mon[17167]: pgmap v5281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:55 smithi016 ceph-mon[22010]: pgmap v5281: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:57 smithi079 ceph-mon[23158]: pgmap v5282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:57 smithi002 ceph-mon[17167]: pgmap v5282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:57 smithi016 ceph-mon[22010]: pgmap v5282: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:53:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9740.073118 secs 2022-03-21T16:53:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9740.073118 secs 2022-03-21T16:53:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9740.073118 secs 2022-03-21T16:53:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:53:59 smithi079 ceph-mon[23158]: pgmap v5283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:53:59 smithi002 ceph-mon[17167]: pgmap v5283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:53:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:53:59 smithi016 ceph-mon[22010]: pgmap v5283: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:01 smithi079 ceph-mon[23158]: pgmap v5284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:01 smithi002 ceph-mon[17167]: pgmap v5284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:01 smithi016 ceph-mon[22010]: pgmap v5284: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:03 smithi079 ceph-mon[23158]: pgmap v5285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9745.073249 secs 2022-03-21T16:54:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:03 smithi002 ceph-mon[17167]: pgmap v5285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9745.073249 secs 2022-03-21T16:54:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:03 smithi016 ceph-mon[22010]: pgmap v5285: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9745.073249 secs 2022-03-21T16:54:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:05 smithi079 ceph-mon[23158]: pgmap v5286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:05 smithi002 ceph-mon[17167]: pgmap v5286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:05 smithi016 ceph-mon[22010]: pgmap v5286: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:07 smithi079 ceph-mon[23158]: pgmap v5287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:07 smithi002 ceph-mon[17167]: pgmap v5287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:07 smithi016 ceph-mon[22010]: pgmap v5287: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9750.073346 secs 2022-03-21T16:54:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9750.073346 secs 2022-03-21T16:54:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9750.073346 secs 2022-03-21T16:54:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:09 smithi079 ceph-mon[23158]: pgmap v5288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:09 smithi002 ceph-mon[17167]: pgmap v5288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:09 smithi016 ceph-mon[22010]: pgmap v5288: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:11 smithi079 ceph-mon[23158]: pgmap v5289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:11 smithi002 ceph-mon[17167]: pgmap v5289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:11 smithi016 ceph-mon[22010]: pgmap v5289: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:13 smithi079 ceph-mon[23158]: pgmap v5290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9755.073464 secs 2022-03-21T16:54:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:13 smithi002 ceph-mon[17167]: pgmap v5290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9755.073464 secs 2022-03-21T16:54:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:13 smithi016 ceph-mon[22010]: pgmap v5290: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9755.073464 secs 2022-03-21T16:54:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:15 smithi079 ceph-mon[23158]: pgmap v5291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:15 smithi002 ceph-mon[17167]: pgmap v5291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:15 smithi016 ceph-mon[22010]: pgmap v5291: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:17 smithi002 ceph-mon[17167]: pgmap v5292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:17 smithi016 ceph-mon[22010]: pgmap v5292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:17 smithi079 ceph-mon[23158]: pgmap v5292: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9760.073586 secs 2022-03-21T16:54:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9760.073586 secs 2022-03-21T16:54:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9760.073586 secs 2022-03-21T16:54:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:19 smithi002 ceph-mon[17167]: pgmap v5293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:19 smithi016 ceph-mon[22010]: pgmap v5293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:19.698 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:19 smithi079 ceph-mon[23158]: pgmap v5293: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:54:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:54:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:54:20.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:54:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:54:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:54:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:54:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:54:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:54:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:54:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:54:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:54:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:21 smithi002 ceph-mon[17167]: pgmap v5294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:21 smithi016 ceph-mon[22010]: pgmap v5294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:21 smithi079 ceph-mon[23158]: pgmap v5294: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:54:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:54:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:54:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:54:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:54:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:54:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:23 smithi002 ceph-mon[17167]: pgmap v5295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9765.073706 secs 2022-03-21T16:54:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:23 smithi016 ceph-mon[22010]: pgmap v5295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9765.073706 secs 2022-03-21T16:54:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:23 smithi079 ceph-mon[23158]: pgmap v5295: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9765.073706 secs 2022-03-21T16:54:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:25 smithi002 ceph-mon[17167]: pgmap v5296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:25 smithi016 ceph-mon[22010]: pgmap v5296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:25 smithi079 ceph-mon[23158]: pgmap v5296: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:27 smithi002 ceph-mon[17167]: pgmap v5297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:27 smithi016 ceph-mon[22010]: pgmap v5297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:27 smithi079 ceph-mon[23158]: pgmap v5297: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9770.073788 secs 2022-03-21T16:54:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9770.073788 secs 2022-03-21T16:54:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9770.073788 secs 2022-03-21T16:54:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:29 smithi002 ceph-mon[17167]: pgmap v5298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:29 smithi016 ceph-mon[22010]: pgmap v5298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:29 smithi079 ceph-mon[23158]: pgmap v5298: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:31 smithi002 ceph-mon[17167]: pgmap v5299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:31 smithi016 ceph-mon[22010]: pgmap v5299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:31 smithi079 ceph-mon[23158]: pgmap v5299: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:33 smithi002 ceph-mon[17167]: pgmap v5300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9775.073906 secs 2022-03-21T16:54:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:33 smithi016 ceph-mon[22010]: pgmap v5300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9775.073906 secs 2022-03-21T16:54:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:33 smithi079 ceph-mon[23158]: pgmap v5300: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9775.073906 secs 2022-03-21T16:54:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:35 smithi002 ceph-mon[17167]: pgmap v5301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:35 smithi016 ceph-mon[22010]: pgmap v5301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:35 smithi079 ceph-mon[23158]: pgmap v5301: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:37 smithi002 ceph-mon[17167]: pgmap v5302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:37 smithi016 ceph-mon[22010]: pgmap v5302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:37 smithi079 ceph-mon[23158]: pgmap v5302: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9780.074028 secs 2022-03-21T16:54:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9780.074028 secs 2022-03-21T16:54:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9780.074028 secs 2022-03-21T16:54:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:39 smithi002 ceph-mon[17167]: pgmap v5303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:39 smithi016 ceph-mon[22010]: pgmap v5303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:39 smithi079 ceph-mon[23158]: pgmap v5303: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:41 smithi002 ceph-mon[17167]: pgmap v5304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:41 smithi016 ceph-mon[22010]: pgmap v5304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:41 smithi079 ceph-mon[23158]: pgmap v5304: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:43 smithi002 ceph-mon[17167]: pgmap v5305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9785.074145 secs 2022-03-21T16:54:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:43 smithi016 ceph-mon[22010]: pgmap v5305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9785.074145 secs 2022-03-21T16:54:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:43 smithi079 ceph-mon[23158]: pgmap v5305: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9785.074145 secs 2022-03-21T16:54:45.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:45 smithi002 ceph-mon[17167]: pgmap v5306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:45 smithi016 ceph-mon[22010]: pgmap v5306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:45 smithi079 ceph-mon[23158]: pgmap v5306: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:47 smithi002 ceph-mon[17167]: pgmap v5307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:47 smithi016 ceph-mon[22010]: pgmap v5307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:47 smithi079 ceph-mon[23158]: pgmap v5307: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9790.074280 secs 2022-03-21T16:54:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9790.074280 secs 2022-03-21T16:54:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9790.074280 secs 2022-03-21T16:54:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:49 smithi002 ceph-mon[17167]: pgmap v5308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:49 smithi016 ceph-mon[22010]: pgmap v5308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:49 smithi079 ceph-mon[23158]: pgmap v5308: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:51 smithi002 ceph-mon[17167]: pgmap v5309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:51 smithi016 ceph-mon[22010]: pgmap v5309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:51 smithi079 ceph-mon[23158]: pgmap v5309: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:53 smithi002 ceph-mon[17167]: pgmap v5310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9795.074395 secs 2022-03-21T16:54:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:53 smithi016 ceph-mon[22010]: pgmap v5310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9795.074395 secs 2022-03-21T16:54:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:53 smithi079 ceph-mon[23158]: pgmap v5310: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9795.074395 secs 2022-03-21T16:54:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:55 smithi002 ceph-mon[17167]: pgmap v5311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:55 smithi016 ceph-mon[22010]: pgmap v5311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:55 smithi079 ceph-mon[23158]: pgmap v5311: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:57 smithi002 ceph-mon[17167]: pgmap v5312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:57 smithi016 ceph-mon[22010]: pgmap v5312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:57 smithi079 ceph-mon[23158]: pgmap v5312: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:54:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9800.074515 secs 2022-03-21T16:54:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9800.074515 secs 2022-03-21T16:54:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9800.074515 secs 2022-03-21T16:54:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:54:59 smithi002 ceph-mon[17167]: pgmap v5313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:54:59 smithi016 ceph-mon[22010]: pgmap v5313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:54:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:54:59 smithi079 ceph-mon[23158]: pgmap v5313: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:00.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:00 smithi002 ceph-mon[17167]: pgmap v5314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:00 smithi016 ceph-mon[22010]: pgmap v5314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:00 smithi079 ceph-mon[23158]: pgmap v5314: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:03 smithi079 ceph-mon[23158]: pgmap v5315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9805.074621 secs 2022-03-21T16:55:03.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:03 smithi002 ceph-mon[17167]: pgmap v5315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9805.074621 secs 2022-03-21T16:55:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:03 smithi016 ceph-mon[22010]: pgmap v5315: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9805.074621 secs 2022-03-21T16:55:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:05 smithi079 ceph-mon[23158]: pgmap v5316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:05 smithi002 ceph-mon[17167]: pgmap v5316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:05 smithi016 ceph-mon[22010]: pgmap v5316: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:07 smithi079 ceph-mon[23158]: pgmap v5317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:07 smithi002 ceph-mon[17167]: pgmap v5317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:07 smithi016 ceph-mon[22010]: pgmap v5317: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9810.074706 secs 2022-03-21T16:55:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9810.074706 secs 2022-03-21T16:55:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9810.074706 secs 2022-03-21T16:55:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:09 smithi079 ceph-mon[23158]: pgmap v5318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:09 smithi002 ceph-mon[17167]: pgmap v5318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:09 smithi016 ceph-mon[22010]: pgmap v5318: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:11 smithi079 ceph-mon[23158]: pgmap v5319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:11 smithi002 ceph-mon[17167]: pgmap v5319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:11 smithi016 ceph-mon[22010]: pgmap v5319: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:13 smithi079 ceph-mon[23158]: pgmap v5320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9815.074814 secs 2022-03-21T16:55:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:13 smithi002 ceph-mon[17167]: pgmap v5320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9815.074814 secs 2022-03-21T16:55:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:13 smithi016 ceph-mon[22010]: pgmap v5320: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9815.074814 secs 2022-03-21T16:55:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:15 smithi079 ceph-mon[23158]: pgmap v5321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:15 smithi002 ceph-mon[17167]: pgmap v5321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:15 smithi016 ceph-mon[22010]: pgmap v5321: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:17 smithi079 ceph-mon[23158]: pgmap v5322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:17 smithi002 ceph-mon[17167]: pgmap v5322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:17 smithi016 ceph-mon[22010]: pgmap v5322: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9820.074922 secs 2022-03-21T16:55:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9820.074922 secs 2022-03-21T16:55:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9820.074922 secs 2022-03-21T16:55:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:19 smithi079 ceph-mon[23158]: pgmap v5323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:19.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:19 smithi002 ceph-mon[17167]: pgmap v5323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:19 smithi016 ceph-mon[22010]: pgmap v5323: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:55:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:55:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:20 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:55:20.538 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:55:20.539 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:55:20.539 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:20 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:55:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:55:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:55:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:20 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:55:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:21 smithi016 ceph-mon[22010]: pgmap v5324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:21 smithi079 ceph-mon[23158]: pgmap v5324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:21 smithi002 ceph-mon[17167]: pgmap v5324: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:21.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:21.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:55:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:55:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:55:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:55:22.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:55:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:55:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:55:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:23 smithi079 ceph-mon[23158]: pgmap v5325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9825.075012 secs 2022-03-21T16:55:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:23 smithi002 ceph-mon[17167]: pgmap v5325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9825.075012 secs 2022-03-21T16:55:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:23 smithi016 ceph-mon[22010]: pgmap v5325: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9825.075012 secs 2022-03-21T16:55:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:25 smithi002 ceph-mon[17167]: pgmap v5326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:25 smithi016 ceph-mon[22010]: pgmap v5326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:25 smithi079 ceph-mon[23158]: pgmap v5326: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:27.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:27 smithi002 ceph-mon[17167]: pgmap v5327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:27 smithi016 ceph-mon[22010]: pgmap v5327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:27 smithi079 ceph-mon[23158]: pgmap v5327: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9830.075151 secs 2022-03-21T16:55:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9830.075151 secs 2022-03-21T16:55:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9830.075151 secs 2022-03-21T16:55:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:29 smithi002 ceph-mon[17167]: pgmap v5328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:29 smithi016 ceph-mon[22010]: pgmap v5328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:29 smithi079 ceph-mon[23158]: pgmap v5328: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:31 smithi002 ceph-mon[17167]: pgmap v5329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:31 smithi016 ceph-mon[22010]: pgmap v5329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:31 smithi079 ceph-mon[23158]: pgmap v5329: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:33 smithi002 ceph-mon[17167]: pgmap v5330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9835.075279 secs 2022-03-21T16:55:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:33 smithi016 ceph-mon[22010]: pgmap v5330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9835.075279 secs 2022-03-21T16:55:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:33 smithi079 ceph-mon[23158]: pgmap v5330: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9835.075279 secs 2022-03-21T16:55:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:35 smithi002 ceph-mon[17167]: pgmap v5331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:35 smithi016 ceph-mon[22010]: pgmap v5331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:35 smithi079 ceph-mon[23158]: pgmap v5331: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:37 smithi002 ceph-mon[17167]: pgmap v5332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:37 smithi016 ceph-mon[22010]: pgmap v5332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:37 smithi079 ceph-mon[23158]: pgmap v5332: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9840.075408 secs 2022-03-21T16:55:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9840.075408 secs 2022-03-21T16:55:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9840.075408 secs 2022-03-21T16:55:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:39 smithi002 ceph-mon[17167]: pgmap v5333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:39 smithi016 ceph-mon[22010]: pgmap v5333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:39 smithi079 ceph-mon[23158]: pgmap v5333: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:41 smithi002 ceph-mon[17167]: pgmap v5334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:41 smithi016 ceph-mon[22010]: pgmap v5334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:41 smithi079 ceph-mon[23158]: pgmap v5334: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:43 smithi002 ceph-mon[17167]: pgmap v5335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9845.075520 secs 2022-03-21T16:55:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:43 smithi016 ceph-mon[22010]: pgmap v5335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9845.075520 secs 2022-03-21T16:55:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:43 smithi079 ceph-mon[23158]: pgmap v5335: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9845.075520 secs 2022-03-21T16:55:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:45 smithi002 ceph-mon[17167]: pgmap v5336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:45 smithi016 ceph-mon[22010]: pgmap v5336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:45 smithi079 ceph-mon[23158]: pgmap v5336: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:47 smithi002 ceph-mon[17167]: pgmap v5337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:47 smithi016 ceph-mon[22010]: pgmap v5337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:47 smithi079 ceph-mon[23158]: pgmap v5337: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9850.075641 secs 2022-03-21T16:55:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9850.075641 secs 2022-03-21T16:55:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9850.075641 secs 2022-03-21T16:55:49.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:49 smithi002 ceph-mon[17167]: pgmap v5338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:49 smithi016 ceph-mon[22010]: pgmap v5338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:49 smithi079 ceph-mon[23158]: pgmap v5338: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:51 smithi002 ceph-mon[17167]: pgmap v5339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:51 smithi016 ceph-mon[22010]: pgmap v5339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:51 smithi079 ceph-mon[23158]: pgmap v5339: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:53 smithi002 ceph-mon[17167]: pgmap v5340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9855.075756 secs 2022-03-21T16:55:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:53 smithi016 ceph-mon[22010]: pgmap v5340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9855.075756 secs 2022-03-21T16:55:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:53 smithi079 ceph-mon[23158]: pgmap v5340: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9855.075756 secs 2022-03-21T16:55:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:55 smithi002 ceph-mon[17167]: pgmap v5341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:55 smithi016 ceph-mon[22010]: pgmap v5341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:55 smithi079 ceph-mon[23158]: pgmap v5341: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:57 smithi002 ceph-mon[17167]: pgmap v5342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:57 smithi016 ceph-mon[22010]: pgmap v5342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:57 smithi079 ceph-mon[23158]: pgmap v5342: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:55:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9860.075854 secs 2022-03-21T16:55:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9860.075854 secs 2022-03-21T16:55:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9860.075854 secs 2022-03-21T16:55:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:55:59 smithi002 ceph-mon[17167]: pgmap v5343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:55:59 smithi016 ceph-mon[22010]: pgmap v5343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:55:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:55:59 smithi079 ceph-mon[23158]: pgmap v5343: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:01 smithi002 ceph-mon[17167]: pgmap v5344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:01 smithi016 ceph-mon[22010]: pgmap v5344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:01 smithi079 ceph-mon[23158]: pgmap v5344: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:03 smithi002 ceph-mon[17167]: pgmap v5345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9865.075972 secs 2022-03-21T16:56:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:03 smithi016 ceph-mon[22010]: pgmap v5345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9865.075972 secs 2022-03-21T16:56:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:03 smithi079 ceph-mon[23158]: pgmap v5345: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9865.075972 secs 2022-03-21T16:56:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:05 smithi002 ceph-mon[17167]: pgmap v5346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:05 smithi016 ceph-mon[22010]: pgmap v5346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:05 smithi079 ceph-mon[23158]: pgmap v5346: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:07 smithi002 ceph-mon[17167]: pgmap v5347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:07 smithi016 ceph-mon[22010]: pgmap v5347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:07 smithi079 ceph-mon[23158]: pgmap v5347: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9870.076089 secs 2022-03-21T16:56:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9870.076089 secs 2022-03-21T16:56:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9870.076089 secs 2022-03-21T16:56:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:09 smithi002 ceph-mon[17167]: pgmap v5348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:09 smithi016 ceph-mon[22010]: pgmap v5348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:09 smithi079 ceph-mon[23158]: pgmap v5348: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:11 smithi002 ceph-mon[17167]: pgmap v5349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:11 smithi016 ceph-mon[22010]: pgmap v5349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:11 smithi079 ceph-mon[23158]: pgmap v5349: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:13 smithi002 ceph-mon[17167]: pgmap v5350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9875.076210 secs 2022-03-21T16:56:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:13 smithi016 ceph-mon[22010]: pgmap v5350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9875.076210 secs 2022-03-21T16:56:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:13 smithi079 ceph-mon[23158]: pgmap v5350: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9875.076210 secs 2022-03-21T16:56:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:15 smithi002 ceph-mon[17167]: pgmap v5351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:15 smithi016 ceph-mon[22010]: pgmap v5351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:15 smithi079 ceph-mon[23158]: pgmap v5351: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:17 smithi002 ceph-mon[17167]: pgmap v5352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:17 smithi016 ceph-mon[22010]: pgmap v5352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:17 smithi079 ceph-mon[23158]: pgmap v5352: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9880.076333 secs 2022-03-21T16:56:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9880.076333 secs 2022-03-21T16:56:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9880.076333 secs 2022-03-21T16:56:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:19 smithi002 ceph-mon[17167]: pgmap v5353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:19 smithi016 ceph-mon[22010]: pgmap v5353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:19 smithi079 ceph-mon[23158]: pgmap v5353: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:20.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:20 smithi002 ceph-mon[17167]: pgmap v5354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:20 smithi016 ceph-mon[22010]: pgmap v5354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:20 smithi079 ceph-mon[23158]: pgmap v5354: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:56:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:56:21.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:56:21.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:56:21.648 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:56:21.648 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:56:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:56:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:56:22.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:56:22.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:22 smithi002 ceph-mon[17167]: pgmap v5355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:56:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:56:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:56:22.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:22 smithi016 ceph-mon[22010]: pgmap v5355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:56:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:56:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:56:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:22 smithi079 ceph-mon[23158]: pgmap v5355: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9885.076441 secs 2022-03-21T16:56:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:56:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9885.076441 secs 2022-03-21T16:56:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:56:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9885.076441 secs 2022-03-21T16:56:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:56:24.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:24 smithi002 ceph-mon[17167]: pgmap v5356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:24 smithi016 ceph-mon[22010]: pgmap v5356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:24 smithi079 ceph-mon[23158]: pgmap v5356: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:27.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:26 smithi002 ceph-mon[17167]: pgmap v5357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:26 smithi016 ceph-mon[22010]: pgmap v5357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:26 smithi079 ceph-mon[23158]: pgmap v5357: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9890.076581 secs 2022-03-21T16:56:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9890.076581 secs 2022-03-21T16:56:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9890.076581 secs 2022-03-21T16:56:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:29 smithi079 ceph-mon[23158]: pgmap v5358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:29 smithi002 ceph-mon[17167]: pgmap v5358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:29 smithi016 ceph-mon[22010]: pgmap v5358: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:31 smithi079 ceph-mon[23158]: pgmap v5359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:31 smithi002 ceph-mon[17167]: pgmap v5359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:31 smithi016 ceph-mon[22010]: pgmap v5359: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:33 smithi079 ceph-mon[23158]: pgmap v5360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9895.076705 secs 2022-03-21T16:56:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:33 smithi002 ceph-mon[17167]: pgmap v5360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9895.076705 secs 2022-03-21T16:56:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:33 smithi016 ceph-mon[22010]: pgmap v5360: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9895.076705 secs 2022-03-21T16:56:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:35 smithi002 ceph-mon[17167]: pgmap v5361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:35 smithi016 ceph-mon[22010]: pgmap v5361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:35 smithi079 ceph-mon[23158]: pgmap v5361: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:37 smithi002 ceph-mon[17167]: pgmap v5362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:37 smithi016 ceph-mon[22010]: pgmap v5362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:37 smithi079 ceph-mon[23158]: pgmap v5362: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9900.076779 secs 2022-03-21T16:56:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9900.076779 secs 2022-03-21T16:56:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9900.076779 secs 2022-03-21T16:56:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:39 smithi002 ceph-mon[17167]: pgmap v5363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:39 smithi016 ceph-mon[22010]: pgmap v5363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:39 smithi079 ceph-mon[23158]: pgmap v5363: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:41 smithi002 ceph-mon[17167]: pgmap v5364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:41 smithi016 ceph-mon[22010]: pgmap v5364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:41 smithi079 ceph-mon[23158]: pgmap v5364: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:43 smithi002 ceph-mon[17167]: pgmap v5365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9905.076895 secs 2022-03-21T16:56:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:43 smithi016 ceph-mon[22010]: pgmap v5365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9905.076895 secs 2022-03-21T16:56:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:43 smithi079 ceph-mon[23158]: pgmap v5365: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9905.076895 secs 2022-03-21T16:56:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:45 smithi002 ceph-mon[17167]: pgmap v5366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:45 smithi016 ceph-mon[22010]: pgmap v5366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:45 smithi079 ceph-mon[23158]: pgmap v5366: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:47 smithi002 ceph-mon[17167]: pgmap v5367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:47 smithi016 ceph-mon[22010]: pgmap v5367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:47 smithi079 ceph-mon[23158]: pgmap v5367: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9910.077015 secs 2022-03-21T16:56:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9910.077015 secs 2022-03-21T16:56:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9910.077015 secs 2022-03-21T16:56:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:49 smithi002 ceph-mon[17167]: pgmap v5368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:49 smithi016 ceph-mon[22010]: pgmap v5368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:49 smithi079 ceph-mon[23158]: pgmap v5368: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:51 smithi002 ceph-mon[17167]: pgmap v5369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:51 smithi016 ceph-mon[22010]: pgmap v5369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:51 smithi079 ceph-mon[23158]: pgmap v5369: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:53 smithi002 ceph-mon[17167]: pgmap v5370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9915.077090 secs 2022-03-21T16:56:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:53 smithi016 ceph-mon[22010]: pgmap v5370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9915.077090 secs 2022-03-21T16:56:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:53 smithi079 ceph-mon[23158]: pgmap v5370: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9915.077090 secs 2022-03-21T16:56:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:55 smithi002 ceph-mon[17167]: pgmap v5371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:55 smithi016 ceph-mon[22010]: pgmap v5371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:55 smithi079 ceph-mon[23158]: pgmap v5371: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:57 smithi002 ceph-mon[17167]: pgmap v5372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:57 smithi016 ceph-mon[22010]: pgmap v5372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:57 smithi079 ceph-mon[23158]: pgmap v5372: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:56:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9920.077207 secs 2022-03-21T16:56:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9920.077207 secs 2022-03-21T16:56:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9920.077207 secs 2022-03-21T16:56:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:56:59 smithi002 ceph-mon[17167]: pgmap v5373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:56:59 smithi016 ceph-mon[22010]: pgmap v5373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:56:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:56:59 smithi079 ceph-mon[23158]: pgmap v5373: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:01 smithi002 ceph-mon[17167]: pgmap v5374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:01 smithi016 ceph-mon[22010]: pgmap v5374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:01 smithi079 ceph-mon[23158]: pgmap v5374: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:03 smithi002 ceph-mon[17167]: pgmap v5375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9925.077326 secs 2022-03-21T16:57:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:03 smithi016 ceph-mon[22010]: pgmap v5375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9925.077326 secs 2022-03-21T16:57:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:03 smithi079 ceph-mon[23158]: pgmap v5375: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9925.077326 secs 2022-03-21T16:57:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:05 smithi002 ceph-mon[17167]: pgmap v5376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:05 smithi016 ceph-mon[22010]: pgmap v5376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:05 smithi079 ceph-mon[23158]: pgmap v5376: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:07 smithi002 ceph-mon[17167]: pgmap v5377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:07 smithi016 ceph-mon[22010]: pgmap v5377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:07 smithi079 ceph-mon[23158]: pgmap v5377: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9930.077440 secs 2022-03-21T16:57:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9930.077440 secs 2022-03-21T16:57:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9930.077440 secs 2022-03-21T16:57:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:09 smithi002 ceph-mon[17167]: pgmap v5378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:09 smithi016 ceph-mon[22010]: pgmap v5378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:09 smithi079 ceph-mon[23158]: pgmap v5378: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:11 smithi002 ceph-mon[17167]: pgmap v5379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:11 smithi016 ceph-mon[22010]: pgmap v5379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:11 smithi079 ceph-mon[23158]: pgmap v5379: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:13.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:13 smithi002 ceph-mon[17167]: pgmap v5380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:13.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9935.077572 secs 2022-03-21T16:57:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:13 smithi016 ceph-mon[22010]: pgmap v5380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9935.077572 secs 2022-03-21T16:57:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:13 smithi079 ceph-mon[23158]: pgmap v5380: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9935.077572 secs 2022-03-21T16:57:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:15 smithi002 ceph-mon[17167]: pgmap v5381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:15 smithi016 ceph-mon[22010]: pgmap v5381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:15 smithi079 ceph-mon[23158]: pgmap v5381: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:17 smithi002 ceph-mon[17167]: pgmap v5382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:17 smithi016 ceph-mon[22010]: pgmap v5382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:17 smithi079 ceph-mon[23158]: pgmap v5382: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9940.077693 secs 2022-03-21T16:57:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9940.077693 secs 2022-03-21T16:57:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9940.077693 secs 2022-03-21T16:57:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:19 smithi002 ceph-mon[17167]: pgmap v5383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:19 smithi016 ceph-mon[22010]: pgmap v5383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:19 smithi079 ceph-mon[23158]: pgmap v5383: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:21 smithi002 ceph-mon[17167]: pgmap v5384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:57:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:57:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:21 smithi016 ceph-mon[22010]: pgmap v5384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:57:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:57:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:21 smithi079 ceph-mon[23158]: pgmap v5384: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:57:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:57:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: pgmap v5385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9945.077809 secs 2022-03-21T16:57:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:57:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:57:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:57:23.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:23 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:57:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: pgmap v5385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9945.077809 secs 2022-03-21T16:57:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:57:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:57:23.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:57:23.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:23 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:57:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: pgmap v5385: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9945.077809 secs 2022-03-21T16:57:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:57:23.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:57:23.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:57:23.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:23 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:57:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:25 smithi002 ceph-mon[17167]: pgmap v5386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:25 smithi016 ceph-mon[22010]: pgmap v5386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:25 smithi079 ceph-mon[23158]: pgmap v5386: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:27 smithi002 ceph-mon[17167]: pgmap v5387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:27 smithi016 ceph-mon[22010]: pgmap v5387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:27 smithi079 ceph-mon[23158]: pgmap v5387: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9950.077963 secs 2022-03-21T16:57:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9950.077963 secs 2022-03-21T16:57:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9950.077963 secs 2022-03-21T16:57:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:29 smithi002 ceph-mon[17167]: pgmap v5388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:29 smithi016 ceph-mon[22010]: pgmap v5388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:29 smithi079 ceph-mon[23158]: pgmap v5388: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:31 smithi002 ceph-mon[17167]: pgmap v5389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:31 smithi016 ceph-mon[22010]: pgmap v5389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:31 smithi079 ceph-mon[23158]: pgmap v5389: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:33 smithi002 ceph-mon[17167]: pgmap v5390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9955.078059 secs 2022-03-21T16:57:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:33 smithi016 ceph-mon[22010]: pgmap v5390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9955.078059 secs 2022-03-21T16:57:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:33 smithi079 ceph-mon[23158]: pgmap v5390: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9955.078059 secs 2022-03-21T16:57:34.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:34 smithi002 ceph-mon[17167]: pgmap v5391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:34 smithi016 ceph-mon[22010]: pgmap v5391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:34 smithi079 ceph-mon[23158]: pgmap v5391: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:37.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:36 smithi002 ceph-mon[17167]: pgmap v5392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:36 smithi016 ceph-mon[22010]: pgmap v5392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:36 smithi079 ceph-mon[23158]: pgmap v5392: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9960.078197 secs 2022-03-21T16:57:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9960.078197 secs 2022-03-21T16:57:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9960.078197 secs 2022-03-21T16:57:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:39 smithi002 ceph-mon[17167]: pgmap v5393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:39 smithi016 ceph-mon[22010]: pgmap v5393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:39 smithi079 ceph-mon[23158]: pgmap v5393: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:41 smithi002 ceph-mon[17167]: pgmap v5394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:41 smithi016 ceph-mon[22010]: pgmap v5394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:41 smithi079 ceph-mon[23158]: pgmap v5394: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:43 smithi002 ceph-mon[17167]: pgmap v5395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9965.078335 secs 2022-03-21T16:57:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:43 smithi016 ceph-mon[22010]: pgmap v5395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9965.078335 secs 2022-03-21T16:57:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:43 smithi079 ceph-mon[23158]: pgmap v5395: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9965.078335 secs 2022-03-21T16:57:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:45 smithi002 ceph-mon[17167]: pgmap v5396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:45 smithi016 ceph-mon[22010]: pgmap v5396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:45 smithi079 ceph-mon[23158]: pgmap v5396: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:47 smithi002 ceph-mon[17167]: pgmap v5397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:47 smithi016 ceph-mon[22010]: pgmap v5397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:47 smithi079 ceph-mon[23158]: pgmap v5397: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9970.078464 secs 2022-03-21T16:57:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9970.078464 secs 2022-03-21T16:57:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9970.078464 secs 2022-03-21T16:57:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:49 smithi002 ceph-mon[17167]: pgmap v5398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:49 smithi016 ceph-mon[22010]: pgmap v5398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:49 smithi079 ceph-mon[23158]: pgmap v5398: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:51 smithi002 ceph-mon[17167]: pgmap v5399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:51 smithi016 ceph-mon[22010]: pgmap v5399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:51 smithi079 ceph-mon[23158]: pgmap v5399: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:53 smithi002 ceph-mon[17167]: pgmap v5400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9975.078570 secs 2022-03-21T16:57:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:53 smithi016 ceph-mon[22010]: pgmap v5400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9975.078570 secs 2022-03-21T16:57:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:53 smithi079 ceph-mon[23158]: pgmap v5400: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9975.078570 secs 2022-03-21T16:57:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:55 smithi002 ceph-mon[17167]: pgmap v5401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:55 smithi016 ceph-mon[22010]: pgmap v5401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:55 smithi079 ceph-mon[23158]: pgmap v5401: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:57 smithi002 ceph-mon[17167]: pgmap v5402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:57 smithi016 ceph-mon[22010]: pgmap v5402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:57 smithi079 ceph-mon[23158]: pgmap v5402: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:57:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9980.078637 secs 2022-03-21T16:57:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9980.078637 secs 2022-03-21T16:57:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9980.078637 secs 2022-03-21T16:57:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:57:59 smithi002 ceph-mon[17167]: pgmap v5403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:57:59 smithi016 ceph-mon[22010]: pgmap v5403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:57:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:57:59 smithi079 ceph-mon[23158]: pgmap v5403: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:01 smithi002 ceph-mon[17167]: pgmap v5404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:01 smithi016 ceph-mon[22010]: pgmap v5404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:01 smithi079 ceph-mon[23158]: pgmap v5404: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:03 smithi002 ceph-mon[17167]: pgmap v5405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9985.078703 secs 2022-03-21T16:58:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:03 smithi016 ceph-mon[22010]: pgmap v5405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9985.078703 secs 2022-03-21T16:58:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:03 smithi079 ceph-mon[23158]: pgmap v5405: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9985.078703 secs 2022-03-21T16:58:05.616 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:05 smithi002 ceph-mon[17167]: pgmap v5406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:05 smithi016 ceph-mon[22010]: pgmap v5406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:05 smithi079 ceph-mon[23158]: pgmap v5406: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:07 smithi002 ceph-mon[17167]: pgmap v5407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:07 smithi016 ceph-mon[22010]: pgmap v5407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:07 smithi079 ceph-mon[23158]: pgmap v5407: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9990.078785 secs 2022-03-21T16:58:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9990.078785 secs 2022-03-21T16:58:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9990.078785 secs 2022-03-21T16:58:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:09 smithi002 ceph-mon[17167]: pgmap v5408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:09 smithi016 ceph-mon[22010]: pgmap v5408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:09 smithi079 ceph-mon[23158]: pgmap v5408: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:11 smithi002 ceph-mon[17167]: pgmap v5409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:11 smithi016 ceph-mon[22010]: pgmap v5409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:11 smithi079 ceph-mon[23158]: pgmap v5409: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:13 smithi002 ceph-mon[17167]: pgmap v5410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 9995.078846 secs 2022-03-21T16:58:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:13 smithi016 ceph-mon[22010]: pgmap v5410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 9995.078846 secs 2022-03-21T16:58:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:13 smithi079 ceph-mon[23158]: pgmap v5410: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 9995.078846 secs 2022-03-21T16:58:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:15 smithi002 ceph-mon[17167]: pgmap v5411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:15 smithi016 ceph-mon[22010]: pgmap v5411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:15 smithi079 ceph-mon[23158]: pgmap v5411: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:17 smithi002 ceph-mon[17167]: pgmap v5412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:17 smithi016 ceph-mon[22010]: pgmap v5412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:17 smithi079 ceph-mon[23158]: pgmap v5412: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10000.078947 secs 2022-03-21T16:58:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10000.078947 secs 2022-03-21T16:58:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10000.078947 secs 2022-03-21T16:58:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:19 smithi002 ceph-mon[17167]: pgmap v5413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:19 smithi016 ceph-mon[22010]: pgmap v5413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:19 smithi079 ceph-mon[23158]: pgmap v5413: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:21 smithi002 ceph-mon[17167]: pgmap v5414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:58:21.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:58:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:21 smithi016 ceph-mon[22010]: pgmap v5414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:58:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:58:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:21 smithi079 ceph-mon[23158]: pgmap v5414: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:58:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:58:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:23 smithi002 ceph-mon[17167]: pgmap v5415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10005.079045 secs 2022-03-21T16:58:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:23 smithi016 ceph-mon[22010]: pgmap v5415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10005.079045 secs 2022-03-21T16:58:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:23 smithi079 ceph-mon[23158]: pgmap v5415: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10005.079045 secs 2022-03-21T16:58:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:58:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:58:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:58:24.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:58:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:58:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:58:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:58:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:58:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:58:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:58:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:58:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:58:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:25 smithi002 ceph-mon[17167]: pgmap v5416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:25 smithi016 ceph-mon[22010]: pgmap v5416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:25 smithi079 ceph-mon[23158]: pgmap v5416: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:27 smithi002 ceph-mon[17167]: pgmap v5417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:27 smithi016 ceph-mon[22010]: pgmap v5417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:27 smithi079 ceph-mon[23158]: pgmap v5417: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10010.079179 secs 2022-03-21T16:58:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10010.079179 secs 2022-03-21T16:58:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10010.079179 secs 2022-03-21T16:58:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:29 smithi002 ceph-mon[17167]: pgmap v5418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:29 smithi016 ceph-mon[22010]: pgmap v5418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:29 smithi079 ceph-mon[23158]: pgmap v5418: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:31.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:31 smithi002 ceph-mon[17167]: pgmap v5419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:31 smithi016 ceph-mon[22010]: pgmap v5419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:31 smithi079 ceph-mon[23158]: pgmap v5419: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:33 smithi002 ceph-mon[17167]: pgmap v5420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10015.079316 secs 2022-03-21T16:58:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:33 smithi016 ceph-mon[22010]: pgmap v5420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10015.079316 secs 2022-03-21T16:58:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:33 smithi079 ceph-mon[23158]: pgmap v5420: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10015.079316 secs 2022-03-21T16:58:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:35 smithi002 ceph-mon[17167]: pgmap v5421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:35 smithi016 ceph-mon[22010]: pgmap v5421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:35 smithi079 ceph-mon[23158]: pgmap v5421: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:37 smithi002 ceph-mon[17167]: pgmap v5422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:37 smithi016 ceph-mon[22010]: pgmap v5422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:37 smithi079 ceph-mon[23158]: pgmap v5422: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10020.079446 secs 2022-03-21T16:58:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10020.079446 secs 2022-03-21T16:58:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10020.079446 secs 2022-03-21T16:58:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:39 smithi016 ceph-mon[22010]: pgmap v5423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:39 smithi079 ceph-mon[23158]: pgmap v5423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:39 smithi002 ceph-mon[17167]: pgmap v5423: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:40 smithi079 ceph-mon[23158]: pgmap v5424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:40.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:40 smithi002 ceph-mon[17167]: pgmap v5424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:40 smithi016 ceph-mon[22010]: pgmap v5424: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:43 smithi002 ceph-mon[17167]: pgmap v5425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10025.079569 secs 2022-03-21T16:58:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:43 smithi016 ceph-mon[22010]: pgmap v5425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10025.079569 secs 2022-03-21T16:58:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:43 smithi079 ceph-mon[23158]: pgmap v5425: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10025.079569 secs 2022-03-21T16:58:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:45 smithi002 ceph-mon[17167]: pgmap v5426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:45 smithi016 ceph-mon[22010]: pgmap v5426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:45 smithi079 ceph-mon[23158]: pgmap v5426: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:47 smithi002 ceph-mon[17167]: pgmap v5427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:47 smithi016 ceph-mon[22010]: pgmap v5427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:47 smithi079 ceph-mon[23158]: pgmap v5427: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10030.079662 secs 2022-03-21T16:58:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10030.079662 secs 2022-03-21T16:58:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10030.079662 secs 2022-03-21T16:58:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:49 smithi002 ceph-mon[17167]: pgmap v5428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:49 smithi016 ceph-mon[22010]: pgmap v5428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:49 smithi079 ceph-mon[23158]: pgmap v5428: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:51 smithi002 ceph-mon[17167]: pgmap v5429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:51 smithi016 ceph-mon[22010]: pgmap v5429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:51 smithi079 ceph-mon[23158]: pgmap v5429: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T16:58:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:53 smithi002 ceph-mon[17167]: pgmap v5430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10035.079728 secs 2022-03-21T16:58:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:53 smithi016 ceph-mon[22010]: pgmap v5430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10035.079728 secs 2022-03-21T16:58:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:53 smithi079 ceph-mon[23158]: pgmap v5430: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10035.079728 secs 2022-03-21T16:58:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:55 smithi002 ceph-mon[17167]: pgmap v5431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:55 smithi016 ceph-mon[22010]: pgmap v5431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:55 smithi079 ceph-mon[23158]: pgmap v5431: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:57 smithi002 ceph-mon[17167]: pgmap v5432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:57 smithi016 ceph-mon[22010]: pgmap v5432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:57 smithi079 ceph-mon[23158]: pgmap v5432: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:58:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10040.079827 secs 2022-03-21T16:58:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10040.079827 secs 2022-03-21T16:58:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10040.079827 secs 2022-03-21T16:58:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:58:59 smithi002 ceph-mon[17167]: pgmap v5433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:58:59 smithi016 ceph-mon[22010]: pgmap v5433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:58:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:58:59 smithi079 ceph-mon[23158]: pgmap v5433: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:01 smithi002 ceph-mon[17167]: pgmap v5434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:01 smithi016 ceph-mon[22010]: pgmap v5434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:01 smithi079 ceph-mon[23158]: pgmap v5434: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:03 smithi002 ceph-mon[17167]: pgmap v5435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10045.079946 secs 2022-03-21T16:59:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:03 smithi016 ceph-mon[22010]: pgmap v5435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10045.079946 secs 2022-03-21T16:59:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:03 smithi079 ceph-mon[23158]: pgmap v5435: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10045.079946 secs 2022-03-21T16:59:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:05 smithi002 ceph-mon[17167]: pgmap v5436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:05 smithi016 ceph-mon[22010]: pgmap v5436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:05 smithi079 ceph-mon[23158]: pgmap v5436: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:07 smithi002 ceph-mon[17167]: pgmap v5437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:07 smithi016 ceph-mon[22010]: pgmap v5437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:07 smithi079 ceph-mon[23158]: pgmap v5437: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10050.080080 secs 2022-03-21T16:59:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10050.080080 secs 2022-03-21T16:59:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10050.080080 secs 2022-03-21T16:59:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:09 smithi002 ceph-mon[17167]: pgmap v5438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:09 smithi016 ceph-mon[22010]: pgmap v5438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:09 smithi079 ceph-mon[23158]: pgmap v5438: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:11 smithi002 ceph-mon[17167]: pgmap v5439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:11 smithi016 ceph-mon[22010]: pgmap v5439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:11 smithi079 ceph-mon[23158]: pgmap v5439: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:13 smithi002 ceph-mon[17167]: pgmap v5440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10055.080198 secs 2022-03-21T16:59:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:13 smithi016 ceph-mon[22010]: pgmap v5440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10055.080198 secs 2022-03-21T16:59:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:13 smithi079 ceph-mon[23158]: pgmap v5440: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10055.080198 secs 2022-03-21T16:59:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:15 smithi002 ceph-mon[17167]: pgmap v5441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:15 smithi016 ceph-mon[22010]: pgmap v5441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:15 smithi079 ceph-mon[23158]: pgmap v5441: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:17 smithi002 ceph-mon[17167]: pgmap v5442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:17 smithi016 ceph-mon[22010]: pgmap v5442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:17 smithi079 ceph-mon[23158]: pgmap v5442: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10060.080324 secs 2022-03-21T16:59:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10060.080324 secs 2022-03-21T16:59:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10060.080324 secs 2022-03-21T16:59:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:19 smithi002 ceph-mon[17167]: pgmap v5443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:19 smithi016 ceph-mon[22010]: pgmap v5443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:19 smithi079 ceph-mon[23158]: pgmap v5443: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:21 smithi002 ceph-mon[17167]: pgmap v5444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:59:21.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:59:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:21 smithi016 ceph-mon[22010]: pgmap v5444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:59:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:59:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:21 smithi079 ceph-mon[23158]: pgmap v5444: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T16:59:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T16:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:23 smithi002 ceph-mon[17167]: pgmap v5445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10065.080449 secs 2022-03-21T16:59:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:23 smithi016 ceph-mon[22010]: pgmap v5445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10065.080449 secs 2022-03-21T16:59:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:23 smithi079 ceph-mon[23158]: pgmap v5445: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10065.080449 secs 2022-03-21T16:59:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:59:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:59:24.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:24 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:59:24.643 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:59:24.644 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:59:24.644 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:24 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:59:24.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T16:59:24.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T16:59:24.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:24 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T16:59:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:25 smithi002 ceph-mon[17167]: pgmap v5446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:59:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:25 smithi016 ceph-mon[22010]: pgmap v5446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:59:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:25 smithi079 ceph-mon[23158]: pgmap v5446: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T16:59:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:27 smithi002 ceph-mon[17167]: pgmap v5447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:27 smithi016 ceph-mon[22010]: pgmap v5447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:27 smithi079 ceph-mon[23158]: pgmap v5447: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:28.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10070.080595 secs 2022-03-21T16:59:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10070.080595 secs 2022-03-21T16:59:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10070.080595 secs 2022-03-21T16:59:29.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:29 smithi002 ceph-mon[17167]: pgmap v5448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:29 smithi016 ceph-mon[22010]: pgmap v5448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:29 smithi079 ceph-mon[23158]: pgmap v5448: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:31 smithi016 ceph-mon[22010]: pgmap v5449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:31 smithi079 ceph-mon[23158]: pgmap v5449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:31 smithi002 ceph-mon[17167]: pgmap v5449: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:33 smithi016 ceph-mon[22010]: pgmap v5450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10075.080710 secs 2022-03-21T16:59:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:33 smithi079 ceph-mon[23158]: pgmap v5450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10075.080710 secs 2022-03-21T16:59:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:33 smithi002 ceph-mon[17167]: pgmap v5450: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10075.080710 secs 2022-03-21T16:59:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:35 smithi016 ceph-mon[22010]: pgmap v5451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:35 smithi079 ceph-mon[23158]: pgmap v5451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:35.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:35 smithi002 ceph-mon[17167]: pgmap v5451: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:37 smithi016 ceph-mon[22010]: pgmap v5452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:37 smithi079 ceph-mon[23158]: pgmap v5452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:37.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:37 smithi002 ceph-mon[17167]: pgmap v5452: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10080.080812 secs 2022-03-21T16:59:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10080.080812 secs 2022-03-21T16:59:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10080.080812 secs 2022-03-21T16:59:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:39 smithi016 ceph-mon[22010]: pgmap v5453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:39 smithi079 ceph-mon[23158]: pgmap v5453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:39 smithi002 ceph-mon[17167]: pgmap v5453: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:40 smithi079 ceph-mon[23158]: pgmap v5454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:40.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:40 smithi002 ceph-mon[17167]: pgmap v5454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:40 smithi016 ceph-mon[22010]: pgmap v5454: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:43 smithi002 ceph-mon[17167]: pgmap v5455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10085.080959 secs 2022-03-21T16:59:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:43 smithi016 ceph-mon[22010]: pgmap v5455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10085.080959 secs 2022-03-21T16:59:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:43 smithi079 ceph-mon[23158]: pgmap v5455: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10085.080959 secs 2022-03-21T16:59:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:45 smithi002 ceph-mon[17167]: pgmap v5456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:45 smithi016 ceph-mon[22010]: pgmap v5456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:45 smithi079 ceph-mon[23158]: pgmap v5456: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:47 smithi002 ceph-mon[17167]: pgmap v5457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:47 smithi016 ceph-mon[22010]: pgmap v5457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:47 smithi079 ceph-mon[23158]: pgmap v5457: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10090.081077 secs 2022-03-21T16:59:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10090.081077 secs 2022-03-21T16:59:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10090.081077 secs 2022-03-21T16:59:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:49 smithi002 ceph-mon[17167]: pgmap v5458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:49 smithi016 ceph-mon[22010]: pgmap v5458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:49 smithi079 ceph-mon[23158]: pgmap v5458: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:51 smithi002 ceph-mon[17167]: pgmap v5459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:51 smithi016 ceph-mon[22010]: pgmap v5459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:51 smithi079 ceph-mon[23158]: pgmap v5459: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:53 smithi002 ceph-mon[17167]: pgmap v5460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10095.081202 secs 2022-03-21T16:59:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:53 smithi016 ceph-mon[22010]: pgmap v5460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10095.081202 secs 2022-03-21T16:59:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:53 smithi079 ceph-mon[23158]: pgmap v5460: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10095.081202 secs 2022-03-21T16:59:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:55 smithi002 ceph-mon[17167]: pgmap v5461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:55 smithi016 ceph-mon[22010]: pgmap v5461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:55 smithi079 ceph-mon[23158]: pgmap v5461: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:57 smithi002 ceph-mon[17167]: pgmap v5462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:57 smithi016 ceph-mon[22010]: pgmap v5462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:57 smithi079 ceph-mon[23158]: pgmap v5462: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T16:59:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10100.081318 secs 2022-03-21T16:59:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10100.081318 secs 2022-03-21T16:59:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10100.081318 secs 2022-03-21T16:59:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 16:59:59 smithi002 ceph-mon[17167]: pgmap v5463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 16:59:59 smithi016 ceph-mon[22010]: pgmap v5463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T16:59:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 16:59:59 smithi079 ceph-mon[23158]: pgmap v5463: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:00.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:00:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:00:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:00:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:01 smithi002 ceph-mon[17167]: pgmap v5464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:01 smithi016 ceph-mon[22010]: pgmap v5464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:01 smithi079 ceph-mon[23158]: pgmap v5464: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:03 smithi002 ceph-mon[17167]: pgmap v5465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10105.081451 secs 2022-03-21T17:00:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:03 smithi016 ceph-mon[22010]: pgmap v5465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10105.081451 secs 2022-03-21T17:00:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:03 smithi079 ceph-mon[23158]: pgmap v5465: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10105.081451 secs 2022-03-21T17:00:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:05 smithi002 ceph-mon[17167]: pgmap v5466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:05 smithi016 ceph-mon[22010]: pgmap v5466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:05 smithi079 ceph-mon[23158]: pgmap v5466: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:07 smithi002 ceph-mon[17167]: pgmap v5467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:07 smithi016 ceph-mon[22010]: pgmap v5467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:07 smithi079 ceph-mon[23158]: pgmap v5467: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:08.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10110.081573 secs 2022-03-21T17:00:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10110.081573 secs 2022-03-21T17:00:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10110.081573 secs 2022-03-21T17:00:09.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:09 smithi002 ceph-mon[17167]: pgmap v5468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:09 smithi016 ceph-mon[22010]: pgmap v5468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:09 smithi079 ceph-mon[23158]: pgmap v5468: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:11.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:11 smithi002 ceph-mon[17167]: pgmap v5469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:11 smithi016 ceph-mon[22010]: pgmap v5469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:11 smithi079 ceph-mon[23158]: pgmap v5469: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:13 smithi002 ceph-mon[17167]: pgmap v5470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10115.081698 secs 2022-03-21T17:00:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:13 smithi016 ceph-mon[22010]: pgmap v5470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10115.081698 secs 2022-03-21T17:00:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:13 smithi079 ceph-mon[23158]: pgmap v5470: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10115.081698 secs 2022-03-21T17:00:15.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:15 smithi002 ceph-mon[17167]: pgmap v5471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:15 smithi016 ceph-mon[22010]: pgmap v5471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:15 smithi079 ceph-mon[23158]: pgmap v5471: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:17 smithi002 ceph-mon[17167]: pgmap v5472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:17 smithi016 ceph-mon[22010]: pgmap v5472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:17 smithi079 ceph-mon[23158]: pgmap v5472: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:18.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10120.081815 secs 2022-03-21T17:00:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10120.081815 secs 2022-03-21T17:00:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10120.081815 secs 2022-03-21T17:00:19.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:19 smithi002 ceph-mon[17167]: pgmap v5473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:19 smithi016 ceph-mon[22010]: pgmap v5473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:19 smithi079 ceph-mon[23158]: pgmap v5473: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:21 smithi016 ceph-mon[22010]: pgmap v5474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:00:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:00:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:21 smithi079 ceph-mon[23158]: pgmap v5474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:00:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:00:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:21 smithi002 ceph-mon[17167]: pgmap v5474: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:00:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:00:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:23 smithi016 ceph-mon[22010]: pgmap v5475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10125.081917 secs 2022-03-21T17:00:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:23 smithi079 ceph-mon[23158]: pgmap v5475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10125.081917 secs 2022-03-21T17:00:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:23 smithi002 ceph-mon[17167]: pgmap v5475: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10125.081917 secs 2022-03-21T17:00:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:25 smithi002 ceph-mon[17167]: pgmap v5476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:00:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:00:25.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:25 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:00:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:25 smithi016 ceph-mon[22010]: pgmap v5476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:00:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:00:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:25 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:00:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:25 smithi079 ceph-mon[23158]: pgmap v5476: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:00:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:00:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:25 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:00:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:00:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:00:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:00:26.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:00:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:00:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:00:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:00:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:00:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: pgmap v5477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:00:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:00:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:00:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:00:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:00:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:00:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: pgmap v5477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:00:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:00:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:00:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:00:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:00:27.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:00:27.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: pgmap v5477: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:27.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:00:27.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:27 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:00:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10130.082037 secs 2022-03-21T17:00:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10130.082037 secs 2022-03-21T17:00:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10130.082037 secs 2022-03-21T17:00:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:29 smithi016 ceph-mon[22010]: pgmap v5478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:29 smithi079 ceph-mon[23158]: pgmap v5478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:29 smithi002 ceph-mon[17167]: pgmap v5478: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:31 smithi016 ceph-mon[22010]: pgmap v5479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:31 smithi079 ceph-mon[23158]: pgmap v5479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:31 smithi002 ceph-mon[17167]: pgmap v5479: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:33 smithi016 ceph-mon[22010]: pgmap v5480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10135.082176 secs 2022-03-21T17:00:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:33 smithi079 ceph-mon[23158]: pgmap v5480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10135.082176 secs 2022-03-21T17:00:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:33 smithi002 ceph-mon[17167]: pgmap v5480: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10135.082176 secs 2022-03-21T17:00:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:35 smithi016 ceph-mon[22010]: pgmap v5481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:35 smithi079 ceph-mon[23158]: pgmap v5481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:35.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:35 smithi002 ceph-mon[17167]: pgmap v5481: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:36 smithi016 ceph-mon[22010]: pgmap v5482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:36 smithi079 ceph-mon[23158]: pgmap v5482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:36.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:36 smithi002 ceph-mon[17167]: pgmap v5482: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:38.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10140.082322 secs 2022-03-21T17:00:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10140.082322 secs 2022-03-21T17:00:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10140.082322 secs 2022-03-21T17:00:39.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:39 smithi002 ceph-mon[17167]: pgmap v5483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:39 smithi016 ceph-mon[22010]: pgmap v5483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:39.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:39 smithi079 ceph-mon[23158]: pgmap v5483: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:41 smithi002 ceph-mon[17167]: pgmap v5484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:41 smithi016 ceph-mon[22010]: pgmap v5484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:41 smithi079 ceph-mon[23158]: pgmap v5484: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:43 smithi002 ceph-mon[17167]: pgmap v5485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10145.082438 secs 2022-03-21T17:00:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:43 smithi016 ceph-mon[22010]: pgmap v5485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10145.082438 secs 2022-03-21T17:00:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:43 smithi079 ceph-mon[23158]: pgmap v5485: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10145.082438 secs 2022-03-21T17:00:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:45 smithi002 ceph-mon[17167]: pgmap v5486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:45 smithi016 ceph-mon[22010]: pgmap v5486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:45 smithi079 ceph-mon[23158]: pgmap v5486: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:47 smithi002 ceph-mon[17167]: pgmap v5487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:47 smithi016 ceph-mon[22010]: pgmap v5487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:47 smithi079 ceph-mon[23158]: pgmap v5487: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10150.082573 secs 2022-03-21T17:00:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10150.082573 secs 2022-03-21T17:00:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10150.082573 secs 2022-03-21T17:00:49.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:49 smithi002 ceph-mon[17167]: pgmap v5488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:49 smithi016 ceph-mon[22010]: pgmap v5488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:49 smithi079 ceph-mon[23158]: pgmap v5488: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:51.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:51 smithi002 ceph-mon[17167]: pgmap v5489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:51 smithi016 ceph-mon[22010]: pgmap v5489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:51 smithi079 ceph-mon[23158]: pgmap v5489: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:53 smithi002 ceph-mon[17167]: pgmap v5490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10155.082694 secs 2022-03-21T17:00:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:53 smithi016 ceph-mon[22010]: pgmap v5490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10155.082694 secs 2022-03-21T17:00:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:53 smithi079 ceph-mon[23158]: pgmap v5490: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10155.082694 secs 2022-03-21T17:00:55.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:55 smithi002 ceph-mon[17167]: pgmap v5491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:55 smithi016 ceph-mon[22010]: pgmap v5491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:55 smithi079 ceph-mon[23158]: pgmap v5491: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:57 smithi002 ceph-mon[17167]: pgmap v5492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:57 smithi016 ceph-mon[22010]: pgmap v5492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:57 smithi079 ceph-mon[23158]: pgmap v5492: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:00:58.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10160.082802 secs 2022-03-21T17:00:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10160.082802 secs 2022-03-21T17:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10160.082802 secs 2022-03-21T17:00:59.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:00:59 smithi002 ceph-mon[17167]: pgmap v5493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:00:59 smithi016 ceph-mon[22010]: pgmap v5493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:00:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:00:59 smithi079 ceph-mon[23158]: pgmap v5493: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:01.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:01 smithi002 ceph-mon[17167]: pgmap v5494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:01 smithi016 ceph-mon[22010]: pgmap v5494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:01 smithi079 ceph-mon[23158]: pgmap v5494: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:03 smithi002 ceph-mon[17167]: pgmap v5495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:03.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10165.082932 secs 2022-03-21T17:01:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:03 smithi016 ceph-mon[22010]: pgmap v5495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10165.082932 secs 2022-03-21T17:01:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:03 smithi079 ceph-mon[23158]: pgmap v5495: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10165.082932 secs 2022-03-21T17:01:05.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:05 smithi002 ceph-mon[17167]: pgmap v5496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:05 smithi016 ceph-mon[22010]: pgmap v5496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:05 smithi079 ceph-mon[23158]: pgmap v5496: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:07.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:07 smithi002 ceph-mon[17167]: pgmap v5497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:07 smithi016 ceph-mon[22010]: pgmap v5497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:07 smithi079 ceph-mon[23158]: pgmap v5497: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10170.083030 secs 2022-03-21T17:01:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10170.083030 secs 2022-03-21T17:01:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10170.083030 secs 2022-03-21T17:01:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:09 smithi016 ceph-mon[22010]: pgmap v5498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:09 smithi079 ceph-mon[23158]: pgmap v5498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:09.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:09 smithi002 ceph-mon[17167]: pgmap v5498: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:11 smithi016 ceph-mon[22010]: pgmap v5499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:11 smithi079 ceph-mon[23158]: pgmap v5499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:11.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:11 smithi002 ceph-mon[17167]: pgmap v5499: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:13 smithi016 ceph-mon[22010]: pgmap v5500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10175.083151 secs 2022-03-21T17:01:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:13 smithi079 ceph-mon[23158]: pgmap v5500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10175.083151 secs 2022-03-21T17:01:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:13 smithi002 ceph-mon[17167]: pgmap v5500: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10175.083151 secs 2022-03-21T17:01:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:15 smithi016 ceph-mon[22010]: pgmap v5501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:15 smithi079 ceph-mon[23158]: pgmap v5501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:15 smithi002 ceph-mon[17167]: pgmap v5501: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:17 smithi016 ceph-mon[22010]: pgmap v5502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:17 smithi079 ceph-mon[23158]: pgmap v5502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:17 smithi002 ceph-mon[17167]: pgmap v5502: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10180.083300 secs 2022-03-21T17:01:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10180.083300 secs 2022-03-21T17:01:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10180.083300 secs 2022-03-21T17:01:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:19 smithi016 ceph-mon[22010]: pgmap v5503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:19 smithi079 ceph-mon[23158]: pgmap v5503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:19 smithi002 ceph-mon[17167]: pgmap v5503: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:21 smithi016 ceph-mon[22010]: pgmap v5504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:01:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:01:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:21 smithi079 ceph-mon[23158]: pgmap v5504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:01:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:01:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:21 smithi002 ceph-mon[17167]: pgmap v5504: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:01:21.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:01:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:23 smithi016 ceph-mon[22010]: pgmap v5505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10185.083402 secs 2022-03-21T17:01:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:23 smithi079 ceph-mon[23158]: pgmap v5505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10185.083402 secs 2022-03-21T17:01:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:23 smithi002 ceph-mon[17167]: pgmap v5505: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:23.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10185.083402 secs 2022-03-21T17:01:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:25 smithi016 ceph-mon[22010]: pgmap v5506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:25 smithi079 ceph-mon[23158]: pgmap v5506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:25 smithi002 ceph-mon[17167]: pgmap v5506: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:01:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:01:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:26 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:01:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:01:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:01:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:26 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:01:26.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:01:26.799 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:01:26.800 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:26 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:27 smithi016 ceph-mon[22010]: pgmap v5507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:01:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:27 smithi079 ceph-mon[23158]: pgmap v5507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:01:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:27 smithi002 ceph-mon[17167]: pgmap v5507: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:01:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10190.083561 secs 2022-03-21T17:01:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:28 smithi079 ceph-mon[23158]: pgmap v5508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10190.083561 secs 2022-03-21T17:01:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:28 smithi002 ceph-mon[17167]: pgmap v5508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10190.083561 secs 2022-03-21T17:01:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:28 smithi016 ceph-mon[22010]: pgmap v5508: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:30 smithi016 ceph-mon[22010]: pgmap v5509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:30 smithi079 ceph-mon[23158]: pgmap v5509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:30 smithi002 ceph-mon[17167]: pgmap v5509: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:33 smithi002 ceph-mon[17167]: pgmap v5510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:33.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10195.083671 secs 2022-03-21T17:01:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:33 smithi016 ceph-mon[22010]: pgmap v5510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10195.083671 secs 2022-03-21T17:01:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:33 smithi079 ceph-mon[23158]: pgmap v5510: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10195.083671 secs 2022-03-21T17:01:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:35 smithi002 ceph-mon[17167]: pgmap v5511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:35 smithi016 ceph-mon[22010]: pgmap v5511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:35 smithi079 ceph-mon[23158]: pgmap v5511: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:37 smithi002 ceph-mon[17167]: pgmap v5512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:37 smithi016 ceph-mon[22010]: pgmap v5512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:37 smithi079 ceph-mon[23158]: pgmap v5512: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10200.083796 secs 2022-03-21T17:01:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10200.083796 secs 2022-03-21T17:01:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10200.083796 secs 2022-03-21T17:01:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:39 smithi002 ceph-mon[17167]: pgmap v5513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:39 smithi016 ceph-mon[22010]: pgmap v5513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:39 smithi079 ceph-mon[23158]: pgmap v5513: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:41 smithi002 ceph-mon[17167]: pgmap v5514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:41 smithi016 ceph-mon[22010]: pgmap v5514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:41 smithi079 ceph-mon[23158]: pgmap v5514: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:43 smithi002 ceph-mon[17167]: pgmap v5515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10205.083929 secs 2022-03-21T17:01:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:43 smithi016 ceph-mon[22010]: pgmap v5515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10205.083929 secs 2022-03-21T17:01:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:43 smithi079 ceph-mon[23158]: pgmap v5515: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10205.083929 secs 2022-03-21T17:01:45.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:45 smithi002 ceph-mon[17167]: pgmap v5516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:45 smithi016 ceph-mon[22010]: pgmap v5516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:45 smithi079 ceph-mon[23158]: pgmap v5516: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:47.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:47 smithi002 ceph-mon[17167]: pgmap v5517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:47 smithi016 ceph-mon[22010]: pgmap v5517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:47 smithi079 ceph-mon[23158]: pgmap v5517: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:48.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10210.084060 secs 2022-03-21T17:01:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10210.084060 secs 2022-03-21T17:01:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10210.084060 secs 2022-03-21T17:01:49.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:49 smithi002 ceph-mon[17167]: pgmap v5518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:49 smithi016 ceph-mon[22010]: pgmap v5518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:49 smithi079 ceph-mon[23158]: pgmap v5518: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:51.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:51 smithi002 ceph-mon[17167]: pgmap v5519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:51 smithi016 ceph-mon[22010]: pgmap v5519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:51 smithi079 ceph-mon[23158]: pgmap v5519: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:53 smithi002 ceph-mon[17167]: pgmap v5520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:53.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10215.084191 secs 2022-03-21T17:01:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:53 smithi016 ceph-mon[22010]: pgmap v5520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10215.084191 secs 2022-03-21T17:01:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:53 smithi079 ceph-mon[23158]: pgmap v5520: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10215.084191 secs 2022-03-21T17:01:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:55 smithi016 ceph-mon[22010]: pgmap v5521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:55 smithi079 ceph-mon[23158]: pgmap v5521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:55 smithi002 ceph-mon[17167]: pgmap v5521: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:57 smithi016 ceph-mon[22010]: pgmap v5522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:57 smithi079 ceph-mon[23158]: pgmap v5522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:57 smithi002 ceph-mon[17167]: pgmap v5522: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:01:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10220.084313 secs 2022-03-21T17:01:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10220.084313 secs 2022-03-21T17:01:58.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10220.084313 secs 2022-03-21T17:01:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:01:59 smithi016 ceph-mon[22010]: pgmap v5523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:01:59 smithi079 ceph-mon[23158]: pgmap v5523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:01:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:01:59 smithi002 ceph-mon[17167]: pgmap v5523: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:01 smithi016 ceph-mon[22010]: pgmap v5524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:01 smithi079 ceph-mon[23158]: pgmap v5524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:01 smithi002 ceph-mon[17167]: pgmap v5524: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:03 smithi016 ceph-mon[22010]: pgmap v5525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10225.084443 secs 2022-03-21T17:02:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:03 smithi079 ceph-mon[23158]: pgmap v5525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10225.084443 secs 2022-03-21T17:02:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:03 smithi002 ceph-mon[17167]: pgmap v5525: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10225.084443 secs 2022-03-21T17:02:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:05 smithi016 ceph-mon[22010]: pgmap v5526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:05 smithi079 ceph-mon[23158]: pgmap v5526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:05 smithi002 ceph-mon[17167]: pgmap v5526: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:07 smithi016 ceph-mon[22010]: pgmap v5527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:07 smithi079 ceph-mon[23158]: pgmap v5527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:07 smithi002 ceph-mon[17167]: pgmap v5527: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10230.084605 secs 2022-03-21T17:02:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10230.084605 secs 2022-03-21T17:02:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10230.084605 secs 2022-03-21T17:02:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:09 smithi016 ceph-mon[22010]: pgmap v5528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:09 smithi079 ceph-mon[23158]: pgmap v5528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:09 smithi002 ceph-mon[17167]: pgmap v5528: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:11 smithi016 ceph-mon[22010]: pgmap v5529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:11 smithi079 ceph-mon[23158]: pgmap v5529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:11.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:11 smithi002 ceph-mon[17167]: pgmap v5529: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:13 smithi016 ceph-mon[22010]: pgmap v5530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10235.084686 secs 2022-03-21T17:02:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:13 smithi079 ceph-mon[23158]: pgmap v5530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10235.084686 secs 2022-03-21T17:02:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:13 smithi002 ceph-mon[17167]: pgmap v5530: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10235.084686 secs 2022-03-21T17:02:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:15 smithi016 ceph-mon[22010]: pgmap v5531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:15 smithi079 ceph-mon[23158]: pgmap v5531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:15 smithi002 ceph-mon[17167]: pgmap v5531: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:17 smithi079 ceph-mon[23158]: pgmap v5532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:17.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:17 smithi002 ceph-mon[17167]: pgmap v5532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:17 smithi016 ceph-mon[22010]: pgmap v5532: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10240.084807 secs 2022-03-21T17:02:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10240.084807 secs 2022-03-21T17:02:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10240.084807 secs 2022-03-21T17:02:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:19 smithi079 ceph-mon[23158]: pgmap v5533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:19 smithi002 ceph-mon[17167]: pgmap v5533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:19 smithi016 ceph-mon[22010]: pgmap v5533: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:20 smithi079 ceph-mon[23158]: pgmap v5534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:20 smithi002 ceph-mon[17167]: pgmap v5534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:20 smithi016 ceph-mon[22010]: pgmap v5534: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:02:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:02:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:02:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:02:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:02:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:02:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:23 smithi002 ceph-mon[17167]: pgmap v5535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:23.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10245.084965 secs 2022-03-21T17:02:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:23 smithi016 ceph-mon[22010]: pgmap v5535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10245.084965 secs 2022-03-21T17:02:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:23 smithi079 ceph-mon[23158]: pgmap v5535: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10245.084965 secs 2022-03-21T17:02:25.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:25 smithi002 ceph-mon[17167]: pgmap v5536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:25 smithi016 ceph-mon[22010]: pgmap v5536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:25 smithi079 ceph-mon[23158]: pgmap v5536: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:27 smithi002 ceph-mon[17167]: pgmap v5537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:02:27.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:02:27.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:27 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:02:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:27 smithi016 ceph-mon[22010]: pgmap v5537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:02:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:02:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:27 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:02:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:27 smithi079 ceph-mon[23158]: pgmap v5537: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:02:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:02:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:27 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:02:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10250.085057 secs 2022-03-21T17:02:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:02:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:28 smithi079 ceph-mon[23158]: pgmap v5538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:29.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10250.085057 secs 2022-03-21T17:02:29.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:02:29.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:28 smithi002 ceph-mon[17167]: pgmap v5538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10250.085057 secs 2022-03-21T17:02:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:02:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:28 smithi016 ceph-mon[22010]: pgmap v5538: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:30 smithi016 ceph-mon[22010]: pgmap v5539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:30 smithi079 ceph-mon[23158]: pgmap v5539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:30 smithi002 ceph-mon[17167]: pgmap v5539: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:33 smithi002 ceph-mon[17167]: pgmap v5540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:33.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10255.085218 secs 2022-03-21T17:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:33 smithi016 ceph-mon[22010]: pgmap v5540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10255.085218 secs 2022-03-21T17:02:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:33 smithi079 ceph-mon[23158]: pgmap v5540: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10255.085218 secs 2022-03-21T17:02:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:35 smithi002 ceph-mon[17167]: pgmap v5541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:35 smithi016 ceph-mon[22010]: pgmap v5541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:35 smithi079 ceph-mon[23158]: pgmap v5541: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:37.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:37 smithi002 ceph-mon[17167]: pgmap v5542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:37 smithi016 ceph-mon[22010]: pgmap v5542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:37 smithi079 ceph-mon[23158]: pgmap v5542: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:38.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10260.085350 secs 2022-03-21T17:02:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10260.085350 secs 2022-03-21T17:02:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10260.085350 secs 2022-03-21T17:02:39.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:39 smithi002 ceph-mon[17167]: pgmap v5543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:39 smithi016 ceph-mon[22010]: pgmap v5543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:39 smithi079 ceph-mon[23158]: pgmap v5543: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:41.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:41 smithi002 ceph-mon[17167]: pgmap v5544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:41 smithi016 ceph-mon[22010]: pgmap v5544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:41 smithi079 ceph-mon[23158]: pgmap v5544: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:43 smithi002 ceph-mon[17167]: pgmap v5545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:43.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10265.085452 secs 2022-03-21T17:02:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:43 smithi016 ceph-mon[22010]: pgmap v5545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10265.085452 secs 2022-03-21T17:02:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:43 smithi079 ceph-mon[23158]: pgmap v5545: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10265.085452 secs 2022-03-21T17:02:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:45 smithi016 ceph-mon[22010]: pgmap v5546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:45 smithi079 ceph-mon[23158]: pgmap v5546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:45 smithi002 ceph-mon[17167]: pgmap v5546: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:47 smithi016 ceph-mon[22010]: pgmap v5547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:47 smithi079 ceph-mon[23158]: pgmap v5547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:47 smithi002 ceph-mon[17167]: pgmap v5547: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10270.085585 secs 2022-03-21T17:02:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10270.085585 secs 2022-03-21T17:02:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10270.085585 secs 2022-03-21T17:02:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:49 smithi016 ceph-mon[22010]: pgmap v5548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:49 smithi079 ceph-mon[23158]: pgmap v5548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:49 smithi002 ceph-mon[17167]: pgmap v5548: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:51 smithi016 ceph-mon[22010]: pgmap v5549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:51 smithi079 ceph-mon[23158]: pgmap v5549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:51.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:51 smithi002 ceph-mon[17167]: pgmap v5549: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:53 smithi016 ceph-mon[22010]: pgmap v5550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10275.085724 secs 2022-03-21T17:02:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:53 smithi079 ceph-mon[23158]: pgmap v5550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10275.085724 secs 2022-03-21T17:02:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:53 smithi002 ceph-mon[17167]: pgmap v5550: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10275.085724 secs 2022-03-21T17:02:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:55 smithi016 ceph-mon[22010]: pgmap v5551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:55 smithi079 ceph-mon[23158]: pgmap v5551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:55 smithi002 ceph-mon[17167]: pgmap v5551: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:57 smithi016 ceph-mon[22010]: pgmap v5552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:57 smithi079 ceph-mon[23158]: pgmap v5552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:57 smithi002 ceph-mon[17167]: pgmap v5552: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:02:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10280.085813 secs 2022-03-21T17:02:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10280.085813 secs 2022-03-21T17:02:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10280.085813 secs 2022-03-21T17:02:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:02:59 smithi016 ceph-mon[22010]: pgmap v5553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:02:59 smithi079 ceph-mon[23158]: pgmap v5553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:02:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:02:59 smithi002 ceph-mon[17167]: pgmap v5553: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:01 smithi016 ceph-mon[22010]: pgmap v5554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:01 smithi079 ceph-mon[23158]: pgmap v5554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:01 smithi002 ceph-mon[17167]: pgmap v5554: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:03 smithi079 ceph-mon[23158]: pgmap v5555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10285.085932 secs 2022-03-21T17:03:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:03 smithi002 ceph-mon[17167]: pgmap v5555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10285.085932 secs 2022-03-21T17:03:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:03 smithi016 ceph-mon[22010]: pgmap v5555: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10285.085932 secs 2022-03-21T17:03:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:05 smithi079 ceph-mon[23158]: pgmap v5556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:05 smithi002 ceph-mon[17167]: pgmap v5556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:05 smithi016 ceph-mon[22010]: pgmap v5556: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:07 smithi079 ceph-mon[23158]: pgmap v5557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:07 smithi002 ceph-mon[17167]: pgmap v5557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:07 smithi016 ceph-mon[22010]: pgmap v5557: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10290.086059 secs 2022-03-21T17:03:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10290.086059 secs 2022-03-21T17:03:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10290.086059 secs 2022-03-21T17:03:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:09 smithi079 ceph-mon[23158]: pgmap v5558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:09 smithi002 ceph-mon[17167]: pgmap v5558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:09 smithi016 ceph-mon[22010]: pgmap v5558: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:10 smithi079 ceph-mon[23158]: pgmap v5559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:10 smithi002 ceph-mon[17167]: pgmap v5559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:10 smithi016 ceph-mon[22010]: pgmap v5559: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:13 smithi002 ceph-mon[17167]: pgmap v5560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:13.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10295.086150 secs 2022-03-21T17:03:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:13 smithi016 ceph-mon[22010]: pgmap v5560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10295.086150 secs 2022-03-21T17:03:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:13 smithi079 ceph-mon[23158]: pgmap v5560: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10295.086150 secs 2022-03-21T17:03:15.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:15 smithi002 ceph-mon[17167]: pgmap v5561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:15 smithi016 ceph-mon[22010]: pgmap v5561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:15 smithi079 ceph-mon[23158]: pgmap v5561: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:17.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:17 smithi002 ceph-mon[17167]: pgmap v5562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:17 smithi016 ceph-mon[22010]: pgmap v5562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:17 smithi079 ceph-mon[23158]: pgmap v5562: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10300.086310 secs 2022-03-21T17:03:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10300.086310 secs 2022-03-21T17:03:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10300.086310 secs 2022-03-21T17:03:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:19 smithi016 ceph-mon[22010]: pgmap v5563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:19 smithi079 ceph-mon[23158]: pgmap v5563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:19 smithi002 ceph-mon[17167]: pgmap v5563: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:21 smithi016 ceph-mon[22010]: pgmap v5564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:03:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:21 smithi079 ceph-mon[23158]: pgmap v5564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:03:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:21 smithi002 ceph-mon[17167]: pgmap v5564: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:03:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:03:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:23 smithi016 ceph-mon[22010]: pgmap v5565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10305.086439 secs 2022-03-21T17:03:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:23 smithi079 ceph-mon[23158]: pgmap v5565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10305.086439 secs 2022-03-21T17:03:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:23 smithi002 ceph-mon[17167]: pgmap v5565: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10305.086439 secs 2022-03-21T17:03:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:25 smithi016 ceph-mon[22010]: pgmap v5566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:25 smithi079 ceph-mon[23158]: pgmap v5566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:25 smithi002 ceph-mon[17167]: pgmap v5566: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:27 smithi016 ceph-mon[22010]: pgmap v5567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:27 smithi079 ceph-mon[23158]: pgmap v5567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:27.768 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:27 smithi002 ceph-mon[17167]: pgmap v5567: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10310.086583 secs 2022-03-21T17:03:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:03:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:03:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:28 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:03:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10310.086583 secs 2022-03-21T17:03:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:03:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:03:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:28 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:03:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10310.086583 secs 2022-03-21T17:03:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:03:28.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:03:28.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:28 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:03:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:29 smithi016 ceph-mon[22010]: pgmap v5568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:29 smithi079 ceph-mon[23158]: pgmap v5568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:29.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:29 smithi002 ceph-mon[17167]: pgmap v5568: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:31.532 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:31 smithi002 ceph-mon[17167]: pgmap v5569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:31.532 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:31.533 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:31 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:31 smithi016 ceph-mon[22010]: pgmap v5569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:31 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:31 smithi079 ceph-mon[23158]: pgmap v5569: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:31 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:32 smithi002 ceph-mon[17167]: pgmap v5570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:32.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:32 smithi016 ceph-mon[22010]: pgmap v5570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:32 smithi079 ceph-mon[23158]: pgmap v5570: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:03:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10315.086697 secs 2022-03-21T17:03:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10315.086697 secs 2022-03-21T17:03:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10315.086697 secs 2022-03-21T17:03:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:34 smithi002 ceph-mon[17167]: pgmap v5571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:34 smithi016 ceph-mon[22010]: pgmap v5571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:34 smithi079 ceph-mon[23158]: pgmap v5571: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:36 smithi079 ceph-mon[23158]: pgmap v5572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:36 smithi002 ceph-mon[17167]: pgmap v5572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:36 smithi016 ceph-mon[22010]: pgmap v5572: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10320.086798 secs 2022-03-21T17:03:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10320.086798 secs 2022-03-21T17:03:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10320.086798 secs 2022-03-21T17:03:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:39 smithi016 ceph-mon[22010]: pgmap v5573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:39 smithi079 ceph-mon[23158]: pgmap v5573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:39 smithi002 ceph-mon[17167]: pgmap v5573: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:41 smithi016 ceph-mon[22010]: pgmap v5574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:41 smithi079 ceph-mon[23158]: pgmap v5574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:41 smithi002 ceph-mon[17167]: pgmap v5574: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:43 smithi016 ceph-mon[22010]: pgmap v5575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10325.086918 secs 2022-03-21T17:03:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:43 smithi079 ceph-mon[23158]: pgmap v5575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10325.086918 secs 2022-03-21T17:03:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:43 smithi002 ceph-mon[17167]: pgmap v5575: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10325.086918 secs 2022-03-21T17:03:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:45 smithi016 ceph-mon[22010]: pgmap v5576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:45 smithi079 ceph-mon[23158]: pgmap v5576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:45 smithi002 ceph-mon[17167]: pgmap v5576: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:47 smithi016 ceph-mon[22010]: pgmap v5577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:47 smithi079 ceph-mon[23158]: pgmap v5577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:47 smithi002 ceph-mon[17167]: pgmap v5577: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10330.087055 secs 2022-03-21T17:03:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10330.087055 secs 2022-03-21T17:03:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10330.087055 secs 2022-03-21T17:03:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:49 smithi016 ceph-mon[22010]: pgmap v5578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:49 smithi079 ceph-mon[23158]: pgmap v5578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:49 smithi002 ceph-mon[17167]: pgmap v5578: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:51 smithi016 ceph-mon[22010]: pgmap v5579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:51 smithi079 ceph-mon[23158]: pgmap v5579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:51 smithi002 ceph-mon[17167]: pgmap v5579: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:53 smithi016 ceph-mon[22010]: pgmap v5580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10335.087155 secs 2022-03-21T17:03:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:53 smithi079 ceph-mon[23158]: pgmap v5580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10335.087155 secs 2022-03-21T17:03:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:53 smithi002 ceph-mon[17167]: pgmap v5580: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10335.087155 secs 2022-03-21T17:03:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:55 smithi016 ceph-mon[22010]: pgmap v5581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:55 smithi079 ceph-mon[23158]: pgmap v5581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:55 smithi002 ceph-mon[17167]: pgmap v5581: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:57 smithi016 ceph-mon[22010]: pgmap v5582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:57 smithi079 ceph-mon[23158]: pgmap v5582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:57 smithi002 ceph-mon[17167]: pgmap v5582: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:03:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10340.087277 secs 2022-03-21T17:03:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10340.087277 secs 2022-03-21T17:03:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10340.087277 secs 2022-03-21T17:03:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:03:59 smithi079 ceph-mon[23158]: pgmap v5583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:03:59 smithi002 ceph-mon[17167]: pgmap v5583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:03:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:03:59 smithi016 ceph-mon[22010]: pgmap v5583: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:01 smithi079 ceph-mon[23158]: pgmap v5584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:01 smithi002 ceph-mon[17167]: pgmap v5584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:01 smithi016 ceph-mon[22010]: pgmap v5584: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:03 smithi079 ceph-mon[23158]: pgmap v5585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10345.087389 secs 2022-03-21T17:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:03 smithi002 ceph-mon[17167]: pgmap v5585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10345.087389 secs 2022-03-21T17:04:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:03 smithi016 ceph-mon[22010]: pgmap v5585: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10345.087389 secs 2022-03-21T17:04:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:05 smithi079 ceph-mon[23158]: pgmap v5586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:05 smithi002 ceph-mon[17167]: pgmap v5586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:05 smithi016 ceph-mon[22010]: pgmap v5586: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:07 smithi079 ceph-mon[23158]: pgmap v5587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:07 smithi002 ceph-mon[17167]: pgmap v5587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:07 smithi016 ceph-mon[22010]: pgmap v5587: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10350.087530 secs 2022-03-21T17:04:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10350.087530 secs 2022-03-21T17:04:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10350.087530 secs 2022-03-21T17:04:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:09 smithi079 ceph-mon[23158]: pgmap v5588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:09 smithi002 ceph-mon[17167]: pgmap v5588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:09 smithi016 ceph-mon[22010]: pgmap v5588: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:10 smithi002 ceph-mon[17167]: pgmap v5589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:10 smithi016 ceph-mon[22010]: pgmap v5589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:10 smithi079 ceph-mon[23158]: pgmap v5589: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:13 smithi016 ceph-mon[22010]: pgmap v5590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10355.087637 secs 2022-03-21T17:04:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:13 smithi079 ceph-mon[23158]: pgmap v5590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10355.087637 secs 2022-03-21T17:04:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:13 smithi002 ceph-mon[17167]: pgmap v5590: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10355.087637 secs 2022-03-21T17:04:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:15 smithi016 ceph-mon[22010]: pgmap v5591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:15 smithi079 ceph-mon[23158]: pgmap v5591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:15 smithi002 ceph-mon[17167]: pgmap v5591: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:17 smithi016 ceph-mon[22010]: pgmap v5592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:17 smithi079 ceph-mon[23158]: pgmap v5592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:17 smithi002 ceph-mon[17167]: pgmap v5592: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10360.087759 secs 2022-03-21T17:04:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10360.087759 secs 2022-03-21T17:04:18.866 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10360.087759 secs 2022-03-21T17:04:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:19 smithi016 ceph-mon[22010]: pgmap v5593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:19 smithi079 ceph-mon[23158]: pgmap v5593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:19 smithi002 ceph-mon[17167]: pgmap v5593: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:21 smithi016 ceph-mon[22010]: pgmap v5594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:04:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:04:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:21 smithi079 ceph-mon[23158]: pgmap v5594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:04:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:04:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:21 smithi002 ceph-mon[17167]: pgmap v5594: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:04:21.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:04:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:23 smithi016 ceph-mon[22010]: pgmap v5595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10365.087840 secs 2022-03-21T17:04:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:23 smithi079 ceph-mon[23158]: pgmap v5595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10365.087840 secs 2022-03-21T17:04:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:23 smithi002 ceph-mon[17167]: pgmap v5595: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10365.087840 secs 2022-03-21T17:04:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:25 smithi016 ceph-mon[22010]: pgmap v5596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:25 smithi079 ceph-mon[23158]: pgmap v5596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:25 smithi002 ceph-mon[17167]: pgmap v5596: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:27 smithi016 ceph-mon[22010]: pgmap v5597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:27 smithi079 ceph-mon[23158]: pgmap v5597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:27 smithi002 ceph-mon[17167]: pgmap v5597: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10370.087910 secs 2022-03-21T17:04:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10370.087910 secs 2022-03-21T17:04:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10370.087910 secs 2022-03-21T17:04:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:29 smithi079 ceph-mon[23158]: pgmap v5598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:29 smithi002 ceph-mon[17167]: pgmap v5598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:29 smithi016 ceph-mon[22010]: pgmap v5598: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:31 smithi079 ceph-mon[23158]: pgmap v5599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:31 smithi002 ceph-mon[17167]: pgmap v5599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:31 smithi016 ceph-mon[22010]: pgmap v5599: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:04:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:04:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:32 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:04:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:04:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:04:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:32 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:04:32.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:04:32.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:04:32.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:32 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:04:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:33 smithi079 ceph-mon[23158]: pgmap v5600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10375.088002 secs 2022-03-21T17:04:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:33 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:33.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:33 smithi002 ceph-mon[17167]: pgmap v5600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:33.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10375.088002 secs 2022-03-21T17:04:33.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:33 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:33 smithi016 ceph-mon[22010]: pgmap v5600: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10375.088002 secs 2022-03-21T17:04:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:33 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:34 smithi002 ceph-mon[17167]: pgmap v5601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:34 smithi016 ceph-mon[22010]: pgmap v5601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:04:35.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:34 smithi079 ceph-mon[23158]: pgmap v5601: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:36 smithi079 ceph-mon[23158]: pgmap v5602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:36 smithi002 ceph-mon[17167]: pgmap v5602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:36 smithi016 ceph-mon[22010]: pgmap v5602: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10380.088141 secs 2022-03-21T17:04:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10380.088141 secs 2022-03-21T17:04:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10380.088141 secs 2022-03-21T17:04:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:39 smithi016 ceph-mon[22010]: pgmap v5603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:39 smithi079 ceph-mon[23158]: pgmap v5603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:39 smithi002 ceph-mon[17167]: pgmap v5603: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:41 smithi016 ceph-mon[22010]: pgmap v5604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:41 smithi079 ceph-mon[23158]: pgmap v5604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:41 smithi002 ceph-mon[17167]: pgmap v5604: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:43 smithi016 ceph-mon[22010]: pgmap v5605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10385.088209 secs 2022-03-21T17:04:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:43 smithi079 ceph-mon[23158]: pgmap v5605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10385.088209 secs 2022-03-21T17:04:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:43 smithi002 ceph-mon[17167]: pgmap v5605: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10385.088209 secs 2022-03-21T17:04:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:45 smithi016 ceph-mon[22010]: pgmap v5606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:45 smithi079 ceph-mon[23158]: pgmap v5606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:45 smithi002 ceph-mon[17167]: pgmap v5606: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:47 smithi016 ceph-mon[22010]: pgmap v5607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:47 smithi079 ceph-mon[23158]: pgmap v5607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:47 smithi002 ceph-mon[17167]: pgmap v5607: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10390.088350 secs 2022-03-21T17:04:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10390.088350 secs 2022-03-21T17:04:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10390.088350 secs 2022-03-21T17:04:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:49 smithi079 ceph-mon[23158]: pgmap v5608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:49 smithi002 ceph-mon[17167]: pgmap v5608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:49 smithi016 ceph-mon[22010]: pgmap v5608: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:51 smithi079 ceph-mon[23158]: pgmap v5609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:51 smithi002 ceph-mon[17167]: pgmap v5609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:51 smithi016 ceph-mon[22010]: pgmap v5609: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:53 smithi079 ceph-mon[23158]: pgmap v5610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10395.088467 secs 2022-03-21T17:04:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:53 smithi002 ceph-mon[17167]: pgmap v5610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10395.088467 secs 2022-03-21T17:04:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:53 smithi016 ceph-mon[22010]: pgmap v5610: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10395.088467 secs 2022-03-21T17:04:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:55 smithi079 ceph-mon[23158]: pgmap v5611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:55 smithi002 ceph-mon[17167]: pgmap v5611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:55 smithi016 ceph-mon[22010]: pgmap v5611: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:57 smithi079 ceph-mon[23158]: pgmap v5612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:57 smithi002 ceph-mon[17167]: pgmap v5612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:57 smithi016 ceph-mon[22010]: pgmap v5612: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:04:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10400.088585 secs 2022-03-21T17:04:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10400.088585 secs 2022-03-21T17:04:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10400.088585 secs 2022-03-21T17:04:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:04:59 smithi079 ceph-mon[23158]: pgmap v5613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:04:59 smithi002 ceph-mon[17167]: pgmap v5613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:04:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:04:59 smithi016 ceph-mon[22010]: pgmap v5613: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:01 smithi079 ceph-mon[23158]: pgmap v5614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:01 smithi002 ceph-mon[17167]: pgmap v5614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:01 smithi016 ceph-mon[22010]: pgmap v5614: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:03 smithi079 ceph-mon[23158]: pgmap v5615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10405.088654 secs 2022-03-21T17:05:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:03 smithi002 ceph-mon[17167]: pgmap v5615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10405.088654 secs 2022-03-21T17:05:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:03 smithi016 ceph-mon[22010]: pgmap v5615: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10405.088654 secs 2022-03-21T17:05:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:05 smithi079 ceph-mon[23158]: pgmap v5616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:05 smithi002 ceph-mon[17167]: pgmap v5616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:05 smithi016 ceph-mon[22010]: pgmap v5616: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:06 smithi079 ceph-mon[23158]: pgmap v5617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:06.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:06 smithi002 ceph-mon[17167]: pgmap v5617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:06 smithi016 ceph-mon[22010]: pgmap v5617: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10410.088722 secs 2022-03-21T17:05:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10410.088722 secs 2022-03-21T17:05:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10410.088722 secs 2022-03-21T17:05:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:09 smithi016 ceph-mon[22010]: pgmap v5618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:09 smithi079 ceph-mon[23158]: pgmap v5618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:09 smithi002 ceph-mon[17167]: pgmap v5618: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:11.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:11 smithi016 ceph-mon[22010]: pgmap v5619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:11 smithi079 ceph-mon[23158]: pgmap v5619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:11.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:11 smithi002 ceph-mon[17167]: pgmap v5619: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:13 smithi079 ceph-mon[23158]: pgmap v5620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10415.088794 secs 2022-03-21T17:05:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:13 smithi002 ceph-mon[17167]: pgmap v5620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10415.088794 secs 2022-03-21T17:05:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:13 smithi016 ceph-mon[22010]: pgmap v5620: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10415.088794 secs 2022-03-21T17:05:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:15 smithi079 ceph-mon[23158]: pgmap v5621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:15 smithi002 ceph-mon[17167]: pgmap v5621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:15 smithi016 ceph-mon[22010]: pgmap v5621: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:17 smithi079 ceph-mon[23158]: pgmap v5622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:17 smithi002 ceph-mon[17167]: pgmap v5622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:17 smithi016 ceph-mon[22010]: pgmap v5622: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10420.088862 secs 2022-03-21T17:05:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10420.088862 secs 2022-03-21T17:05:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10420.088862 secs 2022-03-21T17:05:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:19 smithi079 ceph-mon[23158]: pgmap v5623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:19 smithi002 ceph-mon[17167]: pgmap v5623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:19 smithi016 ceph-mon[22010]: pgmap v5623: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:21 smithi079 ceph-mon[23158]: pgmap v5624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:05:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:05:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:21 smithi002 ceph-mon[17167]: pgmap v5624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:05:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:05:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:21 smithi016 ceph-mon[22010]: pgmap v5624: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:05:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:05:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:23 smithi079 ceph-mon[23158]: pgmap v5625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10425.088982 secs 2022-03-21T17:05:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:23 smithi002 ceph-mon[17167]: pgmap v5625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10425.088982 secs 2022-03-21T17:05:23.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:23 smithi016 ceph-mon[22010]: pgmap v5625: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:23.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10425.088982 secs 2022-03-21T17:05:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:25 smithi079 ceph-mon[23158]: pgmap v5626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:25 smithi002 ceph-mon[17167]: pgmap v5626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:25 smithi016 ceph-mon[22010]: pgmap v5626: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:27 smithi079 ceph-mon[23158]: pgmap v5627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:27.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:27 smithi002 ceph-mon[17167]: pgmap v5627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:27 smithi016 ceph-mon[22010]: pgmap v5627: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10430.089111 secs 2022-03-21T17:05:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10430.089111 secs 2022-03-21T17:05:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10430.089111 secs 2022-03-21T17:05:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:29 smithi079 ceph-mon[23158]: pgmap v5628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:29 smithi002 ceph-mon[17167]: pgmap v5628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:29 smithi016 ceph-mon[22010]: pgmap v5628: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:31 smithi079 ceph-mon[23158]: pgmap v5629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:31 smithi002 ceph-mon[17167]: pgmap v5629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:31 smithi016 ceph-mon[22010]: pgmap v5629: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:33 smithi016 ceph-mon[22010]: pgmap v5630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10435.089243 secs 2022-03-21T17:05:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:33 smithi079 ceph-mon[23158]: pgmap v5630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10435.089243 secs 2022-03-21T17:05:33.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:33 smithi002 ceph-mon[17167]: pgmap v5630: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:33.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10435.089243 secs 2022-03-21T17:05:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:05:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:05:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:05:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:34 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:05:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:05:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:05:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:34 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:05:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:05:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:05:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:34 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:35 smithi002 ceph-mon[17167]: pgmap v5631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:35 smithi079 ceph-mon[23158]: pgmap v5631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:35 smithi016 ceph-mon[22010]: pgmap v5631: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:35.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:35.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:05:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:37 smithi079 ceph-mon[23158]: pgmap v5632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:37.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:37 smithi002 ceph-mon[17167]: pgmap v5632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:37 smithi016 ceph-mon[22010]: pgmap v5632: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10440.089384 secs 2022-03-21T17:05:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10440.089384 secs 2022-03-21T17:05:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10440.089384 secs 2022-03-21T17:05:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:39 smithi079 ceph-mon[23158]: pgmap v5633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:39 smithi002 ceph-mon[17167]: pgmap v5633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:39 smithi016 ceph-mon[22010]: pgmap v5633: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:41 smithi079 ceph-mon[23158]: pgmap v5634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:41 smithi002 ceph-mon[17167]: pgmap v5634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:41 smithi016 ceph-mon[22010]: pgmap v5634: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:43 smithi079 ceph-mon[23158]: pgmap v5635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10445.089507 secs 2022-03-21T17:05:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:43 smithi002 ceph-mon[17167]: pgmap v5635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10445.089507 secs 2022-03-21T17:05:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:43 smithi016 ceph-mon[22010]: pgmap v5635: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10445.089507 secs 2022-03-21T17:05:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:45 smithi079 ceph-mon[23158]: pgmap v5636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:45 smithi002 ceph-mon[17167]: pgmap v5636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:45 smithi016 ceph-mon[22010]: pgmap v5636: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:47 smithi079 ceph-mon[23158]: pgmap v5637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:47 smithi002 ceph-mon[17167]: pgmap v5637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:47 smithi016 ceph-mon[22010]: pgmap v5637: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10450.089637 secs 2022-03-21T17:05:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10450.089637 secs 2022-03-21T17:05:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10450.089637 secs 2022-03-21T17:05:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:49 smithi079 ceph-mon[23158]: pgmap v5638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:49 smithi002 ceph-mon[17167]: pgmap v5638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:49 smithi016 ceph-mon[22010]: pgmap v5638: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:51 smithi079 ceph-mon[23158]: pgmap v5639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:51 smithi002 ceph-mon[17167]: pgmap v5639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:51 smithi016 ceph-mon[22010]: pgmap v5639: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:53 smithi079 ceph-mon[23158]: pgmap v5640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10455.089760 secs 2022-03-21T17:05:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:53 smithi002 ceph-mon[17167]: pgmap v5640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10455.089760 secs 2022-03-21T17:05:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:53 smithi016 ceph-mon[22010]: pgmap v5640: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10455.089760 secs 2022-03-21T17:05:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:55 smithi079 ceph-mon[23158]: pgmap v5641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:55 smithi002 ceph-mon[17167]: pgmap v5641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:55 smithi016 ceph-mon[22010]: pgmap v5641: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:56 smithi079 ceph-mon[23158]: pgmap v5642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:56.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:56 smithi002 ceph-mon[17167]: pgmap v5642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:56 smithi016 ceph-mon[22010]: pgmap v5642: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:05:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10460.089880 secs 2022-03-21T17:05:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10460.089880 secs 2022-03-21T17:05:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10460.089880 secs 2022-03-21T17:05:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:05:59 smithi079 ceph-mon[23158]: pgmap v5643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:05:59 smithi002 ceph-mon[17167]: pgmap v5643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:05:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:05:59 smithi016 ceph-mon[22010]: pgmap v5643: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:01 smithi079 ceph-mon[23158]: pgmap v5644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:01 smithi002 ceph-mon[17167]: pgmap v5644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:01 smithi016 ceph-mon[22010]: pgmap v5644: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:03 smithi079 ceph-mon[23158]: pgmap v5645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10465.089994 secs 2022-03-21T17:06:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:03 smithi002 ceph-mon[17167]: pgmap v5645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:03.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10465.089994 secs 2022-03-21T17:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:03 smithi016 ceph-mon[22010]: pgmap v5645: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10465.089994 secs 2022-03-21T17:06:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:05 smithi079 ceph-mon[23158]: pgmap v5646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:05 smithi002 ceph-mon[17167]: pgmap v5646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:05 smithi016 ceph-mon[22010]: pgmap v5646: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:07 smithi079 ceph-mon[23158]: pgmap v5647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:07 smithi002 ceph-mon[17167]: pgmap v5647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:07 smithi016 ceph-mon[22010]: pgmap v5647: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10470.090059 secs 2022-03-21T17:06:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10470.090059 secs 2022-03-21T17:06:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10470.090059 secs 2022-03-21T17:06:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:09 smithi079 ceph-mon[23158]: pgmap v5648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:09 smithi002 ceph-mon[17167]: pgmap v5648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:09 smithi016 ceph-mon[22010]: pgmap v5648: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:11 smithi079 ceph-mon[23158]: pgmap v5649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:11.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:11 smithi002 ceph-mon[17167]: pgmap v5649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:11 smithi016 ceph-mon[22010]: pgmap v5649: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:13 smithi079 ceph-mon[23158]: pgmap v5650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10475.090191 secs 2022-03-21T17:06:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:13 smithi002 ceph-mon[17167]: pgmap v5650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10475.090191 secs 2022-03-21T17:06:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:13 smithi016 ceph-mon[22010]: pgmap v5650: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10475.090191 secs 2022-03-21T17:06:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:15 smithi079 ceph-mon[23158]: pgmap v5651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:15 smithi002 ceph-mon[17167]: pgmap v5651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:15 smithi016 ceph-mon[22010]: pgmap v5651: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:16.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:16 smithi002 ceph-mon[17167]: pgmap v5652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:16 smithi016 ceph-mon[22010]: pgmap v5652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:16 smithi079 ceph-mon[23158]: pgmap v5652: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10480.090310 secs 2022-03-21T17:06:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10480.090310 secs 2022-03-21T17:06:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10480.090310 secs 2022-03-21T17:06:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:19 smithi079 ceph-mon[23158]: pgmap v5653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:19 smithi002 ceph-mon[17167]: pgmap v5653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:19 smithi016 ceph-mon[22010]: pgmap v5653: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:21 smithi079 ceph-mon[23158]: pgmap v5654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:06:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:06:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:21 smithi002 ceph-mon[17167]: pgmap v5654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:06:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:06:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:21 smithi016 ceph-mon[22010]: pgmap v5654: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:06:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:06:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:23 smithi079 ceph-mon[23158]: pgmap v5655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10485.090412 secs 2022-03-21T17:06:23.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:23 smithi002 ceph-mon[17167]: pgmap v5655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:23.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10485.090412 secs 2022-03-21T17:06:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:23 smithi016 ceph-mon[22010]: pgmap v5655: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10485.090412 secs 2022-03-21T17:06:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:25 smithi079 ceph-mon[23158]: pgmap v5656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:25 smithi002 ceph-mon[17167]: pgmap v5656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:25 smithi016 ceph-mon[22010]: pgmap v5656: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:27 smithi079 ceph-mon[23158]: pgmap v5657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:27.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:27 smithi002 ceph-mon[17167]: pgmap v5657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:27 smithi016 ceph-mon[22010]: pgmap v5657: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10490.090504 secs 2022-03-21T17:06:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10490.090504 secs 2022-03-21T17:06:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10490.090504 secs 2022-03-21T17:06:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:29 smithi079 ceph-mon[23158]: pgmap v5658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:29 smithi002 ceph-mon[17167]: pgmap v5658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:29 smithi016 ceph-mon[22010]: pgmap v5658: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:31 smithi079 ceph-mon[23158]: pgmap v5659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:31 smithi002 ceph-mon[17167]: pgmap v5659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:31 smithi016 ceph-mon[22010]: pgmap v5659: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3 op/s 2022-03-21T17:06:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:33 smithi079 ceph-mon[23158]: pgmap v5660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10495.090634 secs 2022-03-21T17:06:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:33 smithi002 ceph-mon[17167]: pgmap v5660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10495.090634 secs 2022-03-21T17:06:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:33 smithi016 ceph-mon[22010]: pgmap v5660: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10495.090634 secs 2022-03-21T17:06:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:35 smithi002 ceph-mon[17167]: pgmap v5661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:06:35.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:06:35.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:35 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:06:35.648 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:35 smithi079 ceph-mon[23158]: pgmap v5661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:35.649 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:06:35.649 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:06:35.649 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:35 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:06:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:35 smithi016 ceph-mon[22010]: pgmap v5661: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:06:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:06:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:35 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:06:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:37 smithi002 ceph-mon[17167]: pgmap v5662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:06:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:37 smithi016 ceph-mon[22010]: pgmap v5662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:37 smithi079 ceph-mon[23158]: pgmap v5662: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:06:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10500.090755 secs 2022-03-21T17:06:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10500.090755 secs 2022-03-21T17:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10500.090755 secs 2022-03-21T17:06:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:39 smithi079 ceph-mon[23158]: pgmap v5663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:39 smithi002 ceph-mon[17167]: pgmap v5663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:39 smithi016 ceph-mon[22010]: pgmap v5663: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:41 smithi079 ceph-mon[23158]: pgmap v5664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:41.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:41 smithi002 ceph-mon[17167]: pgmap v5664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:41 smithi016 ceph-mon[22010]: pgmap v5664: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:43 smithi079 ceph-mon[23158]: pgmap v5665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10505.090883 secs 2022-03-21T17:06:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:43 smithi002 ceph-mon[17167]: pgmap v5665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10505.090883 secs 2022-03-21T17:06:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:43 smithi016 ceph-mon[22010]: pgmap v5665: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10505.090883 secs 2022-03-21T17:06:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:45 smithi079 ceph-mon[23158]: pgmap v5666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:45 smithi002 ceph-mon[17167]: pgmap v5666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:45 smithi016 ceph-mon[22010]: pgmap v5666: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:47 smithi079 ceph-mon[23158]: pgmap v5667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:47 smithi002 ceph-mon[17167]: pgmap v5667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:47 smithi016 ceph-mon[22010]: pgmap v5667: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10510.090976 secs 2022-03-21T17:06:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10510.090976 secs 2022-03-21T17:06:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10510.090976 secs 2022-03-21T17:06:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:49 smithi079 ceph-mon[23158]: pgmap v5668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:49 smithi002 ceph-mon[17167]: pgmap v5668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:49 smithi016 ceph-mon[22010]: pgmap v5668: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:51 smithi079 ceph-mon[23158]: pgmap v5669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:51 smithi002 ceph-mon[17167]: pgmap v5669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:51 smithi016 ceph-mon[22010]: pgmap v5669: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:53 smithi079 ceph-mon[23158]: pgmap v5670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10515.091099 secs 2022-03-21T17:06:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:53 smithi002 ceph-mon[17167]: pgmap v5670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10515.091099 secs 2022-03-21T17:06:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:53 smithi016 ceph-mon[22010]: pgmap v5670: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10515.091099 secs 2022-03-21T17:06:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:55 smithi002 ceph-mon[17167]: pgmap v5671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:55 smithi016 ceph-mon[22010]: pgmap v5671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:55 smithi079 ceph-mon[23158]: pgmap v5671: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:56.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:56 smithi002 ceph-mon[17167]: pgmap v5672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:56 smithi016 ceph-mon[22010]: pgmap v5672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:56 smithi079 ceph-mon[23158]: pgmap v5672: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:06:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10520.091221 secs 2022-03-21T17:06:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10520.091221 secs 2022-03-21T17:06:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10520.091221 secs 2022-03-21T17:06:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:06:59 smithi079 ceph-mon[23158]: pgmap v5673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:06:59 smithi002 ceph-mon[17167]: pgmap v5673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:06:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:06:59 smithi016 ceph-mon[22010]: pgmap v5673: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:01 smithi079 ceph-mon[23158]: pgmap v5674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:01.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:01 smithi002 ceph-mon[17167]: pgmap v5674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:01 smithi016 ceph-mon[22010]: pgmap v5674: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:03 smithi079 ceph-mon[23158]: pgmap v5675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10525.091297 secs 2022-03-21T17:07:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:03 smithi002 ceph-mon[17167]: pgmap v5675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10525.091297 secs 2022-03-21T17:07:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:03 smithi016 ceph-mon[22010]: pgmap v5675: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10525.091297 secs 2022-03-21T17:07:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:05 smithi079 ceph-mon[23158]: pgmap v5676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:05 smithi002 ceph-mon[17167]: pgmap v5676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:05 smithi016 ceph-mon[22010]: pgmap v5676: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:07 smithi079 ceph-mon[23158]: pgmap v5677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:07 smithi002 ceph-mon[17167]: pgmap v5677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:07 smithi016 ceph-mon[22010]: pgmap v5677: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10530.091415 secs 2022-03-21T17:07:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10530.091415 secs 2022-03-21T17:07:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10530.091415 secs 2022-03-21T17:07:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:09 smithi002 ceph-mon[17167]: pgmap v5678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:09 smithi016 ceph-mon[22010]: pgmap v5678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:09 smithi079 ceph-mon[23158]: pgmap v5678: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:10.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:10 smithi002 ceph-mon[17167]: pgmap v5679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:10 smithi016 ceph-mon[22010]: pgmap v5679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:10 smithi079 ceph-mon[23158]: pgmap v5679: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:13 smithi079 ceph-mon[23158]: pgmap v5680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10535.091563 secs 2022-03-21T17:07:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:13 smithi002 ceph-mon[17167]: pgmap v5680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10535.091563 secs 2022-03-21T17:07:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:13 smithi016 ceph-mon[22010]: pgmap v5680: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10535.091563 secs 2022-03-21T17:07:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:15 smithi079 ceph-mon[23158]: pgmap v5681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:15 smithi002 ceph-mon[17167]: pgmap v5681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:15 smithi016 ceph-mon[22010]: pgmap v5681: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:17 smithi079 ceph-mon[23158]: pgmap v5682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:17 smithi002 ceph-mon[17167]: pgmap v5682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:17 smithi016 ceph-mon[22010]: pgmap v5682: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10540.091706 secs 2022-03-21T17:07:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10540.091706 secs 2022-03-21T17:07:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10540.091706 secs 2022-03-21T17:07:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:19 smithi079 ceph-mon[23158]: pgmap v5683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:19 smithi002 ceph-mon[17167]: pgmap v5683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:19 smithi016 ceph-mon[22010]: pgmap v5683: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:21 smithi079 ceph-mon[23158]: pgmap v5684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:07:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:07:21.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:21 smithi002 ceph-mon[17167]: pgmap v5684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:21.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:07:21.872 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:07:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:21 smithi016 ceph-mon[22010]: pgmap v5684: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:07:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:07:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:23 smithi002 ceph-mon[17167]: pgmap v5685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10545.091792 secs 2022-03-21T17:07:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:23 smithi016 ceph-mon[22010]: pgmap v5685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10545.091792 secs 2022-03-21T17:07:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:23 smithi079 ceph-mon[23158]: pgmap v5685: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10545.091792 secs 2022-03-21T17:07:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:25 smithi002 ceph-mon[17167]: pgmap v5686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:25 smithi016 ceph-mon[22010]: pgmap v5686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:25 smithi079 ceph-mon[23158]: pgmap v5686: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:26 smithi002 ceph-mon[17167]: pgmap v5687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:26 smithi016 ceph-mon[22010]: pgmap v5687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:26 smithi079 ceph-mon[23158]: pgmap v5687: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10550.091919 secs 2022-03-21T17:07:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10550.091919 secs 2022-03-21T17:07:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10550.091919 secs 2022-03-21T17:07:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:29 smithi079 ceph-mon[23158]: pgmap v5688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:29 smithi002 ceph-mon[17167]: pgmap v5688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:29 smithi016 ceph-mon[22010]: pgmap v5688: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:31 smithi079 ceph-mon[23158]: pgmap v5689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:31.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:31 smithi002 ceph-mon[17167]: pgmap v5689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:31 smithi016 ceph-mon[22010]: pgmap v5689: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:33 smithi079 ceph-mon[23158]: pgmap v5690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10555.092042 secs 2022-03-21T17:07:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:33 smithi002 ceph-mon[17167]: pgmap v5690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10555.092042 secs 2022-03-21T17:07:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:33 smithi016 ceph-mon[22010]: pgmap v5690: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10555.092042 secs 2022-03-21T17:07:35.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:35 smithi002 ceph-mon[17167]: pgmap v5691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:35 smithi016 ceph-mon[22010]: pgmap v5691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:35 smithi079 ceph-mon[23158]: pgmap v5691: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:36.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:07:36.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:07:36.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:36 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:07:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:07:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:07:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:36 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:07:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:07:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:07:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:36 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:07:37.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:37 smithi002 ceph-mon[17167]: pgmap v5692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:37.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:07:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:37 smithi016 ceph-mon[22010]: pgmap v5692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:07:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:37 smithi079 ceph-mon[23158]: pgmap v5692: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:07:38.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10560.092146 secs 2022-03-21T17:07:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10560.092146 secs 2022-03-21T17:07:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10560.092146 secs 2022-03-21T17:07:39.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:39 smithi002 ceph-mon[17167]: pgmap v5693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:39 smithi016 ceph-mon[22010]: pgmap v5693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:39 smithi079 ceph-mon[23158]: pgmap v5693: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:40.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:40 smithi002 ceph-mon[17167]: pgmap v5694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:40 smithi016 ceph-mon[22010]: pgmap v5694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:40 smithi079 ceph-mon[23158]: pgmap v5694: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:43 smithi079 ceph-mon[23158]: pgmap v5695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10565.092273 secs 2022-03-21T17:07:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:43 smithi002 ceph-mon[17167]: pgmap v5695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10565.092273 secs 2022-03-21T17:07:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:43 smithi016 ceph-mon[22010]: pgmap v5695: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10565.092273 secs 2022-03-21T17:07:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:45 smithi079 ceph-mon[23158]: pgmap v5696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:45 smithi002 ceph-mon[17167]: pgmap v5696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:45 smithi016 ceph-mon[22010]: pgmap v5696: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:47 smithi002 ceph-mon[17167]: pgmap v5697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:47 smithi016 ceph-mon[22010]: pgmap v5697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:47 smithi079 ceph-mon[23158]: pgmap v5697: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10570.092399 secs 2022-03-21T17:07:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10570.092399 secs 2022-03-21T17:07:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10570.092399 secs 2022-03-21T17:07:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:49 smithi002 ceph-mon[17167]: pgmap v5698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:49 smithi016 ceph-mon[22010]: pgmap v5698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:49 smithi079 ceph-mon[23158]: pgmap v5698: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:51.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:51 smithi002 ceph-mon[17167]: pgmap v5699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:51 smithi016 ceph-mon[22010]: pgmap v5699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:51 smithi079 ceph-mon[23158]: pgmap v5699: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:53 smithi002 ceph-mon[17167]: pgmap v5700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10575.092496 secs 2022-03-21T17:07:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:53 smithi016 ceph-mon[22010]: pgmap v5700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10575.092496 secs 2022-03-21T17:07:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:53 smithi079 ceph-mon[23158]: pgmap v5700: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10575.092496 secs 2022-03-21T17:07:54.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:54 smithi002 ceph-mon[17167]: pgmap v5701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:54 smithi016 ceph-mon[22010]: pgmap v5701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:54 smithi079 ceph-mon[23158]: pgmap v5701: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:57 smithi002 ceph-mon[17167]: pgmap v5702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:57 smithi016 ceph-mon[22010]: pgmap v5702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:57.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:57 smithi079 ceph-mon[23158]: pgmap v5702: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:07:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10580.092624 secs 2022-03-21T17:07:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10580.092624 secs 2022-03-21T17:07:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10580.092624 secs 2022-03-21T17:07:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:07:59 smithi002 ceph-mon[17167]: pgmap v5703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:07:59 smithi016 ceph-mon[22010]: pgmap v5703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:07:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:07:59 smithi079 ceph-mon[23158]: pgmap v5703: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:01.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:01 smithi002 ceph-mon[17167]: pgmap v5704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:01 smithi016 ceph-mon[22010]: pgmap v5704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:01 smithi079 ceph-mon[23158]: pgmap v5704: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:03 smithi002 ceph-mon[17167]: pgmap v5705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:03.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10585.092692 secs 2022-03-21T17:08:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:03 smithi016 ceph-mon[22010]: pgmap v5705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10585.092692 secs 2022-03-21T17:08:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:03 smithi079 ceph-mon[23158]: pgmap v5705: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10585.092692 secs 2022-03-21T17:08:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:05 smithi002 ceph-mon[17167]: pgmap v5706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:05 smithi016 ceph-mon[22010]: pgmap v5706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:05 smithi079 ceph-mon[23158]: pgmap v5706: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:06.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:06 smithi002 ceph-mon[17167]: pgmap v5707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:06 smithi016 ceph-mon[22010]: pgmap v5707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:06 smithi079 ceph-mon[23158]: pgmap v5707: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10590.092791 secs 2022-03-21T17:08:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10590.092791 secs 2022-03-21T17:08:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10590.092791 secs 2022-03-21T17:08:09.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:09 smithi002 ceph-mon[17167]: pgmap v5708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:09 smithi016 ceph-mon[22010]: pgmap v5708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:09 smithi079 ceph-mon[23158]: pgmap v5708: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:11.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:11 smithi002 ceph-mon[17167]: pgmap v5709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:11 smithi016 ceph-mon[22010]: pgmap v5709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:11 smithi079 ceph-mon[23158]: pgmap v5709: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:13 smithi002 ceph-mon[17167]: pgmap v5710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10595.092894 secs 2022-03-21T17:08:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:13 smithi016 ceph-mon[22010]: pgmap v5710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10595.092894 secs 2022-03-21T17:08:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:13 smithi079 ceph-mon[23158]: pgmap v5710: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10595.092894 secs 2022-03-21T17:08:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:15 smithi002 ceph-mon[17167]: pgmap v5711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:15 smithi016 ceph-mon[22010]: pgmap v5711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:15 smithi079 ceph-mon[23158]: pgmap v5711: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:16.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:16 smithi002 ceph-mon[17167]: pgmap v5712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:16 smithi016 ceph-mon[22010]: pgmap v5712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:16 smithi079 ceph-mon[23158]: pgmap v5712: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10600.093022 secs 2022-03-21T17:08:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10600.093022 secs 2022-03-21T17:08:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10600.093022 secs 2022-03-21T17:08:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:19 smithi002 ceph-mon[17167]: pgmap v5713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:19 smithi016 ceph-mon[22010]: pgmap v5713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:19 smithi079 ceph-mon[23158]: pgmap v5713: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:21 smithi002 ceph-mon[17167]: pgmap v5714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:08:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:08:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:21 smithi016 ceph-mon[22010]: pgmap v5714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:08:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:08:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:21 smithi079 ceph-mon[23158]: pgmap v5714: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:08:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:08:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:23 smithi002 ceph-mon[17167]: pgmap v5715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10605.093150 secs 2022-03-21T17:08:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:23 smithi016 ceph-mon[22010]: pgmap v5715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10605.093150 secs 2022-03-21T17:08:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:23 smithi079 ceph-mon[23158]: pgmap v5715: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10605.093150 secs 2022-03-21T17:08:24.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:24 smithi002 ceph-mon[17167]: pgmap v5716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:24 smithi016 ceph-mon[22010]: pgmap v5716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:24 smithi079 ceph-mon[23158]: pgmap v5716: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:27 smithi002 ceph-mon[17167]: pgmap v5717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:27 smithi016 ceph-mon[22010]: pgmap v5717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:27 smithi079 ceph-mon[23158]: pgmap v5717: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10610.093252 secs 2022-03-21T17:08:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10610.093252 secs 2022-03-21T17:08:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10610.093252 secs 2022-03-21T17:08:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:29 smithi002 ceph-mon[17167]: pgmap v5718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:29 smithi016 ceph-mon[22010]: pgmap v5718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:29 smithi079 ceph-mon[23158]: pgmap v5718: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:31.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:31 smithi002 ceph-mon[17167]: pgmap v5719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:31 smithi016 ceph-mon[22010]: pgmap v5719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:31 smithi079 ceph-mon[23158]: pgmap v5719: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:33 smithi002 ceph-mon[17167]: pgmap v5720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10615.093384 secs 2022-03-21T17:08:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:33 smithi016 ceph-mon[22010]: pgmap v5720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10615.093384 secs 2022-03-21T17:08:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:33 smithi079 ceph-mon[23158]: pgmap v5720: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10615.093384 secs 2022-03-21T17:08:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:34 smithi002 ceph-mon[17167]: pgmap v5721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:34 smithi016 ceph-mon[22010]: pgmap v5721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:34 smithi079 ceph-mon[23158]: pgmap v5721: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:37 smithi002 ceph-mon[17167]: pgmap v5722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:08:37.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:08:37.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:37 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:08:37.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:37 smithi079 ceph-mon[23158]: pgmap v5722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:37.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:08:37.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:08:37.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:37 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:08:37.405 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:37 smithi016 ceph-mon[22010]: pgmap v5722: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:37.405 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:08:37.405 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:08:37.406 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:37 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:08:39.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10620.093536 secs 2022-03-21T17:08:39.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:08:39.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:38 smithi002 ceph-mon[17167]: pgmap v5723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10620.093536 secs 2022-03-21T17:08:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:08:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:38 smithi016 ceph-mon[22010]: pgmap v5723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10620.093536 secs 2022-03-21T17:08:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:08:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:38 smithi079 ceph-mon[23158]: pgmap v5723: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:41 smithi002 ceph-mon[17167]: pgmap v5724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:41 smithi016 ceph-mon[22010]: pgmap v5724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:41 smithi079 ceph-mon[23158]: pgmap v5724: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:43 smithi002 ceph-mon[17167]: pgmap v5725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10625.093659 secs 2022-03-21T17:08:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:43 smithi016 ceph-mon[22010]: pgmap v5725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10625.093659 secs 2022-03-21T17:08:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:43 smithi079 ceph-mon[23158]: pgmap v5725: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10625.093659 secs 2022-03-21T17:08:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:45 smithi002 ceph-mon[17167]: pgmap v5726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:45 smithi016 ceph-mon[22010]: pgmap v5726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:45 smithi079 ceph-mon[23158]: pgmap v5726: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:47.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:47 smithi002 ceph-mon[17167]: pgmap v5727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:47 smithi016 ceph-mon[22010]: pgmap v5727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:47 smithi079 ceph-mon[23158]: pgmap v5727: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10630.093760 secs 2022-03-21T17:08:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10630.093760 secs 2022-03-21T17:08:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10630.093760 secs 2022-03-21T17:08:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:49 smithi002 ceph-mon[17167]: pgmap v5728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:49 smithi016 ceph-mon[22010]: pgmap v5728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:49 smithi079 ceph-mon[23158]: pgmap v5728: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:50.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:50 smithi002 ceph-mon[17167]: pgmap v5729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:50 smithi016 ceph-mon[22010]: pgmap v5729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:50 smithi079 ceph-mon[23158]: pgmap v5729: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:53 smithi002 ceph-mon[17167]: pgmap v5730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:53.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10635.093858 secs 2022-03-21T17:08:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:53 smithi016 ceph-mon[22010]: pgmap v5730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10635.093858 secs 2022-03-21T17:08:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:53 smithi079 ceph-mon[23158]: pgmap v5730: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10635.093858 secs 2022-03-21T17:08:55.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:55 smithi002 ceph-mon[17167]: pgmap v5731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:55 smithi016 ceph-mon[22010]: pgmap v5731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:55 smithi079 ceph-mon[23158]: pgmap v5731: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:57.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:57 smithi002 ceph-mon[17167]: pgmap v5732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:57 smithi016 ceph-mon[22010]: pgmap v5732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:57 smithi079 ceph-mon[23158]: pgmap v5732: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:08:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10640.093982 secs 2022-03-21T17:08:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10640.093982 secs 2022-03-21T17:08:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10640.093982 secs 2022-03-21T17:08:59.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:08:59 smithi002 ceph-mon[17167]: pgmap v5733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:08:59 smithi016 ceph-mon[22010]: pgmap v5733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:08:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:08:59 smithi079 ceph-mon[23158]: pgmap v5733: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:00.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:00 smithi002 ceph-mon[17167]: pgmap v5734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:00 smithi016 ceph-mon[22010]: pgmap v5734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:00 smithi079 ceph-mon[23158]: pgmap v5734: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:03.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:03 smithi002 ceph-mon[17167]: pgmap v5735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:03.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10645.094120 secs 2022-03-21T17:09:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:03 smithi016 ceph-mon[22010]: pgmap v5735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10645.094120 secs 2022-03-21T17:09:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:03 smithi079 ceph-mon[23158]: pgmap v5735: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10645.094120 secs 2022-03-21T17:09:05.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:05 smithi002 ceph-mon[17167]: pgmap v5736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:05 smithi016 ceph-mon[22010]: pgmap v5736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:05 smithi079 ceph-mon[23158]: pgmap v5736: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:07.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:07 smithi002 ceph-mon[17167]: pgmap v5737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:07 smithi016 ceph-mon[22010]: pgmap v5737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:07 smithi079 ceph-mon[23158]: pgmap v5737: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:08.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10650.094265 secs 2022-03-21T17:09:08.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:08 smithi002 ceph-mon[17167]: pgmap v5738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10650.094265 secs 2022-03-21T17:09:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:08 smithi016 ceph-mon[22010]: pgmap v5738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10650.094265 secs 2022-03-21T17:09:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:08 smithi079 ceph-mon[23158]: pgmap v5738: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:11 smithi002 ceph-mon[17167]: pgmap v5739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:11 smithi016 ceph-mon[22010]: pgmap v5739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:11 smithi079 ceph-mon[23158]: pgmap v5739: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:13 smithi002 ceph-mon[17167]: pgmap v5740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:13.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10655.094391 secs 2022-03-21T17:09:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:13 smithi016 ceph-mon[22010]: pgmap v5740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10655.094391 secs 2022-03-21T17:09:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:13 smithi079 ceph-mon[23158]: pgmap v5740: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10655.094391 secs 2022-03-21T17:09:15.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:15 smithi002 ceph-mon[17167]: pgmap v5741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:15 smithi016 ceph-mon[22010]: pgmap v5741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:15 smithi079 ceph-mon[23158]: pgmap v5741: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:16.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:16 smithi002 ceph-mon[17167]: pgmap v5742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:16 smithi016 ceph-mon[22010]: pgmap v5742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:16 smithi079 ceph-mon[23158]: pgmap v5742: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10660.094568 secs 2022-03-21T17:09:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10660.094568 secs 2022-03-21T17:09:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10660.094568 secs 2022-03-21T17:09:19.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:19 smithi002 ceph-mon[17167]: pgmap v5743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:19 smithi016 ceph-mon[22010]: pgmap v5743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:19 smithi079 ceph-mon[23158]: pgmap v5743: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:20.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:20 smithi002 ceph-mon[17167]: pgmap v5744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:20 smithi016 ceph-mon[22010]: pgmap v5744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:20 smithi079 ceph-mon[23158]: pgmap v5744: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:09:21.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:21 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:09:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:09:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:21 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:09:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:09:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:21 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:09:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:23 smithi002 ceph-mon[17167]: pgmap v5745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:23.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10665.094639 secs 2022-03-21T17:09:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:23 smithi016 ceph-mon[22010]: pgmap v5745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10665.094639 secs 2022-03-21T17:09:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:23 smithi079 ceph-mon[23158]: pgmap v5745: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10665.094639 secs 2022-03-21T17:09:25.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:25 smithi002 ceph-mon[17167]: pgmap v5746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:25 smithi016 ceph-mon[22010]: pgmap v5746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:25 smithi079 ceph-mon[23158]: pgmap v5746: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:26.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:26 smithi002 ceph-mon[17167]: pgmap v5747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:26 smithi016 ceph-mon[22010]: pgmap v5747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:26 smithi079 ceph-mon[23158]: pgmap v5747: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10670.094762 secs 2022-03-21T17:09:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10670.094762 secs 2022-03-21T17:09:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10670.094762 secs 2022-03-21T17:09:29.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:29 smithi002 ceph-mon[17167]: pgmap v5748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:29 smithi016 ceph-mon[22010]: pgmap v5748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:29 smithi079 ceph-mon[23158]: pgmap v5748: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:30.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:30 smithi002 ceph-mon[17167]: pgmap v5749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:30 smithi016 ceph-mon[22010]: pgmap v5749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:30 smithi079 ceph-mon[23158]: pgmap v5749: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:33 smithi002 ceph-mon[17167]: pgmap v5750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:33.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10675.094868 secs 2022-03-21T17:09:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:33 smithi016 ceph-mon[22010]: pgmap v5750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10675.094868 secs 2022-03-21T17:09:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:33 smithi079 ceph-mon[23158]: pgmap v5750: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10675.094868 secs 2022-03-21T17:09:34.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:34 smithi002 ceph-mon[17167]: pgmap v5751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:34 smithi016 ceph-mon[22010]: pgmap v5751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:34 smithi079 ceph-mon[23158]: pgmap v5751: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:37 smithi002 ceph-mon[17167]: pgmap v5752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:37 smithi016 ceph-mon[22010]: pgmap v5752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:37 smithi079 ceph-mon[23158]: pgmap v5752: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10680.094994 secs 2022-03-21T17:09:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:09:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:09:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:38 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:09:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10680.094994 secs 2022-03-21T17:09:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:09:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:09:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:38 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:09:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10680.094994 secs 2022-03-21T17:09:38.797 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:09:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:09:38.798 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:38 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:09:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:39 smithi016 ceph-mon[22010]: pgmap v5753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:39.832 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:39 smithi002 ceph-mon[17167]: pgmap v5753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:39 smithi079 ceph-mon[23158]: pgmap v5753: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:40.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:40 smithi002 ceph-mon[17167]: pgmap v5754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:40.893 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:40 smithi016 ceph-mon[22010]: pgmap v5754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:40 smithi079 ceph-mon[23158]: pgmap v5754: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:41 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T17:09:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:41 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:42.338 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:41 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi016 2022-03-21T17:09:42.338 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:41 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:41 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T17:09:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:41 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:43 smithi002 ceph-mon[17167]: pgmap v5755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:43.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10685.095117 secs 2022-03-21T17:09:43.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:43 smithi016 ceph-mon[22010]: pgmap v5755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10685.095117 secs 2022-03-21T17:09:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:43 smithi079 ceph-mon[23158]: pgmap v5755: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10685.095117 secs 2022-03-21T17:09:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:09:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:44 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi002 2022-03-21T17:09:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:44 smithi002 ceph-mon[17167]: pgmap v5756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:44 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi002 2022-03-21T17:09:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:44 smithi016 ceph-mon[22010]: pgmap v5756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:44 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi002 2022-03-21T17:09:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:44 smithi079 ceph-mon[23158]: pgmap v5756: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:47 smithi002 ceph-mon[17167]: pgmap v5757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:47 smithi016 ceph-mon[22010]: pgmap v5757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:47 smithi079 ceph-mon[23158]: pgmap v5757: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10690.095254 secs 2022-03-21T17:09:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10690.095254 secs 2022-03-21T17:09:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10690.095254 secs 2022-03-21T17:09:49.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:49 smithi002 ceph-mon[17167]: pgmap v5758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:49 smithi016 ceph-mon[22010]: pgmap v5758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:49 smithi079 ceph-mon[23158]: pgmap v5758: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:50.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:50 smithi002 ceph-mon[17167]: pgmap v5759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:50 smithi016 ceph-mon[22010]: pgmap v5759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:50 smithi079 ceph-mon[23158]: pgmap v5759: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:53 smithi002 ceph-mon[17167]: pgmap v5760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10695.095372 secs 2022-03-21T17:09:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:53 smithi016 ceph-mon[22010]: pgmap v5760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10695.095372 secs 2022-03-21T17:09:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:53 smithi079 ceph-mon[23158]: pgmap v5760: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10695.095372 secs 2022-03-21T17:09:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:55 smithi002 ceph-mon[17167]: pgmap v5761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:55 smithi016 ceph-mon[22010]: pgmap v5761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:55 smithi079 ceph-mon[23158]: pgmap v5761: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:57 smithi002 ceph-mon[17167]: pgmap v5762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:57 smithi016 ceph-mon[22010]: pgmap v5762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:57 smithi079 ceph-mon[23158]: pgmap v5762: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:09:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10700.095499 secs 2022-03-21T17:09:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:09:58 smithi002 ceph-mon[17167]: pgmap v5763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10700.095499 secs 2022-03-21T17:09:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:09:58 smithi016 ceph-mon[22010]: pgmap v5763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:09:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10700.095499 secs 2022-03-21T17:09:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:09:58 smithi079 ceph-mon[23158]: pgmap v5763: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:00.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:00 smithi002 ceph-mon[17167]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:10:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:00 smithi016 ceph-mon[22010]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:10:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:00 smithi079 ceph-mon[23158]: overall HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests 2022-03-21T17:10:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:01 smithi002 ceph-mon[17167]: pgmap v5764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:01 smithi016 ceph-mon[22010]: pgmap v5764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:01 smithi079 ceph-mon[23158]: pgmap v5764: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:03 smithi002 ceph-mon[17167]: pgmap v5765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10705.095619 secs 2022-03-21T17:10:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:03 smithi016 ceph-mon[22010]: pgmap v5765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10705.095619 secs 2022-03-21T17:10:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:03 smithi079 ceph-mon[23158]: pgmap v5765: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10705.095619 secs 2022-03-21T17:10:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:05 smithi002 ceph-mon[17167]: pgmap v5766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:05 smithi016 ceph-mon[22010]: pgmap v5766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:05 smithi079 ceph-mon[23158]: pgmap v5766: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:07 smithi002 ceph-mon[17167]: pgmap v5767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:07 smithi016 ceph-mon[22010]: pgmap v5767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:07 smithi079 ceph-mon[23158]: pgmap v5767: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10710.095713 secs 2022-03-21T17:10:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:08 smithi002 ceph-mon[17167]: pgmap v5768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10710.095713 secs 2022-03-21T17:10:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:08 smithi016 ceph-mon[22010]: pgmap v5768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10710.095713 secs 2022-03-21T17:10:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:08 smithi079 ceph-mon[23158]: pgmap v5768: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:11 smithi002 ceph-mon[17167]: pgmap v5769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:11 smithi016 ceph-mon[22010]: pgmap v5769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:11 smithi079 ceph-mon[23158]: pgmap v5769: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:13 smithi002 ceph-mon[17167]: pgmap v5770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10715.095819 secs 2022-03-21T17:10:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:13 smithi016 ceph-mon[22010]: pgmap v5770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10715.095819 secs 2022-03-21T17:10:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:13 smithi079 ceph-mon[23158]: pgmap v5770: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10715.095819 secs 2022-03-21T17:10:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:15 smithi002 ceph-mon[17167]: pgmap v5771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:15 smithi016 ceph-mon[22010]: pgmap v5771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:15 smithi079 ceph-mon[23158]: pgmap v5771: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:17 smithi002 ceph-mon[17167]: pgmap v5772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:17 smithi016 ceph-mon[22010]: pgmap v5772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:17 smithi079 ceph-mon[23158]: pgmap v5772: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10720.095930 secs 2022-03-21T17:10:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:18 smithi002 ceph-mon[17167]: pgmap v5773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10720.095930 secs 2022-03-21T17:10:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:18 smithi016 ceph-mon[22010]: pgmap v5773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10720.095930 secs 2022-03-21T17:10:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:18 smithi079 ceph-mon[23158]: pgmap v5773: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:21 smithi002 ceph-mon[17167]: pgmap v5774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:21 smithi016 ceph-mon[22010]: pgmap v5774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:21 smithi079 ceph-mon[23158]: pgmap v5774: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:10:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:10:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:10:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:10:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:10:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:10:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:23 smithi002 ceph-mon[17167]: pgmap v5775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10725.096044 secs 2022-03-21T17:10:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:23 smithi016 ceph-mon[22010]: pgmap v5775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10725.096044 secs 2022-03-21T17:10:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:23 smithi079 ceph-mon[23158]: pgmap v5775: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10725.096044 secs 2022-03-21T17:10:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:25 smithi002 ceph-mon[17167]: pgmap v5776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:25 smithi016 ceph-mon[22010]: pgmap v5776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:25 smithi079 ceph-mon[23158]: pgmap v5776: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:27 smithi002 ceph-mon[17167]: pgmap v5777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:27 smithi016 ceph-mon[22010]: pgmap v5777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:27 smithi079 ceph-mon[23158]: pgmap v5777: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10730.096129 secs 2022-03-21T17:10:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:28 smithi002 ceph-mon[17167]: pgmap v5778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10730.096129 secs 2022-03-21T17:10:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:28 smithi016 ceph-mon[22010]: pgmap v5778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10730.096129 secs 2022-03-21T17:10:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:28 smithi079 ceph-mon[23158]: pgmap v5778: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:31 smithi002 ceph-mon[17167]: pgmap v5779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:31 smithi016 ceph-mon[22010]: pgmap v5779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:31 smithi079 ceph-mon[23158]: pgmap v5779: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:33 smithi002 ceph-mon[17167]: pgmap v5780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10735.096268 secs 2022-03-21T17:10:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:33 smithi016 ceph-mon[22010]: pgmap v5780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10735.096268 secs 2022-03-21T17:10:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:33 smithi079 ceph-mon[23158]: pgmap v5780: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10735.096268 secs 2022-03-21T17:10:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:35 smithi002 ceph-mon[17167]: pgmap v5781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:35 smithi016 ceph-mon[22010]: pgmap v5781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:35 smithi079 ceph-mon[23158]: pgmap v5781: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:37.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:37 smithi002 ceph-mon[17167]: pgmap v5782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:37 smithi016 ceph-mon[22010]: pgmap v5782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:37 smithi079 ceph-mon[23158]: pgmap v5782: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10740.096389 secs 2022-03-21T17:10:38.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:38 smithi002 ceph-mon[17167]: pgmap v5783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10740.096389 secs 2022-03-21T17:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:38 smithi016 ceph-mon[22010]: pgmap v5783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10740.096389 secs 2022-03-21T17:10:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:38 smithi079 ceph-mon[23158]: pgmap v5783: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:41.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:41 smithi002 ceph-mon[17167]: pgmap v5784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:41 smithi016 ceph-mon[22010]: pgmap v5784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:41 smithi079 ceph-mon[23158]: pgmap v5784: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:43.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:43 smithi002 ceph-mon[17167]: pgmap v5785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:43.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10745.096502 secs 2022-03-21T17:10:43.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:10:43.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:10:43.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:43 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:10:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:43 smithi016 ceph-mon[22010]: pgmap v5785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10745.096502 secs 2022-03-21T17:10:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:10:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:10:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:43 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:10:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:43 smithi079 ceph-mon[23158]: pgmap v5785: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10745.096502 secs 2022-03-21T17:10:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:10:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:10:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:43 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:10:44.207 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.208 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.208 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.208 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.208 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:10:44.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:10:44.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.210 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:44 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:10:44.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:10:44.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:44 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:10:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:10:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:44 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:45 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:10:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:45 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:10:45.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:45 smithi002 ceph-mon[17167]: pgmap v5786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:45 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:10:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:45 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:10:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:45 smithi016 ceph-mon[22010]: pgmap v5786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:45.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:45 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi002 to 223.3M 2022-03-21T17:10:45.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:45 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi002 to 234232320: error parsing value: Value '234232320' is below minimum 939524096 2022-03-21T17:10:45.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:45 smithi079 ceph-mon[23158]: pgmap v5786: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: Detected new or changed devices on smithi079 2022-03-21T17:10:47.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:10:47.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:10:47.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:46 smithi002 ceph-mon[17167]: pgmap v5787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T17:10:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:10:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:10:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:46 smithi016 ceph-mon[22010]: pgmap v5787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T17:10:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:10:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:10:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:10:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:10:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:46 smithi079 ceph-mon[23158]: pgmap v5787: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10750.096628 secs 2022-03-21T17:10:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:48 smithi002 ceph-mon[17167]: pgmap v5788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10750.096628 secs 2022-03-21T17:10:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:48 smithi016 ceph-mon[22010]: pgmap v5788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10750.096628 secs 2022-03-21T17:10:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:48 smithi079 ceph-mon[23158]: pgmap v5788: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:51 smithi002 ceph-mon[17167]: pgmap v5789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:51 smithi016 ceph-mon[22010]: pgmap v5789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:51 smithi079 ceph-mon[23158]: pgmap v5789: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:53 smithi002 ceph-mon[17167]: pgmap v5790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10755.096729 secs 2022-03-21T17:10:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:53 smithi016 ceph-mon[22010]: pgmap v5790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10755.096729 secs 2022-03-21T17:10:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:53 smithi079 ceph-mon[23158]: pgmap v5790: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10755.096729 secs 2022-03-21T17:10:55.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:55 smithi002 ceph-mon[17167]: pgmap v5791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:55 smithi016 ceph-mon[22010]: pgmap v5791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:55 smithi079 ceph-mon[23158]: pgmap v5791: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:57 smithi002 ceph-mon[17167]: pgmap v5792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:57 smithi016 ceph-mon[22010]: pgmap v5792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:57 smithi079 ceph-mon[23158]: pgmap v5792: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:10:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10760.096816 secs 2022-03-21T17:10:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:10:58 smithi002 ceph-mon[17167]: pgmap v5793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10760.096816 secs 2022-03-21T17:10:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:10:58 smithi016 ceph-mon[22010]: pgmap v5793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:10:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10760.096816 secs 2022-03-21T17:10:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:10:58 smithi079 ceph-mon[23158]: pgmap v5793: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:01 smithi002 ceph-mon[17167]: pgmap v5794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:01 smithi016 ceph-mon[22010]: pgmap v5794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:01 smithi079 ceph-mon[23158]: pgmap v5794: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:03 smithi002 ceph-mon[17167]: pgmap v5795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:03.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:03 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10765.096885 secs 2022-03-21T17:11:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:03 smithi016 ceph-mon[22010]: pgmap v5795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10765.096885 secs 2022-03-21T17:11:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:03 smithi079 ceph-mon[23158]: pgmap v5795: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10765.096885 secs 2022-03-21T17:11:05.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:05 smithi002 ceph-mon[17167]: pgmap v5796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:05 smithi016 ceph-mon[22010]: pgmap v5796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:05 smithi079 ceph-mon[23158]: pgmap v5796: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:07.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:07 smithi002 ceph-mon[17167]: pgmap v5797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:07 smithi016 ceph-mon[22010]: pgmap v5797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:07 smithi079 ceph-mon[23158]: pgmap v5797: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:08 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10770.096986 secs 2022-03-21T17:11:08.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:08 smithi002 ceph-mon[17167]: pgmap v5798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10770.096986 secs 2022-03-21T17:11:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:08 smithi016 ceph-mon[22010]: pgmap v5798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10770.096986 secs 2022-03-21T17:11:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:08 smithi079 ceph-mon[23158]: pgmap v5798: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:11.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:11 smithi002 ceph-mon[17167]: pgmap v5799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:11 smithi016 ceph-mon[22010]: pgmap v5799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:11 smithi079 ceph-mon[23158]: pgmap v5799: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:13.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:13 smithi002 ceph-mon[17167]: pgmap v5800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:13.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:13 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10775.097100 secs 2022-03-21T17:11:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:13 smithi016 ceph-mon[22010]: pgmap v5800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10775.097100 secs 2022-03-21T17:11:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:13 smithi079 ceph-mon[23158]: pgmap v5800: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10775.097100 secs 2022-03-21T17:11:15.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:15 smithi002 ceph-mon[17167]: pgmap v5801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:15 smithi016 ceph-mon[22010]: pgmap v5801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:15 smithi079 ceph-mon[23158]: pgmap v5801: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:17.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:17 smithi002 ceph-mon[17167]: pgmap v5802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:17 smithi016 ceph-mon[22010]: pgmap v5802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:17 smithi079 ceph-mon[23158]: pgmap v5802: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:18 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10780.097242 secs 2022-03-21T17:11:18.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:18 smithi002 ceph-mon[17167]: pgmap v5803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10780.097242 secs 2022-03-21T17:11:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:18 smithi016 ceph-mon[22010]: pgmap v5803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10780.097242 secs 2022-03-21T17:11:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:18 smithi079 ceph-mon[23158]: pgmap v5803: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:21 smithi002 ceph-mon[17167]: pgmap v5804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:21 smithi016 ceph-mon[22010]: pgmap v5804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:21 smithi079 ceph-mon[23158]: pgmap v5804: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:11:22.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:22 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:11:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:11:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:22 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:11:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:11:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:22 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-03-21T17:11:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:23 smithi002 ceph-mon[17167]: pgmap v5805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:23.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:23 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10785.097363 secs 2022-03-21T17:11:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:23 smithi016 ceph-mon[22010]: pgmap v5805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10785.097363 secs 2022-03-21T17:11:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:23 smithi079 ceph-mon[23158]: pgmap v5805: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10785.097363 secs 2022-03-21T17:11:25.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:25 smithi002 ceph-mon[17167]: pgmap v5806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:25 smithi016 ceph-mon[22010]: pgmap v5806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:25 smithi079 ceph-mon[23158]: pgmap v5806: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:27.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:27 smithi002 ceph-mon[17167]: pgmap v5807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:27 smithi016 ceph-mon[22010]: pgmap v5807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:27 smithi079 ceph-mon[23158]: pgmap v5807: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:28 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10790.097509 secs 2022-03-21T17:11:28.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:28 smithi002 ceph-mon[17167]: pgmap v5808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10790.097509 secs 2022-03-21T17:11:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:28 smithi016 ceph-mon[22010]: pgmap v5808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10790.097509 secs 2022-03-21T17:11:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:28 smithi079 ceph-mon[23158]: pgmap v5808: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 124 2022-03-21T17:11:31.058 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2022-03-21T17:11:31.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-03-21T17:11:31.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:31 smithi002 ceph-mon[17167]: pgmap v5809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:31 smithi016 ceph-mon[22010]: pgmap v5809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:31 smithi079 ceph-mon[23158]: pgmap v5809: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:33.124 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/dirfrag.sh) on smithi002 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh' 2022-03-21T17:11:33.228 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ac3c43b14370419cbdb577d900233dd5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d3799b8cfa5046465710eadeea2bed7bb4681409/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/dirfrag.sh) on smithi002 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d3799b8cfa5046465710eadeea2bed7bb4681409 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh' 2022-03-21T17:11:33.231 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2022-03-21T17:11:33.240 DEBUG:tasks.check_counter:Getting stats from a 2022-03-21T17:11:33.243 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.a perf dump 2022-03-21T17:11:33.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:33 smithi002 ceph-mon[17167]: pgmap v5810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:33.369 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:33 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10795.097628 secs 2022-03-21T17:11:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:33 smithi016 ceph-mon[22010]: pgmap v5810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10795.097628 secs 2022-03-21T17:11:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:33 smithi079 ceph-mon[23158]: pgmap v5810: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10795.097628 secs 2022-03-21T17:11:33.915 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T17:11:35.060 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T17:11:35.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:35 smithi002 ceph-mon[17167]: pgmap v5811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:35 smithi016 ceph-mon[22010]: pgmap v5811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:35 smithi079 ceph-mon[23158]: pgmap v5811: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:35.603 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T17:11:35.604 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:35.604 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:35.605 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:35.605 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:35.605 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:35.606 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:35.606 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:35.606 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.004543629, 2022-03-21T17:11:35.607 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:35.607 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000899433, 2022-03-21T17:11:35.608 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:35.608 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:35.609 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:35.609 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:35.609 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:35.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.610 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:35.610 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:35.611 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:35.611 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:35.612 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:35.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.612 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:35.613 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2920, 2022-03-21T17:11:35.613 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2917, 2022-03-21T17:11:35.613 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 970336, 2022-03-21T17:11:35.614 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1682336, 2022-03-21T17:11:35.614 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:35.614 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:35.615 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.555831388, 2022-03-21T17:11:35.615 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.251011521, 2022-03-21T17:11:35.615 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.127848439, 2022-03-21T17:11:35.616 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.083017082, 2022-03-21T17:11:35.616 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:35.616 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2917, 2022-03-21T17:11:35.617 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.238816791, 2022-03-21T17:11:35.617 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000081870 2022-03-21T17:11:35.617 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.618 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:35.618 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:35.618 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:35.619 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:35.619 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:35.619 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.620 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:35.620 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 12, 2022-03-21T17:11:35.621 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2238, 2022-03-21T17:11:35.621 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1920, 2022-03-21T17:11:35.621 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1260032, 2022-03-21T17:11:35.622 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:35.622 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:35.622 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.295448954, 2022-03-21T17:11:35.623 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.237275634, 2022-03-21T17:11:35.623 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.011173534, 2022-03-21T17:11:35.623 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:35.624 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:35.624 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2236, 2022-03-21T17:11:35.624 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.202813261, 2022-03-21T17:11:35.625 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000090703 2022-03-21T17:11:35.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.625 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:35.626 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:35.626 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:35.626 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:35.627 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:35.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.627 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2022-03-21T17:11:35.628 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2022-03-21T17:11:35.628 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2022-03-21T17:11:35.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.629 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2022-03-21T17:11:35.629 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:35.629 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:35.630 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:35.630 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:35.630 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:35.631 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:35.631 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:35.632 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:35.632 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:35.632 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:35.633 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:35.633 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:35.633 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:35.634 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:35.634 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:35.635 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:35.635 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:35.635 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:35.636 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:35.636 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:35.636 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:35.637 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:35.637 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:35.637 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:35.638 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:35.638 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:35.638 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:35.639 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:35.639 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:35.639 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:35.640 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:35.640 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2022-03-21T17:11:35.640 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:35.641 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:35.641 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:35.642 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:35.642 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149655, 2022-03-21T17:11:35.642 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:35.643 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:35.643 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:35.643 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2022-03-21T17:11:35.644 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2022-03-21T17:11:35.645 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:35.645 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:35.645 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:35.646 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:35.646 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:35.646 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2022-03-21T17:11:35.647 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:35.647 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:35.647 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:35.648 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2022-03-21T17:11:35.648 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:35.649 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2022-03-21T17:11:35.649 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:35.650 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2022-03-21T17:11:35.650 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:35.650 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2022-03-21T17:11:35.651 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:35.651 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:35.652 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2022-03-21T17:11:35.652 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2022-03-21T17:11:35.653 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2022-03-21T17:11:35.653 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 2928, 2022-03-21T17:11:35.653 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 554686, 2022-03-21T17:11:35.654 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:35.654 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 2928, 2022-03-21T17:11:35.654 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2022-03-21T17:11:35.655 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2022-03-21T17:11:35.655 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 2928, 2022-03-21T17:11:35.655 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 554686, 2022-03-21T17:11:35.656 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2022-03-21T17:11:35.656 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:35.656 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:35.657 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:35.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:35.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:35.658 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T17:11:36.064 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2022-03-21T17:11:36.065 DEBUG:tasks.check_counter:Getting stats from d 2022-03-21T17:11:36.065 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.d perf dump 2022-03-21T17:11:36.720 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T17:11:37.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:37 smithi002 ceph-mon[17167]: pgmap v5812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:37 smithi016 ceph-mon[22010]: pgmap v5812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:37 smithi079 ceph-mon[23158]: pgmap v5812: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T17:11:38.162 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T17:11:38.163 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:38.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2916, 2022-03-21T17:11:38.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2915, 2022-03-21T17:11:38.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 967440, 2022-03-21T17:11:38.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1681872, 2022-03-21T17:11:38.165 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:38.165 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:38.165 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.572251658, 2022-03-21T17:11:38.166 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.257277028, 2022-03-21T17:11:38.166 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.132742531, 2022-03-21T17:11:38.166 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.083844268, 2022-03-21T17:11:38.167 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:38.167 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2914, 2022-03-21T17:11:38.167 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.230544786, 2022-03-21T17:11:38.168 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000079116 2022-03-21T17:11:38.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.168 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:38.169 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:38.169 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:38.169 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:38.170 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:38.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.170 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:38.171 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 16, 2022-03-21T17:11:38.171 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2240, 2022-03-21T17:11:38.171 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 4816, 2022-03-21T17:11:38.172 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1260480, 2022-03-21T17:11:38.172 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:38.172 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:38.173 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.295776876, 2022-03-21T17:11:38.173 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.236110307, 2022-03-21T17:11:38.174 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.011783206, 2022-03-21T17:11:38.174 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:38.174 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:38.175 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2239, 2022-03-21T17:11:38.175 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.203086019, 2022-03-21T17:11:38.176 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000090703 2022-03-21T17:11:38.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.176 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:38.177 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:38.177 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:38.177 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:38.178 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:38.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.179 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:38.179 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:38.179 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:38.180 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:38.180 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:38.180 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:38.181 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:38.181 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001133725, 2022-03-21T17:11:38.181 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:38.182 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000759460, 2022-03-21T17:11:38.182 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:38.182 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:38.183 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:38.183 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:38.183 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:38.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.184 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:38.185 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:38.185 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:38.185 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:38.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:38.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.186 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2022-03-21T17:11:38.187 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2022-03-21T17:11:38.187 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2022-03-21T17:11:38.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.188 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2022-03-21T17:11:38.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:38.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:38.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:38.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:38.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:38.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:38.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:38.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:38.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:38.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:38.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:38.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:38.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:38.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:38.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:38.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:38.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:38.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:38.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:38.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:38.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:38.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:38.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:38.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:38.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:38.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:38.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:38.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:38.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:38.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:38.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:38.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2022-03-21T17:11:38.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:38.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:38.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:38.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:38.200 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149655, 2022-03-21T17:11:38.201 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:38.201 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:38.201 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:38.202 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2022-03-21T17:11:38.202 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2022-03-21T17:11:38.202 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:38.203 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:38.203 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:38.203 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:38.204 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:38.204 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2022-03-21T17:11:38.204 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:38.205 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:38.205 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:38.205 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2022-03-21T17:11:38.206 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:38.206 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2022-03-21T17:11:38.207 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:38.207 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2022-03-21T17:11:38.207 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:38.208 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2022-03-21T17:11:38.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:38.208 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:38.209 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2022-03-21T17:11:38.209 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2022-03-21T17:11:38.209 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2022-03-21T17:11:38.210 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 2928, 2022-03-21T17:11:38.210 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 554686, 2022-03-21T17:11:38.210 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:38.211 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 2928, 2022-03-21T17:11:38.211 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2022-03-21T17:11:38.211 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2022-03-21T17:11:38.212 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 2928, 2022-03-21T17:11:38.212 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 554686, 2022-03-21T17:11:38.212 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2022-03-21T17:11:38.213 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:38.213 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:38.213 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:38.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:38.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:38.215 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T17:11:38.686 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2022-03-21T17:11:38.687 DEBUG:tasks.check_counter:Getting stats from g 2022-03-21T17:11:38.687 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.g perf dump 2022-03-21T17:11:38.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:38 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10800.097781 secs 2022-03-21T17:11:38.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:38 smithi002 ceph-mon[17167]: pgmap v5813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10800.097781 secs 2022-03-21T17:11:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:38 smithi016 ceph-mon[22010]: pgmap v5813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10800.097781 secs 2022-03-21T17:11:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:38 smithi079 ceph-mon[23158]: pgmap v5813: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T17:11:40.297 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T17:11:40.849 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T17:11:40.850 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:40.850 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2915, 2022-03-21T17:11:40.851 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2914, 2022-03-21T17:11:40.851 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 967168, 2022-03-21T17:11:40.851 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1681296, 2022-03-21T17:11:40.852 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:40.852 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:40.852 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.655622309, 2022-03-21T17:11:40.853 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.350587979, 2022-03-21T17:11:40.853 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.126299457, 2022-03-21T17:11:40.853 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.080120504, 2022-03-21T17:11:40.854 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:40.854 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2913, 2022-03-21T17:11:40.854 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.238584961, 2022-03-21T17:11:40.855 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000081903 2022-03-21T17:11:40.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.855 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:40.856 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:40.856 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:40.856 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:40.857 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:40.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.857 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:40.858 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 13, 2022-03-21T17:11:40.858 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2239, 2022-03-21T17:11:40.859 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2080, 2022-03-21T17:11:40.859 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1332416, 2022-03-21T17:11:40.859 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 15, 2022-03-21T17:11:40.860 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:40.860 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.295974258, 2022-03-21T17:11:40.860 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.236395086, 2022-03-21T17:11:40.861 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.012974199, 2022-03-21T17:11:40.861 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:40.861 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:40.862 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2237, 2022-03-21T17:11:40.862 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.201668491, 2022-03-21T17:11:40.862 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000090151 2022-03-21T17:11:40.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.863 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:40.863 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:40.864 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:40.864 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:40.864 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:40.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.865 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:40.865 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2022-03-21T17:11:40.866 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2022-03-21T17:11:40.866 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2896, 2022-03-21T17:11:40.866 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2022-03-21T17:11:40.867 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:40.867 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:40.867 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001815005, 2022-03-21T17:11:40.868 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000053785, 2022-03-21T17:11:40.868 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001423473, 2022-03-21T17:11:40.869 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:40.869 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:40.869 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2022-03-21T17:11:40.870 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001572825, 2022-03-21T17:11:40.870 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000524275 2022-03-21T17:11:40.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.871 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:40.871 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:40.871 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:40.872 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:40.872 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:40.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.873 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2022-03-21T17:11:40.873 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2022-03-21T17:11:40.873 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2022-03-21T17:11:40.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.874 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2022-03-21T17:11:40.874 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:40.875 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:40.875 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:40.875 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:40.876 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:40.876 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:40.876 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:40.877 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:40.877 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:40.877 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:40.878 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:40.878 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:40.878 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:40.879 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:40.879 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:40.880 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:40.880 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:40.880 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:40.881 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:40.881 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:40.881 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:40.882 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:40.882 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:40.882 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:40.883 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:40.883 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:40.883 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:40.884 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:40.884 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:40.884 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2022-03-21T17:11:40.885 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:40.886 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:40.887 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149655, 2022-03-21T17:11:40.887 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:40.887 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:40.888 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:40.888 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2022-03-21T17:11:40.888 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2022-03-21T17:11:40.889 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:40.889 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:40.889 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:40.890 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:40.890 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:40.890 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2022-03-21T17:11:40.891 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:40.891 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:40.891 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:40.892 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2022-03-21T17:11:40.892 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:40.892 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2022-03-21T17:11:40.893 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:40.893 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2022-03-21T17:11:40.894 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:40.894 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2022-03-21T17:11:40.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:40.895 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:40.895 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2022-03-21T17:11:40.895 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2022-03-21T17:11:40.896 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2022-03-21T17:11:40.896 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 2928, 2022-03-21T17:11:40.896 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 554583, 2022-03-21T17:11:40.897 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:40.897 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 2928, 2022-03-21T17:11:40.897 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2022-03-21T17:11:40.898 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2022-03-21T17:11:40.898 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 2928, 2022-03-21T17:11:40.899 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 554583, 2022-03-21T17:11:40.899 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2022-03-21T17:11:40.899 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:40.900 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:40.900 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:40.900 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:40.901 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:40.901 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T17:11:41.297 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:41 smithi002 ceph-mon[17167]: pgmap v5814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:41.300 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2022-03-21T17:11:41.301 DEBUG:tasks.check_counter:Getting stats from j 2022-03-21T17:11:41.301 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.j perf dump 2022-03-21T17:11:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:41 smithi016 ceph-mon[22010]: pgmap v5814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:41 smithi079 ceph-mon[23158]: pgmap v5814: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:41.957 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.a/config 2022-03-21T17:11:42.915 INFO:teuthology.orchestra.run.smithi002.stderr:Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded. 2022-03-21T17:11:43.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:43 smithi002 ceph-mon[17167]: pgmap v5815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:43.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:43 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10805.097895 secs 2022-03-21T17:11:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:43 smithi016 ceph-mon[22010]: pgmap v5815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10805.097895 secs 2022-03-21T17:11:43.445 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-03-21T17:11:43.445 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:43.446 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2022-03-21T17:11:43.446 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2022-03-21T17:11:43.446 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2896, 2022-03-21T17:11:43.447 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2022-03-21T17:11:43.447 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:43.447 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:43.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.005642588, 2022-03-21T17:11:43.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000050051, 2022-03-21T17:11:43.448 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001407823, 2022-03-21T17:11:43.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:43.449 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:43.450 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2022-03-21T17:11:43.450 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001518819, 2022-03-21T17:11:43.450 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000506273 2022-03-21T17:11:43.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.451 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:43.451 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:43.452 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:43.452 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:43.452 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:43.453 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.453 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:43.453 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2915, 2022-03-21T17:11:43.454 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2914, 2022-03-21T17:11:43.454 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 967168, 2022-03-21T17:11:43.454 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1681296, 2022-03-21T17:11:43.455 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:43.455 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:43.455 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.649900236, 2022-03-21T17:11:43.456 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.348136918, 2022-03-21T17:11:43.456 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.126578720, 2022-03-21T17:11:43.456 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.079944062, 2022-03-21T17:11:43.457 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:43.457 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2913, 2022-03-21T17:11:43.457 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.234673639, 2022-03-21T17:11:43.458 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000080560 2022-03-21T17:11:43.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.458 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:43.459 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:43.459 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:43.459 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:43.460 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:43.460 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.460 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:43.461 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 12, 2022-03-21T17:11:43.461 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2237, 2022-03-21T17:11:43.461 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1920, 2022-03-21T17:11:43.462 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 1259856, 2022-03-21T17:11:43.462 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:43.463 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:43.463 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.299506397, 2022-03-21T17:11:43.463 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.240558350, 2022-03-21T17:11:43.464 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.011891835, 2022-03-21T17:11:43.464 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:43.464 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:43.465 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 2235, 2022-03-21T17:11:43.465 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.204784362, 2022-03-21T17:11:43.465 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000091626 2022-03-21T17:11:43.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.466 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:43.466 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:43.467 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:43.467 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:43.467 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:43.468 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.468 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2022-03-21T17:11:43.469 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2022-03-21T17:11:43.469 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2022-03-21T17:11:43.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.470 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2022-03-21T17:11:43.470 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:43.470 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:43.471 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:43.471 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:43.471 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:43.472 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:43.472 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:43.472 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:43.473 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:43.473 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:43.473 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:43.474 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:43.474 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:43.474 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:43.475 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:43.475 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:43.475 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:43.476 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:43.476 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:43.476 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:43.477 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:43.477 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:43.477 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:43.478 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:43.478 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:43.478 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:43.479 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:43.479 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:43.479 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:43.480 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:43.480 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:43.480 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2022-03-21T17:11:43.481 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:43.481 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:43.481 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:43.482 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:43.482 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 150686, 2022-03-21T17:11:43.483 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 31, 2022-03-21T17:11:43.483 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:43.483 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:43.484 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2022-03-21T17:11:43.484 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2022-03-21T17:11:43.484 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:43.485 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:43.485 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:43.485 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:43.486 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:43.486 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2022-03-21T17:11:43.486 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:43.487 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:43.487 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:43.487 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2022-03-21T17:11:43.488 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:43.488 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2022-03-21T17:11:43.488 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:43.489 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2022-03-21T17:11:43.489 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:43.489 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2022-03-21T17:11:43.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2022-03-21T17:11:43.490 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:43.490 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2022-03-21T17:11:43.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2022-03-21T17:11:43.491 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2022-03-21T17:11:43.492 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 2927, 2022-03-21T17:11:43.492 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 554556, 2022-03-21T17:11:43.492 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:43.493 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 2927, 2022-03-21T17:11:43.493 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2022-03-21T17:11:43.493 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2022-03-21T17:11:43.494 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 2927, 2022-03-21T17:11:43.494 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 554556, 2022-03-21T17:11:43.494 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2022-03-21T17:11:43.495 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2022-03-21T17:11:43.495 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2022-03-21T17:11:43.495 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2022-03-21T17:11:43.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:43.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-03-21T17:11:43.496 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-03-21T17:11:43.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:43 smithi079 ceph-mon[23158]: pgmap v5815: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:43.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10805.097895 secs 2022-03-21T17:11:43.920 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2022-03-21T17:11:43.920 DEBUG:tasks.check_counter:Getting stats from b 2022-03-21T17:11:43.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.b perf dump 2022-03-21T17:11:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.b/config 2022-03-21T17:11:45.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:45 smithi002 ceph-mon[17167]: pgmap v5816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:45 smithi016 ceph-mon[22010]: pgmap v5816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:45 smithi079 ceph-mon[23158]: pgmap v5816: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:45.765 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2022-03-21T17:11:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001557034, 2022-03-21T17:11:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000226719, 2022-03-21T17:11:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 12, 2022-03-21T17:11:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2237, 2022-03-21T17:11:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1920, 2022-03-21T17:11:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1259856, 2022-03-21T17:11:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.138870460, 2022-03-21T17:11:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.101969317, 2022-03-21T17:11:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.005052821, 2022-03-21T17:11:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2235, 2022-03-21T17:11:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.157430670, 2022-03-21T17:11:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000070438 2022-03-21T17:11:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2919, 2022-03-21T17:11:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2916, 2022-03-21T17:11:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 970064, 2022-03-21T17:11:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1681728, 2022-03-21T17:11:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.475256647, 2022-03-21T17:11:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.206336815, 2022-03-21T17:11:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.113909493, 2022-03-21T17:11:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.085214041, 2022-03-21T17:11:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2916, 2022-03-21T17:11:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.165553313, 2022-03-21T17:11:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000056774 2022-03-21T17:11:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2022-03-21T17:11:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2022-03-21T17:11:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2022-03-21T17:11:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2022-03-21T17:11:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2022-03-21T17:11:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148664, 2022-03-21T17:11:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2022-03-21T17:11:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2022-03-21T17:11:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2022-03-21T17:11:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2022-03-21T17:11:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2022-03-21T17:11:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2022-03-21T17:11:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2022-03-21T17:11:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2022-03-21T17:11:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2022-03-21T17:11:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2022-03-21T17:11:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2927, 2022-03-21T17:11:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 554556, 2022-03-21T17:11:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2927, 2022-03-21T17:11:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2022-03-21T17:11:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2022-03-21T17:11:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2927, 2022-03-21T17:11:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 554556, 2022-03-21T17:11:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2022-03-21T17:11:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:45.819 INFO:teuthology.orchestra.run.smithi016.stdout:} 2022-03-21T17:11:46.098 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.b 2022-03-21T17:11:46.099 DEBUG:tasks.check_counter:Getting stats from e 2022-03-21T17:11:46.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.e perf dump 2022-03-21T17:11:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:11:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:11:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:46 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:11:46.359 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:11:46.359 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:11:46.360 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:46 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:11:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:11:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:11:46.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:46 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:11:46.552 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.b/config 2022-03-21T17:11:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:47 smithi002 ceph-mon[17167]: pgmap v5817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:47.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:47 smithi002 ceph-mon[17167]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:11:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:47 smithi016 ceph-mon[22010]: pgmap v5817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:47.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:47 smithi016 ceph-mon[22010]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:11:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:47 smithi079 ceph-mon[23158]: pgmap v5817: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:47 smithi079 ceph-mon[23158]: from='mgr.14150 172.21.15.2:0/3604443276' entity='mgr.x' 2022-03-21T17:11:47.809 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2022-03-21T17:11:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001413321, 2022-03-21T17:11:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000120374, 2022-03-21T17:11:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2915, 2022-03-21T17:11:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2914, 2022-03-21T17:11:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 967168, 2022-03-21T17:11:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1681280, 2022-03-21T17:11:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.481543312, 2022-03-21T17:11:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.203337704, 2022-03-21T17:11:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.119732435, 2022-03-21T17:11:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.088640823, 2022-03-21T17:11:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2913, 2022-03-21T17:11:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.166850644, 2022-03-21T17:11:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000057277 2022-03-21T17:11:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 17, 2022-03-21T17:11:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2240, 2022-03-21T17:11:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 4976, 2022-03-21T17:11:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1332688, 2022-03-21T17:11:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 15, 2022-03-21T17:11:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.140347473, 2022-03-21T17:11:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.103018079, 2022-03-21T17:11:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.005669252, 2022-03-21T17:11:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2239, 2022-03-21T17:11:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.156135938, 2022-03-21T17:11:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000069734 2022-03-21T17:11:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2022-03-21T17:11:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2022-03-21T17:11:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2022-03-21T17:11:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2022-03-21T17:11:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:47.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:47.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:47.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:47.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:47.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:47.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:47.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:47.844 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:47.844 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:47.844 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:47.845 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:47.845 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:47.845 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2022-03-21T17:11:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:47.847 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148619, 2022-03-21T17:11:47.848 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:47.849 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:47.849 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:47.849 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2022-03-21T17:11:47.850 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2022-03-21T17:11:47.850 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:47.850 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:47.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:47.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:47.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:47.852 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2022-03-21T17:11:47.852 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:47.852 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:47.853 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:47.853 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2022-03-21T17:11:47.853 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:47.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2022-03-21T17:11:47.854 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:47.854 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2022-03-21T17:11:47.855 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:47.855 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2022-03-21T17:11:47.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:47.856 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:47.856 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2022-03-21T17:11:47.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2022-03-21T17:11:47.857 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2022-03-21T17:11:47.857 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2928, 2022-03-21T17:11:47.858 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 554583, 2022-03-21T17:11:47.858 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:47.858 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2928, 2022-03-21T17:11:47.859 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2022-03-21T17:11:47.859 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2022-03-21T17:11:47.859 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2928, 2022-03-21T17:11:47.860 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 554583, 2022-03-21T17:11:47.860 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2022-03-21T17:11:47.860 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:47.861 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:47.861 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:47.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:47.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:47.862 INFO:teuthology.orchestra.run.smithi016.stdout:} 2022-03-21T17:11:48.166 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2022-03-21T17:11:48.166 DEBUG:tasks.check_counter:Getting stats from h 2022-03-21T17:11:48.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.h perf dump 2022-03-21T17:11:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.b/config 2022-03-21T17:11:48.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10810.097989 secs 2022-03-21T17:11:48.818 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:48 smithi016 ceph-mon[22010]: pgmap v5818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:48 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10810.097989 secs 2022-03-21T17:11:48.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:48 smithi002 ceph-mon[17167]: pgmap v5818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10810.097989 secs 2022-03-21T17:11:49.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:48 smithi079 ceph-mon[23158]: pgmap v5818: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:49.808 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2022-03-21T17:11:49.809 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:49.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:49.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:49.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001478839, 2022-03-21T17:11:49.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000114412, 2022-03-21T17:11:49.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:49.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:49.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.819 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:49.820 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:49.821 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:49.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:49.823 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 16, 2022-03-21T17:11:49.824 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2239, 2022-03-21T17:11:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 4816, 2022-03-21T17:11:49.825 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1260304, 2022-03-21T17:11:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:49.826 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:49.827 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.137726600, 2022-03-21T17:11:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.100646431, 2022-03-21T17:11:49.828 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.004870793, 2022-03-21T17:11:49.829 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:49.830 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2238, 2022-03-21T17:11:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.159541324, 2022-03-21T17:11:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000071287 2022-03-21T17:11:49.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:49.833 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:49.834 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:49.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.836 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:49.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2915, 2022-03-21T17:11:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2914, 2022-03-21T17:11:49.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 967168, 2022-03-21T17:11:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1681280, 2022-03-21T17:11:49.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.473706737, 2022-03-21T17:11:49.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.206430987, 2022-03-21T17:11:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.112465809, 2022-03-21T17:11:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.085012371, 2022-03-21T17:11:49.841 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2913, 2022-03-21T17:11:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.163445124, 2022-03-21T17:11:49.842 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000056108 2022-03-21T17:11:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.843 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2022-03-21T17:11:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2022-03-21T17:11:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2022-03-21T17:11:49.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2022-03-21T17:11:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:49.847 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:49.848 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:49.849 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:49.850 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:49.851 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:49.852 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:49.853 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:49.854 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:49.855 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:49.856 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:49.857 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2022-03-21T17:11:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148602, 2022-03-21T17:11:49.859 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2022-03-21T17:11:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:49.860 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2022-03-21T17:11:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2022-03-21T17:11:49.861 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:49.862 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:49.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2022-03-21T17:11:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:49.864 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2022-03-21T17:11:49.865 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2022-03-21T17:11:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2022-03-21T17:11:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:49.867 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2022-03-21T17:11:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:49.868 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2022-03-21T17:11:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2022-03-21T17:11:49.869 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2022-03-21T17:11:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2927, 2022-03-21T17:11:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 554556, 2022-03-21T17:11:49.870 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2927, 2022-03-21T17:11:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2022-03-21T17:11:49.871 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2022-03-21T17:11:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2927, 2022-03-21T17:11:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 554556, 2022-03-21T17:11:49.872 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2022-03-21T17:11:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:49.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:49.874 INFO:teuthology.orchestra.run.smithi016.stdout:} 2022-03-21T17:11:50.138 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2022-03-21T17:11:50.139 DEBUG:tasks.check_counter:Getting stats from k 2022-03-21T17:11:50.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.k perf dump 2022-03-21T17:11:50.587 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.b/config 2022-03-21T17:11:51.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:51 smithi016 ceph-mon[22010]: pgmap v5819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:51.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:51 smithi002 ceph-mon[17167]: pgmap v5819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:51 smithi079 ceph-mon[23158]: pgmap v5819: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:51.740 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2022-03-21T17:11:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2022-03-21T17:11:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2022-03-21T17:11:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 2896, 2022-03-21T17:11:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2022-03-21T17:11:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001908190, 2022-03-21T17:11:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000043044, 2022-03-21T17:11:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000584622, 2022-03-21T17:11:51.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:51.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:51.745 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2022-03-21T17:11:51.746 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000732263, 2022-03-21T17:11:51.746 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000244087 2022-03-21T17:11:51.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.747 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:51.747 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:51.747 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:51.748 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:51.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:51.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.749 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:51.749 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 12, 2022-03-21T17:11:51.749 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2237, 2022-03-21T17:11:51.750 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1920, 2022-03-21T17:11:51.750 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1259856, 2022-03-21T17:11:51.750 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 14, 2022-03-21T17:11:51.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:51.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.139360194, 2022-03-21T17:11:51.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.102297112, 2022-03-21T17:11:51.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.004944735, 2022-03-21T17:11:51.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:51.753 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:51.753 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2235, 2022-03-21T17:11:51.753 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.161718280, 2022-03-21T17:11:51.754 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000072357 2022-03-21T17:11:51.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:51.755 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:51.755 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:51.755 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:51.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:51.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.756 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:51.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2915, 2022-03-21T17:11:51.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2914, 2022-03-21T17:11:51.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 967168, 2022-03-21T17:11:51.758 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1681280, 2022-03-21T17:11:51.758 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:51.758 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:51.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.473607350, 2022-03-21T17:11:51.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.203264675, 2022-03-21T17:11:51.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.114912442, 2022-03-21T17:11:51.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.085953776, 2022-03-21T17:11:51.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:51.760 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2913, 2022-03-21T17:11:51.761 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.168273123, 2022-03-21T17:11:51.761 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000057766 2022-03-21T17:11:51.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.762 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:51.762 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:51.762 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:51.763 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:51.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:51.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.764 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2022-03-21T17:11:51.764 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2022-03-21T17:11:51.765 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2022-03-21T17:11:51.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2022-03-21T17:11:51.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:51.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:51.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:51.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:51.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:51.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:51.769 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:51.769 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:51.769 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:51.770 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:51.770 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:51.771 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:51.771 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:51.771 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:51.772 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:51.772 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:51.772 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:51.773 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:51.773 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:51.773 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:51.774 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:51.774 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:51.774 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:51.775 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:51.775 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:51.775 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:51.776 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:51.776 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:51.776 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2022-03-21T17:11:51.777 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:51.777 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:51.777 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:51.778 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:51.778 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148624, 2022-03-21T17:11:51.778 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2022-03-21T17:11:51.779 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:51.779 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:51.780 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2022-03-21T17:11:51.780 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2022-03-21T17:11:51.780 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:51.781 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:51.781 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2022-03-21T17:11:51.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:51.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:51.783 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:51.784 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2022-03-21T17:11:51.784 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:51.785 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2022-03-21T17:11:51.785 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:51.785 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2022-03-21T17:11:51.786 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:51.786 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2022-03-21T17:11:51.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2022-03-21T17:11:51.787 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:51.787 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2022-03-21T17:11:51.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2022-03-21T17:11:51.788 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2022-03-21T17:11:51.788 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2927, 2022-03-21T17:11:51.789 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 554556, 2022-03-21T17:11:51.789 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:51.789 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2927, 2022-03-21T17:11:51.790 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2022-03-21T17:11:51.790 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2022-03-21T17:11:51.790 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2927, 2022-03-21T17:11:51.791 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 554556, 2022-03-21T17:11:51.791 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2022-03-21T17:11:51.791 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2022-03-21T17:11:51.792 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2022-03-21T17:11:51.792 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2022-03-21T17:11:51.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:51.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2022-03-21T17:11:51.793 INFO:teuthology.orchestra.run.smithi016.stdout:} 2022-03-21T17:11:52.091 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2022-03-21T17:11:52.092 DEBUG:tasks.check_counter:Getting stats from c 2022-03-21T17:11:52.092 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.c perf dump 2022-03-21T17:11:52.539 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.c/config 2022-03-21T17:11:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:53 smithi002 ceph-mon[17167]: pgmap v5820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:53.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:53 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10815.098086 secs 2022-03-21T17:11:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:53 smithi016 ceph-mon[22010]: pgmap v5820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10815.098086 secs 2022-03-21T17:11:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:53 smithi079 ceph-mon[23158]: pgmap v5820: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10815.098086 secs 2022-03-21T17:11:53.820 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2022-03-21T17:11:53.820 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:53.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 0, 2022-03-21T17:11:53.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2, 2022-03-21T17:11:53.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 0, 2022-03-21T17:11:53.822 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 0, 2022-03-21T17:11:53.822 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:53.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:53.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.001323387, 2022-03-21T17:11:53.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000000000, 2022-03-21T17:11:53.824 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000252635, 2022-03-21T17:11:53.824 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:53.825 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:53.825 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:53.826 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:53.826 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:53.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.827 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:53.827 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:53.828 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:53.828 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:53.828 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:53.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.829 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:53.829 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 17, 2022-03-21T17:11:53.830 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2240, 2022-03-21T17:11:53.830 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 4976, 2022-03-21T17:11:53.830 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 1332688, 2022-03-21T17:11:53.831 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 15, 2022-03-21T17:11:53.831 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:53.831 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.136816065, 2022-03-21T17:11:53.832 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.100850748, 2022-03-21T17:11:53.832 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.005112559, 2022-03-21T17:11:53.832 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:53.833 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:53.833 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2239, 2022-03-21T17:11:53.833 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.162935087, 2022-03-21T17:11:53.834 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000072771 2022-03-21T17:11:53.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.834 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:53.835 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:53.835 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:53.835 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:53.836 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:53.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.836 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:53.837 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 2916, 2022-03-21T17:11:53.837 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2915, 2022-03-21T17:11:53.837 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 967504, 2022-03-21T17:11:53.838 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 1681440, 2022-03-21T17:11:53.838 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:53.838 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:53.839 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.465591786, 2022-03-21T17:11:53.839 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.205903214, 2022-03-21T17:11:53.839 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.109865351, 2022-03-21T17:11:53.840 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.082681494, 2022-03-21T17:11:53.840 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:53.840 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2914, 2022-03-21T17:11:53.841 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.157619697, 2022-03-21T17:11:53.841 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000054090 2022-03-21T17:11:53.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.842 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:53.842 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:53.842 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:53.843 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:53.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:53.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.844 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2022-03-21T17:11:53.844 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2022-03-21T17:11:53.844 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2022-03-21T17:11:53.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.845 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2022-03-21T17:11:53.845 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:53.846 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:53.846 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:53.846 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:53.847 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:53.847 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:53.847 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:53.848 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:53.848 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:53.848 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:53.849 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:53.849 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:53.849 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:53.850 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:53.850 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:53.850 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:53.851 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:53.851 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:53.852 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:53.852 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:53.852 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:53.853 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:53.853 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:53.853 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:53.854 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:53.854 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:53.854 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:53.855 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:53.855 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:53.855 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:53.856 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:53.856 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2022-03-21T17:11:53.856 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:53.857 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:53.857 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:53.857 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:53.858 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 148664, 2022-03-21T17:11:53.858 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 30, 2022-03-21T17:11:53.858 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:53.859 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:53.859 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2022-03-21T17:11:53.859 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2022-03-21T17:11:53.860 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:53.860 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:53.860 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:53.861 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:53.861 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:53.861 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2022-03-21T17:11:53.862 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:53.862 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:53.862 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:53.863 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2022-03-21T17:11:53.863 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:53.863 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2022-03-21T17:11:53.864 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:53.864 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2022-03-21T17:11:53.864 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:53.865 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2022-03-21T17:11:53.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:53.865 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:53.866 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:53.866 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:53.867 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:53.867 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 2929, 2022-03-21T17:11:53.867 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 554777, 2022-03-21T17:11:53.868 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:53.868 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 2929, 2022-03-21T17:11:53.868 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:53.869 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:53.869 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2929, 2022-03-21T17:11:53.869 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 554777, 2022-03-21T17:11:53.869 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:53.870 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:53.870 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:53.871 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:53.871 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:53.871 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:53.872 INFO:teuthology.orchestra.run.smithi079.stdout:} 2022-03-21T17:11:54.254 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2022-03-21T17:11:54.254 DEBUG:tasks.check_counter:Getting stats from f 2022-03-21T17:11:54.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.f perf dump 2022-03-21T17:11:54.691 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.c/config 2022-03-21T17:11:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:55 smithi079 ceph-mon[23158]: pgmap v5821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:55.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:55 smithi002 ceph-mon[17167]: pgmap v5821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:55 smithi016 ceph-mon[22010]: pgmap v5821: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:55.855 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2022-03-21T17:11:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 4, 2022-03-21T17:11:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 4, 2022-03-21T17:11:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 2896, 2022-03-21T17:11:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 448, 2022-03-21T17:11:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2022-03-21T17:11:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2022-03-21T17:11:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.001569741, 2022-03-21T17:11:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000035934, 2022-03-21T17:11:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000531889, 2022-03-21T17:11:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3, 2022-03-21T17:11:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000731331, 2022-03-21T17:11:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000243777 2022-03-21T17:11:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 2916, 2022-03-21T17:11:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2915, 2022-03-21T17:11:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 967504, 2022-03-21T17:11:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 1681440, 2022-03-21T17:11:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2022-03-21T17:11:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2022-03-21T17:11:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.465009631, 2022-03-21T17:11:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.202109151, 2022-03-21T17:11:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.112971698, 2022-03-21T17:11:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.084836315, 2022-03-21T17:11:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2914, 2022-03-21T17:11:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.158625830, 2022-03-21T17:11:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000054435 2022-03-21T17:11:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 13, 2022-03-21T17:11:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2238, 2022-03-21T17:11:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 2080, 2022-03-21T17:11:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 1332240, 2022-03-21T17:11:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 15, 2022-03-21T17:11:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 2, 2022-03-21T17:11:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.136003640, 2022-03-21T17:11:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.100814227, 2022-03-21T17:11:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.004858888, 2022-03-21T17:11:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2022-03-21T17:11:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2236, 2022-03-21T17:11:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.162109158, 2022-03-21T17:11:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000072499 2022-03-21T17:11:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2022-03-21T17:11:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2022-03-21T17:11:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2022-03-21T17:11:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2022-03-21T17:11:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2022-03-21T17:11:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 148624, 2022-03-21T17:11:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 29, 2022-03-21T17:11:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2022-03-21T17:11:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2022-03-21T17:11:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2022-03-21T17:11:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2022-03-21T17:11:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2022-03-21T17:11:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2022-03-21T17:11:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2022-03-21T17:11:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2022-03-21T17:11:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2022-03-21T17:11:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 2929, 2022-03-21T17:11:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 554777, 2022-03-21T17:11:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 2929, 2022-03-21T17:11:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 2929, 2022-03-21T17:11:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 554777, 2022-03-21T17:11:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:55.912 INFO:teuthology.orchestra.run.smithi079.stdout:} 2022-03-21T17:11:56.202 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2022-03-21T17:11:56.203 DEBUG:tasks.check_counter:Getting stats from i 2022-03-21T17:11:56.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.i perf dump 2022-03-21T17:11:56.643 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.c/config 2022-03-21T17:11:57.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:57 smithi002 ceph-mon[17167]: pgmap v5822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:57.387 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:57 smithi079 ceph-mon[23158]: pgmap v5822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:57 smithi016 ceph-mon[22010]: pgmap v5822: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:11:57.803 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2022-03-21T17:11:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 11668, 2022-03-21T17:11:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 11670, 2022-03-21T17:11:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 66802993, 2022-03-21T17:11:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 3628758, 2022-03-21T17:11:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 40, 2022-03-21T17:11:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 4, 2022-03-21T17:11:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 1.600991879, 2022-03-21T17:11:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.428361310, 2022-03-21T17:11:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.421531247, 2022-03-21T17:11:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.582370476, 2022-03-21T17:11:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 11668, 2022-03-21T17:11:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.717688106, 2022-03-21T17:11:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000061509 2022-03-21T17:11:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 24424, 2022-03-21T17:11:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 24420, 2022-03-21T17:11:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 48783105, 2022-03-21T17:11:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 8368147, 2022-03-21T17:11:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 26, 2022-03-21T17:11:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 4, 2022-03-21T17:11:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 2.277402529, 2022-03-21T17:11:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.898080270, 2022-03-21T17:11:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.520339680, 2022-03-21T17:11:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.620584540, 2022-03-21T17:11:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 24419, 2022-03-21T17:11:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.810900824, 2022-03-21T17:11:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000033207 2022-03-21T17:11:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 10967, 2022-03-21T17:11:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 13190, 2022-03-21T17:11:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 36431774, 2022-03-21T17:11:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 8063609, 2022-03-21T17:11:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 37, 2022-03-21T17:11:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 5, 2022-03-21T17:11:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 1.194228100, 2022-03-21T17:11:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.498578052, 2022-03-21T17:11:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.253104381, 2022-03-21T17:11:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.306076458, 2022-03-21T17:11:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 13189, 2022-03-21T17:11:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.552355654, 2022-03-21T17:11:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000041880 2022-03-21T17:11:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2022-03-21T17:11:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2022-03-21T17:11:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2022-03-21T17:11:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2022-03-21T17:11:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2022-03-21T17:11:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2022-03-21T17:11:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 63699, 2022-03-21T17:11:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 2.334959539, 2022-03-21T17:11:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000036656 2022-03-21T17:11:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2022-03-21T17:11:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2022-03-21T17:11:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2022-03-21T17:11:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2022-03-21T17:11:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 0, 2022-03-21T17:11:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 0, 2022-03-21T17:11:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2022-03-21T17:11:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 0, 2022-03-21T17:11:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 0, 2022-03-21T17:11:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2022-03-21T17:11:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 0, 2022-03-21T17:11:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 0, 2022-03-21T17:11:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 0, 2022-03-21T17:11:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 32441, 2022-03-21T17:11:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 0, 2022-03-21T17:11:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 0, 2022-03-21T17:11:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 32441, 2022-03-21T17:11:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 32441, 2022-03-21T17:11:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 0, 2022-03-21T17:11:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 0, 2022-03-21T17:11:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 0, 2022-03-21T17:11:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 2, 2022-03-21T17:11:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 0, 2022-03-21T17:11:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 0, 2022-03-21T17:11:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 0, 2022-03-21T17:11:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 0, 2022-03-21T17:11:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2022-03-21T17:11:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2022-03-21T17:11:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 0, 2022-03-21T17:11:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2022-03-21T17:11:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2022-03-21T17:11:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 0, 2022-03-21T17:11:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 0, 2022-03-21T17:11:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 0, 2022-03-21T17:11:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 0, 2022-03-21T17:11:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2022-03-21T17:11:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2022-03-21T17:11:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2022-03-21T17:11:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 23465, 2022-03-21T17:11:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 0, 2022-03-21T17:11:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2022-03-21T17:11:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2022-03-21T17:11:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2022-03-21T17:11:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2022-03-21T17:11:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2022-03-21T17:11:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2022-03-21T17:11:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2022-03-21T17:11:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2022-03-21T17:11:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2022-03-21T17:11:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 0, 2022-03-21T17:11:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 0, 2022-03-21T17:11:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 0, 2022-03-21T17:11:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 0, 2022-03-21T17:11:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2022-03-21T17:11:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2022-03-21T17:11:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 0, 2022-03-21T17:11:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 0, 2022-03-21T17:11:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 0, 2022-03-21T17:11:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 0 2022-03-21T17:11:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2022-03-21T17:11:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 0, 2022-03-21T17:11:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2022-03-21T17:11:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2022-03-21T17:11:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 0, 2022-03-21T17:11:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 0, 2022-03-21T17:11:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2022-03-21T17:11:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2022-03-21T17:11:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2022-03-21T17:11:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2022-03-21T17:11:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2022-03-21T17:11:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2022-03-21T17:11:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2022-03-21T17:11:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2022-03-21T17:11:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 0, 2022-03-21T17:11:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2022-03-21T17:11:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 0, 2022-03-21T17:11:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2022-03-21T17:11:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2022-03-21T17:11:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2022-03-21T17:11:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 0, 2022-03-21T17:11:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 0, 2022-03-21T17:11:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 0, 2022-03-21T17:11:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 0, 2022-03-21T17:11:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2022-03-21T17:11:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 0, 2022-03-21T17:11:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 0, 2022-03-21T17:11:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 0, 2022-03-21T17:11:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 0, 2022-03-21T17:11:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 72, 2022-03-21T17:11:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2022-03-21T17:11:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 0, 2022-03-21T17:11:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 4194304, 2022-03-21T17:11:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 0, 2022-03-21T17:11:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 142814805, 2022-03-21T17:11:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2022-03-21T17:11:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 65941 2022-03-21T17:11:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2022-03-21T17:11:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 32336, 2022-03-21T17:11:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 32444, 2022-03-21T17:11:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 108, 2022-03-21T17:11:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 1621, 2022-03-21T17:11:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 1956, 2022-03-21T17:11:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 335, 2022-03-21T17:11:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 32441, 2022-03-21T17:11:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 32446, 2022-03-21T17:11:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 5, 2022-03-21T17:11:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 0, 2022-03-21T17:11:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 0, 2022-03-21T17:11:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 0, 2022-03-21T17:11:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 163740, 2022-03-21T17:11:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 207124 2022-03-21T17:11:57.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.880 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2022-03-21T17:11:57.880 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 0, 2022-03-21T17:11:57.881 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 0, 2022-03-21T17:11:57.881 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 0, 2022-03-21T17:11:57.881 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 0, 2022-03-21T17:11:57.882 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 0, 2022-03-21T17:11:57.882 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2022-03-21T17:11:57.882 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.883 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.883 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.884 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2022-03-21T17:11:57.884 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.885 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.885 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.886 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2022-03-21T17:11:57.886 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.886 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.887 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.887 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2022-03-21T17:11:57.888 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.888 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.888 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.889 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2022-03-21T17:11:57.889 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.890 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.890 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2022-03-21T17:11:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.892 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.892 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.893 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2022-03-21T17:11:57.893 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.893 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.894 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.894 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2022-03-21T17:11:57.895 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.895 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.895 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.896 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2022-03-21T17:11:57.896 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.897 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.897 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.898 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2022-03-21T17:11:57.898 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.899 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.899 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.900 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2022-03-21T17:11:57.900 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.900 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.901 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.901 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2022-03-21T17:11:57.902 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.903 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.903 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.904 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2022-03-21T17:11:57.904 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.905 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.905 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.906 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2022-03-21T17:11:57.906 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.906 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.907 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.907 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2022-03-21T17:11:57.908 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.908 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.908 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.909 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2022-03-21T17:11:57.910 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.910 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.910 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.911 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2022-03-21T17:11:57.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.912 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.912 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.913 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2022-03-21T17:11:57.913 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.913 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.914 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.914 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2022-03-21T17:11:57.915 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.915 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.915 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.916 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2022-03-21T17:11:57.916 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.917 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.917 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.918 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2022-03-21T17:11:57.918 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.918 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.919 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.919 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2022-03-21T17:11:57.920 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.920 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.920 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.921 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2022-03-21T17:11:57.921 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.922 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.922 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.923 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2022-03-21T17:11:57.923 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.924 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.924 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.925 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2022-03-21T17:11:57.925 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.925 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.926 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.926 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2022-03-21T17:11:57.927 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.927 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.927 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.928 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2022-03-21T17:11:57.928 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.929 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.929 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.930 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2022-03-21T17:11:57.930 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.930 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.931 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.931 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2022-03-21T17:11:57.932 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2022-03-21T17:11:57.932 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2022-03-21T17:11:57.932 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.933 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.933 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.933 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.934 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2022-03-21T17:11:57.934 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2022-03-21T17:11:57.935 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 1, 2022-03-21T17:11:57.935 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 0, 2022-03-21T17:11:57.935 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 1, 2022-03-21T17:11:57.936 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2022-03-21T17:11:57.936 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 0, 2022-03-21T17:11:57.936 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 0, 2022-03-21T17:11:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 12446 2022-03-21T17:11:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2022-03-21T17:11:57.938 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:57.938 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:57.938 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:57.939 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:57.939 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:57.939 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:57.940 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:57.940 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:57.940 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:57.941 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:57.941 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:57.941 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:57.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:57.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:57.942 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:57.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:57.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:57.943 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:57.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:57.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:57.944 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:57.946 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:57.946 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:57.946 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:57.947 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:57.947 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:57.947 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:57.948 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:57.948 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2022-03-21T17:11:57.948 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:57.949 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:57.949 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:57.949 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:57.950 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 1331184, 2022-03-21T17:11:57.950 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 32408, 2022-03-21T17:11:57.951 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2022-03-21T17:11:57.951 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2022-03-21T17:11:57.951 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2022-03-21T17:11:57.952 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2022-03-21T17:11:57.952 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:57.952 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:57.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:57.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:57.953 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8008, 2022-03-21T17:11:57.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 173, 2022-03-21T17:11:57.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:57.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:57.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:57.955 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2022-03-21T17:11:57.955 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 96391828, 2022-03-21T17:11:57.956 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 881701, 2022-03-21T17:11:57.956 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:57.956 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2022-03-21T17:11:57.957 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:57.957 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2022-03-21T17:11:57.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.958 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2022-03-21T17:11:57.958 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2022-03-21T17:11:57.958 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2022-03-21T17:11:57.959 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 44115, 2022-03-21T17:11:57.959 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 26, 2022-03-21T17:11:57.959 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 4, 2022-03-21T17:11:57.960 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 44115, 2022-03-21T17:11:57.960 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2022-03-21T17:11:57.960 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 44115, 2022-03-21T17:11:57.961 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 44116 2022-03-21T17:11:57.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.961 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 44115, 2022-03-21T17:11:57.962 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 44115, 2022-03-21T17:11:57.962 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 0, 2022-03-21T17:11:57.962 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2022-03-21T17:11:57.963 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2022-03-21T17:11:57.963 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 32880, 2022-03-21T17:11:57.963 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 0, 2022-03-21T17:11:57.964 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 11232, 2022-03-21T17:11:57.964 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 0, 2022-03-21T17:11:57.964 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 0, 2022-03-21T17:11:57.965 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2022-03-21T17:11:57.965 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2022-03-21T17:11:57.965 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2022-03-21T17:11:57.966 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 0, 2022-03-21T17:11:57.966 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 0, 2022-03-21T17:11:57.966 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2022-03-21T17:11:57.967 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2022-03-21T17:11:57.967 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2022-03-21T17:11:57.967 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 2, 2022-03-21T17:11:57.968 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 0, 2022-03-21T17:11:57.968 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2022-03-21T17:11:57.968 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2022-03-21T17:11:57.969 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2022-03-21T17:11:57.969 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2022-03-21T17:11:57.969 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2022-03-21T17:11:57.970 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2022-03-21T17:11:57.970 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2022-03-21T17:11:57.970 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2022-03-21T17:11:57.971 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2022-03-21T17:11:57.971 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 0, 2022-03-21T17:11:57.971 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2022-03-21T17:11:57.972 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2022-03-21T17:11:57.972 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2022-03-21T17:11:57.972 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2022-03-21T17:11:57.973 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2022-03-21T17:11:57.973 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2022-03-21T17:11:57.974 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2022-03-21T17:11:57.974 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2022-03-21T17:11:57.974 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2022-03-21T17:11:57.975 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2022-03-21T17:11:57.975 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2022-03-21T17:11:57.975 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2022-03-21T17:11:57.976 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2022-03-21T17:11:57.976 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2022-03-21T17:11:57.976 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2022-03-21T17:11:57.977 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2022-03-21T17:11:57.977 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 86, 2022-03-21T17:11:57.977 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2022-03-21T17:11:57.978 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 4, 2022-03-21T17:11:57.978 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 10, 2022-03-21T17:11:57.978 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 84, 2022-03-21T17:11:57.979 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 74, 2022-03-21T17:11:57.979 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2022-03-21T17:11:57.979 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 0, 2022-03-21T17:11:57.980 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 2, 2022-03-21T17:11:57.980 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 0 2022-03-21T17:11:57.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.981 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2022-03-21T17:11:57.981 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 0, 2022-03-21T17:11:57.981 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2022-03-21T17:11:57.982 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2022-03-21T17:11:57.982 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2022-03-21T17:11:57.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.983 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2022-03-21T17:11:57.983 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2022-03-21T17:11:57.983 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 0, 2022-03-21T17:11:57.984 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2022-03-21T17:11:57.984 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 0, 2022-03-21T17:11:57.984 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 0, 2022-03-21T17:11:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2022-03-21T17:11:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.985 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:57.986 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 47055, 2022-03-21T17:11:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 147628315, 2022-03-21T17:11:57.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:57.988 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 47055, 2022-03-21T17:11:57.988 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:57.988 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:57.989 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 47055, 2022-03-21T17:11:57.989 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 147628315, 2022-03-21T17:11:57.989 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:57.990 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.990 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.991 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.992 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2022-03-21T17:11:57.992 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:57.992 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:57.993 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:57.993 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:57.993 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:57.994 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:57.994 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:57.994 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 44115, 2022-03-21T17:11:57.995 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 138656046, 2022-03-21T17:11:57.995 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 44115, 2022-03-21T17:11:57.995 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 138656046, 2022-03-21T17:11:57.996 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:57.996 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:57.996 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:57.997 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:57.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:57.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:57.998 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2022-03-21T17:11:57.998 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:57.998 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2022-03-21T17:11:57.999 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:57.999 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:57.999 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:58.000 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:58.000 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:58.000 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 44115, 2022-03-21T17:11:58.001 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 44115, 2022-03-21T17:11:58.001 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 44115, 2022-03-21T17:11:58.001 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 44115, 2022-03-21T17:11:58.002 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:58.002 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:58.002 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:58.003 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:58.003 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:58.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:58.004 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2022-03-21T17:11:58.004 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:58.004 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2022-03-21T17:11:58.005 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:58.005 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:58.005 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:58.006 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:58.006 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:58.006 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:58.007 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:58.007 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2022-03-21T17:11:58.007 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2022-03-21T17:11:58.008 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:58.008 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:58.008 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:58.009 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:58.009 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:58.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:58.010 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x562750788000": { 2022-03-21T17:11:58.010 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:58.010 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2022-03-21T17:11:58.011 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:58.011 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:58.011 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:58.012 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:58.012 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:58.012 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:58.013 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:58.013 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 0, 2022-03-21T17:11:58.013 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 0, 2022-03-21T17:11:58.014 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:58.014 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:58.014 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:58.015 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:58.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:58.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:58.016 INFO:teuthology.orchestra.run.smithi079.stdout:} 2022-03-21T17:11:58.155 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2022-03-21T17:11:58.156 DEBUG:tasks.check_counter:Getting stats from l 2022-03-21T17:11:58.157 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d3799b8cfa5046465710eadeea2bed7bb4681409 shell --fsid ee5e78f2-a91e-11ec-8c35-001a4aab830c -- ceph daemon mds.l perf dump 2022-03-21T17:11:58.595 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/ee5e78f2-a91e-11ec-8c35-001a4aab830c/mon.c/config 2022-03-21T17:11:58.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10820.098176 secs 2022-03-21T17:11:58.833 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:11:58 smithi079 ceph-mon[23158]: pgmap v5823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:58 smithi002 ceph-mon[17167]: 40 slow requests, 0 included below; oldest blocked for > 10820.098176 secs 2022-03-21T17:11:58.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:11:58 smithi002 ceph-mon[17167]: pgmap v5823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10820.098176 secs 2022-03-21T17:11:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:11:58 smithi016 ceph-mon[22010]: pgmap v5823: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:11:59.784 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2022-03-21T17:11:59.785 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2022-03-21T17:11:59.785 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 1150, 2022-03-21T17:11:59.786 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 1152, 2022-03-21T17:11:59.786 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 318192, 2022-03-21T17:11:59.787 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 63363895, 2022-03-21T17:11:59.787 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 52, 2022-03-21T17:11:59.787 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2022-03-21T17:11:59.788 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.278923342, 2022-03-21T17:11:59.788 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.175308371, 2022-03-21T17:11:59.788 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.032351334, 2022-03-21T17:11:59.789 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.061163254, 2022-03-21T17:11:59.789 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:59.789 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1150, 2022-03-21T17:11:59.790 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.302653729, 2022-03-21T17:11:59.790 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000263177 2022-03-21T17:11:59.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.791 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:59.791 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.791 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.792 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.792 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.793 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2022-03-21T17:11:59.793 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 1316, 2022-03-21T17:11:59.794 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 3541, 2022-03-21T17:11:59.794 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 372529, 2022-03-21T17:11:59.794 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 68578968, 2022-03-21T17:11:59.795 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 63, 2022-03-21T17:11:59.795 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 5, 2022-03-21T17:11:59.795 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.421018601, 2022-03-21T17:11:59.796 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.283076157, 2022-03-21T17:11:59.796 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.036666643, 2022-03-21T17:11:59.796 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.066534826, 2022-03-21T17:11:59.797 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:59.797 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3539, 2022-03-21T17:11:59.797 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.357619187, 2022-03-21T17:11:59.798 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000101050 2022-03-21T17:11:59.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.798 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:59.799 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.799 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.799 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.800 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.800 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2022-03-21T17:11:59.801 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 236835, 2022-03-21T17:11:59.801 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 444393, 2022-03-21T17:11:59.801 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 74935400, 2022-03-21T17:11:59.802 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 177452068, 2022-03-21T17:11:59.802 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 41, 2022-03-21T17:11:59.802 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 6, 2022-03-21T17:11:59.803 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 10.092011886, 2022-03-21T17:11:59.803 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 4.671829191, 2022-03-21T17:11:59.803 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 4.538213711, 2022-03-21T17:11:59.804 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.970797161, 2022-03-21T17:11:59.804 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2022-03-21T17:11:59.804 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 444393, 2022-03-21T17:11:59.805 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 4.967156897, 2022-03-21T17:11:59.805 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000011177 2022-03-21T17:11:59.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.806 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2022-03-21T17:11:59.806 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 353974, 2022-03-21T17:11:59.806 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.049367339, 2022-03-21T17:11:59.807 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000139 2022-03-21T17:11:59.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.808 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2022-03-21T17:11:59.808 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 1, 2022-03-21T17:11:59.808 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2022-03-21T17:11:59.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.809 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-MDSRank": { 2022-03-21T17:11:59.809 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2022-03-21T17:11:59.810 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2022-03-21T17:11:59.810 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1807, 2022-03-21T17:11:59.810 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 28.025691783, 2022-03-21T17:11:59.811 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.015509513 2022-03-21T17:11:59.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.812 INFO:teuthology.orchestra.run.smithi079.stdout: "finisher-PurgeQueue": { 2022-03-21T17:11:59.812 INFO:teuthology.orchestra.run.smithi079.stdout: "queue_len": 0, 2022-03-21T17:11:59.813 INFO:teuthology.orchestra.run.smithi079.stdout: "complete_latency": { 2022-03-21T17:11:59.813 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 251, 2022-03-21T17:11:59.813 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.014265300, 2022-03-21T17:11:59.814 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000056833 2022-03-21T17:11:59.814 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.815 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2022-03-21T17:11:59.815 INFO:teuthology.orchestra.run.smithi079.stdout: "request": 32709, 2022-03-21T17:11:59.815 INFO:teuthology.orchestra.run.smithi079.stdout: "reply": 32670, 2022-03-21T17:11:59.816 INFO:teuthology.orchestra.run.smithi079.stdout: "reply_latency": { 2022-03-21T17:11:59.816 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 32670, 2022-03-21T17:11:59.816 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1405.308901172, 2022-03-21T17:11:59.817 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.043015270 2022-03-21T17:11:59.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.817 INFO:teuthology.orchestra.run.smithi079.stdout: "forward": 0, 2022-03-21T17:11:59.818 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_complete": 0, 2022-03-21T17:11:59.818 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_fetch_keys": 0, 2022-03-21T17:11:59.818 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_commit": 1857, 2022-03-21T17:11:59.819 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_split": 230, 2022-03-21T17:11:59.819 INFO:teuthology.orchestra.run.smithi079.stdout: "dir_merge": 0, 2022-03-21T17:11:59.819 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes": 32343, 2022-03-21T17:11:59.820 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_top": 0, 2022-03-21T17:11:59.820 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_bottom": 32343, 2022-03-21T17:11:59.820 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pin_tail": 0, 2022-03-21T17:11:59.821 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_pinned": 32343, 2022-03-21T17:11:59.821 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_expired": 0, 2022-03-21T17:11:59.821 INFO:teuthology.orchestra.run.smithi079.stdout: "inodes_with_caps": 167, 2022-03-21T17:11:59.822 INFO:teuthology.orchestra.run.smithi079.stdout: "caps": 167, 2022-03-21T17:11:59.822 INFO:teuthology.orchestra.run.smithi079.stdout: "subtrees": 2, 2022-03-21T17:11:59.822 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse": 110056, 2022-03-21T17:11:59.823 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_hit": 109723, 2022-03-21T17:11:59.823 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_forward": 0, 2022-03-21T17:11:59.823 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_discover": 0, 2022-03-21T17:11:59.824 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_dir_fetch": 0, 2022-03-21T17:11:59.824 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_remote_ino": 0, 2022-03-21T17:11:59.824 INFO:teuthology.orchestra.run.smithi079.stdout: "traverse_lock": 0, 2022-03-21T17:11:59.825 INFO:teuthology.orchestra.run.smithi079.stdout: "load_cent": 0, 2022-03-21T17:11:59.825 INFO:teuthology.orchestra.run.smithi079.stdout: "q": 0, 2022-03-21T17:11:59.825 INFO:teuthology.orchestra.run.smithi079.stdout: "exported": 0, 2022-03-21T17:11:59.826 INFO:teuthology.orchestra.run.smithi079.stdout: "exported_inodes": 0, 2022-03-21T17:11:59.826 INFO:teuthology.orchestra.run.smithi079.stdout: "imported": 0, 2022-03-21T17:11:59.826 INFO:teuthology.orchestra.run.smithi079.stdout: "imported_inodes": 0, 2022-03-21T17:11:59.827 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_dir_fetch": 0, 2022-03-21T17:11:59.827 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_backtrace_fetch": 0, 2022-03-21T17:11:59.828 INFO:teuthology.orchestra.run.smithi079.stdout: "openino_peer_discover": 0, 2022-03-21T17:11:59.828 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rfiles": 23465, 2022-03-21T17:11:59.828 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rbytes": 0, 2022-03-21T17:11:59.829 INFO:teuthology.orchestra.run.smithi079.stdout: "root_rsnaps": 0, 2022-03-21T17:11:59.829 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_fetch": 0, 2022-03-21T17:11:59.829 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_set_tag": 0, 2022-03-21T17:11:59.830 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_backtrace_repaired": 0, 2022-03-21T17:11:59.830 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_inotable_repaired": 0, 2022-03-21T17:11:59.830 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_inodes": 0, 2022-03-21T17:11:59.831 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dir_base_inodes": 0, 2022-03-21T17:11:59.831 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_dirfrag_rstats": 0, 2022-03-21T17:11:59.831 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_file_inodes": 0, 2022-03-21T17:11:59.832 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_inode_file_caps": 0, 2022-03-21T17:11:59.832 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_revoke": 154463, 2022-03-21T17:11:59.832 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_grant": 186685, 2022-03-21T17:11:59.833 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_trunc": 1, 2022-03-21T17:11:59.833 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flushsnap_ack": 0, 2022-03-21T17:11:59.834 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_cap_op_flush_ack": 0, 2022-03-21T17:11:59.834 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps": 187056, 2022-03-21T17:11:59.834 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_caps_dirty": 32374, 2022-03-21T17:11:59.835 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_cap_release": 208, 2022-03-21T17:11:59.835 INFO:teuthology.orchestra.run.smithi079.stdout: "process_request_cap_release": 128 2022-03-21T17:11:59.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.836 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache": { 2022-03-21T17:11:59.836 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays": 0, 2022-03-21T17:11:59.836 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_delayed": 0, 2022-03-21T17:11:59.837 INFO:teuthology.orchestra.run.smithi079.stdout: "num_strays_enqueuing": 0, 2022-03-21T17:11:59.837 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_created": 108, 2022-03-21T17:11:59.837 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_enqueued": 108, 2022-03-21T17:11:59.838 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_reintegrated": 0, 2022-03-21T17:11:59.838 INFO:teuthology.orchestra.run.smithi079.stdout: "strays_migrated": 0, 2022-03-21T17:11:59.838 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_processing": 0, 2022-03-21T17:11:59.839 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_enqueued": 0, 2022-03-21T17:11:59.839 INFO:teuthology.orchestra.run.smithi079.stdout: "num_recovering_prioritized": 0, 2022-03-21T17:11:59.839 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_started": 0, 2022-03-21T17:11:59.840 INFO:teuthology.orchestra.run.smithi079.stdout: "recovery_completed": 0, 2022-03-21T17:11:59.840 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_enqueue_scrub": 0, 2022-03-21T17:11:59.840 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_exportdir": 0, 2022-03-21T17:11:59.841 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_flush": 0, 2022-03-21T17:11:59.841 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragmentdir": 241, 2022-03-21T17:11:59.841 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_fragstats": 0, 2022-03-21T17:11:59.842 INFO:teuthology.orchestra.run.smithi079.stdout: "ireq_inodestats": 0 2022-03-21T17:11:59.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.842 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log": { 2022-03-21T17:11:59.843 INFO:teuthology.orchestra.run.smithi079.stdout: "evadd": 65941, 2022-03-21T17:11:59.843 INFO:teuthology.orchestra.run.smithi079.stdout: "evex": 0, 2022-03-21T17:11:59.843 INFO:teuthology.orchestra.run.smithi079.stdout: "evtrm": 0, 2022-03-21T17:11:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: "ev": 65941, 2022-03-21T17:11:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: "evexg": 0, 2022-03-21T17:11:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: "evexd": 0, 2022-03-21T17:11:59.845 INFO:teuthology.orchestra.run.smithi079.stdout: "segadd": 72, 2022-03-21T17:11:59.845 INFO:teuthology.orchestra.run.smithi079.stdout: "segex": 0, 2022-03-21T17:11:59.845 INFO:teuthology.orchestra.run.smithi079.stdout: "segtrm": 0, 2022-03-21T17:11:59.846 INFO:teuthology.orchestra.run.smithi079.stdout: "seg": 72, 2022-03-21T17:11:59.846 INFO:teuthology.orchestra.run.smithi079.stdout: "segexg": 0, 2022-03-21T17:11:59.846 INFO:teuthology.orchestra.run.smithi079.stdout: "segexd": 0, 2022-03-21T17:11:59.847 INFO:teuthology.orchestra.run.smithi079.stdout: "expos": 4194304, 2022-03-21T17:11:59.847 INFO:teuthology.orchestra.run.smithi079.stdout: "wrpos": 142816856, 2022-03-21T17:11:59.847 INFO:teuthology.orchestra.run.smithi079.stdout: "rdpos": 0, 2022-03-21T17:11:59.848 INFO:teuthology.orchestra.run.smithi079.stdout: "jlat": { 2022-03-21T17:11:59.848 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 694, 2022-03-21T17:11:59.848 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 15.886536009, 2022-03-21T17:11:59.849 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.022891262 2022-03-21T17:11:59.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.849 INFO:teuthology.orchestra.run.smithi079.stdout: "replayed": 0 2022-03-21T17:11:59.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.850 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mem": { 2022-03-21T17:11:59.850 INFO:teuthology.orchestra.run.smithi079.stdout: "ino": 32345, 2022-03-21T17:11:59.851 INFO:teuthology.orchestra.run.smithi079.stdout: "ino+": 32453, 2022-03-21T17:11:59.851 INFO:teuthology.orchestra.run.smithi079.stdout: "ino-": 108, 2022-03-21T17:11:59.851 INFO:teuthology.orchestra.run.smithi079.stdout: "dir": 1630, 2022-03-21T17:11:59.852 INFO:teuthology.orchestra.run.smithi079.stdout: "dir+": 1965, 2022-03-21T17:11:59.852 INFO:teuthology.orchestra.run.smithi079.stdout: "dir-": 335, 2022-03-21T17:11:59.853 INFO:teuthology.orchestra.run.smithi079.stdout: "dn": 32343, 2022-03-21T17:11:59.853 INFO:teuthology.orchestra.run.smithi079.stdout: "dn+": 32456, 2022-03-21T17:11:59.853 INFO:teuthology.orchestra.run.smithi079.stdout: "dn-": 113, 2022-03-21T17:11:59.854 INFO:teuthology.orchestra.run.smithi079.stdout: "cap": 167, 2022-03-21T17:11:59.854 INFO:teuthology.orchestra.run.smithi079.stdout: "cap+": 32442, 2022-03-21T17:11:59.854 INFO:teuthology.orchestra.run.smithi079.stdout: "cap-": 32275, 2022-03-21T17:11:59.855 INFO:teuthology.orchestra.run.smithi079.stdout: "rss": 251512, 2022-03-21T17:11:59.855 INFO:teuthology.orchestra.run.smithi079.stdout: "heap": 207124 2022-03-21T17:11:59.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.856 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server": { 2022-03-21T17:11:59.856 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_client_request": 187146, 2022-03-21T17:11:59.856 INFO:teuthology.orchestra.run.smithi079.stdout: "dispatch_server_request": 0, 2022-03-21T17:11:59.857 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_request": 32709, 2022-03-21T17:11:59.857 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_client_session": 144, 2022-03-21T17:11:59.857 INFO:teuthology.orchestra.run.smithi079.stdout: "handle_peer_request": 0, 2022-03-21T17:11:59.858 INFO:teuthology.orchestra.run.smithi079.stdout: "req_create_latency": { 2022-03-21T17:11:59.858 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 32327, 2022-03-21T17:11:59.858 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 1344.962330811, 2022-03-21T17:11:59.859 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.041604922 2022-03-21T17:11:59.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.859 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getattr_latency": { 2022-03-21T17:11:59.860 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 6, 2022-03-21T17:11:59.860 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.001458899, 2022-03-21T17:11:59.860 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000243149 2022-03-21T17:11:59.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.861 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getfilelock_latency": { 2022-03-21T17:11:59.861 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.862 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.862 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.863 INFO:teuthology.orchestra.run.smithi079.stdout: "req_link_latency": { 2022-03-21T17:11:59.863 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.864 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.864 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.865 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookup_latency": { 2022-03-21T17:11:59.865 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 110, 2022-03-21T17:11:59.865 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.034555988, 2022-03-21T17:11:59.866 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000314145 2022-03-21T17:11:59.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.866 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookuphash_latency": { 2022-03-21T17:11:59.867 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.867 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.867 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.868 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupino_latency": { 2022-03-21T17:11:59.868 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.869 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.869 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.870 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupname_latency": { 2022-03-21T17:11:59.870 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.871 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.871 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.872 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupparent_latency": { 2022-03-21T17:11:59.872 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.872 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.873 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.873 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lookupsnap_latency": { 2022-03-21T17:11:59.874 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.874 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.874 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.875 INFO:teuthology.orchestra.run.smithi079.stdout: "req_lssnap_latency": { 2022-03-21T17:11:59.875 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.876 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.876 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.877 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mkdir_latency": { 2022-03-21T17:11:59.877 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 113, 2022-03-21T17:11:59.877 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.103640061, 2022-03-21T17:11:59.878 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000917168 2022-03-21T17:11:59.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.878 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mknod_latency": { 2022-03-21T17:11:59.879 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.879 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.879 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.880 INFO:teuthology.orchestra.run.smithi079.stdout: "req_mksnap_latency": { 2022-03-21T17:11:59.880 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.881 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.881 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.882 INFO:teuthology.orchestra.run.smithi079.stdout: "req_open_latency": { 2022-03-21T17:11:59.882 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.882 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.883 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.883 INFO:teuthology.orchestra.run.smithi079.stdout: "req_readdir_latency": { 2022-03-21T17:11:59.884 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3, 2022-03-21T17:11:59.884 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.008784042, 2022-03-21T17:11:59.884 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.002928014 2022-03-21T17:11:59.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.885 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rename_latency": { 2022-03-21T17:11:59.885 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.886 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.886 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.887 INFO:teuthology.orchestra.run.smithi079.stdout: "req_renamesnap_latency": { 2022-03-21T17:11:59.887 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.887 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.888 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.888 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmdir_latency": { 2022-03-21T17:11:59.889 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 105, 2022-03-21T17:11:59.889 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.189492954, 2022-03-21T17:11:59.889 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.001804694 2022-03-21T17:11:59.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.890 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmsnap_latency": { 2022-03-21T17:11:59.890 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.891 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.891 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.892 INFO:teuthology.orchestra.run.smithi079.stdout: "req_rmxattr_latency": { 2022-03-21T17:11:59.892 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.892 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.893 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.893 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setattr_latency": { 2022-03-21T17:11:59.894 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 2, 2022-03-21T17:11:59.894 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 60.004345182, 2022-03-21T17:11:59.894 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 30.002172591 2022-03-21T17:11:59.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.895 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setdirlayout_latency": { 2022-03-21T17:11:59.895 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.896 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.896 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.897 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setfilelock_latency": { 2022-03-21T17:11:59.897 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.897 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.898 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.898 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setlayout_latency": { 2022-03-21T17:11:59.899 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.899 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.899 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.900 INFO:teuthology.orchestra.run.smithi079.stdout: "req_setxattr_latency": { 2022-03-21T17:11:59.900 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 1, 2022-03-21T17:11:59.901 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000866924, 2022-03-21T17:11:59.901 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000866924 2022-03-21T17:11:59.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.902 INFO:teuthology.orchestra.run.smithi079.stdout: "req_symlink_latency": { 2022-03-21T17:11:59.902 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.902 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.903 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.903 INFO:teuthology.orchestra.run.smithi079.stdout: "req_unlink_latency": { 2022-03-21T17:11:59.904 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3, 2022-03-21T17:11:59.904 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.003426311, 2022-03-21T17:11:59.904 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.001142103 2022-03-21T17:11:59.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.905 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_revoke_eviction": 0, 2022-03-21T17:11:59.905 INFO:teuthology.orchestra.run.smithi079.stdout: "cap_acquisition_throttle": 0, 2022-03-21T17:11:59.906 INFO:teuthology.orchestra.run.smithi079.stdout: "req_getvxattr_latency": { 2022-03-21T17:11:59.906 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.906 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.907 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.907 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.908 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessions": { 2022-03-21T17:11:59.908 INFO:teuthology.orchestra.run.smithi079.stdout: "session_count": 1, 2022-03-21T17:11:59.908 INFO:teuthology.orchestra.run.smithi079.stdout: "session_add": 1, 2022-03-21T17:11:59.909 INFO:teuthology.orchestra.run.smithi079.stdout: "session_remove": 0, 2022-03-21T17:11:59.909 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_open": 1, 2022-03-21T17:11:59.910 INFO:teuthology.orchestra.run.smithi079.stdout: "sessions_stale": 0, 2022-03-21T17:11:59.910 INFO:teuthology.orchestra.run.smithi079.stdout: "total_load": 13036, 2022-03-21T17:11:59.910 INFO:teuthology.orchestra.run.smithi079.stdout: "average_load": 13036, 2022-03-21T17:11:59.911 INFO:teuthology.orchestra.run.smithi079.stdout: "avg_session_uptime": 11010 2022-03-21T17:11:59.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.911 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2022-03-21T17:11:59.912 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2022-03-21T17:11:59.912 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2022-03-21T17:11:59.912 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2022-03-21T17:11:59.913 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2022-03-21T17:11:59.913 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2022-03-21T17:11:59.913 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2022-03-21T17:11:59.914 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2022-03-21T17:11:59.914 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2022-03-21T17:11:59.914 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2022-03-21T17:11:59.915 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2022-03-21T17:11:59.915 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2022-03-21T17:11:59.915 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2022-03-21T17:11:59.916 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_bytes": 0, 2022-03-21T17:11:59.916 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Buffer_items": 0, 2022-03-21T17:11:59.916 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_bytes": 0, 2022-03-21T17:11:59.917 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Extent_items": 0, 2022-03-21T17:11:59.917 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_bytes": 0, 2022-03-21T17:11:59.917 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_Blob_items": 0, 2022-03-21T17:11:59.918 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_bytes": 0, 2022-03-21T17:11:59.918 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_SharedBlob_items": 0, 2022-03-21T17:11:59.918 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2022-03-21T17:11:59.919 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2022-03-21T17:11:59.919 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2022-03-21T17:11:59.919 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2022-03-21T17:11:59.920 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2022-03-21T17:11:59.920 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2022-03-21T17:11:59.920 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2022-03-21T17:11:59.920 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2022-03-21T17:11:59.921 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2022-03-21T17:11:59.921 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2022-03-21T17:11:59.921 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2022-03-21T17:11:59.922 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2022-03-21T17:11:59.922 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2022-03-21T17:11:59.922 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2022-03-21T17:11:59.923 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2022-03-21T17:11:59.923 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2022-03-21T17:11:59.924 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 1401250, 2022-03-21T17:11:59.924 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 32674, 2022-03-21T17:11:59.924 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 3960, 2022-03-21T17:11:59.925 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 45, 2022-03-21T17:11:59.925 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2022-03-21T17:11:59.925 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2022-03-21T17:11:59.926 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2022-03-21T17:11:59.926 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2022-03-21T17:11:59.926 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2022-03-21T17:11:59.927 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2022-03-21T17:11:59.927 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 8008, 2022-03-21T17:11:59.927 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 173, 2022-03-21T17:11:59.928 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2022-03-21T17:11:59.928 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2022-03-21T17:11:59.928 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2022-03-21T17:11:59.929 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2022-03-21T17:11:59.929 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 99293452, 2022-03-21T17:11:59.929 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 1240046, 2022-03-21T17:11:59.930 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2022-03-21T17:11:59.930 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2022-03-21T17:11:59.930 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2022-03-21T17:11:59.931 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2022-03-21T17:11:59.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.931 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter": { 2022-03-21T17:11:59.932 INFO:teuthology.orchestra.run.smithi079.stdout: "op_active": 0, 2022-03-21T17:11:59.932 INFO:teuthology.orchestra.run.smithi079.stdout: "op_laggy": 0, 2022-03-21T17:11:59.932 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send": 3099, 2022-03-21T17:11:59.933 INFO:teuthology.orchestra.run.smithi079.stdout: "op_send_bytes": 159888052, 2022-03-21T17:11:59.933 INFO:teuthology.orchestra.run.smithi079.stdout: "op_resend": 37, 2022-03-21T17:11:59.933 INFO:teuthology.orchestra.run.smithi079.stdout: "op_reply": 3099, 2022-03-21T17:11:59.934 INFO:teuthology.orchestra.run.smithi079.stdout: "oplen_avg": { 2022-03-21T17:11:59.934 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3099, 2022-03-21T17:11:59.934 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 5026 2022-03-21T17:11:59.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.935 INFO:teuthology.orchestra.run.smithi079.stdout: "op": 3099, 2022-03-21T17:11:59.936 INFO:teuthology.orchestra.run.smithi079.stdout: "op_r": 29, 2022-03-21T17:11:59.936 INFO:teuthology.orchestra.run.smithi079.stdout: "op_w": 3070, 2022-03-21T17:11:59.936 INFO:teuthology.orchestra.run.smithi079.stdout: "op_rmw": 0, 2022-03-21T17:11:59.937 INFO:teuthology.orchestra.run.smithi079.stdout: "op_pg": 0, 2022-03-21T17:11:59.937 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_stat": 0, 2022-03-21T17:11:59.937 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_create": 11, 2022-03-21T17:11:59.938 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_read": 29, 2022-03-21T17:11:59.938 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_write": 723, 2022-03-21T17:11:59.938 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writefull": 43, 2022-03-21T17:11:59.939 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_writesame": 0, 2022-03-21T17:11:59.939 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_append": 0, 2022-03-21T17:11:59.939 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_zero": 0, 2022-03-21T17:11:59.940 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_truncate": 5, 2022-03-21T17:11:59.940 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_delete": 378, 2022-03-21T17:11:59.940 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_mapext": 0, 2022-03-21T17:11:59.941 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_sparse_read": 0, 2022-03-21T17:11:59.941 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_clonerange": 0, 2022-03-21T17:11:59.943 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_getxattr": 0, 2022-03-21T17:11:59.944 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_setxattr": 22, 2022-03-21T17:11:59.944 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_cmpxattr": 0, 2022-03-21T17:11:59.945 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_rmxattr": 0, 2022-03-21T17:11:59.945 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_resetxattrs": 0, 2022-03-21T17:11:59.945 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_call": 0, 2022-03-21T17:11:59.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_watch": 0, 2022-03-21T17:11:59.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_notify": 0, 2022-03-21T17:11:59.946 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_src_cmpxattr": 0, 2022-03-21T17:11:59.947 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls": 0, 2022-03-21T17:11:59.947 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_pgls_filter": 0, 2022-03-21T17:11:59.947 INFO:teuthology.orchestra.run.smithi079.stdout: "osdop_other": 1, 2022-03-21T17:11:59.948 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_active": 0, 2022-03-21T17:11:59.948 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_send": 0, 2022-03-21T17:11:59.948 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_resend": 0, 2022-03-21T17:11:59.949 INFO:teuthology.orchestra.run.smithi079.stdout: "linger_ping": 0, 2022-03-21T17:11:59.949 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_active": 0, 2022-03-21T17:11:59.949 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_send": 0, 2022-03-21T17:11:59.950 INFO:teuthology.orchestra.run.smithi079.stdout: "poolop_resend": 0, 2022-03-21T17:11:59.950 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_active": 0, 2022-03-21T17:11:59.950 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_send": 0, 2022-03-21T17:11:59.951 INFO:teuthology.orchestra.run.smithi079.stdout: "poolstat_resend": 0, 2022-03-21T17:11:59.951 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_active": 0, 2022-03-21T17:11:59.951 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_send": 0, 2022-03-21T17:11:59.952 INFO:teuthology.orchestra.run.smithi079.stdout: "statfs_resend": 0, 2022-03-21T17:11:59.952 INFO:teuthology.orchestra.run.smithi079.stdout: "command_active": 0, 2022-03-21T17:11:59.952 INFO:teuthology.orchestra.run.smithi079.stdout: "command_send": 0, 2022-03-21T17:11:59.953 INFO:teuthology.orchestra.run.smithi079.stdout: "command_resend": 0, 2022-03-21T17:11:59.953 INFO:teuthology.orchestra.run.smithi079.stdout: "map_epoch": 86, 2022-03-21T17:11:59.953 INFO:teuthology.orchestra.run.smithi079.stdout: "map_full": 0, 2022-03-21T17:11:59.954 INFO:teuthology.orchestra.run.smithi079.stdout: "map_inc": 5, 2022-03-21T17:11:59.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_sessions": 11, 2022-03-21T17:11:59.954 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_open": 130, 2022-03-21T17:11:59.955 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_session_close": 119, 2022-03-21T17:11:59.955 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_laggy": 0, 2022-03-21T17:11:59.955 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_wr": 3785, 2022-03-21T17:11:59.956 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_rd": 0, 2022-03-21T17:11:59.956 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_del": 29 2022-03-21T17:11:59.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.957 INFO:teuthology.orchestra.run.smithi079.stdout: "oft": { 2022-03-21T17:11:59.957 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_objs": 1, 2022-03-21T17:11:59.957 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_kv_pairs": 0, 2022-03-21T17:11:59.958 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_updates": 0, 2022-03-21T17:11:59.958 INFO:teuthology.orchestra.run.smithi079.stdout: "omap_total_removes": 0 2022-03-21T17:11:59.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.959 INFO:teuthology.orchestra.run.smithi079.stdout: "purge_queue": { 2022-03-21T17:11:59.959 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops": 0, 2022-03-21T17:11:59.959 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_ops_high_water": 17, 2022-03-21T17:11:59.960 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing": 0, 2022-03-21T17:11:59.960 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executing_high_water": 17, 2022-03-21T17:11:59.960 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_executed": 108, 2022-03-21T17:11:59.961 INFO:teuthology.orchestra.run.smithi079.stdout: "pq_item_in_journal": 0 2022-03-21T17:11:59.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.961 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2022-03-21T17:11:59.962 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:59.962 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:59.962 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:59.963 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 239297, 2022-03-21T17:11:59.963 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 57540981, 2022-03-21T17:11:59.964 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:59.964 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 239297, 2022-03-21T17:11:59.964 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:59.965 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:59.965 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 239297, 2022-03-21T17:11:59.965 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 57540981, 2022-03-21T17:11:59.966 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:59.966 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.966 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.967 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.968 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_bytes": { 2022-03-21T17:11:59.968 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:59.968 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2022-03-21T17:11:59.969 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:59.969 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:59.969 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:59.970 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:59.970 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:59.970 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 3099, 2022-03-21T17:11:59.971 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 159898960, 2022-03-21T17:11:59.971 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 3099, 2022-03-21T17:11:59.971 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 159898960, 2022-03-21T17:11:59.972 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:59.972 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.972 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.973 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.973 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.974 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-objecter_ops": { 2022-03-21T17:11:59.974 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:59.974 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 1024, 2022-03-21T17:11:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 0, 2022-03-21T17:11:59.975 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 0, 2022-03-21T17:11:59.976 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:59.976 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 0, 2022-03-21T17:11:59.976 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 3099, 2022-03-21T17:11:59.977 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 3099, 2022-03-21T17:11:59.977 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 3099, 2022-03-21T17:11:59.977 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 3099, 2022-03-21T17:11:59.978 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:59.978 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.978 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.979 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.979 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.980 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle": { 2022-03-21T17:11:59.980 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:59.980 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2022-03-21T17:11:59.981 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:59.981 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 108, 2022-03-21T17:11:59.981 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 10908, 2022-03-21T17:11:59.982 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:59.982 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 108, 2022-03-21T17:11:59.982 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:59.983 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:59.983 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 29, 2022-03-21T17:11:59.983 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 10908, 2022-03-21T17:11:59.984 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:59.984 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.984 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.985 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2022-03-21T17:11:59.986 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-write_buf_throttle-0x56289ec15e00": { 2022-03-21T17:11:59.986 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2022-03-21T17:11:59.986 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 3758096384, 2022-03-21T17:11:59.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2022-03-21T17:11:59.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 65941, 2022-03-21T17:11:59.987 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 138622552, 2022-03-21T17:11:59.988 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2022-03-21T17:11:59.988 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 65941, 2022-03-21T17:11:59.988 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2022-03-21T17:11:59.989 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2022-03-21T17:11:59.989 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 694, 2022-03-21T17:11:59.990 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 138622552, 2022-03-21T17:11:59.990 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2022-03-21T17:11:59.990 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2022-03-21T17:11:59.991 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2022-03-21T17:11:59.991 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2022-03-21T17:11:59.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2022-03-21T17:11:59.992 INFO:teuthology.orchestra.run.smithi079.stdout:} 2022-03-21T17:12:00.210 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=230 2022-03-21T17:12:00.210 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2022-03-21T17:12:00.222 INFO:tasks.kclient:Unmounting kernel clients... 2022-03-21T17:12:00.223 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-03-21T17:12:00.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-03-21T17:12:00.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo umount /home/ubuntu/cephtest/mnt.0 2022-03-21T17:12:01.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 21 17:12:01 smithi002 ceph-mon[17167]: pgmap v5824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:12:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:01 smithi016 ceph-mon[22010]: pgmap v5824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:12:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:01 smithi079 ceph-mon[23158]: pgmap v5824: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3 op/s 2022-03-21T17:12:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:23 smithi016 ceph-mon[22010]: pgmap v5825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:12:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:23 smithi079 ceph-mon[23158]: pgmap v5825: 97 pgs: 97 active+clean; 133 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4 op/s 2022-03-21T17:12:24.596 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:24 smithi079 conmon[34304]: 2022-03-21T17:12:24.311+0000 7f8e7c9d9700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10825.098301 secs 2022-03-21T17:12:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10830.098422 secs 2022-03-21T17:12:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10835.098572 secs 2022-03-21T17:12:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mon.b calling monitor election 2022-03-21T17:12:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-03-21T17:12:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T17:12:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T17:12:24.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: osdmap e86: 12 total, 12 up, 12 in 2022-03-21T17:12:24.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mgrmap e16: x(active, since 3h), standbys: y, z 2022-03-21T17:12:24.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-03-21T17:12:24.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests; 1/3 mons down, quorum b,c 2022-03-21T17:12:24.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release 2022-03-21T17:12:24.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mds.l(mds.0): Client smithi002:0 failing to respond to capability release client_id: 14964 2022-03-21T17:12:24.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T17:12:24.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mds.l(mds.0): 40 slow requests are blocked > 180 secs 2022-03-21T17:12:24.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:12:24.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:24 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:12:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10825.098301 secs 2022-03-21T17:12:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10830.098422 secs 2022-03-21T17:12:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10835.098572 secs 2022-03-21T17:12:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mon.b calling monitor election 2022-03-21T17:12:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-03-21T17:12:24.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} 2022-03-21T17:12:24.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T17:12:24.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: osdmap e86: 12 total, 12 up, 12 in 2022-03-21T17:12:24.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mgrmap e16: x(active, since 3h), standbys: y, z 2022-03-21T17:12:24.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-03-21T17:12:24.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests; 1/3 mons down, quorum b,c 2022-03-21T17:12:24.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release 2022-03-21T17:12:24.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mds.l(mds.0): Client smithi002:0 failing to respond to capability release client_id: 14964 2022-03-21T17:12:24.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T17:12:24.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mds.l(mds.0): 40 slow requests are blocked > 180 secs 2022-03-21T17:12:24.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:12:24.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:24 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:12:25.595 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 17:12:25 smithi016 conmon[43966]: 2022-03-21T17:12:25.310+0000 7f40bd348700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:00.999939+0000 front 2022-03-21T17:12:14.903192+0000 (oldest deadline 2022-03-21T17:12:25.099995+0000) 2022-03-21T17:12:25.596 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:25 smithi079 conmon[34304]: 2022-03-21T17:12:25.332+0000 7f8e7c9d9700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:12:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T17:12:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:12:25 smithi016 conmon[32727]: 2022-03-21T17:12:25.724+0000 7ff70c0ba700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:00.660420+0000 front 2022-03-21T17:12:05.361501+0000 (oldest deadline 2022-03-21T17:12:25.360328+0000) 2022-03-21T17:12:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:12:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-03-21T17:12:26.665 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 17:12:26 smithi016 conmon[43966]: 2022-03-21T17:12:26.348+0000 7f40bd348700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:00.999939+0000 front 2022-03-21T17:12:14.903192+0000 (oldest deadline 2022-03-21T17:12:25.099995+0000) 2022-03-21T17:12:26.724 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:26 smithi079 conmon[34304]: 2022-03-21T17:12:26.360+0000 7f8e7c9d9700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:26.725 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 17:12:26 smithi079 conmon[45530]: 2022-03-21T17:12:26.557+0000 7f239a4b4700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:14.405300+0000 front 2022-03-21T17:12:01.503319+0000 (oldest deadline 2022-03-21T17:12:25.603109+0000) 2022-03-21T17:12:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:12:26 smithi016 conmon[32727]: 2022-03-21T17:12:26.754+0000 7ff70c0ba700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:00.660420+0000 front 2022-03-21T17:12:05.361501+0000 (oldest deadline 2022-03-21T17:12:25.360328+0000) 2022-03-21T17:12:27.586 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:27 smithi079 conmon[34304]: 2022-03-21T17:12:27.325+0000 7f8e7c9d9700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:27.586 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 17:12:27 smithi079 conmon[39896]: 2022-03-21T17:12:27.272+0000 7f72da57e700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:12:06.859405+0000 front 2022-03-21T17:12:03.958812+0000 (oldest deadline 2022-03-21T17:12:26.858845+0000) 2022-03-21T17:12:27.638 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 17:12:27 smithi016 conmon[43966]: 2022-03-21T17:12:27.310+0000 7f40bd348700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:00.999939+0000 front 2022-03-21T17:12:14.903192+0000 (oldest deadline 2022-03-21T17:12:25.099995+0000) 2022-03-21T17:12:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:27 smithi016 ceph-mon[22010]: osd.9 reported failed by osd.11 2022-03-21T17:12:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:12:27 smithi016 conmon[32727]: 2022-03-21T17:12:27.767+0000 7ff70c0ba700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:00.660420+0000 front 2022-03-21T17:12:05.361501+0000 (oldest deadline 2022-03-21T17:12:25.360328+0000) 2022-03-21T17:12:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:27 smithi079 ceph-mon[23158]: osd.9 reported failed by osd.11 2022-03-21T17:12:27.975 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 17:12:27 smithi079 conmon[45530]: 2022-03-21T17:12:27.583+0000 7f239a4b4700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:14.405300+0000 front 2022-03-21T17:12:01.503319+0000 (oldest deadline 2022-03-21T17:12:25.603109+0000) 2022-03-21T17:12:28.552 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 17:12:28 smithi079 conmon[39896]: 2022-03-21T17:12:28.255+0000 7f72da57e700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:12:06.859405+0000 front 2022-03-21T17:12:03.958812+0000 (oldest deadline 2022-03-21T17:12:26.858845+0000) 2022-03-21T17:12:28.553 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:28 smithi079 conmon[34304]: 2022-03-21T17:12:28.324+0000 7f8e7c9d9700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:28.665 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 17:12:28 smithi016 conmon[43966]: 2022-03-21T17:12:28.347+0000 7f40bd348700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:00.999939+0000 front 2022-03-21T17:12:14.903192+0000 (oldest deadline 2022-03-21T17:12:25.099995+0000) 2022-03-21T17:12:28.974 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 17:12:28 smithi079 conmon[45530]: 2022-03-21T17:12:28.550+0000 7f239a4b4700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:14.405300+0000 front 2022-03-21T17:12:01.503319+0000 (oldest deadline 2022-03-21T17:12:25.603109+0000) 2022-03-21T17:12:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10850.098906 secs 2022-03-21T17:12:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:28 smithi079 ceph-mon[23158]: osd.9 reported failed by osd.4 2022-03-21T17:12:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:28 smithi079 ceph-mon[23158]: osd.9 failed (root=default,host=smithi002) (2 reporters from different osd after 26.183863 >= grace 20.000000) 2022-03-21T17:12:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:28 smithi079 ceph-mon[23158]: osd.3 reported failed by osd.5 2022-03-21T17:12:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:28 smithi079 ceph-mon[23158]: osd.0 reported failed by osd.10 2022-03-21T17:12:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10850.098906 secs 2022-03-21T17:12:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:28 smithi016 ceph-mon[22010]: osd.9 reported failed by osd.4 2022-03-21T17:12:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:28 smithi016 ceph-mon[22010]: osd.9 failed (root=default,host=smithi002) (2 reporters from different osd after 26.183863 >= grace 20.000000) 2022-03-21T17:12:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:28 smithi016 ceph-mon[22010]: osd.3 reported failed by osd.5 2022-03-21T17:12:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:28 smithi016 ceph-mon[22010]: osd.0 reported failed by osd.10 2022-03-21T17:12:29.170 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:12:28 smithi016 conmon[32727]: 2022-03-21T17:12:28.772+0000 7ff70c0ba700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:00.660420+0000 front 2022-03-21T17:12:05.361501+0000 (oldest deadline 2022-03-21T17:12:25.360328+0000) 2022-03-21T17:12:29.580 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:29 smithi079 conmon[34304]: 2022-03-21T17:12:29.339+0000 7f8e7c9d9700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:11:58.394226+0000 front 2022-03-21T17:12:14.796440+0000 (oldest deadline 2022-03-21T17:12:23.694128+0000) 2022-03-21T17:12:29.581 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 17:12:29 smithi079 conmon[39896]: 2022-03-21T17:12:29.293+0000 7f72da57e700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:06.859487+0000 front 2022-03-21T17:12:14.361201+0000 (oldest deadline 2022-03-21T17:12:28.559426+0000) 2022-03-21T17:12:29.582 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 17:12:29 smithi079 conmon[39896]: 2022-03-21T17:12:29.293+0000 7f72da57e700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-03-21T17:12:06.859405+0000 front 2022-03-21T17:12:03.958812+0000 (oldest deadline 2022-03-21T17:12:26.858845+0000) 2022-03-21T17:12:29.583 INFO:journalctl@ceph.osd.8.smithi079.stdout:Mar 21 17:12:29 smithi079 conmon[39896]: 2022-03-21T17:12:29.293+0000 7f72da57e700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:14.361487+0000 front 2022-03-21T17:12:06.859577+0000 (oldest deadline 2022-03-21T17:12:28.559426+0000) 2022-03-21T17:12:29.665 INFO:journalctl@ceph.osd.10.smithi016.stdout:Mar 21 17:12:29 smithi016 conmon[43966]: 2022-03-21T17:12:29.347+0000 7f40bd348700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:00.999939+0000 front 2022-03-21T17:12:14.903192+0000 (oldest deadline 2022-03-21T17:12:25.099995+0000) 2022-03-21T17:12:29.666 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:29 smithi016 conmon[27157]: 2022-03-21T17:12:29.545+0000 7f2db43b4700 -1 osd.1 87 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-03-21T17:12:17.020804+0000 front 2022-03-21T17:12:05.318735+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:29.666 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:29 smithi016 conmon[27157]: 2022-03-21T17:12:29.545+0000 7f2db43b4700 -1 osd.1 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:29.975 INFO:journalctl@ceph.osd.11.smithi079.stdout:Mar 21 17:12:29 smithi079 conmon[45530]: 2022-03-21T17:12:29.577+0000 7f239a4b4700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-03-21T17:12:14.405300+0000 front 2022-03-21T17:12:01.503319+0000 (oldest deadline 2022-03-21T17:12:25.603109+0000) 2022-03-21T17:12:30.212 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: Health check failed: 1 osds down (OSD_DOWN) 2022-03-21T17:12:30.213 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: osdmap e87: 12 total, 11 up, 12 in 2022-03-21T17:12:30.213 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: osd.0 reported failed by osd.8 2022-03-21T17:12:30.214 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: osd.0 failed (root=default,host=smithi002) (2 reporters from different osd after 22.000287 >= grace 20.000000) 2022-03-21T17:12:30.214 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: osd.3 reported failed by osd.8 2022-03-21T17:12:30.215 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:29 smithi079 ceph-mon[23158]: osd.3 failed (root=default,host=smithi002) (2 reporters from different osd after 25.000526 >= grace 20.000000) 2022-03-21T17:12:30.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: Health check failed: 1 osds down (OSD_DOWN) 2022-03-21T17:12:30.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: osdmap e87: 12 total, 11 up, 12 in 2022-03-21T17:12:30.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: osd.0 reported failed by osd.8 2022-03-21T17:12:30.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: osd.0 failed (root=default,host=smithi002) (2 reporters from different osd after 22.000287 >= grace 20.000000) 2022-03-21T17:12:30.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: osd.3 reported failed by osd.8 2022-03-21T17:12:30.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:29 smithi016 ceph-mon[22010]: osd.3 failed (root=default,host=smithi002) (2 reporters from different osd after 25.000526 >= grace 20.000000) 2022-03-21T17:12:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:30 smithi079 ceph-mon[23158]: osdmap e88: 12 total, 9 up, 12 in 2022-03-21T17:12:30.726 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:30 smithi079 conmon[34304]: 2022-03-21T17:12:30.379+0000 7f8e7c9d9700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:04.195211+0000 front 2022-03-21T17:12:14.796460+0000 (oldest deadline 2022-03-21T17:12:30.095190+0000) 2022-03-21T17:12:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:30 smithi016 ceph-mon[22010]: osdmap e88: 12 total, 9 up, 12 in 2022-03-21T17:12:30.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:30 smithi016 conmon[27157]: 2022-03-21T17:12:30.506+0000 7f2db43b4700 -1 osd.1 88 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:31.415 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 17:12:31 smithi016 conmon[38333]: 2022-03-21T17:12:31.020+0000 7f7e343ba700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:07.820573+0000 front 2022-03-21T17:12:16.622032+0000 (oldest deadline 2022-03-21T17:12:30.720544+0000) 2022-03-21T17:12:31.722 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:31 smithi079 conmon[34304]: 2022-03-21T17:12:31.393+0000 7f8e7c9d9700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:04.195211+0000 front 2022-03-21T17:12:14.796460+0000 (oldest deadline 2022-03-21T17:12:30.095190+0000) 2022-03-21T17:12:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:31 smithi016 ceph-mon[22010]: osdmap e89: 12 total, 9 up, 12 in 2022-03-21T17:12:31.916 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:31 smithi016 conmon[27157]: 2022-03-21T17:12:31.485+0000 7f2db43b4700 -1 osd.1 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:31 smithi079 ceph-mon[23158]: osdmap e89: 12 total, 9 up, 12 in 2022-03-21T17:12:32.415 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 17:12:32 smithi016 conmon[38333]: 2022-03-21T17:12:32.027+0000 7f7e343ba700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:07.820573+0000 front 2022-03-21T17:12:16.622032+0000 (oldest deadline 2022-03-21T17:12:30.720544+0000) 2022-03-21T17:12:32.724 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:32 smithi079 conmon[34304]: 2022-03-21T17:12:32.431+0000 7f8e7c9d9700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:04.195211+0000 front 2022-03-21T17:12:14.796460+0000 (oldest deadline 2022-03-21T17:12:30.095190+0000) 2022-03-21T17:12:32.915 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:32 smithi016 conmon[27157]: 2022-03-21T17:12:32.529+0000 7f2db43b4700 -1 osd.1 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:33.415 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 17:12:33 smithi016 conmon[38333]: 2022-03-21T17:12:33.003+0000 7f7e343ba700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:07.820573+0000 front 2022-03-21T17:12:16.622032+0000 (oldest deadline 2022-03-21T17:12:30.720544+0000) 2022-03-21T17:12:33.706 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:33 smithi079 conmon[34304]: 2022-03-21T17:12:33.398+0000 7f8e7c9d9700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:04.195211+0000 front 2022-03-21T17:12:14.796460+0000 (oldest deadline 2022-03-21T17:12:30.095190+0000) 2022-03-21T17:12:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10855.099008 secs 2022-03-21T17:12:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:33 smithi016 ceph-mon[22010]: osd.6 reported failed by osd.5 2022-03-21T17:12:33.917 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:33 smithi016 conmon[27157]: 2022-03-21T17:12:33.483+0000 7f2db43b4700 -1 osd.1 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10855.099008 secs 2022-03-21T17:12:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:33 smithi079 ceph-mon[23158]: osd.6 reported failed by osd.5 2022-03-21T17:12:34.415 INFO:journalctl@ceph.osd.7.smithi016.stdout:Mar 21 17:12:34 smithi016 conmon[38333]: 2022-03-21T17:12:34.012+0000 7f7e343ba700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:07.820573+0000 front 2022-03-21T17:12:16.622032+0000 (oldest deadline 2022-03-21T17:12:30.720544+0000) 2022-03-21T17:12:34.707 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 21 17:12:34 smithi016 conmon[27157]: 2022-03-21T17:12:34.445+0000 7f2db43b4700 -1 osd.1 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:05.319134+0000 front 2022-03-21T17:12:05.319105+0000 (oldest deadline 2022-03-21T17:12:29.418958+0000) 2022-03-21T17:12:34.708 INFO:journalctl@ceph.osd.5.smithi079.stdout:Mar 21 17:12:34 smithi079 conmon[34304]: 2022-03-21T17:12:34.432+0000 7f8e7c9d9700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-03-21T17:12:04.195211+0000 front 2022-03-21T17:12:14.796460+0000 (oldest deadline 2022-03-21T17:12:30.095190+0000) 2022-03-21T17:12:34.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:34 smithi079 ceph-mon[23158]: osd.6 reported failed by osd.7 2022-03-21T17:12:34.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:34 smithi079 ceph-mon[23158]: osd.6 failed (root=default,host=smithi002) (2 reporters from different osd after 26.000219 >= grace 20.000000) 2022-03-21T17:12:34.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:34 smithi079 ceph-mon[23158]: osd.6 reported failed by osd.1 2022-03-21T17:12:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:34 smithi016 ceph-mon[22010]: osd.6 reported failed by osd.7 2022-03-21T17:12:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:34 smithi016 ceph-mon[22010]: osd.6 failed (root=default,host=smithi002) (2 reporters from different osd after 26.000219 >= grace 20.000000) 2022-03-21T17:12:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:34 smithi016 ceph-mon[22010]: osd.6 reported failed by osd.1 2022-03-21T17:12:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:35 smithi079 ceph-mon[23158]: Health check update: 4 osds down (OSD_DOWN) 2022-03-21T17:12:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:35 smithi079 ceph-mon[23158]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-03-21T17:12:35.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:35 smithi079 ceph-mon[23158]: osdmap e90: 12 total, 8 up, 12 in 2022-03-21T17:12:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:35 smithi016 ceph-mon[22010]: Health check update: 4 osds down (OSD_DOWN) 2022-03-21T17:12:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:35 smithi016 ceph-mon[22010]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-03-21T17:12:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:35 smithi016 ceph-mon[22010]: osdmap e90: 12 total, 8 up, 12 in 2022-03-21T17:12:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:36 smithi079 ceph-mon[23158]: osdmap e91: 12 total, 8 up, 12 in 2022-03-21T17:12:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:36 smithi016 ceph-mon[22010]: osdmap e91: 12 total, 8 up, 12 in 2022-03-21T17:12:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10860.099104 secs 2022-03-21T17:12:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10860.099104 secs 2022-03-21T17:12:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10865.099222 secs 2022-03-21T17:12:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10865.099222 secs 2022-03-21T17:12:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10870.099375 secs 2022-03-21T17:12:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10870.099375 secs 2022-03-21T17:12:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10875.099503 secs 2022-03-21T17:12:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10875.099503 secs 2022-03-21T17:12:58.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-03-21T17:12:58.115 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: osdmap e92: 12 total, 8 up, 12 in 2022-03-21T17:12:58.116 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: mgrmap e17: y(active, starting, since 0.0100316s), standbys: z 2022-03-21T17:12:58.116 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T17:12:58.117 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T17:12:58.117 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T17:12:58.117 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-03-21T17:12:58.118 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-03-21T17:12:58.118 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-03-21T17:12:58.118 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-03-21T17:12:58.119 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-03-21T17:12:58.119 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-03-21T17:12:58.119 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-03-21T17:12:58.120 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-03-21T17:12:58.120 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-03-21T17:12:58.120 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-03-21T17:12:58.121 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T17:12:58.121 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-03-21T17:12:58.121 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-03-21T17:12:58.122 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-03-21T17:12:58.122 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T17:12:58.122 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T17:12:58.123 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T17:12:58.123 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T17:12:58.123 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T17:12:58.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T17:12:58.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T17:12:58.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T17:12:58.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T17:12:58.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T17:12:58.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T17:12:58.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T17:12:58.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-03-21T17:12:58.127 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-03-21T17:12:58.127 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-03-21T17:12:58.128 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.947+0000 7f92596ad700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.128 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.947+0000 7f92596ad700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.129 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.947+0000 7f92596ad700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.129 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.947+0000 7f92596ad700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.129 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.947+0000 7f92596ad700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.130 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.956+0000 7f92556a5700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.130 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.956+0000 7f92556a5700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.130 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.956+0000 7f92556a5700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.131 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.956+0000 7f92556a5700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.131 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:57 smithi016 conmon[23230]: 2022-03-21T17:12:57.956+0000 7f92556a5700 -1 client.0 error registering admin socket command: (17) File exists 2022-03-21T17:12:58.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-03-21T17:12:58.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: osdmap e92: 12 total, 8 up, 12 in 2022-03-21T17:12:58.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: mgrmap e17: y(active, starting, since 0.0100316s), standbys: z 2022-03-21T17:12:58.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-03-21T17:12:58.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-03-21T17:12:58.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-03-21T17:12:58.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-03-21T17:12:58.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-03-21T17:12:58.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-03-21T17:12:58.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-03-21T17:12:58.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-03-21T17:12:58.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-03-21T17:12:58.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-03-21T17:12:58.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-03-21T17:12:58.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-03-21T17:12:58.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-03-21T17:12:58.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T17:12:58.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-03-21T17:12:58.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-03-21T17:12:58.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-03-21T17:12:58.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-03-21T17:12:58.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-03-21T17:12:58.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-03-21T17:12:58.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-03-21T17:12:58.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-03-21T17:12:58.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-03-21T17:12:58.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-03-21T17:12:58.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-03-21T17:12:58.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-03-21T17:12:58.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-03-21T17:12:58.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-03-21T17:12:58.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-03-21T17:12:58.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-03-21T17:12:58.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-03-21T17:12:58.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-03-21T17:12:58.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:58 smithi016 conmon[23230]: [21/Mar/2022:17:12:58] ENGINE Bus STARTING 2022-03-21T17:12:58.763 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:58 smithi016 conmon[23230]: [21/Mar/2022:17:12:58] ENGINE Serving on https://172.21.15.16:7150 2022-03-21T17:12:58.764 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:12:58 smithi016 conmon[23230]: [21/Mar/2022:17:12:58] ENGINE Bus STARTED 2022-03-21T17:12:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10880.099630 secs 2022-03-21T17:12:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: Manager daemon y is now available 2022-03-21T17:12:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:12:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:12:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:12:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:12:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:12:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:12:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:12:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:12:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:12:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10880.099630 secs 2022-03-21T17:12:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: Manager daemon y is now available 2022-03-21T17:12:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:12:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:12:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:12:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:12:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:12:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:12:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:12:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:12:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:59 smithi016 ceph-mon[22010]: [21/Mar/2022:17:12:58] ENGINE Bus STARTING 2022-03-21T17:13:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:59 smithi016 ceph-mon[22010]: [21/Mar/2022:17:12:58] ENGINE Serving on https://172.21.15.16:7150 2022-03-21T17:13:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:59 smithi016 ceph-mon[22010]: [21/Mar/2022:17:12:58] ENGINE Bus STARTED 2022-03-21T17:13:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:59 smithi016 ceph-mon[22010]: mgrmap e18: y(active, since 1.06202s), standbys: z 2022-03-21T17:13:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:12:59 smithi016 ceph-mon[22010]: pgmap v5: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:59 smithi079 ceph-mon[23158]: [21/Mar/2022:17:12:58] ENGINE Bus STARTING 2022-03-21T17:13:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:59 smithi079 ceph-mon[23158]: [21/Mar/2022:17:12:58] ENGINE Serving on https://172.21.15.16:7150 2022-03-21T17:13:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:59 smithi079 ceph-mon[23158]: [21/Mar/2022:17:12:58] ENGINE Bus STARTED 2022-03-21T17:13:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:59 smithi079 ceph-mon[23158]: mgrmap e18: y(active, since 1.06202s), standbys: z 2022-03-21T17:13:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:12:59 smithi079 ceph-mon[23158]: pgmap v5: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:00 smithi079 ceph-mon[23158]: pgmap v6: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:00 smithi079 ceph-mon[23158]: Health check failed: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:00 smithi079 ceph-mon[23158]: Health check failed: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded (PG_DEGRADED) 2022-03-21T17:13:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:00 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:00 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:01.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:00 smithi016 ceph-mon[22010]: pgmap v6: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:01.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:00 smithi016 ceph-mon[22010]: Health check failed: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:01.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:00 smithi016 ceph-mon[22010]: Health check failed: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded (PG_DEGRADED) 2022-03-21T17:13:01.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:00 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:01.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:00 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: mgrmap e19: y(active, since 3s), standbys: z 2022-03-21T17:13:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:01.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: mgrmap e19: y(active, since 3s), standbys: z 2022-03-21T17:13:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:13:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:13:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:13:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T17:13:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:13:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:13:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Updating smithi016:/etc/ceph/ceph.conf 2022-03-21T17:13:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2022-03-21T17:13:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2022-03-21T17:13:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: pgmap v7: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:02 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:13:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:13:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Updating smithi079:/etc/ceph/ceph.conf 2022-03-21T17:13:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:13:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:13:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Updating smithi016:/etc/ceph/ceph.conf 2022-03-21T17:13:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2022-03-21T17:13:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2022-03-21T17:13:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: pgmap v7: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:02 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:13:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10885.099734 secs 2022-03-21T17:13:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10885.099734 secs 2022-03-21T17:13:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:04 smithi016 ceph-mon[22010]: pgmap v8: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:04 smithi079 ceph-mon[23158]: pgmap v8: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:06 smithi016 ceph-mon[22010]: pgmap v9: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:06 smithi079 ceph-mon[23158]: pgmap v9: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10890.099848 secs 2022-03-21T17:13:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:08 smithi079 ceph-mon[23158]: pgmap v10: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10890.099848 secs 2022-03-21T17:13:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:08 smithi016 ceph-mon[22010]: pgmap v10: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:10 smithi016 ceph-mon[22010]: pgmap v11: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:10 smithi079 ceph-mon[23158]: pgmap v11: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:12 smithi016 ceph-mon[22010]: pgmap v12: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:12 smithi079 ceph-mon[23158]: pgmap v12: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10895.099996 secs 2022-03-21T17:13:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10895.099996 secs 2022-03-21T17:13:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:14 smithi016 ceph-mon[22010]: pgmap v13: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:14 smithi079 ceph-mon[23158]: pgmap v13: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:16 smithi016 ceph-mon[22010]: pgmap v14: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:16 smithi079 ceph-mon[23158]: pgmap v14: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10900.100133 secs 2022-03-21T17:13:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:18 smithi079 ceph-mon[23158]: pgmap v15: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10900.100133 secs 2022-03-21T17:13:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:18 smithi016 ceph-mon[22010]: pgmap v15: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:20 smithi016 ceph-mon[22010]: pgmap v16: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:20 smithi079 ceph-mon[23158]: pgmap v16: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:22 smithi016 ceph-mon[22010]: pgmap v17: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:22 smithi079 ceph-mon[23158]: pgmap v17: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10905.100249 secs 2022-03-21T17:13:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10905.100249 secs 2022-03-21T17:13:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:24 smithi016 ceph-mon[22010]: pgmap v18: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:24 smithi079 ceph-mon[23158]: pgmap v18: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:26 smithi016 ceph-mon[22010]: pgmap v19: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:26 smithi079 ceph-mon[23158]: pgmap v19: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:27 smithi016 ceph-mon[22010]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:27 smithi016 ceph-mon[22010]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:27 smithi016 ceph-mon[22010]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:27 smithi016 ceph-mon[22010]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:27 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:13:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:27 smithi079 ceph-mon[23158]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:27 smithi079 ceph-mon[23158]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:27 smithi079 ceph-mon[23158]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:27 smithi079 ceph-mon[23158]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-03-21T17:13:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:27 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:13:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10910.100376 secs 2022-03-21T17:13:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:28 smithi016 ceph-mon[22010]: pgmap v20: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10910.100376 secs 2022-03-21T17:13:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:28 smithi079 ceph-mon[23158]: pgmap v20: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:29 smithi016 ceph-mon[22010]: Health check update: Reduced data availability: 22 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:29 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-03-21T17:13:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:29 smithi079 ceph-mon[23158]: Health check update: Reduced data availability: 22 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:29 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-03-21T17:13:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:30 smithi016 ceph-mon[22010]: pgmap v21: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:30 smithi079 ceph-mon[23158]: pgmap v21: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:32 smithi016 ceph-mon[22010]: pgmap v22: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:33.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: 2022-03-21T17:13:33.007+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:33.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: yield 2022-03-21T17:13:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:33.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:33.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:33.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:33.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:33.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:32 smithi079 ceph-mon[23158]: pgmap v22: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10915.100519 secs 2022-03-21T17:13:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:33 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:34.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:34.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:34.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:34.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:34.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:34.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:34.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:34.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:34.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:34.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:34.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:34.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:34.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:34.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:34.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:34.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:34.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:33 smithi016 ceph-mon[22010]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T17:13:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10915.100519 secs 2022-03-21T17:13:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:33 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:34.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:34.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:34.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:34.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:34.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:34.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:34.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:34.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:34.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:34.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:34.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:34.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:34.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:34.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:33 smithi079 ceph-mon[23158]: Updating smithi002:/etc/ceph/ceph.conf 2022-03-21T17:13:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:34 smithi016 ceph-mon[22010]: pgmap v23: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:34 smithi079 ceph-mon[23158]: pgmap v23: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:35 smithi079 ceph-mon[23158]: Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:35 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2022-03-21T17:13:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:35 smithi016 ceph-mon[22010]: Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:13:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:35 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 1483/4890 objects degraded (30.327%), 22 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2022-03-21T17:13:36.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.077+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield 2022-03-21T17:13:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.078+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield 2022-03-21T17:13:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:13:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise 2022-03-21T17:13:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:13:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:13:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.079+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield 2022-03-21T17:13:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:36.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:36.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:13:36.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:36.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:36.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:36.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:36.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:36.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:36.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:36.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise 2022-03-21T17:13:36.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:36.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:13:36.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:36.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:13:36.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:36.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:13:36.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: 2022-03-21T17:13:36.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:36.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:36.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:13:36.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:36.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:13:36.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files 2022-03-21T17:13:36.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-03-21T17:13:36.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 237, in write_remote_file 2022-03-21T17:13:36.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: host, path, content, mode, uid, gid, addr)) 2022-03-21T17:13:36.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:36.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:13:36.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:36.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:36.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:36.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:13:36.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:36.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:13:36.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 225, in _write_remote_file 2022-03-21T17:13:36.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:13:36.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:36 smithi079 ceph-mon[23158]: pgmap v24: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:13:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:13:37.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:13:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:36 smithi016 ceph-mon[22010]: pgmap v24: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:13:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:13:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:13:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:38.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:37 smithi079 ceph-mon[23158]: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:38.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:13:38.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:37 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:38.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:38.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:38.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:38.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:38.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:38.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:38.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:13:38.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:38.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:38.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:38.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:38.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:38.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:13:38.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:38.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:13:38.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files 2022-03-21T17:13:38.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-03-21T17:13:38.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 237, in write_remote_file 2022-03-21T17:13:38.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: host, path, content, mode, uid, gid, addr)) 2022-03-21T17:13:38.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:38.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:13:38.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:38.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:38.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:38.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:13:38.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:38.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:13:38.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 225, in _write_remote_file 2022-03-21T17:13:38.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.264 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:37 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:13:38.264 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:37 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10920.100634 secs 2022-03-21T17:13:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:38.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:37 smithi016 ceph-mon[22010]: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:38.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:38.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:38.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:38.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:38.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:38.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:38.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:38.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:38.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:13:38.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:38.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:38.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:38.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:37 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:38.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:38.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:38.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:38.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:38.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:38.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:38.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:38.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:38.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:38.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:38.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:38.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:38.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:38.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:38.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:38.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:38.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 203, in _write_remote_file 2022-03-21T17:13:38.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-03-21T17:13:38.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:38.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:38.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:38.452 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:38.452 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:38.453 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:13:38.454 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:38.455 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:38.456 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:38.456 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.457 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.457 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.458 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:38.459 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:38.461 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:38.461 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:38.462 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:13:38.463 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:38.463 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:13:38.464 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files 2022-03-21T17:13:38.464 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-03-21T17:13:38.465 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 237, in write_remote_file 2022-03-21T17:13:38.466 INFO:journalctl@ceph.mon.b.smithi016.stdout: host, path, content, mode, uid, gid, addr)) 2022-03-21T17:13:38.466 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:38.467 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:13:38.468 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:38.468 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:38.469 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:38.469 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:13:38.470 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:38.471 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:13:38.471 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 225, in _write_remote_file 2022-03-21T17:13:38.471 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:38.472 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:38.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:37 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:13:38.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:37 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10920.100634 secs 2022-03-21T17:13:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:38 smithi079 ceph-mon[23158]: pgmap v25: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:38 smithi016 ceph-mon[22010]: pgmap v25: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:39.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 2022-03-21T17:13:39.150+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:39.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:39.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:39.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: yield 2022-03-21T17:13:39.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:39.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:39.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:39.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:39.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:39.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:39.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:39.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:39.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:39.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:39.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:39.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:39.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:39.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:39.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:39.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 2022-03-21T17:13:39.150+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:39.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:39.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:39.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: yield 2022-03-21T17:13:39.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:39.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:39.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:39.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:13:39.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:39.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:13:39.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:39.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:13:39.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:39.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:13:39.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:39.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:13:39.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:39.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:39.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:39.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 2022-03-21T17:13:39.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:13:39.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: 2022-03-21T17:13:39.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:13:39.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:39.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:13:39.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:39.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:13:39.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:13:39.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:13:39.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:13:39.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:13:39.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:39.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:13:39.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:39.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:39.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:39.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:13:39.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:39.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:13:39.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:39.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:39.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:39.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:39.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:39.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise 2022-03-21T17:13:39.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:39.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:13:39.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:39.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:13:39.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:13:39 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:40 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:41.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:41.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:41.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:41.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:41.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:41.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:41.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:40 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:41.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:41.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:13:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:41.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:13:41.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:13:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:13:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:41.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:13:41.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:41.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:41.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:13:41.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:41.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:13:41.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:41.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:13:41.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:13:41.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:13:41.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:13:41.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:13:41.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:41.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:13:41.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:13:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:13:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:13:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:41.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:41.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:41.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:41.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:40 smithi079 ceph-mon[23158]: pgmap v26: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:40 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:41.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:41.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:41.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:41.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:41.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:41.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:40 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:13:41.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:13:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:13:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:13:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:13:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:13:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:13:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:13:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:13:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:13:41.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:13:41.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:13:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:13:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:13:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:13:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:13:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:13:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:13:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:13:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:13:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:13:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:13:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:13:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:13:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:13:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:13:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:13:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:13:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:13:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:13:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:13:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:13:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:13:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:13:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:13:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:13:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:13:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:13:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:13:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:13:41.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:13:41.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:13:41.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:13:41.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:13:41.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:13:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:13:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:13:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:40 smithi016 ceph-mon[22010]: pgmap v26: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:42 smithi079 ceph-mon[23158]: pgmap v27: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:42 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10925.100731 secs 2022-03-21T17:13:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:42 smithi016 ceph-mon[22010]: pgmap v27: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:42 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10925.100731 secs 2022-03-21T17:13:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:44 smithi079 ceph-mon[23158]: pgmap v28: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:44 smithi016 ceph-mon[22010]: pgmap v28: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:46 smithi079 ceph-mon[23158]: pgmap v29: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:46 smithi016 ceph-mon[22010]: pgmap v29: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10930.100848 secs 2022-03-21T17:13:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:48 smithi016 ceph-mon[22010]: pgmap v30: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10930.100848 secs 2022-03-21T17:13:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:48 smithi079 ceph-mon[23158]: pgmap v30: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:50 smithi016 ceph-mon[22010]: pgmap v31: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:50 smithi079 ceph-mon[23158]: pgmap v31: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:52 smithi016 ceph-mon[22010]: pgmap v32: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:52 smithi079 ceph-mon[23158]: pgmap v32: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10935.100912 secs 2022-03-21T17:13:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10935.100912 secs 2022-03-21T17:13:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:54 smithi016 ceph-mon[22010]: pgmap v33: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:54 smithi079 ceph-mon[23158]: pgmap v33: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:56 smithi016 ceph-mon[22010]: pgmap v34: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:56 smithi079 ceph-mon[23158]: pgmap v34: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10940.100996 secs 2022-03-21T17:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: pgmap v35: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:13:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:13:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:13:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10940.100996 secs 2022-03-21T17:13:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: pgmap v35: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:13:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:14:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:00 smithi016 ceph-mon[22010]: pgmap v36: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:00 smithi079 ceph-mon[23158]: pgmap v36: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:02 smithi016 ceph-mon[22010]: pgmap v37: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:02 smithi079 ceph-mon[23158]: pgmap v37: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10945.101112 secs 2022-03-21T17:14:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10945.101112 secs 2022-03-21T17:14:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:04 smithi016 ceph-mon[22010]: pgmap v38: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:04 smithi079 ceph-mon[23158]: pgmap v38: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:06 smithi016 ceph-mon[22010]: pgmap v39: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:06 smithi079 ceph-mon[23158]: pgmap v39: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10950.101231 secs 2022-03-21T17:14:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:08 smithi016 ceph-mon[22010]: pgmap v40: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10950.101231 secs 2022-03-21T17:14:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:08 smithi079 ceph-mon[23158]: pgmap v40: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:10 smithi016 ceph-mon[22010]: pgmap v41: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:10 smithi079 ceph-mon[23158]: pgmap v41: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:12 smithi016 ceph-mon[22010]: pgmap v42: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:12 smithi079 ceph-mon[23158]: pgmap v42: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10955.101349 secs 2022-03-21T17:14:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10955.101349 secs 2022-03-21T17:14:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:14 smithi016 ceph-mon[22010]: pgmap v43: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:14 smithi079 ceph-mon[23158]: pgmap v43: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:16 smithi016 ceph-mon[22010]: pgmap v44: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:16 smithi079 ceph-mon[23158]: pgmap v44: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10960.101503 secs 2022-03-21T17:14:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:18 smithi016 ceph-mon[22010]: pgmap v45: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10960.101503 secs 2022-03-21T17:14:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:18 smithi079 ceph-mon[23158]: pgmap v45: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:20 smithi016 ceph-mon[22010]: pgmap v46: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:20 smithi079 ceph-mon[23158]: pgmap v46: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:22 smithi016 ceph-mon[22010]: pgmap v47: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:22 smithi079 ceph-mon[23158]: pgmap v47: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10965.101622 secs 2022-03-21T17:14:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10965.101622 secs 2022-03-21T17:14:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:24 smithi016 ceph-mon[22010]: pgmap v48: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:24 smithi079 ceph-mon[23158]: pgmap v48: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:26 smithi016 ceph-mon[22010]: pgmap v49: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:26 smithi079 ceph-mon[23158]: pgmap v49: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10970.101716 secs 2022-03-21T17:14:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:28 smithi016 ceph-mon[22010]: pgmap v50: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10970.101716 secs 2022-03-21T17:14:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:28 smithi079 ceph-mon[23158]: pgmap v50: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:29 smithi016 ceph-mon[22010]: mgrmap e20: y(active, since 91s), standbys: z 2022-03-21T17:14:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:29 smithi079 ceph-mon[23158]: mgrmap e20: y(active, since 91s), standbys: z 2022-03-21T17:14:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:30 smithi016 ceph-mon[22010]: pgmap v51: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:30 smithi079 ceph-mon[23158]: pgmap v51: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:32 smithi016 ceph-mon[22010]: pgmap v52: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:32 smithi079 ceph-mon[23158]: pgmap v52: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10975.101828 secs 2022-03-21T17:14:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10975.101828 secs 2022-03-21T17:14:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:34 smithi016 ceph-mon[22010]: pgmap v53: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:34 smithi079 ceph-mon[23158]: pgmap v53: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:36 smithi016 ceph-mon[22010]: pgmap v54: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:36 smithi079 ceph-mon[23158]: pgmap v54: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10980.101958 secs 2022-03-21T17:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:38 smithi016 ceph-mon[22010]: pgmap v55: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:39.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10980.101958 secs 2022-03-21T17:14:39.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:38 smithi079 ceph-mon[23158]: pgmap v55: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:14:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:14:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:14:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:14:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:14:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:14:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:40 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:14:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:40 smithi016 ceph-mon[22010]: pgmap v56: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:40 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:14:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:40 smithi079 ceph-mon[23158]: pgmap v56: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:42.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 2022-03-21T17:14:42.253+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:42.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:14:42.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:42.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: yield 2022-03-21T17:14:42.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:42.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:42.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:42.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:14:42.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:42.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:14:42.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:42.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:14:42.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:42.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:14:42.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:42.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:14:42.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:42.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:42.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:42.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 2022-03-21T17:14:42.254+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:14:42.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:14:42.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:42.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: yield 2022-03-21T17:14:42.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:42.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:42.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:42.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:14:42.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:42.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:14:42.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:42.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:14:42.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:42.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:14:42.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:42.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:14:42.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:42.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:42.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:42.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 2022-03-21T17:14:42.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:14:42.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: 2022-03-21T17:14:42.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:14:42.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:14:42.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:14:42.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:14:42.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:14:42.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:14:42.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:14:42.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:14:42.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:14:42.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:14:42.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:14:42.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:14:42.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:14:42.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:14:42.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:14:42.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:14:42.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:14:42.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:14:42.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:14:42.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:14:42.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:14:42.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:14:42.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise 2022-03-21T17:14:42.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:14:42.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:14:42.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:14:42.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:14:42.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:14:42 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:42 smithi016 ceph-mon[22010]: pgmap v57: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:42 smithi079 ceph-mon[23158]: pgmap v57: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:14:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:14:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:14:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:14:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:44.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:14:44.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:44.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:14:44.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:44.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:44.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:43 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:14:44.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:44.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:14:44.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:44.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:44.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:44.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:14:44.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:44.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:14:44.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:44.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:14:44.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:44.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:14:44.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:44.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:14:44.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:44.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:44.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:14:44.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:14:44.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:14:44.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:14:44.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:14:44.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:14:44.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:14:44.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:14:44.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:14:44.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:14:44.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:14:44.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:14:44.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:14:44.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:14:44.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:14:44.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:14:44.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:14:44.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:14:44.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:14:44.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:14:44.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:14:44.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:14:44.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:14:44.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:14:44.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:14:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:14:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:14:44.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:14:44.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:14:44.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.197 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10985.102079 secs 2022-03-21T17:14:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:14:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:14:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:14:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:14:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:14:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:14:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:43 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:14:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:14:44.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:14:44.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:14:44.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:14:44.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:14:44.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:14:44.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:14:44.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:14:44.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:14:44.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:14:44.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:14:44.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:14:44.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:14:44.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:14:44.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:14:44.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:14:44.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:14:44.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:14:44.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:14:44.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:14:44.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:14:44.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:14:44.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:14:44.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:14:44.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:14:44.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:14:44.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:14:44.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:14:44.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:14:44.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:14:44.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:14:44.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:14:44.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:14:44.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:14:44.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:14:44.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:14:44.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:14:44.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:14:44.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:14:44.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:14:44.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:14:44.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:14:44.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:14:44.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:14:44.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:14:44.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:14:44.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:14:44.258 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10985.102079 secs 2022-03-21T17:14:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:44 smithi016 ceph-mon[22010]: pgmap v58: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:44 smithi079 ceph-mon[23158]: pgmap v58: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:46 smithi016 ceph-mon[22010]: pgmap v59: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:46 smithi079 ceph-mon[23158]: pgmap v59: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10990.102183 secs 2022-03-21T17:14:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:48 smithi016 ceph-mon[22010]: pgmap v60: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10990.102183 secs 2022-03-21T17:14:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:48 smithi079 ceph-mon[23158]: pgmap v60: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:50 smithi016 ceph-mon[22010]: pgmap v61: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:50 smithi079 ceph-mon[23158]: pgmap v61: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:52 smithi016 ceph-mon[22010]: pgmap v62: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:52 smithi079 ceph-mon[23158]: pgmap v62: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 10995.102278 secs 2022-03-21T17:14:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 10995.102278 secs 2022-03-21T17:14:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:54 smithi016 ceph-mon[22010]: pgmap v63: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:54 smithi079 ceph-mon[23158]: pgmap v63: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:56 smithi016 ceph-mon[22010]: pgmap v64: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:56 smithi079 ceph-mon[23158]: pgmap v64: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11000.102397 secs 2022-03-21T17:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: pgmap v65: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11000.102397 secs 2022-03-21T17:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: pgmap v65: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:14:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:15:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:00 smithi016 ceph-mon[22010]: pgmap v66: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:00 smithi079 ceph-mon[23158]: pgmap v66: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:02 smithi016 ceph-mon[22010]: pgmap v67: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:02 smithi079 ceph-mon[23158]: pgmap v67: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11005.102519 secs 2022-03-21T17:15:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11005.102519 secs 2022-03-21T17:15:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:04 smithi016 ceph-mon[22010]: pgmap v68: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:04 smithi079 ceph-mon[23158]: pgmap v68: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:06 smithi016 ceph-mon[22010]: pgmap v69: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:06 smithi079 ceph-mon[23158]: pgmap v69: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11010.102626 secs 2022-03-21T17:15:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:08 smithi016 ceph-mon[22010]: pgmap v70: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11010.102626 secs 2022-03-21T17:15:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:08 smithi079 ceph-mon[23158]: pgmap v70: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:10 smithi016 ceph-mon[22010]: pgmap v71: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:10 smithi079 ceph-mon[23158]: pgmap v71: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:12 smithi016 ceph-mon[22010]: pgmap v72: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:12 smithi079 ceph-mon[23158]: pgmap v72: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11015.102738 secs 2022-03-21T17:15:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11015.102738 secs 2022-03-21T17:15:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:14 smithi016 ceph-mon[22010]: pgmap v73: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:14 smithi079 ceph-mon[23158]: pgmap v73: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:16 smithi016 ceph-mon[22010]: pgmap v74: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:16 smithi079 ceph-mon[23158]: pgmap v74: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11020.102847 secs 2022-03-21T17:15:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:18 smithi016 ceph-mon[22010]: pgmap v75: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11020.102847 secs 2022-03-21T17:15:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:18 smithi079 ceph-mon[23158]: pgmap v75: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:20 smithi016 ceph-mon[22010]: pgmap v76: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:20 smithi079 ceph-mon[23158]: pgmap v76: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:22 smithi016 ceph-mon[22010]: pgmap v77: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:22 smithi079 ceph-mon[23158]: pgmap v77: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11025.102940 secs 2022-03-21T17:15:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11025.102940 secs 2022-03-21T17:15:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:24 smithi016 ceph-mon[22010]: pgmap v78: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:24 smithi079 ceph-mon[23158]: pgmap v78: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:26 smithi016 ceph-mon[22010]: pgmap v79: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:26 smithi079 ceph-mon[23158]: pgmap v79: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11030.103061 secs 2022-03-21T17:15:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:28 smithi016 ceph-mon[22010]: pgmap v80: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11030.103061 secs 2022-03-21T17:15:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:28 smithi079 ceph-mon[23158]: pgmap v80: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:30 smithi016 ceph-mon[22010]: pgmap v81: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:30 smithi079 ceph-mon[23158]: pgmap v81: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:32 smithi016 ceph-mon[22010]: pgmap v82: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:32 smithi079 ceph-mon[23158]: pgmap v82: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11035.103210 secs 2022-03-21T17:15:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11035.103210 secs 2022-03-21T17:15:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:34 smithi016 ceph-mon[22010]: pgmap v83: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:34 smithi079 ceph-mon[23158]: pgmap v83: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:36 smithi016 ceph-mon[22010]: pgmap v84: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:36 smithi079 ceph-mon[23158]: pgmap v84: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11040.103328 secs 2022-03-21T17:15:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:38 smithi016 ceph-mon[22010]: pgmap v85: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11040.103328 secs 2022-03-21T17:15:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:38 smithi079 ceph-mon[23158]: pgmap v85: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:40 smithi016 ceph-mon[22010]: pgmap v86: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:40 smithi079 ceph-mon[23158]: pgmap v86: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: pgmap v87: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:15:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:15:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:15:42.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:15:42.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:42 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:15:43.106 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: pgmap v87: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:43.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:15:43.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:15:43.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:15:43.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:15:43.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:42 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:15:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11045.103399 secs 2022-03-21T17:15:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11045.103399 secs 2022-03-21T17:15:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:44 smithi016 ceph-mon[22010]: pgmap v88: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:44 smithi079 ceph-mon[23158]: pgmap v88: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:45.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: 2022-03-21T17:15:45.357+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:45.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:15:45.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: yield 2022-03-21T17:15:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:45.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:45.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:15:45.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:45.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:15:45.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:45.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:15:45.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:45.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:15:45.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:45.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:15:45.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:45.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:45.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:15:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:15:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:15:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:15:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:15:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:15:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:15:47.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:46 smithi016 ceph-mon[22010]: pgmap v89: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:15:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:15:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:15:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:15:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:15:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:15:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:15:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:46 smithi079 ceph-mon[23158]: pgmap v89: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:48.760 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 2022-03-21T17:15:48.429+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:48.760 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:15:48.761 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:48.761 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: yield 2022-03-21T17:15:48.761 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:48.762 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:48.762 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:48.762 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:15:48.763 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:48.763 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:15:48.763 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:48.764 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:15:48.764 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:48.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:15:48.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:48.765 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:15:48.766 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:48.766 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:48.766 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:48.767 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 2022-03-21T17:15:48.430+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:15:48.767 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:15:48.767 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:48.768 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: yield 2022-03-21T17:15:48.768 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:48.769 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:48.769 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:48.769 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:15:48.770 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:48.770 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:15:48.770 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:48.771 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:15:48.771 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:48.771 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:15:48.772 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:48.772 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:15:48.772 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:48.773 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:48.773 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:48.774 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 2022-03-21T17:15:48.774 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:15:48.774 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: 2022-03-21T17:15:48.775 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:15:48.775 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:15:48.775 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:15:48.776 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:15:48.776 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:15:48.777 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:15:48.777 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:15:48.777 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:15:48.778 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:15:48.778 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:15:48.778 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:15:48.779 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:15:48.779 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:15:48.779 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:15:48.780 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:15:48.781 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:15:48.782 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:15:48.782 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:15:48.783 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:15:48.783 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:15:48.783 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:15:48.784 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:15:48.784 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise 2022-03-21T17:15:48.784 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:15:48.785 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:15:48.785 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:15:48.785 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:15:48.786 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:15:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11050.103538 secs 2022-03-21T17:15:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:48 smithi016 ceph-mon[22010]: pgmap v90: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11050.103538 secs 2022-03-21T17:15:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:48 smithi079 ceph-mon[23158]: pgmap v90: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:50.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:15:50.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:50.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:50.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:50.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:15:50.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:50.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:15:50.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:50.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:15:50.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:50.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:50.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:50.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:50.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:49 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:15:50.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:50.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:15:50.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:50.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:50.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:50.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:15:50.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:50.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:15:50.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:50.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:15:50.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:50.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:50.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:50.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:15:50.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:50.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:50.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:15:50.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:15:50.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:15:50.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:15:50.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:15:50.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:15:50.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:15:50.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:15:50.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:15:50.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:15:50.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:15:50.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:15:50.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:15:50.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:15:50.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:15:50.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:15:50.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:15:50.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:15:50.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:15:50.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:15:50.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:15:50.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:15:50.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:15:50.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:15:50.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:15:50.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:15:50.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:15:50.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:15:50.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:15:50.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:15:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:15:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:15:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:15:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:15:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:49 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:15:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:15:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:15:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:15:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:15:50.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:15:50.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:15:50.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:15:50.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:15:50.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:15:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:15:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:15:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:15:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:15:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:15:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:15:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:15:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:15:50.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:15:50.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:15:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:15:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:15:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:15:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:15:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:15:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:15:50.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:15:50.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:15:50.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:15:50.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:15:50.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:15:50.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:15:50.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:15:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:15:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:15:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:15:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:15:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:15:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:15:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:15:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:15:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:15:50.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:15:50.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:15:50.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:15:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:15:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:15:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:15:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:50 smithi016 ceph-mon[22010]: pgmap v91: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:50 smithi079 ceph-mon[23158]: pgmap v91: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:52 smithi016 ceph-mon[22010]: pgmap v92: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:52 smithi079 ceph-mon[23158]: pgmap v92: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11055.103652 secs 2022-03-21T17:15:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11055.103652 secs 2022-03-21T17:15:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:54 smithi016 ceph-mon[22010]: pgmap v93: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:54 smithi079 ceph-mon[23158]: pgmap v93: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:56 smithi016 ceph-mon[22010]: pgmap v94: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:56 smithi079 ceph-mon[23158]: pgmap v94: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11060.103777 secs 2022-03-21T17:15:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: pgmap v95: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:15:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:15:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:15:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:15:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:15:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11060.103777 secs 2022-03-21T17:15:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: pgmap v95: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:15:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:15:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:15:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:15:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:15:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:16:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:00 smithi016 ceph-mon[22010]: pgmap v96: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:00 smithi079 ceph-mon[23158]: pgmap v96: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:02.224 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 17:16:01 smithi079 conmon[49648]: ignoring --setuser ceph since I am not root 2022-03-21T17:16:02.225 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 17:16:01 smithi079 conmon[49648]: ignoring --setgroup ceph since I am not root 2022-03-21T17:16:02.225 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 17:16:01 smithi079 ceph-mds[49672]: -- 172.21.15.79:0/1486354863 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d88dd4c340 con 0x55d88dd24800 2022-03-21T17:16:02.225 INFO:journalctl@ceph.mds.i.smithi079.stdout:Mar 21 17:16:01 smithi079 conmon[49648]: starting mds.i at 2022-03-21T17:16:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:02 smithi016 ceph-mon[22010]: osdmap e93: 12 total, 8 up, 12 in 2022-03-21T17:16:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:02 smithi016 ceph-mon[22010]: pgmap v98: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:02 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 6 up:standby 2022-03-21T17:16:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:02 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 5 up:standby 2022-03-21T17:16:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:02 smithi079 ceph-mon[23158]: osdmap e93: 12 total, 8 up, 12 in 2022-03-21T17:16:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:02 smithi079 ceph-mon[23158]: pgmap v98: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:02.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:02 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 6 up:standby 2022-03-21T17:16:02.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:02 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 5 up:standby 2022-03-21T17:16:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11065.103894 secs 2022-03-21T17:16:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:03 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6836/1378810835,v1:172.21.15.79:6837/1378810835] up:boot 2022-03-21T17:16:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:03 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:16:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T17:16:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11065.103894 secs 2022-03-21T17:16:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:03 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6836/1378810835,v1:172.21.15.79:6837/1378810835] up:boot 2022-03-21T17:16:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:03 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:16:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-03-21T17:16:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:04 smithi016 ceph-mon[22010]: pgmap v99: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:04 smithi079 ceph-mon[23158]: pgmap v99: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:06 smithi016 ceph-mon[22010]: pgmap v100: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:06 smithi079 ceph-mon[23158]: pgmap v100: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11070.104015 secs 2022-03-21T17:16:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:08 smithi016 ceph-mon[22010]: pgmap v101: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11070.104015 secs 2022-03-21T17:16:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:08 smithi079 ceph-mon[23158]: pgmap v101: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:10 smithi016 ceph-mon[22010]: pgmap v102: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:10 smithi079 ceph-mon[23158]: pgmap v102: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:12 smithi016 ceph-mon[22010]: pgmap v103: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:12 smithi079 ceph-mon[23158]: pgmap v103: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:13 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11075.104122 secs 2022-03-21T17:16:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:13 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11075.104122 secs 2022-03-21T17:16:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:14 smithi016 ceph-mon[22010]: pgmap v104: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:14 smithi079 ceph-mon[23158]: pgmap v104: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:16 smithi016 ceph-mon[22010]: pgmap v105: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:16 smithi079 ceph-mon[23158]: pgmap v105: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11080.104226 secs 2022-03-21T17:16:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:18 smithi016 ceph-mon[22010]: pgmap v106: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11080.104226 secs 2022-03-21T17:16:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:18 smithi079 ceph-mon[23158]: pgmap v106: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:20 smithi016 ceph-mon[22010]: pgmap v107: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:20 smithi079 ceph-mon[23158]: pgmap v107: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:22 smithi016 ceph-mon[22010]: pgmap v108: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:22 smithi079 ceph-mon[23158]: pgmap v108: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:23 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11085.104327 secs 2022-03-21T17:16:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:23 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11085.104327 secs 2022-03-21T17:16:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:24 smithi016 ceph-mon[22010]: pgmap v109: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:24 smithi079 ceph-mon[23158]: pgmap v109: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:26 smithi016 ceph-mon[22010]: pgmap v110: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:26 smithi079 ceph-mon[23158]: pgmap v110: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:28 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11090.104433 secs 2022-03-21T17:16:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:28 smithi016 ceph-mon[22010]: pgmap v111: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:28 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11090.104433 secs 2022-03-21T17:16:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:28 smithi079 ceph-mon[23158]: pgmap v111: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:30 smithi016 ceph-mon[22010]: pgmap v112: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:30 smithi079 ceph-mon[23158]: pgmap v112: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:32 smithi016 ceph-mon[22010]: pgmap v113: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:32 smithi079 ceph-mon[23158]: pgmap v113: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:33 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11095.104569 secs 2022-03-21T17:16:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:33 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11095.104569 secs 2022-03-21T17:16:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:34 smithi016 ceph-mon[22010]: pgmap v114: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:34 smithi079 ceph-mon[23158]: pgmap v114: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:36 smithi016 ceph-mon[22010]: pgmap v115: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:36 smithi079 ceph-mon[23158]: pgmap v115: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:38 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11100.104692 secs 2022-03-21T17:16:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:38 smithi016 ceph-mon[22010]: pgmap v116: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:38 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11100.104692 secs 2022-03-21T17:16:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:38 smithi079 ceph-mon[23158]: pgmap v116: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:40 smithi016 ceph-mon[22010]: pgmap v117: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:40 smithi079 ceph-mon[23158]: pgmap v117: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:42 smithi016 ceph-mon[22010]: pgmap v118: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:42 smithi079 ceph-mon[23158]: pgmap v118: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:43 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11105.104812 secs 2022-03-21T17:16:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:43 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11105.104812 secs 2022-03-21T17:16:45.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:44 smithi016 ceph-mon[22010]: pgmap v119: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:44 smithi079 ceph-mon[23158]: pgmap v119: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:46 smithi016 ceph-mon[22010]: pgmap v120: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:46 smithi079 ceph-mon[23158]: pgmap v120: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:48 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11110.104935 secs 2022-03-21T17:16:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:48 smithi016 ceph-mon[22010]: pgmap v121: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:16:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:16:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:16:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:48 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11110.104935 secs 2022-03-21T17:16:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:48 smithi079 ceph-mon[23158]: pgmap v121: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:49.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:16:49.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:16:49.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:16:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:50 smithi016 ceph-mon[22010]: pgmap v122: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:50 smithi079 ceph-mon[23158]: pgmap v122: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:51.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: 2022-03-21T17:16:51.532+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:51.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:16:51.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: yield 2022-03-21T17:16:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:16:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:16:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:16:51.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:51.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:16:51.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:16:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:51.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:52 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:16:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:16:53.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:53.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:53.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:53.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:16:53.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:53.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:16:53.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:53.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:16:53.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:53.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:53.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:53.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:16:53.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:53.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:53.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:53.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:52 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:16:53.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:52 smithi016 ceph-mon[22010]: pgmap v123: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:52 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:16:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:16:53.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:53.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:53.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:53.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:16:53.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:53.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:16:53.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:53.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:16:53.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:53.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:53.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:53.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:16:53.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:53.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:53.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:53.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:52 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:16:53.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:52 smithi079 ceph-mon[23158]: pgmap v123: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:53 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11115.105067 secs 2022-03-21T17:16:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:53 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11115.105067 secs 2022-03-21T17:16:54.907 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 2022-03-21T17:16:54.604+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:54.908 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:16:54.908 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:54.908 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: yield 2022-03-21T17:16:54.909 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:54.909 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:54.909 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:54.910 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:16:54.910 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:54.910 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:16:54.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:54.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:16:54.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:54.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:16:54.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:54.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:16:54.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:54.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:54.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:54.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 2022-03-21T17:16:54.605+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:16:54.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:16:54.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:54.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: yield 2022-03-21T17:16:54.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:54.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:16:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:16:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:16:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:16:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:16:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 2022-03-21T17:16:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:16:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: 2022-03-21T17:16:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:16:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:16:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:16:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:16:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:16:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:16:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:16:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:16:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:16:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:16:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:16:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:16:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:16:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:16:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:16:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:16:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:16:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:16:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:16:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:16:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:16:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:16:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise 2022-03-21T17:16:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:16:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:16:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:16:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:16:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:16:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:54 smithi016 ceph-mon[22010]: pgmap v124: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:54 smithi079 ceph-mon[23158]: pgmap v124: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:56.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:16:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:16:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:16:56.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:56.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:16:56.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:16:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:55 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:16:56.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:56.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:16:56.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:56.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:56.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:56.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:16:56.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:56.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:16:56.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:56.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:16:56.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:56.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:56.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:56.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:16:56.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:56.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:56.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:16:56.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:16:56.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:16:56.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:16:56.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:16:56.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:16:56.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:16:56.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:16:56.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:16:56.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:16:56.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:16:56.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:16:56.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:16:56.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:16:56.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:16:56.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:16:56.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:16:56.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:16:56.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:16:56.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:16:56.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:16:56.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:16:56.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:16:56.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:16:56.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:16:56.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:16:56.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:16:56.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:16:56.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:16:56.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:16:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:16:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:16:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:16:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:16:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:55 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:16:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:16:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:16:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:16:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:16:56.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:16:56.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:16:56.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:16:56.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:16:56.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:16:56.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:16:56.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:16:56.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:16:56.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:16:56.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:16:56.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:16:56.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:16:56.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:56.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:16:56.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:16:56.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:16:56.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:16:56.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:16:56.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:16:56.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:16:56.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:16:56.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:16:56.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:16:56.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:16:56.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:16:56.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:16:56.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:16:56.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:16:56.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:16:56.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:16:56.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:16:56.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:16:56.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:16:56.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:16:56.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:16:56.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:16:56.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:16:56.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:16:56.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:16:56.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:16:56.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:16:56.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:16:56.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:16:56.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:16:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:56 smithi016 ceph-mon[22010]: pgmap v125: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:56 smithi079 ceph-mon[23158]: pgmap v125: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11120.105187 secs 2022-03-21T17:16:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: pgmap v126: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:16:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:16:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:16:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:16:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11120.105187 secs 2022-03-21T17:16:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: pgmap v126: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:16:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:16:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:16:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:16:59.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:17:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:00 smithi016 ceph-mon[22010]: pgmap v127: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:00 smithi079 ceph-mon[23158]: pgmap v127: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:02 smithi016 ceph-mon[22010]: pgmap v128: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:02 smithi079 ceph-mon[23158]: pgmap v128: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:03 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11125.105338 secs 2022-03-21T17:17:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:03 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11125.105338 secs 2022-03-21T17:17:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:04 smithi016 ceph-mon[22010]: pgmap v129: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:04 smithi079 ceph-mon[23158]: pgmap v129: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:06 smithi016 ceph-mon[22010]: pgmap v130: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:07.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:06 smithi079 ceph-mon[23158]: pgmap v130: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:08 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11130.105457 secs 2022-03-21T17:17:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:08 smithi016 ceph-mon[22010]: pgmap v131: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:08 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11130.105457 secs 2022-03-21T17:17:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:08 smithi079 ceph-mon[23158]: pgmap v131: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:10 smithi016 ceph-mon[22010]: pgmap v132: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:10 smithi079 ceph-mon[23158]: pgmap v132: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:12 smithi016 ceph-mon[22010]: pgmap v133: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:12 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11135.105538 secs 2022-03-21T17:17:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:12 smithi079 ceph-mon[23158]: pgmap v133: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:12 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11135.105538 secs 2022-03-21T17:17:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:14 smithi016 ceph-mon[22010]: pgmap v134: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:14 smithi079 ceph-mon[23158]: pgmap v134: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:16 smithi016 ceph-mon[22010]: pgmap v135: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:16 smithi079 ceph-mon[23158]: pgmap v135: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:18 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11140.105659 secs 2022-03-21T17:17:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:18 smithi016 ceph-mon[22010]: pgmap v136: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:18 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11140.105659 secs 2022-03-21T17:17:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:18 smithi079 ceph-mon[23158]: pgmap v136: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:20 smithi016 ceph-mon[22010]: pgmap v137: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:20 smithi079 ceph-mon[23158]: pgmap v137: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:22 smithi016 ceph-mon[22010]: pgmap v138: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:22 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11145.105758 secs 2022-03-21T17:17:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:22 smithi079 ceph-mon[23158]: pgmap v138: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:22 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11145.105758 secs 2022-03-21T17:17:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:24 smithi079 ceph-mon[23158]: pgmap v139: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:24 smithi016 ceph-mon[22010]: pgmap v139: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:26 smithi079 ceph-mon[23158]: pgmap v140: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:26 smithi016 ceph-mon[22010]: pgmap v140: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:27 smithi079 ceph-mon[23158]: 40 slow requests, 0 included below; oldest blocked for > 11150.105875 secs 2022-03-21T17:17:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:27 smithi079 ceph-mon[23158]: evicting unresponsive client smithi002:0 (14964), after 337.722 seconds 2022-03-21T17:17:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:27 smithi079 ceph-mon[23158]: Evicting (and blocklisting) client session 14964 (v1:192.168.0.1:0/1390208746) 2022-03-21T17:17:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:27 smithi079 ceph-mon[23158]: from='mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]: dispatch 2022-03-21T17:17:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:27 smithi079 ceph-mon[23158]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]: dispatch 2022-03-21T17:17:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:27 smithi016 ceph-mon[22010]: 40 slow requests, 0 included below; oldest blocked for > 11150.105875 secs 2022-03-21T17:17:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:27 smithi016 ceph-mon[22010]: evicting unresponsive client smithi002:0 (14964), after 337.722 seconds 2022-03-21T17:17:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:27 smithi016 ceph-mon[22010]: Evicting (and blocklisting) client session 14964 (v1:192.168.0.1:0/1390208746) 2022-03-21T17:17:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:27 smithi016 ceph-mon[22010]: from='mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]: dispatch 2022-03-21T17:17:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:27 smithi016 ceph-mon[22010]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]: dispatch 2022-03-21T17:17:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:28 smithi079 ceph-mon[23158]: pgmap v141: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:28 smithi079 ceph-mon[23158]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]': finished 2022-03-21T17:17:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:28 smithi079 ceph-mon[23158]: osdmap e94: 12 total, 8 up, 12 in 2022-03-21T17:17:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:28 smithi016 ceph-mon[22010]: pgmap v141: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 1483/4890 objects degraded (30.327%) 2022-03-21T17:17:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:28 smithi016 ceph-mon[22010]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"v1:192.168.0.1:0/1390208746"}]': finished 2022-03-21T17:17:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:28 smithi016 ceph-mon[22010]: osdmap e94: 12 total, 8 up, 12 in 2022-03-21T17:17:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:29 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2022-03-21T17:17:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:29 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2022-03-21T17:17:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:30 smithi016 ceph-mon[22010]: pgmap v143: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:30 smithi079 ceph-mon[23158]: pgmap v143: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:32 smithi016 ceph-mon[22010]: pgmap v144: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:32 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11154.989888 secs 2022-03-21T17:17:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:32 smithi079 ceph-mon[23158]: pgmap v144: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:32 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11154.989888 secs 2022-03-21T17:17:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:34 smithi016 ceph-mon[22010]: pgmap v145: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:34 smithi079 ceph-mon[23158]: pgmap v145: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:35 smithi016 ceph-mon[22010]: MDS health message cleared (mds.?): Client smithi002:0 failing to respond to capability release 2022-03-21T17:17:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:35 smithi079 ceph-mon[23158]: MDS health message cleared (mds.?): Client smithi002:0 failing to respond to capability release 2022-03-21T17:17:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:36 smithi016 ceph-mon[22010]: pgmap v146: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:36 smithi016 ceph-mon[22010]: Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2022-03-21T17:17:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:17:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:17:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:36 smithi079 ceph-mon[23158]: pgmap v146: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:36 smithi079 ceph-mon[23158]: Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2022-03-21T17:17:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:17:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:17:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11159.990025 secs 2022-03-21T17:17:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:38 smithi016 ceph-mon[22010]: pgmap v147: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11159.990025 secs 2022-03-21T17:17:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:38 smithi079 ceph-mon[23158]: pgmap v147: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:40 smithi079 ceph-mon[23158]: pgmap v148: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:40 smithi016 ceph-mon[22010]: pgmap v148: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s wr, 1 op/s; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:42 smithi079 ceph-mon[23158]: pgmap v149: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:42 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11164.990131 secs 2022-03-21T17:17:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:42 smithi016 ceph-mon[22010]: pgmap v149: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:42 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11164.990131 secs 2022-03-21T17:17:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:44 smithi079 ceph-mon[23158]: pgmap v150: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:44 smithi016 ceph-mon[22010]: pgmap v150: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:46 smithi079 ceph-mon[23158]: pgmap v151: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:46 smithi016 ceph-mon[22010]: pgmap v151: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11169.990252 secs 2022-03-21T17:17:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:48 smithi016 ceph-mon[22010]: pgmap v152: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11169.990252 secs 2022-03-21T17:17:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:48 smithi079 ceph-mon[23158]: pgmap v152: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:50 smithi079 ceph-mon[23158]: pgmap v153: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:50 smithi016 ceph-mon[22010]: pgmap v153: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:52 smithi079 ceph-mon[23158]: pgmap v154: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:52 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11174.990330 secs 2022-03-21T17:17:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:52 smithi016 ceph-mon[22010]: pgmap v154: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:52 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11174.990330 secs 2022-03-21T17:17:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:54 smithi079 ceph-mon[23158]: pgmap v155: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:55.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:17:55.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:17:55.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:54 smithi016 ceph-mon[22010]: pgmap v155: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:17:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:17:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:17:57.200 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:56 smithi016 ceph-mon[22010]: pgmap v156: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:56 smithi079 ceph-mon[23158]: pgmap v156: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:57.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: 2022-03-21T17:17:57.196+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:17:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:17:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: yield 2022-03-21T17:17:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:17:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:17:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:17:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:17:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:17:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:17:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:17:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:17:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:17:57.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:17:57.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:17:57.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:17:57.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:17:57.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:17:57.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:17:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:17:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:17:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:17:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:17:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:17:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:17:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:17:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:17:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:17:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:17:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:17:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:17:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:17:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:17:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:17:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:17:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:17:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:17:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11179.990430 secs 2022-03-21T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: pgmap v157: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:17:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:17:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:17:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:17:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:17:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:17:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:17:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:17:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:17:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:17:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:17:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:17:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:17:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:17:59.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:17:59.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:17:59.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:17:59.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:17:59.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:17:59.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:17:59.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:17:59.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:17:59.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:17:59.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11179.990430 secs 2022-03-21T17:17:59.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: pgmap v157: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:17:59.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:17:59.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:17:59.235 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:17:59.235 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:18:00.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 2022-03-21T17:18:00.268+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:00.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:18:00.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:00.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: yield 2022-03-21T17:18:00.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:00.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:00.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:00.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:18:00.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:00.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:18:00.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:00.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:18:00.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:00.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:18:00.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:00.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:18:00.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:00.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:00.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:00.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 2022-03-21T17:18:00.269+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:18:00.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:18:00.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:00.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: yield 2022-03-21T17:18:00.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:00.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:00.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:00.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:18:00.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:00.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:18:00.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:00.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:18:00.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:00.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:18:00.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:00.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:18:00.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:00.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:00.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:00.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 2022-03-21T17:18:00.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:18:00.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: 2022-03-21T17:18:00.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:18:00.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:18:00.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:18:00.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:18:00.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:18:00.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:18:00.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:18:00.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:18:00.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:18:00.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:18:00.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:18:00.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:18:00.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:18:00.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:18:00.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:18:00.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:18:00.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:18:00.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:18:00.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:18:00.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:18:00.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:18:00.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:18:00.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise 2022-03-21T17:18:00.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:18:00.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:18:00.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:18:00.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:18:00.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:18:00 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:00 smithi079 ceph-mon[23158]: pgmap v158: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:00 smithi016 ceph-mon[22010]: pgmap v158: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:18:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:18:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:18:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:18:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:18:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:18:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:01 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:18:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:18:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:18:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:18:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:02.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:18:02.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:02.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:18:02.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:02.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:18:02.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:18:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:18:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:18:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:18:02.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:18:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:18:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:18:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:18:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:18:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:18:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:18:02.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:18:02.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:18:02.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:18:02.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:18:02.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:18:02.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:18:02.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:18:02.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:18:02.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:18:02.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:18:02.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:18:02.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:18:02.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:18:02.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:18:02.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:18:02.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:18:02.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:18:02.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:18:02.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:18:02.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:02.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:02.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:02.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:18:02.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:02.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:18:02.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:02.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:18:02.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:02.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:18:02.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:02.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:18:02.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:02.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:02.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:01 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:18:02.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:18:02.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:18:02.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:18:02.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:18:02.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:18:02.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:18:02.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:18:02.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:18:02.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:18:02.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:18:02.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:18:02.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:18:02.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:18:02.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:18:02.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:18:02.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:18:02.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:02.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:18:02.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:18:02.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:18:02.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:18:02.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:18:02.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:18:02.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:18:02.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:18:02.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:18:02.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:18:02.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:18:02.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:18:02.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:18:02.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:18:02.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:18:02.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:18:02.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:18:02.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:18:02.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:18:02.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:18:02.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:18:02.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:18:02.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:18:02.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:18:02.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:18:02.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:18:02.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:18:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:02 smithi079 ceph-mon[23158]: pgmap v159: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:02 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11184.990547 secs 2022-03-21T17:18:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:02 smithi016 ceph-mon[22010]: pgmap v159: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:02 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11184.990547 secs 2022-03-21T17:18:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:04 smithi079 ceph-mon[23158]: pgmap v160: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:04 smithi016 ceph-mon[22010]: pgmap v160: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:06 smithi079 ceph-mon[23158]: pgmap v161: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:06 smithi016 ceph-mon[22010]: pgmap v161: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:08 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11189.990684 secs 2022-03-21T17:18:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:08 smithi016 ceph-mon[22010]: pgmap v162: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:08 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11189.990684 secs 2022-03-21T17:18:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:08 smithi079 ceph-mon[23158]: pgmap v162: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:10 smithi079 ceph-mon[23158]: pgmap v163: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:10 smithi016 ceph-mon[22010]: pgmap v163: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:12 smithi079 ceph-mon[23158]: pgmap v164: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:12 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11194.990821 secs 2022-03-21T17:18:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:12 smithi016 ceph-mon[22010]: pgmap v164: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:12 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11194.990821 secs 2022-03-21T17:18:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:14 smithi079 ceph-mon[23158]: pgmap v165: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:14 smithi016 ceph-mon[22010]: pgmap v165: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:16 smithi079 ceph-mon[23158]: pgmap v166: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:16 smithi016 ceph-mon[22010]: pgmap v166: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:18 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11199.990947 secs 2022-03-21T17:18:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:18 smithi016 ceph-mon[22010]: pgmap v167: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:18 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11199.990947 secs 2022-03-21T17:18:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:18 smithi079 ceph-mon[23158]: pgmap v167: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:20 smithi079 ceph-mon[23158]: pgmap v168: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:20 smithi016 ceph-mon[22010]: pgmap v168: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:22 smithi079 ceph-mon[23158]: pgmap v169: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:22 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11204.991065 secs 2022-03-21T17:18:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:22 smithi016 ceph-mon[22010]: pgmap v169: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:22 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11204.991065 secs 2022-03-21T17:18:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:24 smithi079 ceph-mon[23158]: pgmap v170: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:24 smithi016 ceph-mon[22010]: pgmap v170: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:26 smithi079 ceph-mon[23158]: pgmap v171: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:26 smithi016 ceph-mon[22010]: pgmap v171: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:28 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11209.991195 secs 2022-03-21T17:18:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:28 smithi016 ceph-mon[22010]: pgmap v172: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:28 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11209.991195 secs 2022-03-21T17:18:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:28 smithi079 ceph-mon[23158]: pgmap v172: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:30 smithi079 ceph-mon[23158]: pgmap v173: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:30 smithi016 ceph-mon[22010]: pgmap v173: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:32 smithi079 ceph-mon[23158]: pgmap v174: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:32 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11214.991303 secs 2022-03-21T17:18:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:32 smithi016 ceph-mon[22010]: pgmap v174: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:32 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11214.991303 secs 2022-03-21T17:18:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:34 smithi079 ceph-mon[23158]: pgmap v175: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:34 smithi016 ceph-mon[22010]: pgmap v175: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:36 smithi079 ceph-mon[23158]: pgmap v176: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:36 smithi016 ceph-mon[22010]: pgmap v176: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11219.991422 secs 2022-03-21T17:18:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:38 smithi016 ceph-mon[22010]: pgmap v177: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11219.991422 secs 2022-03-21T17:18:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:38 smithi079 ceph-mon[23158]: pgmap v177: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:40 smithi079 ceph-mon[23158]: pgmap v178: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:40 smithi016 ceph-mon[22010]: pgmap v178: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:42 smithi079 ceph-mon[23158]: pgmap v179: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:42 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11224.991537 secs 2022-03-21T17:18:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:42 smithi016 ceph-mon[22010]: pgmap v179: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:42 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11224.991537 secs 2022-03-21T17:18:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:44 smithi079 ceph-mon[23158]: pgmap v180: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:44 smithi016 ceph-mon[22010]: pgmap v180: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:46 smithi079 ceph-mon[23158]: pgmap v181: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:46 smithi016 ceph-mon[22010]: pgmap v181: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11229.991656 secs 2022-03-21T17:18:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:48 smithi016 ceph-mon[22010]: pgmap v182: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11229.991656 secs 2022-03-21T17:18:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:48 smithi079 ceph-mon[23158]: pgmap v182: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:50 smithi079 ceph-mon[23158]: pgmap v183: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:50 smithi016 ceph-mon[22010]: pgmap v183: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:52 smithi079 ceph-mon[23158]: pgmap v184: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:52 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11234.991771 secs 2022-03-21T17:18:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:52 smithi016 ceph-mon[22010]: pgmap v184: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:53.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:52 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11234.991771 secs 2022-03-21T17:18:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:54 smithi079 ceph-mon[23158]: pgmap v185: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:54 smithi016 ceph-mon[22010]: pgmap v185: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:56 smithi079 ceph-mon[23158]: pgmap v186: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:56 smithi016 ceph-mon[22010]: pgmap v186: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11239.991910 secs 2022-03-21T17:18:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:18:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:18:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: pgmap v187: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:18:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:18:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11239.991910 secs 2022-03-21T17:18:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:18:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:18:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: pgmap v187: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:18:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:18:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:19:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:00 smithi079 ceph-mon[23158]: pgmap v188: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:00 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:19:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:00 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:19:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:00 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:19:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:00 smithi016 ceph-mon[22010]: pgmap v188: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:00 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:19:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:00 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:19:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:00 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:19:02.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: 2022-03-21T17:19:01.708+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:02.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:19:02.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:02.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: yield 2022-03-21T17:19:02.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:02.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:02.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:02.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:19:02.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:02.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:19:02.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:02.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:19:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:19:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:19:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:02.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:19:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:19:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:19:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:19:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:19:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:19:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:02 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:19:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:02 smithi079 ceph-mon[23158]: pgmap v189: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:02 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11244.992052 secs 2022-03-21T17:19:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:19:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:19:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:19:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:19:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:19:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:03.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:03.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:19:03.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:03.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:03.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:03.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:02 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:19:03.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:02 smithi016 ceph-mon[22010]: pgmap v189: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:03.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:02 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11244.992052 secs 2022-03-21T17:19:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:04 smithi016 ceph-mon[22010]: pgmap v190: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:05.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 2022-03-21T17:19:04.781+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:05.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:19:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: yield 2022-03-21T17:19:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:19:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:19:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:19:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:19:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:19:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 2022-03-21T17:19:04.781+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:19:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:19:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: yield 2022-03-21T17:19:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:05.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:19:05.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:05.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:19:05.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:05.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:19:05.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:05.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:19:05.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:19:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 2022-03-21T17:19:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:19:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: 2022-03-21T17:19:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:19:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:19:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:19:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:19:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:19:05.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:19:05.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:19:05.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:19:05.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:19:05.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:19:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:19:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:19:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:19:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:19:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:19:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:19:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:19:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:19:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:19:05.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:19:05.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:19:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:19:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise 2022-03-21T17:19:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:19:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:19:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:19:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:19:05.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:19:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:04 smithi079 ceph-mon[23158]: pgmap v190: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:19:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:19:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:19:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:19:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:19:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:05 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:19:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:19:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:19:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:19:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:19:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:19:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:19:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:19:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:19:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:19:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:19:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:19:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:19:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:19:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:19:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:19:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:19:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:19:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:19:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:19:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:19:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:19:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:19:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:19:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:19:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:19:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:19:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:19:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:19:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:19:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:19:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:19:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:19:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:19:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:19:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:19:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:19:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:19:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:19:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:19:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:05 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:19:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:19:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:19:06.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:19:06.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:19:06.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:19:06.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:19:06.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:19:06.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:19:06.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:19:06.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:19:06.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:19:06.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:19:06.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:19:06.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:19:06.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:19:06.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:19:06.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:06.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:19:06.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:19:06.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:19:06.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:19:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:19:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:19:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:19:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:19:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:19:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:19:06.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:19:06.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:19:06.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:19:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:19:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:19:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:19:06.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:19:06.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:19:06.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:19:06.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:19:06.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:19:06.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:19:06.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:19:06.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:19:06.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:19:06.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:19:06.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:19:06.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:19:06.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:19:06.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:19:06.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:19:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:06 smithi079 ceph-mon[23158]: pgmap v191: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:06 smithi016 ceph-mon[22010]: pgmap v191: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:08 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11249.992171 secs 2022-03-21T17:19:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:08 smithi016 ceph-mon[22010]: pgmap v192: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:08 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11249.992171 secs 2022-03-21T17:19:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:08 smithi079 ceph-mon[23158]: pgmap v192: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:10 smithi079 ceph-mon[23158]: pgmap v193: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:10 smithi016 ceph-mon[22010]: pgmap v193: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:12 smithi079 ceph-mon[23158]: pgmap v194: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:12 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11254.992285 secs 2022-03-21T17:19:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:12 smithi016 ceph-mon[22010]: pgmap v194: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:12 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11254.992285 secs 2022-03-21T17:19:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:14 smithi079 ceph-mon[23158]: pgmap v195: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:14 smithi016 ceph-mon[22010]: pgmap v195: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:16 smithi079 ceph-mon[23158]: pgmap v196: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:16 smithi016 ceph-mon[22010]: pgmap v196: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:18 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11259.992427 secs 2022-03-21T17:19:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:18 smithi016 ceph-mon[22010]: pgmap v197: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:18 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11259.992427 secs 2022-03-21T17:19:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:18 smithi079 ceph-mon[23158]: pgmap v197: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:20 smithi079 ceph-mon[23158]: pgmap v198: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:20 smithi016 ceph-mon[22010]: pgmap v198: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:22 smithi079 ceph-mon[23158]: pgmap v199: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:22 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11264.992546 secs 2022-03-21T17:19:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:22 smithi016 ceph-mon[22010]: pgmap v199: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:22 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11264.992546 secs 2022-03-21T17:19:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:24 smithi079 ceph-mon[23158]: pgmap v200: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:24 smithi016 ceph-mon[22010]: pgmap v200: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:26 smithi016 ceph-mon[22010]: pgmap v201: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:26 smithi079 ceph-mon[23158]: pgmap v201: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:28 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11269.992672 secs 2022-03-21T17:19:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:28 smithi016 ceph-mon[22010]: pgmap v202: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:28 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11269.992672 secs 2022-03-21T17:19:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:28 smithi079 ceph-mon[23158]: pgmap v202: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:30 smithi079 ceph-mon[23158]: pgmap v203: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:30 smithi016 ceph-mon[22010]: pgmap v203: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:32 smithi079 ceph-mon[23158]: pgmap v204: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:32 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11274.992786 secs 2022-03-21T17:19:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:32 smithi016 ceph-mon[22010]: pgmap v204: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:32 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11274.992786 secs 2022-03-21T17:19:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:34 smithi016 ceph-mon[22010]: pgmap v205: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:34 smithi079 ceph-mon[23158]: pgmap v205: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:36 smithi016 ceph-mon[22010]: pgmap v206: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:36 smithi079 ceph-mon[23158]: pgmap v206: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11279.992927 secs 2022-03-21T17:19:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:38 smithi016 ceph-mon[22010]: pgmap v207: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11279.992927 secs 2022-03-21T17:19:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:38 smithi079 ceph-mon[23158]: pgmap v207: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:40 smithi016 ceph-mon[22010]: pgmap v208: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:40 smithi079 ceph-mon[23158]: pgmap v208: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:42 smithi016 ceph-mon[22010]: pgmap v209: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:42 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11284.993044 secs 2022-03-21T17:19:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:42 smithi079 ceph-mon[23158]: pgmap v209: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:42 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11284.993044 secs 2022-03-21T17:19:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:44 smithi016 ceph-mon[22010]: pgmap v210: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:44 smithi079 ceph-mon[23158]: pgmap v210: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:46 smithi016 ceph-mon[22010]: pgmap v211: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:47.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:46 smithi079 ceph-mon[23158]: pgmap v211: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11289.993168 secs 2022-03-21T17:19:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:48 smithi016 ceph-mon[22010]: pgmap v212: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11289.993168 secs 2022-03-21T17:19:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:48 smithi079 ceph-mon[23158]: pgmap v212: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:50 smithi016 ceph-mon[22010]: pgmap v213: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:50 smithi079 ceph-mon[23158]: pgmap v213: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:52 smithi016 ceph-mon[22010]: pgmap v214: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:52 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11294.993291 secs 2022-03-21T17:19:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:52 smithi079 ceph-mon[23158]: pgmap v214: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:52 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11294.993291 secs 2022-03-21T17:19:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:54 smithi016 ceph-mon[22010]: pgmap v215: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:54 smithi079 ceph-mon[23158]: pgmap v215: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:56 smithi016 ceph-mon[22010]: pgmap v216: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:56 smithi079 ceph-mon[23158]: pgmap v216: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11299.993407 secs 2022-03-21T17:19:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:19:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:19:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:19:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:19:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:19:58 smithi016 ceph-mon[22010]: pgmap v217: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11299.993407 secs 2022-03-21T17:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:19:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:19:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:19:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:19:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:19:58 smithi079 ceph-mon[23158]: pgmap v217: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pgmap v218: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 24 pgs inactive; Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized 2022-03-21T17:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T17:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: mds.l(mds.0): 1 slow requests are blocked > 180 secs 2022-03-21T17:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:20:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:20:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] OSD_DOWN: 4 osds down 2022-03-21T17:20:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: osd.0 (root=default,host=smithi002) is down 2022-03-21T17:20:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: osd.3 (root=default,host=smithi002) is down 2022-03-21T17:20:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: osd.6 (root=default,host=smithi002) is down 2022-03-21T17:20:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: osd.9 (root=default,host=smithi002) is down 2022-03-21T17:20:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-03-21T17:20:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: host smithi002 (root=default) (4 osds) is down 2022-03-21T17:20:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 24 pgs inactive 2022-03-21T17:20:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:20:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.4 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [2] 2022-03-21T17:20:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.5 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.6 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.9 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [10] 2022-03-21T17:20:01.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.e is stuck inactive for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.19 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.1e is stuck inactive for 7m, current state undersized+degraded+peered, last acting [5] 2022-03-21T17:20:01.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.4 is stuck inactive for 3h, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.5 is stuck inactive for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:20:01.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.b is stuck inactive for 3h, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.12 is stuck inactive for 3h, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.17 is stuck inactive for 3h, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.18 is stuck inactive for 3h, current state undersized+peered, last acting [11] 2022-03-21T17:20:01.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.19 is stuck inactive for 3h, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1f is stuck inactive for 3h, current state undersized+peered, last acting [1] 2022-03-21T17:20:01.427 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.22 is stuck inactive for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.427 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.23 is stuck inactive for 3h, current state undersized+peered, last acting [4] 2022-03-21T17:20:01.427 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.24 is stuck inactive for 3h, current state undersized+peered, last acting [4] 2022-03-21T17:20:01.428 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.28 is stuck inactive for 7m, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.428 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2b is stuck inactive for 3h, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2d is stuck inactive for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3b is stuck inactive for 7m, current state undersized+peered, last acting [8] 2022-03-21T17:20:01.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: [WRN] PG_DEGRADED: Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized 2022-03-21T17:20:01.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.2 is stuck undersized for 7m, current state active+undersized+degraded, last acting [7,1] 2022-03-21T17:20:01.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.3 is stuck undersized for 7m, current state active+undersized+degraded, last acting [4,10] 2022-03-21T17:20:01.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.4 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [2] 2022-03-21T17:20:01.431 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.5 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.431 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.6 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.431 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.7 is stuck undersized for 7m, current state active+undersized+degraded, last acting [4,11] 2022-03-21T17:20:01.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.8 is stuck undersized for 7m, current state active+undersized+degraded, last acting [1,2] 2022-03-21T17:20:01.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.9 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [10] 2022-03-21T17:20:01.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.a is stuck undersized for 7m, current state active+undersized+degraded, last acting [2,8] 2022-03-21T17:20:01.433 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.10 is stuck undersized for 7m, current state active+undersized+degraded, last acting [8,2] 2022-03-21T17:20:01.433 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.12 is stuck undersized for 7m, current state active+undersized+degraded, last acting [11,7] 2022-03-21T17:20:01.433 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.13 is stuck undersized for 7m, current state active+undersized+degraded, last acting [2,10] 2022-03-21T17:20:01.434 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.15 is stuck undersized for 7m, current state active+undersized+degraded, last acting [5,10] 2022-03-21T17:20:01.434 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.17 is stuck undersized for 7m, current state active+undersized+degraded, last acting [10,5] 2022-03-21T17:20:01.434 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.19 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.435 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.1a is stuck undersized for 7m, current state active+undersized+degraded, last acting [10,8] 2022-03-21T17:20:01.435 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.1e is stuck undersized for 7m, current state undersized+degraded+peered, last acting [5] 2022-03-21T17:20:01.435 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 2.1f is stuck undersized for 7m, current state active+undersized+degraded, last acting [7,1] 2022-03-21T17:20:01.436 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1 is stuck undersized for 7m, current state active+undersized, last acting [4,2] 2022-03-21T17:20:01.436 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2 is stuck undersized for 7m, current state active+undersized, last acting [7,1] 2022-03-21T17:20:01.437 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3 is stuck undersized for 7m, current state active+undersized, last acting [10,4] 2022-03-21T17:20:01.437 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.4 is stuck undersized for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.437 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.5 is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.438 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.7 is stuck undersized for 7m, current state active+undersized, last acting [11,7] 2022-03-21T17:20:01.438 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.8 is stuck undersized for 7m, current state active+undersized, last acting [11,2] 2022-03-21T17:20:01.438 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.b is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.12 is stuck undersized for 7m, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.13 is stuck undersized for 7m, current state active+undersized, last acting [7,4] 2022-03-21T17:20:01.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.16 is stuck undersized for 7m, current state active+undersized, last acting [8,10] 2022-03-21T17:20:01.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.17 is stuck undersized for 7m, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.18 is stuck undersized for 7m, current state undersized+peered, last acting [11] 2022-03-21T17:20:01.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.19 is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1a is stuck undersized for 7m, current state active+undersized, last acting [8,7] 2022-03-21T17:20:01.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1b is stuck undersized for 7m, current state active+undersized, last acting [5,4] 2022-03-21T17:20:01.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1c is stuck undersized for 7m, current state active+undersized, last acting [1,10] 2022-03-21T17:20:01.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1e is stuck undersized for 7m, current state active+undersized, last acting [2,5] 2022-03-21T17:20:01.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.1f is stuck undersized for 7m, current state undersized+peered, last acting [1] 2022-03-21T17:20:01.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2d is stuck undersized for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.443 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2e is stuck undersized for 7m, current state active+undersized, last acting [1,10] 2022-03-21T17:20:01.443 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.2f is stuck undersized for 7m, current state active+undersized, last acting [11,1] 2022-03-21T17:20:01.444 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.30 is stuck undersized for 7m, current state active+undersized, last acting [1,7] 2022-03-21T17:20:01.444 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.31 is stuck undersized for 7m, current state active+undersized, last acting [4,2] 2022-03-21T17:20:01.444 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.32 is stuck undersized for 7m, current state active+undersized, last acting [8,10] 2022-03-21T17:20:01.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.33 is stuck undersized for 7m, current state active+undersized, last acting [10,5] 2022-03-21T17:20:01.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.38 is stuck undersized for 7m, current state active+undersized, last acting [5,10] 2022-03-21T17:20:01.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.39 is stuck undersized for 7m, current state active+undersized, last acting [7,5] 2022-03-21T17:20:01.446 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3a is stuck undersized for 7m, current state active+undersized, last acting [10,8] 2022-03-21T17:20:01.446 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3b is stuck undersized for 7m, current state undersized+peered, last acting [8] 2022-03-21T17:20:01.446 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3d is stuck undersized for 7m, current state active+undersized, last acting [2,1] 2022-03-21T17:20:01.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3e is stuck undersized for 7m, current state active+undersized, last acting [11,10] 2022-03-21T17:20:01.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:00 smithi016 ceph-mon[22010]: pg 3.3f is stuck undersized for 7m, current state active+undersized, last acting [11,1] 2022-03-21T17:20:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pgmap v218: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 24 pgs inactive; Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized 2022-03-21T17:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2022-03-21T17:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: mds.l(mds.0): 1 slow requests are blocked > 180 secs 2022-03-21T17:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] OSD_DOWN: 4 osds down 2022-03-21T17:20:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: osd.0 (root=default,host=smithi002) is down 2022-03-21T17:20:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: osd.3 (root=default,host=smithi002) is down 2022-03-21T17:20:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: osd.6 (root=default,host=smithi002) is down 2022-03-21T17:20:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: osd.9 (root=default,host=smithi002) is down 2022-03-21T17:20:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-03-21T17:20:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: host smithi002 (root=default) (4 osds) is down 2022-03-21T17:20:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 24 pgs inactive 2022-03-21T17:20:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:20:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.4 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [2] 2022-03-21T17:20:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.5 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.6 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.9 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [10] 2022-03-21T17:20:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.e is stuck inactive for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.19 is stuck inactive for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.1e is stuck inactive for 7m, current state undersized+degraded+peered, last acting [5] 2022-03-21T17:20:01.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.4 is stuck inactive for 3h, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.5 is stuck inactive for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:20:01.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.b is stuck inactive for 3h, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.12 is stuck inactive for 3h, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.17 is stuck inactive for 3h, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.18 is stuck inactive for 3h, current state undersized+peered, last acting [11] 2022-03-21T17:20:01.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.19 is stuck inactive for 3h, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.485 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1f is stuck inactive for 3h, current state undersized+peered, last acting [1] 2022-03-21T17:20:01.485 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.22 is stuck inactive for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.485 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.23 is stuck inactive for 3h, current state undersized+peered, last acting [4] 2022-03-21T17:20:01.486 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.24 is stuck inactive for 3h, current state undersized+peered, last acting [4] 2022-03-21T17:20:01.486 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.28 is stuck inactive for 7m, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.486 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2b is stuck inactive for 3h, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2d is stuck inactive for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3b is stuck inactive for 7m, current state undersized+peered, last acting [8] 2022-03-21T17:20:01.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: [WRN] PG_DEGRADED: Degraded data redundancy: 1487/4908 objects degraded (30.297%), 22 pgs degraded, 65 pgs undersized 2022-03-21T17:20:01.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.2 is stuck undersized for 7m, current state active+undersized+degraded, last acting [7,1] 2022-03-21T17:20:01.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.3 is stuck undersized for 7m, current state active+undersized+degraded, last acting [4,10] 2022-03-21T17:20:01.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.4 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [2] 2022-03-21T17:20:01.489 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.5 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.490 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.6 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [8] 2022-03-21T17:20:01.490 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.7 is stuck undersized for 7m, current state active+undersized+degraded, last acting [4,11] 2022-03-21T17:20:01.490 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.8 is stuck undersized for 7m, current state active+undersized+degraded, last acting [1,2] 2022-03-21T17:20:01.491 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.9 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [10] 2022-03-21T17:20:01.491 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.a is stuck undersized for 7m, current state active+undersized+degraded, last acting [2,8] 2022-03-21T17:20:01.491 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.10 is stuck undersized for 7m, current state active+undersized+degraded, last acting [8,2] 2022-03-21T17:20:01.492 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.12 is stuck undersized for 7m, current state active+undersized+degraded, last acting [11,7] 2022-03-21T17:20:01.492 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.13 is stuck undersized for 7m, current state active+undersized+degraded, last acting [2,10] 2022-03-21T17:20:01.492 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.15 is stuck undersized for 7m, current state active+undersized+degraded, last acting [5,10] 2022-03-21T17:20:01.493 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.17 is stuck undersized for 7m, current state active+undersized+degraded, last acting [10,5] 2022-03-21T17:20:01.493 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.19 is stuck undersized for 7m, current state undersized+degraded+peered, last acting [4] 2022-03-21T17:20:01.493 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.1a is stuck undersized for 7m, current state active+undersized+degraded, last acting [10,8] 2022-03-21T17:20:01.494 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.1e is stuck undersized for 7m, current state undersized+degraded+peered, last acting [5] 2022-03-21T17:20:01.494 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 2.1f is stuck undersized for 7m, current state active+undersized+degraded, last acting [7,1] 2022-03-21T17:20:01.494 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1 is stuck undersized for 7m, current state active+undersized, last acting [4,2] 2022-03-21T17:20:01.495 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2 is stuck undersized for 7m, current state active+undersized, last acting [7,1] 2022-03-21T17:20:01.495 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3 is stuck undersized for 7m, current state active+undersized, last acting [10,4] 2022-03-21T17:20:01.495 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.4 is stuck undersized for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.496 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.5 is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.496 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.7 is stuck undersized for 7m, current state active+undersized, last acting [11,7] 2022-03-21T17:20:01.496 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.8 is stuck undersized for 7m, current state active+undersized, last acting [11,2] 2022-03-21T17:20:01.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.b is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.12 is stuck undersized for 7m, current state undersized+peered, last acting [7] 2022-03-21T17:20:01.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.13 is stuck undersized for 7m, current state active+undersized, last acting [7,4] 2022-03-21T17:20:01.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.16 is stuck undersized for 7m, current state active+undersized, last acting [8,10] 2022-03-21T17:20:01.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.17 is stuck undersized for 7m, current state undersized+peered, last acting [5] 2022-03-21T17:20:01.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.18 is stuck undersized for 7m, current state undersized+peered, last acting [11] 2022-03-21T17:20:01.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.19 is stuck undersized for 7m, current state undersized+peered, last acting [10] 2022-03-21T17:20:01.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1a is stuck undersized for 7m, current state active+undersized, last acting [8,7] 2022-03-21T17:20:01.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1b is stuck undersized for 7m, current state active+undersized, last acting [5,4] 2022-03-21T17:20:01.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1c is stuck undersized for 7m, current state active+undersized, last acting [1,10] 2022-03-21T17:20:01.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1e is stuck undersized for 7m, current state active+undersized, last acting [2,5] 2022-03-21T17:20:01.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.1f is stuck undersized for 7m, current state undersized+peered, last acting [1] 2022-03-21T17:20:01.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2d is stuck undersized for 7m, current state undersized+peered, last acting [2] 2022-03-21T17:20:01.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2e is stuck undersized for 7m, current state active+undersized, last acting [1,10] 2022-03-21T17:20:01.503 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.2f is stuck undersized for 7m, current state active+undersized, last acting [11,1] 2022-03-21T17:20:01.503 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.30 is stuck undersized for 7m, current state active+undersized, last acting [1,7] 2022-03-21T17:20:01.503 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.31 is stuck undersized for 7m, current state active+undersized, last acting [4,2] 2022-03-21T17:20:01.504 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.32 is stuck undersized for 7m, current state active+undersized, last acting [8,10] 2022-03-21T17:20:01.504 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.33 is stuck undersized for 7m, current state active+undersized, last acting [10,5] 2022-03-21T17:20:01.504 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.38 is stuck undersized for 7m, current state active+undersized, last acting [5,10] 2022-03-21T17:20:01.505 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.39 is stuck undersized for 7m, current state active+undersized, last acting [7,5] 2022-03-21T17:20:01.505 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3a is stuck undersized for 7m, current state active+undersized, last acting [10,8] 2022-03-21T17:20:01.505 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3b is stuck undersized for 7m, current state undersized+peered, last acting [8] 2022-03-21T17:20:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3d is stuck undersized for 7m, current state active+undersized, last acting [2,1] 2022-03-21T17:20:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3e is stuck undersized for 7m, current state active+undersized, last acting [11,10] 2022-03-21T17:20:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:00 smithi079 ceph-mon[23158]: pg 3.3f is stuck undersized for 7m, current state active+undersized, last acting [11,1] 2022-03-21T17:20:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:02 smithi016 ceph-mon[22010]: pgmap v219: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:02 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11304.993523 secs 2022-03-21T17:20:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:02 smithi079 ceph-mon[23158]: pgmap v219: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:02 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11304.993523 secs 2022-03-21T17:20:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:04 smithi016 ceph-mon[22010]: pgmap v220: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:20:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:20:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:20:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:04 smithi079 ceph-mon[23158]: pgmap v220: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:20:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:20:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:20:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:06 smithi016 ceph-mon[22010]: pgmap v221: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:06 smithi079 ceph-mon[23158]: pgmap v221: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:08.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: 2022-03-21T17:20:07.884+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:08.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:20:08.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:08.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: yield 2022-03-21T17:20:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:08.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:20:08.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:08.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:20:08.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:08.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:20:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:20:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:20:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:08.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:08 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11309.993677 secs 2022-03-21T17:20:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:20:09.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:09.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:20:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:09.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:20:09.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:20:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:20:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:20:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:08 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:20:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:08 smithi016 ceph-mon[22010]: pgmap v222: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:08 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11309.993677 secs 2022-03-21T17:20:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:20:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:20:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:09.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:20:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:20:09.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:09.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:20:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:20:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:09.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:09.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:08 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:20:09.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:08 smithi079 ceph-mon[23158]: pgmap v222: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:10 smithi016 ceph-mon[22010]: pgmap v223: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 2022-03-21T17:20:10.956+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:20:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: yield 2022-03-21T17:20:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:20:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:20:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:20:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:20:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:20:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:11.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 2022-03-21T17:20:10.957+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:20:11.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:20:11.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:11.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: yield 2022-03-21T17:20:11.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:11.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:11.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:11.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:20:11.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:11.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:20:11.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:11.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:20:11.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:11.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:20:11.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:11.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:20:11.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:11.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:11.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:11.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 2022-03-21T17:20:11.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:20:11.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: 2022-03-21T17:20:11.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:20:11.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:20:11.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:20:11.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:20:11.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:20:11.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:20:11.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:20:11.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:20:11.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:20:11.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:20:11.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:20:11.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:20:11.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:20:11.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:20:11.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:20:11.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:20:11.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:20:11.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:20:11.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:20:11.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:20:11.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:20:11.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:20:11.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise 2022-03-21T17:20:11.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:20:11.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:20:11.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:20:11.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:20:11.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:20:10 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:10 smithi079 ceph-mon[23158]: pgmap v223: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:20:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:12.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:20:12.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:12.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:20:12.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:12.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:20:12.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:12.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:12.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:12.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:20:12.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:12.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:12.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:11 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:20:12.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:12.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:20:12.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:12.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:12.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:12.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:20:12.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:12.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:20:12.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:12.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:20:12.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:12.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:12.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:12.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:20:12.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:12.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:12.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:20:12.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:20:12.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:20:12.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:20:12.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:20:12.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:20:12.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:20:12.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:20:12.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:20:12.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:20:12.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:20:12.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:20:12.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:20:12.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:20:12.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:20:12.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:20:12.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:20:12.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:20:12.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:20:12.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:20:12.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:20:12.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:20:12.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:20:12.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:20:12.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:20:12.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:20:12.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:20:12.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:20:12.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:20:12.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:20:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:20:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:20:12.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:12.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:20:12.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:12.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:12.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:20:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:12.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:11 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:20:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:20:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:20:12.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:20:12.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:20:12.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:20:12.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:20:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:20:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:20:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:20:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:20:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:20:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:20:12.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:20:12.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:20:12.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:20:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:20:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:20:12.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:20:12.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:20:12.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:20:12.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:20:12.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:20:12.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:20:12.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:20:12.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:20:12.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:20:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:20:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:20:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:20:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:20:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:20:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:20:12.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:20:12.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:20:12.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:20:12.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:20:12.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:20:12.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:20:12.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:20:12.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:20:12.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:20:12.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:20:12.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:20:12.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:20:12.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:20:12.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:20:12.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:20:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:12 smithi016 ceph-mon[22010]: pgmap v224: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:13.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:12 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11314.993743 secs 2022-03-21T17:20:13.569 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:12 smithi079 ceph-mon[23158]: pgmap v224: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:13.570 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:12 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11314.993743 secs 2022-03-21T17:20:15.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:14 smithi016 ceph-mon[22010]: pgmap v225: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:14 smithi079 ceph-mon[23158]: pgmap v225: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:16 smithi016 ceph-mon[22010]: pgmap v226: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:16 smithi079 ceph-mon[23158]: pgmap v226: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:18 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11319.993860 secs 2022-03-21T17:20:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:18 smithi016 ceph-mon[22010]: pgmap v227: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:18 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11319.993860 secs 2022-03-21T17:20:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:18 smithi079 ceph-mon[23158]: pgmap v227: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:20 smithi016 ceph-mon[22010]: pgmap v228: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:20 smithi079 ceph-mon[23158]: pgmap v228: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:22 smithi016 ceph-mon[22010]: pgmap v229: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:22 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11324.993975 secs 2022-03-21T17:20:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:22 smithi079 ceph-mon[23158]: pgmap v229: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:22 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11324.993975 secs 2022-03-21T17:20:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:24 smithi016 ceph-mon[22010]: pgmap v230: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:24 smithi079 ceph-mon[23158]: pgmap v230: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:26 smithi016 ceph-mon[22010]: pgmap v231: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:26 smithi079 ceph-mon[23158]: pgmap v231: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:28 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11329.994099 secs 2022-03-21T17:20:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:28 smithi016 ceph-mon[22010]: pgmap v232: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:28 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11329.994099 secs 2022-03-21T17:20:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:28 smithi079 ceph-mon[23158]: pgmap v232: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:30 smithi016 ceph-mon[22010]: pgmap v233: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:30 smithi079 ceph-mon[23158]: pgmap v233: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:32 smithi016 ceph-mon[22010]: pgmap v234: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:32 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11334.994217 secs 2022-03-21T17:20:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:32 smithi079 ceph-mon[23158]: pgmap v234: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:32 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11334.994217 secs 2022-03-21T17:20:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:35 smithi016 ceph-mon[22010]: pgmap v235: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:35 smithi079 ceph-mon[23158]: pgmap v235: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:36 smithi016 ceph-mon[22010]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-03-21T17:20:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:36 smithi079 ceph-mon[23158]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-03-21T17:20:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:37 smithi016 ceph-mon[22010]: pgmap v236: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:37 smithi079 ceph-mon[23158]: pgmap v236: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11339.994344 secs 2022-03-21T17:20:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:38 smithi016 ceph-mon[22010]: pgmap v237: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11339.994344 secs 2022-03-21T17:20:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:38 smithi079 ceph-mon[23158]: pgmap v237: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:41.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:40 smithi016 ceph-mon[22010]: pgmap v238: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:40 smithi079 ceph-mon[23158]: pgmap v238: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:43 smithi016 ceph-mon[22010]: pgmap v239: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:43 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11344.994464 secs 2022-03-21T17:20:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:43 smithi079 ceph-mon[23158]: pgmap v239: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:43 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11344.994464 secs 2022-03-21T17:20:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:45 smithi016 ceph-mon[22010]: pgmap v240: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:45 smithi079 ceph-mon[23158]: pgmap v240: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:47 smithi016 ceph-mon[22010]: pgmap v241: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:47 smithi079 ceph-mon[23158]: pgmap v241: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11349.994514 secs 2022-03-21T17:20:49.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:48 smithi016 ceph-mon[22010]: pgmap v242: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:49.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:48 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:49.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:48 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11349.994514 secs 2022-03-21T17:20:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:48 smithi079 ceph-mon[23158]: pgmap v242: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:48 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:48 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:51 smithi016 ceph-mon[22010]: pgmap v243: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:51 smithi079 ceph-mon[23158]: pgmap v243: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:53 smithi016 ceph-mon[22010]: pgmap v244: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:53 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11354.994635 secs 2022-03-21T17:20:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:53 smithi079 ceph-mon[23158]: pgmap v244: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:53 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11354.994635 secs 2022-03-21T17:20:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:55 smithi016 ceph-mon[22010]: pgmap v245: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:55 smithi079 ceph-mon[23158]: pgmap v245: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:57 smithi016 ceph-mon[22010]: pgmap v246: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:57 smithi079 ceph-mon[23158]: pgmap v246: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:20:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:20:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11359.994764 secs 2022-03-21T17:20:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:20:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:20:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:20:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:20:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:20:58 smithi016 ceph-mon[22010]: pgmap v247: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:20:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11359.994764 secs 2022-03-21T17:20:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:20:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:20:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:20:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:20:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:20:58 smithi079 ceph-mon[23158]: pgmap v247: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:00 smithi016 ceph-mon[22010]: pgmap v248: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:00 smithi079 ceph-mon[23158]: pgmap v248: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:03 smithi016 ceph-mon[22010]: pgmap v249: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:03 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11364.994888 secs 2022-03-21T17:21:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:03 smithi079 ceph-mon[23158]: pgmap v249: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:03 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11364.994888 secs 2022-03-21T17:21:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:05 smithi016 ceph-mon[22010]: pgmap v250: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:05 smithi079 ceph-mon[23158]: pgmap v250: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:07 smithi016 ceph-mon[22010]: pgmap v251: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:07 smithi079 ceph-mon[23158]: pgmap v251: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:08 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11369.994986 secs 2022-03-21T17:21:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:08 smithi016 ceph-mon[22010]: pgmap v252: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:08 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:08 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:08 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11369.994986 secs 2022-03-21T17:21:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:08 smithi079 ceph-mon[23158]: pgmap v252: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:08 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:08 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:11.415 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:11 smithi079 ceph-mon[23158]: pgmap v253: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:11.416 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:21:11.416 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:21:11.417 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:21:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:11 smithi016 ceph-mon[22010]: pgmap v253: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:11.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:21:11.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:21:11.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:21:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:13 smithi016 ceph-mon[22010]: pgmap v254: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:13 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11374.995108 secs 2022-03-21T17:21:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:13 smithi079 ceph-mon[23158]: pgmap v254: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:13 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11374.995108 secs 2022-03-21T17:21:14.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: 2022-03-21T17:21:14.060+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:14.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:21:14.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:14.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: yield 2022-03-21T17:21:14.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:14.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:14.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:14.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:21:14.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:14.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:21:14.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:14.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:21:14.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:14.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:21:14.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:14.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:21:14.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:14.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:14.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:15 smithi016 ceph-mon[22010]: pgmap v255: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:21:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:21:15.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:15.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:15.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:15.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:21:15.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:15.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:21:15.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:15.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:21:15.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:15.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:15.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:15.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:21:15.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:15.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:15.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:15.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:15 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:21:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:15 smithi079 ceph-mon[23158]: pgmap v255: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:15.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:21:15.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:15.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:21:15.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:15.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:15.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:15.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:21:15.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:15.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:21:15.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:15.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:21:15.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:15.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:15.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:15.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:21:15.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:15.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:15.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:15.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:15 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:21:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:17 smithi016 ceph-mon[22010]: pgmap v256: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:17.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:17.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 2022-03-21T17:21:17.132+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:17.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:21:17.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:17.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: yield 2022-03-21T17:21:17.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:17.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:17.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:17.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:21:17.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:17.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:21:17.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:17.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:21:17.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:17.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:21:17.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:17.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:21:17.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:17.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:17.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:17.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 2022-03-21T17:21:17.133+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:21:17.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:21:17.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:17.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: yield 2022-03-21T17:21:17.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:17.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:17.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:17.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:21:17.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:17.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:21:17.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:17.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:21:17.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:17.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:21:17.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:17.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:21:17.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:17.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:17.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:17.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 2022-03-21T17:21:17.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:21:17.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: 2022-03-21T17:21:17.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:21:17.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:21:17.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:21:17.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:21:17.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:21:17.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:21:17.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:21:17.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:21:17.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:21:17.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:21:17.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:21:17.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:21:17.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:21:17.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:21:17.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:21:17.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:21:17.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:21:17.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:21:17.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:21:17.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:21:17.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:21:17.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:21:17.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise 2022-03-21T17:21:17.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:21:17.459 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:21:17.459 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:21:17.459 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:21:17.460 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:21:17 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:17 smithi079 ceph-mon[23158]: pgmap v256: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:19.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:21:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:21:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:21:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:21:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:19.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:19.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:19.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:21:19.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:19.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:19.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:18 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:21:19.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:21:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:19.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:19.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:19.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:21:19.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:19.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:21:19.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:19.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:21:19.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:19.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:19.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:19.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:21:19.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:19.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:19.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:21:19.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:21:19.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:21:19.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:21:19.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:21:19.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:21:19.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:21:19.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:21:19.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:21:19.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:21:19.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:21:19.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:21:19.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:21:19.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:21:19.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:21:19.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:21:19.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:21:19.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:21:19.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:21:19.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:21:19.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:21:19.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:21:19.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:21:19.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:21:19.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:21:19.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:21:19.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:21:19.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:21:19.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:21:19.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:18 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11379.995234 secs 2022-03-21T17:21:19.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:18 smithi016 ceph-mon[22010]: pgmap v257: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:21:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:21:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:21:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:21:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:21:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:18 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:21:19.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:21:19.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:21:19.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:21:19.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:21:19.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:21:19.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:21:19.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:21:19.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:21:19.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:21:19.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:21:19.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:21:19.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:21:19.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:21:19.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:21:19.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:21:19.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:21:19.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:21:19.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:21:19.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:21:19.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:21:19.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:21:19.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:21:19.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:21:19.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:21:19.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:21:19.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:21:19.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:21:19.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:21:19.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:21:19.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:21:19.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:21:19.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:21:19.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:21:19.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:21:19.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:21:19.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:21:19.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:21:19.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:21:19.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:21:19.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:21:19.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:21:19.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:21:19.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:21:19.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:21:19.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:21:19.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:21:19.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:21:19.252 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:18 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11379.995234 secs 2022-03-21T17:21:19.252 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:18 smithi079 ceph-mon[23158]: pgmap v257: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:20 smithi016 ceph-mon[22010]: pgmap v258: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:20 smithi079 ceph-mon[23158]: pgmap v258: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:23 smithi016 ceph-mon[22010]: pgmap v259: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:23 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11384.995353 secs 2022-03-21T17:21:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:23 smithi079 ceph-mon[23158]: pgmap v259: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:23 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11384.995353 secs 2022-03-21T17:21:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:25 smithi016 ceph-mon[22010]: pgmap v260: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:25 smithi079 ceph-mon[23158]: pgmap v260: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:27 smithi016 ceph-mon[22010]: pgmap v261: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:27 smithi079 ceph-mon[23158]: pgmap v261: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:28 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11389.995484 secs 2022-03-21T17:21:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:28 smithi016 ceph-mon[22010]: pgmap v262: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:28 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11389.995484 secs 2022-03-21T17:21:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:28 smithi079 ceph-mon[23158]: pgmap v262: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:31 smithi016 ceph-mon[22010]: pgmap v263: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:31 smithi079 ceph-mon[23158]: pgmap v263: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:33 smithi016 ceph-mon[22010]: pgmap v264: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:33 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11394.995610 secs 2022-03-21T17:21:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:33 smithi079 ceph-mon[23158]: pgmap v264: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:33 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11394.995610 secs 2022-03-21T17:21:35.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:35 smithi016 ceph-mon[22010]: pgmap v265: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:35 smithi079 ceph-mon[23158]: pgmap v265: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:37 smithi016 ceph-mon[22010]: pgmap v266: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:37 smithi079 ceph-mon[23158]: pgmap v266: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11399.995735 secs 2022-03-21T17:21:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:38 smithi016 ceph-mon[22010]: pgmap v267: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11399.995735 secs 2022-03-21T17:21:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:38 smithi079 ceph-mon[23158]: pgmap v267: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:40 smithi016 ceph-mon[22010]: pgmap v268: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:40 smithi079 ceph-mon[23158]: pgmap v268: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:43 smithi016 ceph-mon[22010]: pgmap v269: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:43 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11404.995828 secs 2022-03-21T17:21:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:43 smithi079 ceph-mon[23158]: pgmap v269: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:43 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11404.995828 secs 2022-03-21T17:21:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:45 smithi016 ceph-mon[22010]: pgmap v270: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:45 smithi079 ceph-mon[23158]: pgmap v270: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:47 smithi016 ceph-mon[22010]: pgmap v271: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:47 smithi079 ceph-mon[23158]: pgmap v271: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11409.995978 secs 2022-03-21T17:21:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:48 smithi016 ceph-mon[22010]: pgmap v272: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:48 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:48 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11409.995978 secs 2022-03-21T17:21:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:48 smithi079 ceph-mon[23158]: pgmap v272: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:48 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:48 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:51 smithi016 ceph-mon[22010]: pgmap v273: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:51 smithi079 ceph-mon[23158]: pgmap v273: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:53 smithi016 ceph-mon[22010]: pgmap v274: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:53 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11414.996119 secs 2022-03-21T17:21:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:53 smithi079 ceph-mon[23158]: pgmap v274: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:53 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11414.996119 secs 2022-03-21T17:21:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:55 smithi016 ceph-mon[22010]: pgmap v275: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:55 smithi079 ceph-mon[23158]: pgmap v275: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:57 smithi016 ceph-mon[22010]: pgmap v276: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:57 smithi079 ceph-mon[23158]: pgmap v276: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:21:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:21:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11419.996270 secs 2022-03-21T17:21:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:21:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:21:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:21:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:21:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:21:58 smithi016 ceph-mon[22010]: pgmap v277: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:21:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11419.996270 secs 2022-03-21T17:21:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:21:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:21:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:21:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:21:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:21:58 smithi079 ceph-mon[23158]: pgmap v277: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:00 smithi016 ceph-mon[22010]: pgmap v278: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:00 smithi079 ceph-mon[23158]: pgmap v278: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:03 smithi016 ceph-mon[22010]: pgmap v279: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:03 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11424.996412 secs 2022-03-21T17:22:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:03 smithi079 ceph-mon[23158]: pgmap v279: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:03 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11424.996412 secs 2022-03-21T17:22:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:05 smithi016 ceph-mon[22010]: pgmap v280: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:05 smithi079 ceph-mon[23158]: pgmap v280: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:07 smithi016 ceph-mon[22010]: pgmap v281: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:07 smithi079 ceph-mon[23158]: pgmap v281: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:08 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11429.996540 secs 2022-03-21T17:22:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:08 smithi016 ceph-mon[22010]: pgmap v282: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:08 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:08 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:08 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11429.996540 secs 2022-03-21T17:22:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:08 smithi079 ceph-mon[23158]: pgmap v282: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:08 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:08 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:11 smithi016 ceph-mon[22010]: pgmap v283: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:11 smithi079 ceph-mon[23158]: pgmap v283: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:13 smithi016 ceph-mon[22010]: pgmap v284: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:13 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11434.996673 secs 2022-03-21T17:22:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:13 smithi079 ceph-mon[23158]: pgmap v284: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:13 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11434.996673 secs 2022-03-21T17:22:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:15 smithi016 ceph-mon[22010]: pgmap v285: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:15 smithi079 ceph-mon[23158]: pgmap v285: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:17 smithi016 ceph-mon[22010]: pgmap v286: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:17 smithi079 ceph-mon[23158]: pgmap v286: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:22:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:22:18.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:22:18.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:18 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11439.996740 secs 2022-03-21T17:22:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:22:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:22:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:22:18.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:18 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11439.996740 secs 2022-03-21T17:22:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:19 smithi016 ceph-mon[22010]: pgmap v287: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:19 smithi079 ceph-mon[23158]: pgmap v287: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:20.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: 2022-03-21T17:22:20.237+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:22:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:20.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: yield 2022-03-21T17:22:20.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:20.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:20.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:20.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:22:20.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:20.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:22:20.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:20.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:22:20.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:20.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:22:20.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:20.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:22:20.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:20.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:20.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:21 smithi016 ceph-mon[22010]: pgmap v288: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:21 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:21 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:21 smithi079 ceph-mon[23158]: pgmap v288: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:21 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:21 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:22 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:22:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:22:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:22:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:22:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:22:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:22:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:22 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:22:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:22 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:22:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:22.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:22:22.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:22.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:22.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:22.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:22:22.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:22.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:22:22.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:22.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:22:22.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:22.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:22.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:22.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:22:22.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:22.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:22.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:22.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:22 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:22:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:23 smithi016 ceph-mon[22010]: pgmap v289: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:23 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11444.996874 secs 2022-03-21T17:22:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 2022-03-21T17:22:23.309+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:22:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: yield 2022-03-21T17:22:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:22:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:22:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:22:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:22:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:22:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 2022-03-21T17:22:23.310+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:22:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:22:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: yield 2022-03-21T17:22:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:22:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:22:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:22:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:22:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:22:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 2022-03-21T17:22:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:22:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: 2022-03-21T17:22:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:22:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:22:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:22:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:22:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:22:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:22:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:22:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:22:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:22:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:22:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:22:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:22:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:22:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:22:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:22:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:22:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:22:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:22:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:22:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:22:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:22:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:22:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise 2022-03-21T17:22:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:22:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:22:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:22:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:22:23.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:22:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:23.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:23 smithi079 ceph-mon[23158]: pgmap v289: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:23.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:23 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11444.996874 secs 2022-03-21T17:22:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:25 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:25.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:22:25.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:25.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:25.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:25.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:22:25.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:25.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:22:25.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:25.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:22:25.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:25.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:25.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:25.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:22:25.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:25.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:25.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:25 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:22:25.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:25.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:22:25.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:25.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:25.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:25.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:22:25.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:25.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:22:25.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:25.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:22:25.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:25.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:25.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:25.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:22:25.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:25.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:25.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:22:25.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:22:25.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:22:25.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:22:25.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:22:25.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:22:25.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:22:25.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:22:25.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:22:25.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:22:25.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:22:25.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:22:25.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:22:25.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:22:25.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:22:25.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:22:25.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:22:25.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:22:25.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:22:25.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:22:25.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:22:25.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:22:25.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:22:25.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:22:25.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:22:25.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:22:25.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:22:25.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:22:25.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:22:25.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:25 smithi016 ceph-mon[22010]: pgmap v290: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:25.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:25.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:25 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:22:25.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:25.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:25.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:25.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:22:25.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:25.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:22:25.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:25.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:22:25.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:25.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:25.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:25.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:22:25.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:25.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:25.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:25 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:22:25.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:22:25.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:22:25.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:22:25.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:22:25.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:22:25.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:22:25.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:22:25.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:22:25.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:22:25.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:22:25.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:22:25.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:22:25.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:22:25.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:22:25.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:22:25.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:22:25.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:22:25.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:22:25.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:22:25.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:22:25.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:22:25.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:22:25.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:22:25.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:22:25.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:22:25.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:22:25.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:22:25.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:22:25.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:22:25.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:22:25.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:22:25.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:22:25.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:22:25.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:22:25.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:22:25.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:22:25.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:22:25.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:22:25.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:22:25.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:22:25.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:22:25.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:22:25.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:22:25.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:22:25.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:22:25.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:22:25.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:22:25.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:25 smithi079 ceph-mon[23158]: pgmap v290: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:25.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:25.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:27 smithi016 ceph-mon[22010]: pgmap v291: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:27 smithi079 ceph-mon[23158]: pgmap v291: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:28 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11449.996965 secs 2022-03-21T17:22:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:28 smithi016 ceph-mon[22010]: pgmap v292: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:28 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11449.996965 secs 2022-03-21T17:22:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:28 smithi079 ceph-mon[23158]: pgmap v292: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:31 smithi016 ceph-mon[22010]: pgmap v293: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:31 smithi079 ceph-mon[23158]: pgmap v293: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: pgmap v294: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11454.997100 secs 2022-03-21T17:22:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: Marking osd.0 out (has been down for 603 seconds) 2022-03-21T17:22:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: Marking osd.3 out (has been down for 603 seconds) 2022-03-21T17:22:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: Marking osd.9 out (has been down for 604 seconds) 2022-03-21T17:22:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: Health check update: 1 osds down (OSD_DOWN) 2022-03-21T17:22:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:33 smithi016 ceph-mon[22010]: osdmap e95: 12 total, 8 up, 9 in 2022-03-21T17:22:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: pgmap v294: 97 pgs: 7 undersized+degraded+peered, 15 active+undersized+degraded, 2 unknown, 15 undersized+peered, 28 active+undersized, 30 active+clean; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 1487/4908 objects degraded (30.297%) 2022-03-21T17:22:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11454.997100 secs 2022-03-21T17:22:33.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: Marking osd.0 out (has been down for 603 seconds) 2022-03-21T17:22:33.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: Marking osd.3 out (has been down for 603 seconds) 2022-03-21T17:22:33.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: Marking osd.9 out (has been down for 604 seconds) 2022-03-21T17:22:33.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: Health check update: 1 osds down (OSD_DOWN) 2022-03-21T17:22:33.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:33 smithi079 ceph-mon[23158]: osdmap e95: 12 total, 8 up, 9 in 2022-03-21T17:22:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:34 smithi016 ceph-mon[22010]: osdmap e96: 12 total, 8 up, 9 in 2022-03-21T17:22:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:34 smithi016 ceph-mon[22010]: pgmap v297: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:34 smithi079 ceph-mon[23158]: osdmap e96: 12 total, 8 up, 9 in 2022-03-21T17:22:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:34 smithi079 ceph-mon[23158]: pgmap v297: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:35 smithi016 ceph-mon[22010]: Health check update: Reduced data availability: 24 pgs inactive, 48 pgs peering (PG_AVAILABILITY) 2022-03-21T17:22:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:35 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 387/4908 objects degraded (7.885%), 7 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2022-03-21T17:22:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:35 smithi016 ceph-mon[22010]: osdmap e97: 12 total, 8 up, 9 in 2022-03-21T17:22:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:35 smithi079 ceph-mon[23158]: Health check update: Reduced data availability: 24 pgs inactive, 48 pgs peering (PG_AVAILABILITY) 2022-03-21T17:22:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:35 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 387/4908 objects degraded (7.885%), 7 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2022-03-21T17:22:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:35 smithi079 ceph-mon[23158]: osdmap e97: 12 total, 8 up, 9 in 2022-03-21T17:22:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:37.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:37.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:36 smithi016 ceph-mon[22010]: osdmap e98: 12 total, 8 up, 9 in 2022-03-21T17:22:37.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:36 smithi016 ceph-mon[22010]: pgmap v300: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:37.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:37.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:37.630 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:36 smithi079 ceph-mon[23158]: osdmap e98: 12 total, 8 up, 9 in 2022-03-21T17:22:37.630 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:36 smithi079 ceph-mon[23158]: pgmap v300: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:37 smithi016 ceph-mon[22010]: osdmap e99: 12 total, 8 up, 9 in 2022-03-21T17:22:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:37 smithi016 ceph-mon[22010]: Marking osd.6 out (has been down for 603 seconds) 2022-03-21T17:22:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:37 smithi016 ceph-mon[22010]: Health check cleared: OSD_DOWN (was: 1 osds down) 2022-03-21T17:22:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:37 smithi016 ceph-mon[22010]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-03-21T17:22:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:37 smithi016 ceph-mon[22010]: osdmap e100: 12 total, 8 up, 8 in 2022-03-21T17:22:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:37 smithi079 ceph-mon[23158]: osdmap e99: 12 total, 8 up, 9 in 2022-03-21T17:22:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:37 smithi079 ceph-mon[23158]: Marking osd.6 out (has been down for 603 seconds) 2022-03-21T17:22:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:37 smithi079 ceph-mon[23158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2022-03-21T17:22:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:37 smithi079 ceph-mon[23158]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-03-21T17:22:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:37 smithi079 ceph-mon[23158]: osdmap e100: 12 total, 8 up, 8 in 2022-03-21T17:22:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:38 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11459.997188 secs 2022-03-21T17:22:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:38 smithi016 ceph-mon[22010]: pgmap v303: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:38 smithi016 ceph-mon[22010]: osdmap e101: 12 total, 8 up, 8 in 2022-03-21T17:22:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:38 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11459.997188 secs 2022-03-21T17:22:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:38 smithi079 ceph-mon[23158]: pgmap v303: 97 pgs: 48 peering, 30 active+clean, 10 active+undersized, 2 unknown, 7 active+undersized+degraded; 133 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 387/4908 objects degraded (7.885%) 2022-03-21T17:22:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:38 smithi079 ceph-mon[23158]: osdmap e101: 12 total, 8 up, 8 in 2022-03-21T17:22:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:40 smithi016 ceph-mon[22010]: osdmap e102: 12 total, 8 up, 8 in 2022-03-21T17:22:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:40 smithi016 ceph-mon[22010]: pgmap v306: 97 pgs: 24 active+clean+wait, 1 active+recovery_wait+undersized+degraded+remapped+wait, 1 active+recovering+undersized+degraded+remapped+wait, 34 peering, 1 active+recovering+degraded+wait, 4 active+recovery_wait+degraded+wait, 30 active+clean, 2 unknown; 133 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 357/4908 objects degraded (7.274%); 5.1 MiB/s, 1.40k keys/s, 108 objects/s recovering 2022-03-21T17:22:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:40 smithi079 ceph-mon[23158]: osdmap e102: 12 total, 8 up, 8 in 2022-03-21T17:22:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:40 smithi079 ceph-mon[23158]: pgmap v306: 97 pgs: 24 active+clean+wait, 1 active+recovery_wait+undersized+degraded+remapped+wait, 1 active+recovering+undersized+degraded+remapped+wait, 34 peering, 1 active+recovering+degraded+wait, 4 active+recovery_wait+degraded+wait, 30 active+clean, 2 unknown; 133 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 357/4908 objects degraded (7.274%); 5.1 MiB/s, 1.40k keys/s, 108 objects/s recovering 2022-03-21T17:22:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:41 smithi016 ceph-mon[22010]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:22:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:41 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 357/4908 objects degraded (7.274%), 7 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:41 smithi079 ceph-mon[23158]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2022-03-21T17:22:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:41 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 357/4908 objects degraded (7.274%), 7 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:42 smithi016 ceph-mon[22010]: pgmap v307: 97 pgs: 24 active+clean+wait, 1 active+recovery_wait+undersized+degraded+remapped+wait, 1 active+recovering+undersized+degraded+remapped+wait, 34 peering, 1 active+recovering+degraded+wait, 4 active+recovery_wait+degraded+wait, 30 active+clean, 2 unknown; 133 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 357/4908 objects degraded (7.274%); 3.9 MiB/s, 1.09k keys/s, 84 objects/s recovering 2022-03-21T17:22:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:42 smithi016 ceph-mon[22010]: osdmap e103: 12 total, 8 up, 8 in 2022-03-21T17:22:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:42 smithi079 ceph-mon[23158]: pgmap v307: 97 pgs: 24 active+clean+wait, 1 active+recovery_wait+undersized+degraded+remapped+wait, 1 active+recovering+undersized+degraded+remapped+wait, 34 peering, 1 active+recovering+degraded+wait, 4 active+recovery_wait+degraded+wait, 30 active+clean, 2 unknown; 133 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 357/4908 objects degraded (7.274%); 3.9 MiB/s, 1.09k keys/s, 84 objects/s recovering 2022-03-21T17:22:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:42 smithi079 ceph-mon[23158]: osdmap e103: 12 total, 8 up, 8 in 2022-03-21T17:22:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:43 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11464.997269 secs 2022-03-21T17:22:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:43 smithi016 ceph-mon[22010]: osdmap e104: 12 total, 8 up, 8 in 2022-03-21T17:22:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:43 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11464.997269 secs 2022-03-21T17:22:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:43 smithi079 ceph-mon[23158]: osdmap e104: 12 total, 8 up, 8 in 2022-03-21T17:22:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:44 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:44 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:44 smithi016 ceph-mon[22010]: pgmap v310: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1.6 MiB/s, 434 keys/s, 33 objects/s recovering 2022-03-21T17:22:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:44 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:44 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:44 smithi079 ceph-mon[23158]: pgmap v310: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1.6 MiB/s, 434 keys/s, 33 objects/s recovering 2022-03-21T17:22:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:47 smithi016 ceph-mon[22010]: pgmap v311: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1.3 MiB/s, 363 keys/s, 27 objects/s recovering 2022-03-21T17:22:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:47 smithi079 ceph-mon[23158]: pgmap v311: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1.3 MiB/s, 363 keys/s, 27 objects/s recovering 2022-03-21T17:22:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:48 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11469.997372 secs 2022-03-21T17:22:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:48 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 986/4908 objects degraded (20.090%), 18 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:48 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11469.997372 secs 2022-03-21T17:22:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:48 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 986/4908 objects degraded (20.090%), 18 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:49 smithi016 ceph-mon[22010]: pgmap v312: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1024 KiB/s, 279 keys/s, 21 objects/s recovering 2022-03-21T17:22:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:49.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:49 smithi079 ceph-mon[23158]: pgmap v312: 97 pgs: 45 active+clean+wait, 4 active+recovery_wait+undersized+degraded+remapped+wait, 1 peering, 3 active+recovering+degraded+wait, 11 active+recovery_wait+degraded+wait, 31 active+clean, 2 unknown; 133 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 986/4908 objects degraded (20.090%); 1024 KiB/s, 279 keys/s, 21 objects/s recovering 2022-03-21T17:22:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:51 smithi016 ceph-mon[22010]: pgmap v313: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+wait, 1 active+recovering+undersized+degraded+remapped+wait, 6 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped+wait, 2 active+recovering+degraded+wait, 3 active+recovery_wait+degraded+wait, 60 active+clean, 2 unknown; 133 MiB data, 502 MiB used, 715 GiB / 715 GiB avail; 784/4908 objects degraded (15.974%); 1.5 MiB/s, 524 keys/s, 38 objects/s recovering 2022-03-21T17:22:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:51 smithi079 ceph-mon[23158]: pgmap v313: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+wait, 1 active+recovering+undersized+degraded+remapped+wait, 6 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped+wait, 2 active+recovering+degraded+wait, 3 active+recovery_wait+degraded+wait, 60 active+clean, 2 unknown; 133 MiB data, 502 MiB used, 715 GiB / 715 GiB avail; 784/4908 objects degraded (15.974%); 1.5 MiB/s, 524 keys/s, 38 objects/s recovering 2022-03-21T17:22:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:53 smithi016 ceph-mon[22010]: pgmap v314: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+wait, 1 active+recovering+undersized+degraded+remapped+wait, 6 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped+wait, 2 active+recovering+degraded+wait, 3 active+recovery_wait+degraded+wait, 60 active+clean, 2 unknown; 133 MiB data, 502 MiB used, 715 GiB / 715 GiB avail; 784/4908 objects degraded (15.974%); 1.3 MiB/s, 458 keys/s, 33 objects/s recovering 2022-03-21T17:22:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:53 smithi016 ceph-mon[22010]: 1 slow requests, 0 included below; oldest blocked for > 11474.997443 secs 2022-03-21T17:22:53.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:53 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 784/4908 objects degraded (15.974%), 15 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:53 smithi079 ceph-mon[23158]: pgmap v314: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+wait, 1 active+recovering+undersized+degraded+remapped+wait, 6 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped+wait, 2 active+recovering+degraded+wait, 3 active+recovery_wait+degraded+wait, 60 active+clean, 2 unknown; 133 MiB data, 502 MiB used, 715 GiB / 715 GiB avail; 784/4908 objects degraded (15.974%); 1.3 MiB/s, 458 keys/s, 33 objects/s recovering 2022-03-21T17:22:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:53 smithi079 ceph-mon[23158]: 1 slow requests, 0 included below; oldest blocked for > 11474.997443 secs 2022-03-21T17:22:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:53 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 784/4908 objects degraded (15.974%), 15 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:54 smithi016 ceph-mon[22010]: osdmap e105: 12 total, 8 up, 8 in 2022-03-21T17:22:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:54 smithi016 ceph-mon[22010]: pgmap v316: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.8 KiB/s wr, 3 op/s; 634/4911 objects degraded (12.910%); 1.2 MiB/s, 399 keys/s, 28 objects/s recovering 2022-03-21T17:22:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:54 smithi079 ceph-mon[23158]: osdmap e105: 12 total, 8 up, 8 in 2022-03-21T17:22:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:54 smithi079 ceph-mon[23158]: pgmap v316: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.8 KiB/s wr, 3 op/s; 634/4911 objects degraded (12.910%); 1.2 MiB/s, 399 keys/s, 28 objects/s recovering 2022-03-21T17:22:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:55 smithi016 ceph-mon[22010]: osdmap e106: 12 total, 8 up, 8 in 2022-03-21T17:22:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:55 smithi079 ceph-mon[23158]: osdmap e106: 12 total, 8 up, 8 in 2022-03-21T17:22:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:56 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:56 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:56 smithi016 ceph-mon[22010]: pgmap v318: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.7 KiB/s wr, 4 op/s; 634/4911 objects degraded (12.910%); 1.5 MiB/s, 499 keys/s, 36 objects/s recovering 2022-03-21T17:22:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:56 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:22:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:56 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:22:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:56 smithi079 ceph-mon[23158]: pgmap v318: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.7 KiB/s wr, 4 op/s; 634/4911 objects degraded (12.910%); 1.5 MiB/s, 499 keys/s, 36 objects/s recovering 2022-03-21T17:22:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:57 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 634/4911 objects degraded (12.910%), 12 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:57 smithi016 ceph-mon[22010]: osdmap e107: 12 total, 8 up, 8 in 2022-03-21T17:22:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:57 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 634/4911 objects degraded (12.910%), 12 pgs degraded (PG_DEGRADED) 2022-03-21T17:22:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:57 smithi079 ceph-mon[23158]: osdmap e107: 12 total, 8 up, 8 in 2022-03-21T17:22:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:22:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:22:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:22:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:22:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: pgmap v320: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 6.3 KiB/s wr, 5 op/s; 634/4911 objects degraded (12.910%); 1.3 MiB/s, 339 keys/s, 25 objects/s recovering 2022-03-21T17:22:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:58 smithi016 ceph-mon[22010]: osdmap e108: 12 total, 8 up, 8 in 2022-03-21T17:22:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:22:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:22:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:22:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:22:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: pgmap v320: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 6.3 KiB/s wr, 5 op/s; 634/4911 objects degraded (12.910%); 1.3 MiB/s, 339 keys/s, 25 objects/s recovering 2022-03-21T17:22:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:58 smithi079 ceph-mon[23158]: osdmap e108: 12 total, 8 up, 8 in 2022-03-21T17:23:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:22:59 smithi016 ceph-mon[22010]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-03-21T17:23:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:22:59 smithi079 ceph-mon[23158]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-03-21T17:23:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:00 smithi016 ceph-mon[22010]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2022-03-21T17:23:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:00 smithi016 ceph-mon[22010]: pgmap v322: 97 pgs: 1 peering, 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 7 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 518 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 170 B/s wr, 2 op/s; 561/4911 objects degraded (11.423%); 682 KiB/s, 153 keys/s, 11 objects/s recovering 2022-03-21T17:23:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:00 smithi079 ceph-mon[23158]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2022-03-21T17:23:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:00 smithi079 ceph-mon[23158]: pgmap v322: 97 pgs: 1 peering, 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 7 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 518 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 170 B/s wr, 2 op/s; 561/4911 objects degraded (11.423%); 682 KiB/s, 153 keys/s, 11 objects/s recovering 2022-03-21T17:23:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:03 smithi016 ceph-mon[22010]: pgmap v323: 97 pgs: 1 peering, 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 7 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 518 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 143 B/s wr, 2 op/s; 561/4911 objects degraded (11.423%); 573 KiB/s, 129 keys/s, 9 objects/s recovering 2022-03-21T17:23:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:03 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 561/4911 objects degraded (11.423%), 11 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:03 smithi079 ceph-mon[23158]: pgmap v323: 97 pgs: 1 peering, 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 7 active+recovery_wait+degraded, 82 active+clean, 2 unknown; 133 MiB data, 518 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 143 B/s wr, 2 op/s; 561/4911 objects degraded (11.423%); 573 KiB/s, 129 keys/s, 9 objects/s recovering 2022-03-21T17:23:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:03 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 561/4911 objects degraded (11.423%), 11 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:05 smithi016 ceph-mon[22010]: pgmap v324: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 127 B/s wr, 3 op/s; 449/4911 objects degraded (9.143%); 512 KiB/s, 308 keys/s, 23 objects/s recovering 2022-03-21T17:23:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:05 smithi079 ceph-mon[23158]: pgmap v324: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 127 B/s wr, 3 op/s; 449/4911 objects degraded (9.143%); 512 KiB/s, 308 keys/s, 23 objects/s recovering 2022-03-21T17:23:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:07 smithi016 ceph-mon[22010]: pgmap v325: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 125 B/s wr, 3 op/s; 449/4911 objects degraded (9.143%); 502 KiB/s, 302 keys/s, 22 objects/s recovering 2022-03-21T17:23:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:07 smithi079 ceph-mon[23158]: pgmap v325: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 125 B/s wr, 3 op/s; 449/4911 objects degraded (9.143%); 502 KiB/s, 302 keys/s, 22 objects/s recovering 2022-03-21T17:23:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:08 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 449/4911 objects degraded (9.143%), 9 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:08 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 449/4911 objects degraded (9.143%), 9 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:09.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:09 smithi016 ceph-mon[22010]: pgmap v326: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 4.0 MiB/s rd, 102 B/s wr, 2 op/s; 449/4911 objects degraded (9.143%); 410 KiB/s, 246 keys/s, 18 objects/s recovering 2022-03-21T17:23:09.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:09.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:09 smithi079 ceph-mon[23158]: pgmap v326: 97 pgs: 1 active+clean+wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 85 active+clean, 2 unknown; 133 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 4.0 MiB/s rd, 102 B/s wr, 2 op/s; 449/4911 objects degraded (9.143%); 410 KiB/s, 246 keys/s, 18 objects/s recovering 2022-03-21T17:23:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:09.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:11 smithi016 ceph-mon[22010]: pgmap v327: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 89 active+clean, 2 unknown; 133 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 1 op/s; 329/4911 objects degraded (6.699%); 735 KiB/s, 276 keys/s, 20 objects/s recovering 2022-03-21T17:23:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:11 smithi079 ceph-mon[23158]: pgmap v327: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 89 active+clean, 2 unknown; 133 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 1 op/s; 329/4911 objects degraded (6.699%); 735 KiB/s, 276 keys/s, 20 objects/s recovering 2022-03-21T17:23:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:13 smithi016 ceph-mon[22010]: pgmap v328: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 89 active+clean, 2 unknown; 133 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 329/4911 objects degraded (6.699%); 682 KiB/s, 257 keys/s, 19 objects/s recovering 2022-03-21T17:23:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:13 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 329/4911 objects degraded (6.699%), 6 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:13 smithi016 ceph-mon[22010]: osdmap e109: 12 total, 8 up, 8 in 2022-03-21T17:23:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:13 smithi079 ceph-mon[23158]: pgmap v328: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 89 active+clean, 2 unknown; 133 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 329/4911 objects degraded (6.699%); 682 KiB/s, 257 keys/s, 19 objects/s recovering 2022-03-21T17:23:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:13 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 329/4911 objects degraded (6.699%), 6 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:13 smithi079 ceph-mon[23158]: osdmap e109: 12 total, 8 up, 8 in 2022-03-21T17:23:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:14 smithi016 ceph-mon[22010]: osdmap e110: 12 total, 8 up, 8 in 2022-03-21T17:23:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:14 smithi016 ceph-mon[22010]: pgmap v331: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 198/4911 objects degraded (4.032%); 2.0 MiB/s, 421 keys/s, 30 objects/s recovering 2022-03-21T17:23:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:14 smithi079 ceph-mon[23158]: osdmap e110: 12 total, 8 up, 8 in 2022-03-21T17:23:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:14 smithi079 ceph-mon[23158]: pgmap v331: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 198/4911 objects degraded (4.032%); 2.0 MiB/s, 421 keys/s, 30 objects/s recovering 2022-03-21T17:23:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:16 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:16 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:16 smithi016 ceph-mon[22010]: pgmap v332: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 198/4911 objects degraded (4.032%); 2.0 MiB/s, 421 keys/s, 30 objects/s recovering 2022-03-21T17:23:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:16 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:16 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:16 smithi079 ceph-mon[23158]: pgmap v332: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 198/4911 objects degraded (4.032%); 2.0 MiB/s, 421 keys/s, 30 objects/s recovering 2022-03-21T17:23:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:17 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 198/4911 objects degraded (4.032%), 4 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:17 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 198/4911 objects degraded (4.032%), 4 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:18 smithi016 ceph-mon[22010]: pgmap v333: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s; 198/4911 objects degraded (4.032%); 1024 KiB/s, 228 keys/s, 16 objects/s recovering 2022-03-21T17:23:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:18 smithi079 ceph-mon[23158]: pgmap v333: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+degraded, 1 active+recovery_wait+degraded, 90 active+clean, 2 unknown; 133 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s; 198/4911 objects degraded (4.032%); 1024 KiB/s, 228 keys/s, 16 objects/s recovering 2022-03-21T17:23:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:19 smithi016 ceph-mon[22010]: osdmap e111: 12 total, 8 up, 8 in 2022-03-21T17:23:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:19 smithi079 ceph-mon[23158]: osdmap e111: 12 total, 8 up, 8 in 2022-03-21T17:23:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:20 smithi016 ceph-mon[22010]: osdmap e112: 12 total, 8 up, 8 in 2022-03-21T17:23:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:20 smithi016 ceph-mon[22010]: pgmap v336: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 92 active+clean, 2 unknown; 133 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 1 op/s; 72/4911 objects degraded (1.466%); 2.6 MiB/s, 306 keys/s, 22 objects/s recovering 2022-03-21T17:23:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:20 smithi079 ceph-mon[23158]: osdmap e112: 12 total, 8 up, 8 in 2022-03-21T17:23:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:20 smithi079 ceph-mon[23158]: pgmap v336: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 92 active+clean, 2 unknown; 133 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 1 op/s; 72/4911 objects degraded (1.466%); 2.6 MiB/s, 306 keys/s, 22 objects/s recovering 2022-03-21T17:23:23.356 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:23 smithi016 ceph-mon[22010]: pgmap v337: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 92 active+clean, 2 unknown; 133 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s; 72/4911 objects degraded (1.466%); 2.0 MiB/s, 235 keys/s, 17 objects/s recovering 2022-03-21T17:23:23.357 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:23 smithi016 ceph-mon[22010]: Health check update: Degraded data redundancy: 72/4911 objects degraded (1.466%), 2 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:23.358 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:23 smithi079 ceph-mon[23158]: pgmap v337: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 92 active+clean, 2 unknown; 133 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s; 72/4911 objects degraded (1.466%); 2.0 MiB/s, 235 keys/s, 17 objects/s recovering 2022-03-21T17:23:23.358 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:23 smithi079 ceph-mon[23158]: Health check update: Degraded data redundancy: 72/4911 objects degraded (1.466%), 2 pgs degraded (PG_DEGRADED) 2022-03-21T17:23:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:23:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:23:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:23:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:23:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:23:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:23:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:25 smithi016 ceph-mon[22010]: pgmap v338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 3.0 MiB/s, 353 keys/s, 26 objects/s recovering 2022-03-21T17:23:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:25 smithi016 ceph-mon[22010]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 72/4911 objects degraded (1.466%), 2 pgs degraded) 2022-03-21T17:23:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:25 smithi079 ceph-mon[23158]: pgmap v338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 3.0 MiB/s, 353 keys/s, 26 objects/s recovering 2022-03-21T17:23:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:25 smithi079 ceph-mon[23158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 72/4911 objects degraded (1.466%), 2 pgs degraded) 2022-03-21T17:23:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:26.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: 2022-03-21T17:23:26.413+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:26.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:23:26.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:26.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: yield 2022-03-21T17:23:26.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:26.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:26.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:26.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:23:26.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:26.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:23:26.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:26.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:23:26.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:26.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:23:26.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:26.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:23:26.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:26.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:26.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:23:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:23:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: pgmap v339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 3.0 MiB/s, 353 keys/s, 26 objects/s recovering 2022-03-21T17:23:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:26 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:23:27.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:23:27.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: pgmap v339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 3.0 MiB/s, 353 keys/s, 26 objects/s recovering 2022-03-21T17:23:27.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:27.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:26 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:23:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:23:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:23:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:23:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:23:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:23:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:23:28.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:28.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:28.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:28.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:23:28.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:28.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:23:28.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:28.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:23:28.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:28.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:28.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:28.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:23:28.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:28.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:28.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:28.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:27 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:23:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:23:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:23:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:23:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:23:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:28.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:23:28.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:28.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:23:28.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:28.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:28.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:28.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:23:28.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:28.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:23:28.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:28.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:23:28.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:28.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:28.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:28.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:23:28.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:28.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:28.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:28.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:27 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:23:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:28 smithi016 ceph-mon[22010]: pgmap v340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 560 B/s rd, 1 op/s; 897 KiB/s, 103 keys/s, 7 objects/s recovering 2022-03-21T17:23:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:28 smithi079 ceph-mon[23158]: pgmap v340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 560 B/s rd, 1 op/s; 897 KiB/s, 103 keys/s, 7 objects/s recovering 2022-03-21T17:23:29.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 2022-03-21T17:23:29.485+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:29.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:23:29.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:29.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: yield 2022-03-21T17:23:29.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:29.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:23:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:23:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:23:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:23:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:23:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:29.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:29.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 2022-03-21T17:23:29.486+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:23:29.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:23:29.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:29.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: yield 2022-03-21T17:23:29.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:29.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:29.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:29.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:23:29.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:29.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:23:29.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:29.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:23:29.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:29.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:23:29.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:29.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:23:29.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:29.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:29.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:29.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 2022-03-21T17:23:29.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:23:29.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: 2022-03-21T17:23:29.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:23:29.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:23:29.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:23:29.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:23:29.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:23:29.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:23:29.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:23:29.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:23:29.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:23:29.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:23:29.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:23:29.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:23:29.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:23:29.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:23:29.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:23:29.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:23:29.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:23:29.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:23:29.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:23:29.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:23:29.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:23:29.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:23:29.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise 2022-03-21T17:23:29.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:23:29.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:23:29.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:23:29.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:23:29.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:23:29 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:23:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:23:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:23:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:23:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:23:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:23:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:23:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:23:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:23:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:23:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:23:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:23:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:23:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:23:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:23:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:23:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:23:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:23:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:23:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:23:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:23:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:23:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:23:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:23:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:23:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:23:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:23:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:23:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:23:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:23:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:23:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:23:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:23:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:23:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:23:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:23:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:23:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:23:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:23:31.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:23:31.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:31 smithi016 ceph-mon[22010]: pgmap v341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 1 op/s; 809 KiB/s, 93 keys/s, 7 objects/s recovering 2022-03-21T17:23:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:23:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:23:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:23:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:23:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:23:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:23:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:23:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:23:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:23:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:23:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:23:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:23:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:23:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:23:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:23:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:23:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:23:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:23:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:23:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:23:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:23:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:23:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:23:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:23:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:23:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:23:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:23:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:23:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:23:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:23:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:23:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:23:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:23:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:23:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:23:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:23:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:23:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:23:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:23:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:23:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:23:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:23:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:23:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:23:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:23:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:23:31.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:23:31.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:23:31.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:23:31.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:23:31.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:23:31.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:23:31.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:23:31.501 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:31 smithi079 ceph-mon[23158]: pgmap v341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 1 op/s; 809 KiB/s, 93 keys/s, 7 objects/s recovering 2022-03-21T17:23:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:33 smithi016 ceph-mon[22010]: pgmap v342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 682 KiB/s, 78 keys/s, 5 objects/s recovering 2022-03-21T17:23:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:33 smithi079 ceph-mon[23158]: pgmap v342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 682 KiB/s, 78 keys/s, 5 objects/s recovering 2022-03-21T17:23:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:35 smithi016 ceph-mon[22010]: pgmap v343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 682 KiB/s, 78 keys/s, 5 objects/s recovering 2022-03-21T17:23:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:35 smithi079 ceph-mon[23158]: pgmap v343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 682 KiB/s, 78 keys/s, 5 objects/s recovering 2022-03-21T17:23:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:37 smithi016 ceph-mon[22010]: pgmap v344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:37 smithi079 ceph-mon[23158]: pgmap v344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:38 smithi016 ceph-mon[22010]: pgmap v345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:38 smithi079 ceph-mon[23158]: pgmap v345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:40 smithi016 ceph-mon[22010]: pgmap v346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:40 smithi079 ceph-mon[23158]: pgmap v346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:43 smithi016 ceph-mon[22010]: pgmap v347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:43 smithi079 ceph-mon[23158]: pgmap v347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:45 smithi016 ceph-mon[22010]: pgmap v348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:45 smithi079 ceph-mon[23158]: pgmap v348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:47 smithi016 ceph-mon[22010]: pgmap v349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:47 smithi079 ceph-mon[23158]: pgmap v349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:49 smithi016 ceph-mon[22010]: pgmap v350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:49 smithi079 ceph-mon[23158]: pgmap v350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:51 smithi016 ceph-mon[22010]: pgmap v351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:51 smithi079 ceph-mon[23158]: pgmap v351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:53 smithi016 ceph-mon[22010]: pgmap v352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:53 smithi079 ceph-mon[23158]: pgmap v352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:55 smithi016 ceph-mon[22010]: pgmap v353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:55 smithi079 ceph-mon[23158]: pgmap v353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:23:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:57 smithi016 ceph-mon[22010]: pgmap v354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:57 smithi079 ceph-mon[23158]: pgmap v354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:23:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:23:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:23:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:23:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:23:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:23:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:23:58 smithi016 ceph-mon[22010]: pgmap v355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:23:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:23:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:23:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:23:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:23:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:23:58 smithi079 ceph-mon[23158]: pgmap v355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:00 smithi016 ceph-mon[22010]: pgmap v356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:00 smithi079 ceph-mon[23158]: pgmap v356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:03 smithi016 ceph-mon[22010]: pgmap v357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:03 smithi079 ceph-mon[23158]: pgmap v357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:05 smithi016 ceph-mon[22010]: pgmap v358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:05 smithi079 ceph-mon[23158]: pgmap v358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:07 smithi016 ceph-mon[22010]: pgmap v359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:07 smithi079 ceph-mon[23158]: pgmap v359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:09 smithi016 ceph-mon[22010]: pgmap v360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:09.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:09.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:09.571 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:09 smithi079 ceph-mon[23158]: pgmap v360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:09.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:09.572 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:11 smithi016 ceph-mon[22010]: pgmap v361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:11.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:11 smithi079 ceph-mon[23158]: pgmap v361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:13 smithi016 ceph-mon[22010]: pgmap v362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:13 smithi079 ceph-mon[23158]: pgmap v362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:15 smithi016 ceph-mon[22010]: pgmap v363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:15 smithi079 ceph-mon[23158]: pgmap v363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:17 smithi016 ceph-mon[22010]: pgmap v364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:17 smithi079 ceph-mon[23158]: pgmap v364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:18 smithi016 ceph-mon[22010]: pgmap v365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:18 smithi079 ceph-mon[23158]: pgmap v365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:20 smithi016 ceph-mon[22010]: pgmap v366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:20 smithi079 ceph-mon[23158]: pgmap v366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:23 smithi016 ceph-mon[22010]: pgmap v367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:23 smithi079 ceph-mon[23158]: pgmap v367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:25 smithi016 ceph-mon[22010]: pgmap v368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:25 smithi079 ceph-mon[23158]: pgmap v368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:27 smithi016 ceph-mon[22010]: pgmap v369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:27 smithi079 ceph-mon[23158]: pgmap v369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:29 smithi016 ceph-mon[22010]: pgmap v370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:29 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:29 smithi079 ceph-mon[23158]: pgmap v370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:29 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:24:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:24:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:24:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:24:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:24:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:24:30.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: 2022-03-21T17:24:30.413+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:30.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:24:30.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:30.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: yield 2022-03-21T17:24:30.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:30.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:30.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:30.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:24:30.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:30.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:24:30.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:30.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:24:30.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:30.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:24:30.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:30.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:24:30.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:30.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:30.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:31 smithi016 ceph-mon[22010]: pgmap v371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:31 smithi079 ceph-mon[23158]: pgmap v371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:32 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:24:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:24:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:32.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:32.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:32.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:24:32.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:32.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:24:32.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:32.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:24:32.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:32.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:32.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:32.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:24:32.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:32.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:32.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:32.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:32 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:24:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:32 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:32.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:24:32.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:32.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:24:32.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:32.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:32.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:32.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:24:32.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:32.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:24:32.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:32.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:24:32.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:32.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:32.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:32.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:24:32.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:32.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:32.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:32.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:32 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:24:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:33 smithi016 ceph-mon[22010]: pgmap v372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:33 smithi079 ceph-mon[23158]: pgmap v372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:33.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 2022-03-21T17:24:33.485+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:33.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:24:33.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:33.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: yield 2022-03-21T17:24:33.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:33.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:24:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:33.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:24:33.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:33.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:24:33.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:33.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:24:33.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:33.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:24:33.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:33.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:33.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:33.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 2022-03-21T17:24:33.486+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:24:33.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:24:33.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:33.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: yield 2022-03-21T17:24:33.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:33.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:33.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:33.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:24:33.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:33.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:24:33.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:33.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:24:33.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:33.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:24:33.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:33.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:24:33.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:33.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:33.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:33.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 2022-03-21T17:24:33.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:24:33.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: 2022-03-21T17:24:33.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:24:33.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:24:33.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:24:33.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:24:33.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:24:33.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:24:33.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:24:33.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:24:33.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:24:33.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:24:33.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:24:33.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:24:33.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:24:33.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:24:33.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:24:33.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:24:33.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:24:33.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:24:33.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:24:33.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:24:33.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:24:33.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:24:33.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise 2022-03-21T17:24:33.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:24:33.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:24:33.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:24:33.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:24:33.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:24:33 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:35 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:24:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:35.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:35.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:35.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:24:35.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:35.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:24:35.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:35.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:24:35.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:24:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:35 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:24:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:24:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:24:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:35.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:24:35.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:35.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:24:35.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:35.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:35.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:35.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:24:35.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:35.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:35.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:24:35.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:24:35.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:24:35.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:24:35.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:24:35.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:24:35.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:24:35.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:24:35.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:24:35.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:24:35.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:24:35.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:24:35.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:24:35.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:24:35.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:24:35.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:24:35.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:24:35.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:24:35.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:24:35.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:24:35.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:24:35.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:24:35.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:24:35.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:24:35.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:24:35.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:24:35.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:24:35.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:24:35.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:24:35.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:35 smithi016 ceph-mon[22010]: pgmap v373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:35 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:24:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:24:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:24:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:24:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:24:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:35 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:24:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:24:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:24:35.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:24:35.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:24:35.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:24:35.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:24:35.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:24:35.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:24:35.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:24:35.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:24:35.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:24:35.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:24:35.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:24:35.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:24:35.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:24:35.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:24:35.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:24:35.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:24:35.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:24:35.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:24:35.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:24:35.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:24:35.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:24:35.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:24:35.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:24:35.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:24:35.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:24:35.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:24:35.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:24:35.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:24:35.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:24:35.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:24:35.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:24:35.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:24:35.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:24:35.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:24:35.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:24:35.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:24:35.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:24:35.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:24:35.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:24:35.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:24:35.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:24:35.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:24:35.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:24:35.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:24:35.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:24:35.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:35 smithi079 ceph-mon[23158]: pgmap v373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:37 smithi016 ceph-mon[22010]: pgmap v374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:37 smithi079 ceph-mon[23158]: pgmap v374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:38 smithi016 ceph-mon[22010]: pgmap v375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:38 smithi079 ceph-mon[23158]: pgmap v375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:40 smithi016 ceph-mon[22010]: pgmap v376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:40 smithi079 ceph-mon[23158]: pgmap v376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:43 smithi016 ceph-mon[22010]: pgmap v377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:43 smithi079 ceph-mon[23158]: pgmap v377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:45 smithi016 ceph-mon[22010]: pgmap v378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:45 smithi079 ceph-mon[23158]: pgmap v378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:47 smithi016 ceph-mon[22010]: pgmap v379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:47 smithi079 ceph-mon[23158]: pgmap v379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:49 smithi016 ceph-mon[22010]: pgmap v380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:49 smithi079 ceph-mon[23158]: pgmap v380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:51 smithi016 ceph-mon[22010]: pgmap v381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:51 smithi079 ceph-mon[23158]: pgmap v381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:53 smithi016 ceph-mon[22010]: pgmap v382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:53 smithi079 ceph-mon[23158]: pgmap v382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:55 smithi016 ceph-mon[22010]: pgmap v383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:55 smithi079 ceph-mon[23158]: pgmap v383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:24:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:57 smithi016 ceph-mon[22010]: pgmap v384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:57 smithi079 ceph-mon[23158]: pgmap v384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:24:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:24:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:24:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:24:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:24:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:24:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:24:58 smithi016 ceph-mon[22010]: pgmap v385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:24:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:24:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:24:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:24:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:24:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:24:58 smithi079 ceph-mon[23158]: pgmap v385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:00 smithi016 ceph-mon[22010]: pgmap v386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:00 smithi079 ceph-mon[23158]: pgmap v386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:03 smithi016 ceph-mon[22010]: pgmap v387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:03 smithi079 ceph-mon[23158]: pgmap v387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:05 smithi016 ceph-mon[22010]: pgmap v388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:05 smithi079 ceph-mon[23158]: pgmap v388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:07 smithi016 ceph-mon[22010]: pgmap v389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:07 smithi079 ceph-mon[23158]: pgmap v389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:09 smithi016 ceph-mon[22010]: pgmap v390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:09 smithi079 ceph-mon[23158]: pgmap v390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:11 smithi016 ceph-mon[22010]: pgmap v391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:11 smithi079 ceph-mon[23158]: pgmap v391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:13 smithi016 ceph-mon[22010]: pgmap v392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:13 smithi079 ceph-mon[23158]: pgmap v392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:15 smithi016 ceph-mon[22010]: pgmap v393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:15 smithi079 ceph-mon[23158]: pgmap v393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:17 smithi016 ceph-mon[22010]: pgmap v394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:17 smithi079 ceph-mon[23158]: pgmap v394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:18 smithi016 ceph-mon[22010]: pgmap v395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:18 smithi079 ceph-mon[23158]: pgmap v395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:20 smithi016 ceph-mon[22010]: pgmap v396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:20 smithi079 ceph-mon[23158]: pgmap v396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:23 smithi016 ceph-mon[22010]: pgmap v397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:23 smithi079 ceph-mon[23158]: pgmap v397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:25 smithi016 ceph-mon[22010]: pgmap v398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:25 smithi079 ceph-mon[23158]: pgmap v398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:27 smithi016 ceph-mon[22010]: pgmap v399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:27 smithi079 ceph-mon[23158]: pgmap v399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:29 smithi016 ceph-mon[22010]: pgmap v400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:29 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:29 smithi079 ceph-mon[23158]: pgmap v400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:29 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:31 smithi016 ceph-mon[22010]: pgmap v401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:31 smithi079 ceph-mon[23158]: pgmap v401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:33 smithi016 ceph-mon[22010]: pgmap v402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:33 smithi079 ceph-mon[23158]: pgmap v402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:33 smithi016 conmon[32727]: 2022-03-21T17:25:33.719+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:34 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:25:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:34 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:25:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:34 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:25:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:34 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:25:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:34 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:25:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:34 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:25:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:35.123 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:34 smithi016 conmon[32727]: 2022-03-21T17:25:34.746+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:35 smithi016 ceph-mon[22010]: pgmap v403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:35 smithi016 ceph-mon[22010]: Health check failed: 9 slow ops, oldest one blocked for 180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:35 smithi079 ceph-mon[23158]: pgmap v403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:35 smithi079 ceph-mon[23158]: Health check failed: 9 slow ops, oldest one blocked for 180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:36.128 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:35 smithi016 conmon[32727]: 2022-03-21T17:25:35.715+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:36 smithi016 ceph-mon[22010]: pgmap v404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:36 smithi079 ceph-mon[23158]: pgmap v404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[32727]: 2022-03-21T17:25:36.689+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:36.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: 2022-03-21T17:25:36.589+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:36.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:25:36.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: yield 2022-03-21T17:25:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:25:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:25:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:25:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:25:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:25:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:25:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:25:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:25:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:25:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:25:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:37.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:25:37.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:37.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:37.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:37.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:37 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:25:37.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:25:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:37.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:25:37.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:37.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:37.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:37.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:25:37.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:37.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:25:37.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:37.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:25:37.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:37.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:37.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:37.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:25:37.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:37.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:37.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:37.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:37 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:25:37.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:38.131 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:37 smithi016 conmon[32727]: 2022-03-21T17:25:37.725+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:38 smithi016 ceph-mon[22010]: pgmap v405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:38 smithi079 ceph-mon[23158]: pgmap v405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:39.132 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:38 smithi016 conmon[32727]: 2022-03-21T17:25:38.689+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:39.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[32727]: 2022-03-21T17:25:39.688+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:39.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 2022-03-21T17:25:39.661+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:39.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:25:39.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:39.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: yield 2022-03-21T17:25:39.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:39.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:39.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:39.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:25:39.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:39.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:25:39.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:39.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:25:39.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:39.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:25:39.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:39.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:25:39.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:39.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:39.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:39.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 2022-03-21T17:25:39.662+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:25:39.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:25:39.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:39.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: yield 2022-03-21T17:25:39.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:39.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:39.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:39.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:25:39.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:39.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:25:39.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:39.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:25:39.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:39.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:25:39.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:39.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:25:39.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:39.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:39.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:39.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 2022-03-21T17:25:39.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:25:39.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: 2022-03-21T17:25:39.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:25:39.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:25:39.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:25:39.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:25:39.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:25:39.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:25:39.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:25:39.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:25:39.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:25:39.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:25:39.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:25:39.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:25:39.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:25:39.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:25:39.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:25:39.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:25:39.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:25:39.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:25:39.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:25:39.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:25:39.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:25:39.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:25:39.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise 2022-03-21T17:25:39.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:25:39.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:25:39.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:25:39.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:25:39.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:25:39 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:40 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:25:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:25:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:25:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:25:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:40.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:40.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:40.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:25:40.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:40.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:40.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:40 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:25:40.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:40.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:25:40.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:25:40.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:40.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:25:40.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:40.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:25:40.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:40.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:40.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:40.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:25:40.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:40.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:40.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:25:40.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:25:40.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:25:40.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:25:40.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:25:40.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:25:40.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:25:40.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:25:40.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:25:40.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:25:40.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:25:40.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:25:40.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:25:40.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:25:40.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:25:40.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:25:40.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:25:40.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:25:40.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:25:40.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:25:40.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:25:40.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:25:40.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:25:40.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:25:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:25:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:25:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:25:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:25:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:25:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:40.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:40 smithi016 ceph-mon[22010]: pgmap v406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:40 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:25:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:25:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:25:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:25:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:25:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:40 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:25:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:25:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:25:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:25:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:25:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:25:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:25:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:25:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:25:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:25:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:25:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:25:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:25:40.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:25:40.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:25:40.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:25:40.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:25:40.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:25:40.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:25:40.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:25:40.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:25:40.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:25:40.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:25:40.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:25:40.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:25:40.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:25:40.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:25:40.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:25:40.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:25:40.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:25:40.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:25:40.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:25:40.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:25:40.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:25:40.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:25:40.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:25:40.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:25:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:25:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:25:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:25:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:25:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:25:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:25:40.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:25:40.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:25:40.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:25:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:25:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:25:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:40.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:40 smithi079 ceph-mon[23158]: pgmap v406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:40 smithi016 conmon[32727]: 2022-03-21T17:25:40.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:41 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:41 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:41 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:41.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:41 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:41.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:41 smithi016 conmon[32727]: 2022-03-21T17:25:41.630+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:42 smithi016 ceph-mon[22010]: pgmap v407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:42 smithi079 ceph-mon[23158]: pgmap v407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:43.145 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:42 smithi016 conmon[32727]: 2022-03-21T17:25:42.664+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:44.151 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:43 smithi016 conmon[32727]: 2022-03-21T17:25:43.694+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:44 smithi016 ceph-mon[22010]: pgmap v408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:44 smithi079 ceph-mon[23158]: pgmap v408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:44 smithi016 conmon[32727]: 2022-03-21T17:25:44.661+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:45 smithi016 conmon[32727]: 2022-03-21T17:25:45.630+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:46 smithi016 ceph-mon[22010]: pgmap v409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:46 smithi079 ceph-mon[23158]: pgmap v409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:47.159 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:46 smithi016 conmon[32727]: 2022-03-21T17:25:46.666+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:47 smithi016 conmon[32727]: 2022-03-21T17:25:47.632+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:48 smithi016 ceph-mon[22010]: pgmap v410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:48 smithi079 ceph-mon[23158]: pgmap v410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:48 smithi016 conmon[32727]: 2022-03-21T17:25:48.669+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:49.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:49.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:49.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:49 smithi016 conmon[32727]: 2022-03-21T17:25:49.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:50.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:50 smithi079 ceph-mon[23158]: pgmap v411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:50 smithi016 ceph-mon[22010]: pgmap v411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:50 smithi016 conmon[32727]: 2022-03-21T17:25:50.671+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:51.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:51 smithi016 conmon[32727]: 2022-03-21T17:25:51.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:52 smithi079 ceph-mon[23158]: pgmap v412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:52.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:52.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:52 smithi016 ceph-mon[22010]: pgmap v412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:52 smithi016 conmon[32727]: 2022-03-21T17:25:52.642+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:53.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:53.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:53 smithi016 conmon[32727]: 2022-03-21T17:25:53.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:54 smithi079 ceph-mon[23158]: pgmap v413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:54.573 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:54.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:54 smithi016 ceph-mon[22010]: pgmap v413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:25:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:54 smithi016 conmon[32727]: 2022-03-21T17:25:54.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:55.559 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:55 smithi016 conmon[32727]: 2022-03-21T17:25:55.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:56 smithi079 ceph-mon[23158]: pgmap v414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:56.589 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:56.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:56 smithi016 ceph-mon[22010]: pgmap v414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:56 smithi016 conmon[32727]: 2022-03-21T17:25:56.587+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:57.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:25:57.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:25:57.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:57 smithi016 conmon[32727]: 2022-03-21T17:25:57.619+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:25:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:25:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:25:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:25:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:58 smithi079 ceph-mon[23158]: pgmap v415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:58.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:58.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:25:58.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:25:58.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:25:58.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:25:58.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:25:58.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:58 smithi016 ceph-mon[22010]: pgmap v415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:25:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:58 smithi016 conmon[32727]: 2022-03-21T17:25:58.634+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:25:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:25:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:59.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:25:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:25:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:25:59 smithi016 conmon[32727]: 2022-03-21T17:25:59.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:00 smithi079 ceph-mon[23158]: pgmap v416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:00 smithi016 ceph-mon[22010]: pgmap v416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:00 smithi016 conmon[32727]: 2022-03-21T17:26:00.668+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:01 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:01 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:01 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:01 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:01 smithi016 conmon[32727]: 2022-03-21T17:26:01.678+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:02 smithi079 ceph-mon[23158]: pgmap v417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:02 smithi016 ceph-mon[22010]: pgmap v417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:02 smithi016 conmon[32727]: 2022-03-21T17:26:02.667+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:03 smithi016 conmon[32727]: 2022-03-21T17:26:03.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:04 smithi079 ceph-mon[23158]: pgmap v418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:04 smithi016 ceph-mon[22010]: pgmap v418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:04 smithi016 conmon[32727]: 2022-03-21T17:26:04.697+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:05.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:05.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:05.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:05 smithi016 conmon[32727]: 2022-03-21T17:26:05.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:06 smithi079 ceph-mon[23158]: pgmap v419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:06.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:06.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:06 smithi016 ceph-mon[22010]: pgmap v419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:06 smithi016 conmon[32727]: 2022-03-21T17:26:06.658+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:07 smithi016 conmon[32727]: 2022-03-21T17:26:07.699+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:08 smithi079 ceph-mon[23158]: pgmap v420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:08 smithi016 ceph-mon[22010]: pgmap v420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:08 smithi016 conmon[32727]: 2022-03-21T17:26:08.726+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:09 smithi016 conmon[32727]: 2022-03-21T17:26:09.739+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:10 smithi016 ceph-mon[22010]: pgmap v421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:10 smithi079 ceph-mon[23158]: pgmap v421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:10 smithi016 conmon[32727]: 2022-03-21T17:26:10.701+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:11 smithi016 conmon[32727]: 2022-03-21T17:26:11.723+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:12 smithi016 ceph-mon[22010]: pgmap v422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:12.830 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:12 smithi079 ceph-mon[23158]: pgmap v422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:12 smithi016 conmon[32727]: 2022-03-21T17:26:12.676+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:13 smithi016 conmon[32727]: 2022-03-21T17:26:13.699+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:14 smithi016 ceph-mon[22010]: pgmap v423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:14 smithi079 ceph-mon[23158]: pgmap v423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:14 smithi016 conmon[32727]: 2022-03-21T17:26:14.717+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:15 smithi016 conmon[32727]: 2022-03-21T17:26:15.759+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:16 smithi016 ceph-mon[22010]: pgmap v424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:16 smithi079 ceph-mon[23158]: pgmap v424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:16 smithi016 conmon[32727]: 2022-03-21T17:26:16.740+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:17 smithi016 conmon[32727]: 2022-03-21T17:26:17.711+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:18 smithi016 ceph-mon[22010]: pgmap v425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:18 smithi079 ceph-mon[23158]: pgmap v425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:18 smithi016 conmon[32727]: 2022-03-21T17:26:18.718+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:19 smithi016 conmon[32727]: 2022-03-21T17:26:19.698+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:20 smithi016 ceph-mon[22010]: pgmap v426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:20 smithi079 ceph-mon[23158]: pgmap v426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:20 smithi016 conmon[32727]: 2022-03-21T17:26:20.688+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:21.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:21 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:21.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:21 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:21.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:21 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:21 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:21 smithi016 conmon[32727]: 2022-03-21T17:26:21.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:22.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:22.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:22 smithi016 ceph-mon[22010]: pgmap v427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:22 smithi079 ceph-mon[23158]: pgmap v427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:22 smithi016 conmon[32727]: 2022-03-21T17:26:22.641+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:23.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:23.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:23 smithi016 conmon[32727]: 2022-03-21T17:26:23.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:24.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:24.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:24 smithi016 ceph-mon[22010]: pgmap v428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:24 smithi079 ceph-mon[23158]: pgmap v428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:24 smithi016 conmon[32727]: 2022-03-21T17:26:24.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:25.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:25.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:25.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:25 smithi016 conmon[32727]: 2022-03-21T17:26:25.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:26.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:26.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:26 smithi016 ceph-mon[22010]: pgmap v429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:26 smithi079 ceph-mon[23158]: pgmap v429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:26 smithi016 conmon[32727]: 2022-03-21T17:26:26.577+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:27.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:27 smithi016 conmon[32727]: 2022-03-21T17:26:27.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:28.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:28.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:28.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:28 smithi016 ceph-mon[22010]: pgmap v430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:28 smithi079 ceph-mon[23158]: pgmap v430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:28.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:28 smithi016 conmon[32727]: 2022-03-21T17:26:28.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:29.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:29.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:29 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:29.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:29 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:29 smithi016 conmon[32727]: 2022-03-21T17:26:29.554+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:30.561 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:30.562 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:30 smithi016 ceph-mon[22010]: pgmap v431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:30 smithi079 ceph-mon[23158]: pgmap v431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:30 smithi016 conmon[32727]: 2022-03-21T17:26:30.559+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:31.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:31.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:31 smithi016 conmon[32727]: 2022-03-21T17:26:31.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:32 smithi016 ceph-mon[22010]: pgmap v432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:32 smithi016 conmon[32727]: 2022-03-21T17:26:32.492+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:32 smithi079 ceph-mon[23158]: pgmap v432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:33.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:33 smithi016 conmon[32727]: 2022-03-21T17:26:33.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:33.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:34 smithi016 ceph-mon[22010]: pgmap v433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:34 smithi016 conmon[32727]: 2022-03-21T17:26:34.498+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:34 smithi079 ceph-mon[23158]: pgmap v433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:35 smithi016 conmon[32727]: 2022-03-21T17:26:35.466+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:36 smithi016 ceph-mon[22010]: pgmap v434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:36 smithi016 conmon[32727]: 2022-03-21T17:26:36.445+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:36 smithi079 ceph-mon[23158]: pgmap v434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:37 smithi016 conmon[32727]: 2022-03-21T17:26:37.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:37.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:37.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:37.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:38 smithi016 ceph-mon[22010]: pgmap v435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:38 smithi016 conmon[32727]: 2022-03-21T17:26:38.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:38 smithi079 ceph-mon[23158]: pgmap v435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:39 smithi016 conmon[32727]: 2022-03-21T17:26:39.430+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:39.709 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:40.387 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:40.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:26:40.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:26:40.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:26:40.390 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: pgmap v436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:40.390 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:26:40.390 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:40 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:26:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:40 smithi016 conmon[32727]: 2022-03-21T17:26:40.385+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:26:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:26:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:26:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: pgmap v436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:40.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:26:40.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:40 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:26:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:41 smithi016 conmon[32727]: 2022-03-21T17:26:41.371+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:41 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:41 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[32727]: 2022-03-21T17:26:42.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:42 smithi016 ceph-mon[22010]: pgmap v437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:42 smithi079 ceph-mon[23158]: pgmap v437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:43.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: 2022-03-21T17:26:42.765+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:26:43.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:43.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: yield 2022-03-21T17:26:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:43.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:26:43.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:43.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:26:43.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:43.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:26:43.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:43.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:26:43.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:43.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:26:43.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:43.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:43.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:43 smithi016 conmon[32727]: 2022-03-21T17:26:43.332+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:26:43.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:43.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:26:43.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:43.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:43.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:43.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:26:43.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:43.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:26:43.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:43.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:26:43.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:43.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:43.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:43.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:26:43.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:43.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:43.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:43 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:26:43.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:26:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:26:43.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:43.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:43.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:43.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:26:43.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:43.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:26:43.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:43.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:26:43.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:43.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:43.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:43.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:26:43.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:43.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:43.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:43.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:43 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:26:43.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:44 smithi016 ceph-mon[22010]: pgmap v438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:44 smithi016 conmon[32727]: 2022-03-21T17:26:44.359+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:44 smithi079 ceph-mon[23158]: pgmap v438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[32727]: 2022-03-21T17:26:45.384+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:46.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 2022-03-21T17:26:45.837+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:26:46.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: yield 2022-03-21T17:26:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:26:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:26:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:26:46.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:26:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 2022-03-21T17:26:45.839+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:26:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:26:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: yield 2022-03-21T17:26:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:26:46.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:26:46.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:26:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:26:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 2022-03-21T17:26:46.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:26:46.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: 2022-03-21T17:26:46.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:26:46.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:26:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:26:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:26:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:26:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:26:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:26:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:26:46.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:26:46.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:26:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:26:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:26:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:26:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:26:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:26:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:26:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:26:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:26:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:26:46.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:26:46.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:26:46.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:26:46.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise 2022-03-21T17:26:46.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:26:46.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:26:46.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:26:46.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:26:46.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:26:45 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:46 smithi016 conmon[32727]: 2022-03-21T17:26:46.386+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:26:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:26:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:26:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:26:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:26:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:46 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:26:46.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:26:46.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:26:46.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:26:46.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:26:46.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:26:46.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:26:46.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:26:46.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:26:46.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:26:46.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:26:46.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:26:46.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:26:46.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:26:46.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:26:46.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:26:46.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:26:46.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:26:46.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:26:46.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:26:46.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:26:46.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:26:46.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:26:46.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:26:46.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:26:46.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:26:46.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:26:46.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:26:46.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:26:46.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:26:46.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:26:46.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:26:46.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:26:46.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:26:46.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:26:46.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.694 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:46 smithi016 ceph-mon[22010]: pgmap v439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:26:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:26:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:26:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:26:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:26:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:46 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:26:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:26:46.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:26:46.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:26:46.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:26:46.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:26:46.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:26:46.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:26:46.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:26:46.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:26:46.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:26:46.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:26:46.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:26:46.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:26:46.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:26:46.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:26:46.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:26:46.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:26:46.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:26:46.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:26:46.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:26:46.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:26:46.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:26:46.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:26:46.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:26:46.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:26:46.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:26:46.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:26:46.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:26:46.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:26:46.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:26:46.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:26:46.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:26:46.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:26:46.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:26:46.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:26:46.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:26:46.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:26:46.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:26:46.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:26:46.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:26:46.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:26:46.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:26:46.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:26:46.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:26:46.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:26:46.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:26:46.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:26:46.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:46 smithi079 ceph-mon[23158]: pgmap v439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:47 smithi016 conmon[32727]: 2022-03-21T17:26:47.365+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:48 smithi016 conmon[32727]: 2022-03-21T17:26:48.351+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:48 smithi016 ceph-mon[22010]: pgmap v440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:48 smithi079 ceph-mon[23158]: pgmap v440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:49 smithi016 conmon[32727]: 2022-03-21T17:26:49.398+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:50 smithi016 conmon[32727]: 2022-03-21T17:26:50.389+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:50 smithi016 ceph-mon[22010]: pgmap v441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:50 smithi079 ceph-mon[23158]: pgmap v441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:51 smithi016 conmon[32727]: 2022-03-21T17:26:51.399+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:52 smithi016 ceph-mon[22010]: pgmap v442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:52.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:52 smithi016 conmon[32727]: 2022-03-21T17:26:52.372+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:52 smithi079 ceph-mon[23158]: pgmap v442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:53 smithi016 conmon[32727]: 2022-03-21T17:26:53.333+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:54 smithi016 ceph-mon[22010]: pgmap v443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:54 smithi016 conmon[32727]: 2022-03-21T17:26:54.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:54 smithi079 ceph-mon[23158]: pgmap v443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:26:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:55 smithi016 conmon[32727]: 2022-03-21T17:26:55.346+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:56 smithi016 ceph-mon[22010]: pgmap v444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:56 smithi016 conmon[32727]: 2022-03-21T17:26:56.354+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:56 smithi079 ceph-mon[23158]: pgmap v444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:57 smithi016 conmon[32727]: 2022-03-21T17:26:57.368+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:26:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:26:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:58 smithi016 conmon[32727]: 2022-03-21T17:26:58.373+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:26:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:26:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:26:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:26:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:58 smithi016 ceph-mon[22010]: pgmap v445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:26:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:26:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:26:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:26:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:26:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:58 smithi079 ceph-mon[23158]: pgmap v445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:26:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:26:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:26:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:26:59 smithi016 conmon[32727]: 2022-03-21T17:26:59.369+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:26:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:26:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:00.233 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-03-21T17:27:00.249 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-03-21T17:27:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:00 smithi016 ceph-mon[22010]: pgmap v446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:00 smithi016 conmon[32727]: 2022-03-21T17:27:00.334+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:00 smithi079 ceph-mon[23158]: pgmap v446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:01 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:01 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:01 smithi016 conmon[32727]: 2022-03-21T17:27:01.343+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:01 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:01.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:01 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:02 smithi016 ceph-mon[22010]: pgmap v447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:02.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:02 smithi016 conmon[32727]: 2022-03-21T17:27:02.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:02 smithi079 ceph-mon[23158]: pgmap v447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:03.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:03 smithi016 conmon[32727]: 2022-03-21T17:27:03.340+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:04 smithi016 ceph-mon[22010]: pgmap v448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:04 smithi016 conmon[32727]: 2022-03-21T17:27:04.380+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:04 smithi079 ceph-mon[23158]: pgmap v448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:05 smithi016 conmon[32727]: 2022-03-21T17:27:05.385+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:05.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:06 smithi016 ceph-mon[22010]: pgmap v449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:06 smithi016 conmon[32727]: 2022-03-21T17:27:06.366+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:06 smithi079 ceph-mon[23158]: pgmap v449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:07 smithi016 conmon[32727]: 2022-03-21T17:27:07.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:08 smithi016 ceph-mon[22010]: pgmap v450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:08.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:08 smithi016 conmon[32727]: 2022-03-21T17:27:08.348+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:08 smithi079 ceph-mon[23158]: pgmap v450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:09 smithi016 conmon[32727]: 2022-03-21T17:27:09.337+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:09.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:10 smithi016 ceph-mon[22010]: pgmap v451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:10 smithi016 conmon[32727]: 2022-03-21T17:27:10.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:10 smithi079 ceph-mon[23158]: pgmap v451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:11 smithi016 conmon[32727]: 2022-03-21T17:27:11.315+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:12 smithi016 ceph-mon[22010]: pgmap v452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:12 smithi016 conmon[32727]: 2022-03-21T17:27:12.335+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:12 smithi079 ceph-mon[23158]: pgmap v452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:13 smithi016 conmon[32727]: 2022-03-21T17:27:13.376+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:14 smithi079 ceph-mon[23158]: pgmap v453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:14 smithi016 ceph-mon[22010]: pgmap v453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:14 smithi016 conmon[32727]: 2022-03-21T17:27:14.413+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:15 smithi016 conmon[32727]: 2022-03-21T17:27:15.366+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:16 smithi016 ceph-mon[22010]: pgmap v454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:16 smithi016 conmon[32727]: 2022-03-21T17:27:16.366+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:16 smithi079 ceph-mon[23158]: pgmap v454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:17 smithi016 conmon[32727]: 2022-03-21T17:27:17.410+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:17.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:18.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:18 smithi079 ceph-mon[23158]: pgmap v455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:18 smithi016 ceph-mon[22010]: pgmap v455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:18 smithi016 conmon[32727]: 2022-03-21T17:27:18.424+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:19 smithi016 conmon[32727]: 2022-03-21T17:27:19.461+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:20 smithi079 ceph-mon[23158]: pgmap v456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:20 smithi016 ceph-mon[22010]: pgmap v456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:20.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:20 smithi016 conmon[32727]: 2022-03-21T17:27:20.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:21 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:21 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:21 smithi016 conmon[32727]: 2022-03-21T17:27:21.441+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:21 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:21 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:22 smithi016 ceph-mon[22010]: pgmap v457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:22.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:22 smithi016 conmon[32727]: 2022-03-21T17:27:22.406+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:22 smithi079 ceph-mon[23158]: pgmap v457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:23.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:23 smithi016 conmon[32727]: 2022-03-21T17:27:23.435+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:24 smithi079 ceph-mon[23158]: pgmap v458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:24 smithi016 ceph-mon[22010]: pgmap v458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:24 smithi016 conmon[32727]: 2022-03-21T17:27:24.421+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:25 smithi016 conmon[32727]: 2022-03-21T17:27:25.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:26 smithi079 ceph-mon[23158]: pgmap v459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:26 smithi016 ceph-mon[22010]: pgmap v459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:26 smithi016 conmon[32727]: 2022-03-21T17:27:26.442+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:27.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:27 smithi016 conmon[32727]: 2022-03-21T17:27:27.413+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:28 smithi016 conmon[32727]: 2022-03-21T17:27:28.447+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:28 smithi016 ceph-mon[22010]: pgmap v460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:28 smithi079 ceph-mon[23158]: pgmap v460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:29 smithi016 conmon[32727]: 2022-03-21T17:27:29.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:30 smithi016 ceph-mon[22010]: pgmap v461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:30 smithi016 conmon[32727]: 2022-03-21T17:27:30.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:30 smithi079 ceph-mon[23158]: pgmap v461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:31 smithi016 conmon[32727]: 2022-03-21T17:27:31.513+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:32 smithi016 ceph-mon[22010]: pgmap v462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:32 smithi016 conmon[32727]: 2022-03-21T17:27:32.557+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:32 smithi079 ceph-mon[23158]: pgmap v462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:33 smithi016 conmon[32727]: 2022-03-21T17:27:33.544+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:34 smithi016 ceph-mon[22010]: pgmap v463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:34 smithi016 conmon[32727]: 2022-03-21T17:27:34.588+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:34 smithi079 ceph-mon[23158]: pgmap v463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:35 smithi016 conmon[32727]: 2022-03-21T17:27:35.568+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:36 smithi016 ceph-mon[22010]: pgmap v464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:36.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:36 smithi016 conmon[32727]: 2022-03-21T17:27:36.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:36 smithi079 ceph-mon[23158]: pgmap v464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:37 smithi016 conmon[32727]: 2022-03-21T17:27:37.576+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:38 smithi016 ceph-mon[22010]: pgmap v465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:38.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:38 smithi016 conmon[32727]: 2022-03-21T17:27:38.573+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:38 smithi079 ceph-mon[23158]: pgmap v465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:39 smithi016 conmon[32727]: 2022-03-21T17:27:39.543+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:40 smithi016 ceph-mon[22010]: pgmap v466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:40.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:40 smithi016 conmon[32727]: 2022-03-21T17:27:40.536+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:40 smithi079 ceph-mon[23158]: pgmap v466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:41 smithi016 conmon[32727]: 2022-03-21T17:27:41.503+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:42 smithi016 conmon[32727]: 2022-03-21T17:27:42.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:42 smithi016 ceph-mon[22010]: pgmap v467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:42 smithi079 ceph-mon[23158]: pgmap v467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:43 smithi016 conmon[32727]: 2022-03-21T17:27:43.553+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:44.981 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:44 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:44.981 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:44 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:44.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:44 smithi016 ceph-mon[22010]: pgmap v468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:44.982 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:44 smithi016 conmon[32727]: 2022-03-21T17:27:44.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:44.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:44.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:44 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:44.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:44 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:44.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:44 smithi079 ceph-mon[23158]: pgmap v468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:45.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:45.887 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:45 smithi016 conmon[32727]: 2022-03-21T17:27:45.548+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:45.887 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:46.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:46.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:27:46.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:27:46.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:27:46.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:46 smithi016 ceph-mon[22010]: pgmap v469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:27:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:27:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:27:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:46 smithi079 ceph-mon[23158]: pgmap v469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:46 smithi016 conmon[32727]: 2022-03-21T17:27:46.588+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:47 smithi016 conmon[32727]: 2022-03-21T17:27:47.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[32727]: 2022-03-21T17:27:48.499+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:48 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:48 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:48 smithi016 ceph-mon[22010]: pgmap v470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:48 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:48 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:48.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:48 smithi079 ceph-mon[23158]: pgmap v470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:49.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: 2022-03-21T17:27:48.941+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:27:49.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:49.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: yield 2022-03-21T17:27:49.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:49.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:49.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:49.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:27:49.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:49.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:27:49.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:49.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:27:49.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:49.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:27:49.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:49.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:27:49.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:49.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:49.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:49 smithi016 conmon[32727]: 2022-03-21T17:27:49.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:49.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:49.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:27:49.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:49.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:27:49.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:49.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:49.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:49.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:27:49.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:49.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:27:49.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:49.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:27:49.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:49.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:49.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:49.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:27:49.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:49.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:49.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:49 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:27:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:27:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:49.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:27:49.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:49.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:49.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:49.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:27:49.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:49.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:27:49.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:49.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:27:49.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:49.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:49.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:49.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:27:49.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:49.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:49.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:49.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:49 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:27:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:50 smithi016 ceph-mon[22010]: pgmap v471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:50.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:50 smithi016 conmon[32727]: 2022-03-21T17:27:50.465+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:50 smithi079 ceph-mon[23158]: pgmap v471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:51 smithi016 conmon[32727]: 2022-03-21T17:27:51.435+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:52.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 2022-03-21T17:27:52.013+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:27:52.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: yield 2022-03-21T17:27:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:27:52.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:27:52.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:27:52.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:27:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 2022-03-21T17:27:52.014+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:27:52.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:27:52.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: yield 2022-03-21T17:27:52.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:27:52.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:27:52.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:27:52.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:27:52.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 2022-03-21T17:27:52.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:27:52.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: 2022-03-21T17:27:52.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:27:52.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:27:52.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:27:52.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:27:52.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:27:52.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:27:52.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:27:52.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:27:52.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:27:52.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:27:52.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:27:52.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:27:52.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:27:52.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:27:52.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:27:52.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:27:52.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:27:52.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:27:52.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:27:52.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:27:52.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:27:52.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:27:52.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise 2022-03-21T17:27:52.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:27:52.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:27:52.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:27:52.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:27:52.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:52 smithi016 conmon[32727]: 2022-03-21T17:27:52.419+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:52 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:27:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:27:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:27:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:27:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:27:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:52 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:27:52.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:27:52.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:27:52.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:27:52.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:27:52.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:27:52.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:27:52.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:27:52.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:27:52.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:27:52.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:27:52.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:27:52.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:27:52.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:27:52.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:27:52.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:27:52.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:27:52.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:27:52.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:27:52.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:27:52.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:27:52.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:27:52.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:27:52.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:27:52.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:27:52.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:27:52.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:27:52.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:27:52.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:27:52.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:27:52.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:27:52.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:27:52.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:27:52.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:27:52.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:27:52.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.944 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:52 smithi016 ceph-mon[22010]: pgmap v472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:52 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:27:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:27:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:27:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:27:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:27:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:52 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:27:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:27:52.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:27:52.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:27:52.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:27:52.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:27:52.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:27:52.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:27:52.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:27:52.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:27:52.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:27:52.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:27:52.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:27:52.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:27:52.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:27:52.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:27:52.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:27:52.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:52.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:27:52.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:27:52.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:27:52.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:27:52.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:27:52.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:27:52.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:27:52.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:27:52.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:27:52.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:27:52.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:27:52.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:27:52.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:27:52.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:27:52.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:27:52.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:27:52.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:27:52.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:27:52.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:27:52.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:27:52.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:27:52.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:27:52.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:27:52.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:27:52.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:27:52.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:27:52.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:27:52.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:27:52.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:27:52.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:27:52.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:27:53.000 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:52 smithi079 ceph-mon[23158]: pgmap v472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:53 smithi016 conmon[32727]: 2022-03-21T17:27:53.398+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:54 smithi016 ceph-mon[22010]: pgmap v473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:54.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:54 smithi016 conmon[32727]: 2022-03-21T17:27:54.383+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:54 smithi079 ceph-mon[23158]: pgmap v473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:27:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:55 smithi016 conmon[32727]: 2022-03-21T17:27:55.408+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:56 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:56 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:56 smithi016 ceph-mon[22010]: pgmap v474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:56.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:56 smithi016 conmon[32727]: 2022-03-21T17:27:56.400+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:56 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:27:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:56 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:27:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:56 smithi079 ceph-mon[23158]: pgmap v474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:57 smithi016 conmon[32727]: 2022-03-21T17:27:57.404+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:58 smithi016 conmon[32727]: 2022-03-21T17:27:58.432+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:27:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:27:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:27:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:27:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:58 smithi016 ceph-mon[22010]: pgmap v475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:27:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:27:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:27:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:27:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:27:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:58 smithi079 ceph-mon[23158]: pgmap v475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:27:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:27:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:27:59.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:27:59 smithi016 conmon[32727]: 2022-03-21T17:27:59.425+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:27:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:27:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:00 smithi016 ceph-mon[22010]: pgmap v476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:00.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:00 smithi016 conmon[32727]: 2022-03-21T17:28:00.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:00 smithi079 ceph-mon[23158]: pgmap v476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:01.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:01.670 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:01 smithi016 conmon[32727]: 2022-03-21T17:28:01.406+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:02 smithi016 ceph-mon[22010]: pgmap v477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:02 smithi016 conmon[32727]: 2022-03-21T17:28:02.404+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:02 smithi079 ceph-mon[23158]: pgmap v477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:03 smithi016 conmon[32727]: 2022-03-21T17:28:03.417+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:04 smithi016 conmon[32727]: 2022-03-21T17:28:04.425+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:04 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:04 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:04 smithi016 ceph-mon[22010]: pgmap v478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:04 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:04 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:04 smithi079 ceph-mon[23158]: pgmap v478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:05 smithi016 conmon[32727]: 2022-03-21T17:28:05.389+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:06.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:06.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:06 smithi016 ceph-mon[22010]: pgmap v479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:06.670 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:06 smithi016 conmon[32727]: 2022-03-21T17:28:06.390+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:06 smithi079 ceph-mon[23158]: pgmap v479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:07 smithi016 conmon[32727]: 2022-03-21T17:28:07.375+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:08.594 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:08 smithi016 conmon[32727]: 2022-03-21T17:28:08.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:08.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:08.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:08 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:08 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:08 smithi016 ceph-mon[22010]: pgmap v480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:08 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:08 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:08 smithi079 ceph-mon[23158]: pgmap v480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:09 smithi016 conmon[32727]: 2022-03-21T17:28:09.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:10 smithi016 conmon[32727]: 2022-03-21T17:28:10.378+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:10.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:10 smithi016 ceph-mon[22010]: pgmap v481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:10 smithi079 ceph-mon[23158]: pgmap v481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:11 smithi016 conmon[32727]: 2022-03-21T17:28:11.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:12 smithi016 ceph-mon[22010]: pgmap v482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:12.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:12 smithi016 conmon[32727]: 2022-03-21T17:28:12.394+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:12 smithi079 ceph-mon[23158]: pgmap v482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:13 smithi016 conmon[32727]: 2022-03-21T17:28:13.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:13.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:14.612 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:14 smithi016 conmon[32727]: 2022-03-21T17:28:14.346+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:14 smithi016 ceph-mon[22010]: pgmap v483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:14 smithi079 ceph-mon[23158]: pgmap v483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:15.614 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:15 smithi016 conmon[32727]: 2022-03-21T17:28:15.326+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:16.622 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:16 smithi016 conmon[32727]: 2022-03-21T17:28:16.358+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:16 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:16 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:16 smithi016 ceph-mon[22010]: pgmap v484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:16 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:16 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:16 smithi079 ceph-mon[23158]: pgmap v484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:17.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:17.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:17 smithi016 conmon[32727]: 2022-03-21T17:28:17.382+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:18 smithi016 conmon[32727]: 2022-03-21T17:28:18.392+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:18 smithi016 ceph-mon[22010]: pgmap v485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:18 smithi079 ceph-mon[23158]: pgmap v485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:19 smithi016 conmon[32727]: 2022-03-21T17:28:19.404+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:20.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:20 smithi016 conmon[32727]: 2022-03-21T17:28:20.375+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:20 smithi016 ceph-mon[22010]: pgmap v486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:20.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:20.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:20 smithi079 ceph-mon[23158]: pgmap v486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:21 smithi016 conmon[32727]: 2022-03-21T17:28:21.396+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:22 smithi016 ceph-mon[22010]: pgmap v487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:22 smithi016 conmon[32727]: 2022-03-21T17:28:22.431+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:22.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:22.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:22 smithi079 ceph-mon[23158]: pgmap v487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:23.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:23 smithi016 conmon[32727]: 2022-03-21T17:28:23.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:24 smithi016 conmon[32727]: 2022-03-21T17:28:24.480+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:24 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:24 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:24 smithi016 ceph-mon[22010]: pgmap v488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:24 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:24 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:24 smithi079 ceph-mon[23158]: pgmap v488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:25 smithi016 conmon[32727]: 2022-03-21T17:28:25.436+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:26 smithi016 ceph-mon[22010]: pgmap v489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:26 smithi016 conmon[32727]: 2022-03-21T17:28:26.458+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:26 smithi079 ceph-mon[23158]: pgmap v489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:27 smithi016 conmon[32727]: 2022-03-21T17:28:27.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:28 smithi016 ceph-mon[22010]: pgmap v490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:28.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:28 smithi016 conmon[32727]: 2022-03-21T17:28:28.548+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:28 smithi079 ceph-mon[23158]: pgmap v490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:29.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:29 smithi016 conmon[32727]: 2022-03-21T17:28:29.556+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:30 smithi016 ceph-mon[22010]: pgmap v491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:30 smithi016 conmon[32727]: 2022-03-21T17:28:30.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:30 smithi079 ceph-mon[23158]: pgmap v491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:31 smithi016 conmon[32727]: 2022-03-21T17:28:31.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:32 smithi016 ceph-mon[22010]: pgmap v492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:32 smithi016 conmon[32727]: 2022-03-21T17:28:32.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:32 smithi079 ceph-mon[23158]: pgmap v492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:33 smithi016 conmon[32727]: 2022-03-21T17:28:33.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:34 smithi079 ceph-mon[23158]: pgmap v493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:34 smithi016 ceph-mon[22010]: pgmap v493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:34 smithi016 conmon[32727]: 2022-03-21T17:28:34.666+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:36.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:35 smithi016 conmon[32727]: 2022-03-21T17:28:35.681+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:36 smithi079 ceph-mon[23158]: pgmap v494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:36 smithi016 ceph-mon[22010]: pgmap v494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:37.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:36 smithi016 conmon[32727]: 2022-03-21T17:28:36.682+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:37 smithi016 conmon[32727]: 2022-03-21T17:28:37.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:38 smithi016 ceph-mon[22010]: pgmap v495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:38.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:38 smithi016 conmon[32727]: 2022-03-21T17:28:38.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:38 smithi079 ceph-mon[23158]: pgmap v495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:39 smithi016 conmon[32727]: 2022-03-21T17:28:39.619+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:40 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:40 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:40 smithi016 ceph-mon[22010]: pgmap v496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:40.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:40 smithi016 conmon[32727]: 2022-03-21T17:28:40.656+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:40 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:40 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:40 smithi079 ceph-mon[23158]: pgmap v496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:41.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:41 smithi016 conmon[32727]: 2022-03-21T17:28:41.622+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:42 smithi016 ceph-mon[22010]: pgmap v497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:42 smithi016 conmon[32727]: 2022-03-21T17:28:42.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:42 smithi079 ceph-mon[23158]: pgmap v497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:43 smithi016 conmon[32727]: 2022-03-21T17:28:43.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:44 smithi016 conmon[32727]: 2022-03-21T17:28:44.528+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:44 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:44 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:44 smithi016 ceph-mon[22010]: pgmap v498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:44 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:44 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:44 smithi079 ceph-mon[23158]: pgmap v498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:45 smithi016 conmon[32727]: 2022-03-21T17:28:45.532+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:45.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:46 smithi016 ceph-mon[22010]: pgmap v499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:46 smithi016 conmon[32727]: 2022-03-21T17:28:46.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:46 smithi079 ceph-mon[23158]: pgmap v499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:47 smithi016 conmon[32727]: 2022-03-21T17:28:47.534+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:48 smithi016 conmon[32727]: 2022-03-21T17:28:48.514+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:48 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:48 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:48 smithi016 ceph-mon[22010]: pgmap v500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:48.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:48 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:48.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:48 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:48.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:48.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:48 smithi079 ceph-mon[23158]: pgmap v500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:49 smithi016 conmon[32727]: 2022-03-21T17:28:49.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:50 smithi016 ceph-mon[22010]: pgmap v501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:50 smithi016 conmon[32727]: 2022-03-21T17:28:50.488+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:50 smithi079 ceph-mon[23158]: pgmap v501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:51 smithi016 conmon[32727]: 2022-03-21T17:28:51.526+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:52.062 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:52.728 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:52 smithi016 conmon[32727]: 2022-03-21T17:28:52.528+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:52 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:28:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:52 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:28:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:52 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:28:53.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:52 smithi016 ceph-mon[22010]: pgmap v502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:52 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:28:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:52 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:28:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:52 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:28:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:52 smithi079 ceph-mon[23158]: pgmap v502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:53.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:53 smithi016 conmon[32727]: 2022-03-21T17:28:53.500+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:54 smithi016 ceph-mon[22010]: pgmap v503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:54 smithi016 conmon[32727]: 2022-03-21T17:28:54.487+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:54 smithi079 ceph-mon[23158]: pgmap v503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:28:55.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: 2022-03-21T17:28:55.117+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:28:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:28:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: yield 2022-03-21T17:28:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:28:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:28:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:28:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:28:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:28:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:28:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:28:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:28:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:28:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:28:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:28:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:28:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:28:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:28:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:55 smithi016 conmon[32727]: 2022-03-21T17:28:55.523+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:28:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:28:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:28:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:28:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:28:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:28:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:28:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:28:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:28:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:28:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:28:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:28:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:28:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:28:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:28:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:28:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:28:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:55 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:28:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:28:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:28:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:28:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:28:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:28:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:28:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:28:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:28:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:28:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:28:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:28:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:28:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:28:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:28:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:28:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:28:56.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:28:56.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:56.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:55 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:28:56.746 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:56 smithi016 conmon[32727]: 2022-03-21T17:28:56.492+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:56 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:56 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:56 smithi016 ceph-mon[22010]: pgmap v504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:56 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:28:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:56 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:28:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:56 smithi079 ceph-mon[23158]: pgmap v504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:57.747 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:57 smithi016 conmon[32727]: 2022-03-21T17:28:57.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[32727]: 2022-03-21T17:28:58.428+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:28:58.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 2022-03-21T17:28:58.189+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:58.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:28:58.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:28:58.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: yield 2022-03-21T17:28:58.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:28:58.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:28:58.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:28:58.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:28:58.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:28:58.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:28:58.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:28:58.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:28:58.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:28:58.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:28:58.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:28:58.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:28:58.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:28:58.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:28:58.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:58.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 2022-03-21T17:28:58.190+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:28:58.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:28:58.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:28:58.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: yield 2022-03-21T17:28:58.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:28:58.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:28:58.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:28:58.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:28:58.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:28:58.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:28:58.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:28:58.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:28:58.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:28:58.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:28:58.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:28:58.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:28:58.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:28:58.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:28:58.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:58.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 2022-03-21T17:28:58.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:28:58.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: 2022-03-21T17:28:58.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:28:58.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:28:58.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:28:58.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:28:58.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:28:58.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:28:58.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:28:58.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:28:58.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:28:58.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:28:58.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:28:58.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:28:58.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:28:58.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:28:58.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:28:58.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:28:58.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:28:58.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:28:58.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:28:58.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:28:58.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:28:58.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:28:58.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise 2022-03-21T17:28:58.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:28:58.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:28:58.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:28:58.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:28:58.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:28:58 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:28:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:28:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:28:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:28:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:28:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:58 smithi016 ceph-mon[22010]: pgmap v505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:28:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:28:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:28:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:58 smithi079 ceph-mon[23158]: pgmap v505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:28:59.751 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:28:59 smithi016 conmon[32727]: 2022-03-21T17:28:59.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:59 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:29:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:29:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:29:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:29:00.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:29:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:29:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:29:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:29:00.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:29:00.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:29:00.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:29:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:29:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:29:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:29:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:29:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:29:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:59 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:29:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:29:00.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:29:00.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:29:00.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:29:00.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:29:00.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:29:00.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:29:00.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:29:00.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:29:00.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:29:00.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:29:00.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:29:00.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:29:00.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:29:00.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:29:00.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:29:00.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:29:00.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:29:00.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:29:00.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:29:00.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:29:00.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:29:00.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:29:00.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:29:00.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:29:00.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:29:00.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:29:00.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:29:00.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:29:00.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:29:00.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:29:00.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:29:00.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:29:00.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:29:00.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:29:00.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:29:00.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:29:00.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:29:00.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:29:00.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:29:00.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:29:00.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:29:00.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:29:00.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:29:00.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:29:00.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.199 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:28:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:59 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:29:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:29:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:29:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:29:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:29:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:29:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:29:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:29:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:29:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:29:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:29:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:29:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:29:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:29:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:29:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:29:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:59 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:29:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:29:00.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:29:00.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:29:00.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:29:00.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:29:00.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:29:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:29:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:29:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:29:00.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:29:00.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:29:00.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:29:00.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:29:00.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:29:00.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:29:00.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:29:00.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:29:00.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:29:00.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:29:00.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:29:00.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:29:00.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:29:00.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:29:00.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:29:00.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:29:00.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:29:00.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:29:00.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:29:00.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:29:00.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:29:00.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:29:00.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:29:00.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:29:00.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:29:00.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:29:00.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:29:00.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:29:00.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:29:00.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:29:00.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:29:00.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:29:00.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:29:00.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:29:00.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:29:00.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:29:00.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:29:00.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:29:00.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:28:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:00 smithi016 conmon[32727]: 2022-03-21T17:29:00.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:00 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:00 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:00.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:00 smithi016 ceph-mon[22010]: pgmap v506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:00 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:00 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:00 smithi079 ceph-mon[23158]: pgmap v506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:01 smithi016 conmon[32727]: 2022-03-21T17:29:01.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:02.758 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:02 smithi016 conmon[32727]: 2022-03-21T17:29:02.503+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:02 smithi016 ceph-mon[22010]: pgmap v507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:02 smithi079 ceph-mon[23158]: pgmap v507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:03 smithi016 conmon[32727]: 2022-03-21T17:29:03.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:04 smithi016 conmon[32727]: 2022-03-21T17:29:04.530+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:04 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:04 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:04 smithi016 ceph-mon[22010]: pgmap v508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:04 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:04 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:04 smithi079 ceph-mon[23158]: pgmap v508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:05 smithi016 conmon[32727]: 2022-03-21T17:29:05.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:06 smithi016 ceph-mon[22010]: pgmap v509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:06 smithi016 conmon[32727]: 2022-03-21T17:29:06.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:06 smithi079 ceph-mon[23158]: pgmap v509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:07.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:07 smithi016 conmon[32727]: 2022-03-21T17:29:07.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:08 smithi016 conmon[32727]: 2022-03-21T17:29:08.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:08 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:08 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:08 smithi016 ceph-mon[22010]: pgmap v510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:08 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:08 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:08 smithi079 ceph-mon[23158]: pgmap v510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:09 smithi016 conmon[32727]: 2022-03-21T17:29:09.648+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:10 smithi016 ceph-mon[22010]: pgmap v511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:10 smithi016 conmon[32727]: 2022-03-21T17:29:10.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:10 smithi079 ceph-mon[23158]: pgmap v511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:11 smithi016 conmon[32727]: 2022-03-21T17:29:11.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:12 smithi016 ceph-mon[22010]: pgmap v512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:12 smithi016 conmon[32727]: 2022-03-21T17:29:12.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:12 smithi079 ceph-mon[23158]: pgmap v512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:13 smithi016 conmon[32727]: 2022-03-21T17:29:13.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:14 smithi016 ceph-mon[22010]: pgmap v513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:14 smithi016 conmon[32727]: 2022-03-21T17:29:14.656+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:14 smithi079 ceph-mon[23158]: pgmap v513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:15 smithi016 conmon[32727]: 2022-03-21T17:29:15.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:16.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:16 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:16.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:16 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:16.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:16 smithi016 ceph-mon[22010]: pgmap v514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:16.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:16 smithi016 conmon[32727]: 2022-03-21T17:29:16.629+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:16 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:16 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:16 smithi079 ceph-mon[23158]: pgmap v514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:17 smithi016 conmon[32727]: 2022-03-21T17:29:17.589+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:18 smithi016 ceph-mon[22010]: pgmap v515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:18 smithi016 conmon[32727]: 2022-03-21T17:29:18.616+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:18 smithi079 ceph-mon[23158]: pgmap v515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:19 smithi016 conmon[32727]: 2022-03-21T17:29:19.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:20 smithi016 conmon[32727]: 2022-03-21T17:29:20.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:20 smithi016 ceph-mon[22010]: pgmap v516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:21.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:21.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:21.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:21.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:20 smithi079 ceph-mon[23158]: pgmap v516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:21 smithi016 conmon[32727]: 2022-03-21T17:29:21.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:22 smithi016 ceph-mon[22010]: pgmap v517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:22 smithi016 conmon[32727]: 2022-03-21T17:29:22.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:22 smithi079 ceph-mon[23158]: pgmap v517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:23 smithi016 conmon[32727]: 2022-03-21T17:29:23.599+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:24 smithi016 conmon[32727]: 2022-03-21T17:29:24.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:24 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:24 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:24 smithi016 ceph-mon[22010]: pgmap v518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:24 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:24 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:24 smithi079 ceph-mon[23158]: pgmap v518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:25 smithi016 conmon[32727]: 2022-03-21T17:29:25.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:26 smithi016 conmon[32727]: 2022-03-21T17:29:26.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:26 smithi079 ceph-mon[23158]: pgmap v519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:26 smithi016 ceph-mon[22010]: pgmap v519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:27 smithi016 conmon[32727]: 2022-03-21T17:29:27.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:27 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:27 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:28.955 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:28 smithi016 conmon[32727]: 2022-03-21T17:29:28.670+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:28 smithi079 ceph-mon[23158]: pgmap v520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:28 smithi016 ceph-mon[22010]: pgmap v520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:29 smithi016 conmon[32727]: 2022-03-21T17:29:29.626+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:30 smithi016 conmon[32727]: 2022-03-21T17:29:30.645+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:30 smithi079 ceph-mon[23158]: pgmap v521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:30 smithi016 ceph-mon[22010]: pgmap v521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:31.966 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:31 smithi016 conmon[32727]: 2022-03-21T17:29:31.680+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:32.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:32 smithi016 conmon[32727]: 2022-03-21T17:29:32.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:32 smithi079 ceph-mon[23158]: pgmap v522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:32 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:32 smithi016 ceph-mon[22010]: pgmap v522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:33.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:33.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:32 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:33.971 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:33 smithi016 conmon[32727]: 2022-03-21T17:29:33.666+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:34 smithi016 conmon[32727]: 2022-03-21T17:29:34.662+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:34 smithi016 ceph-mon[22010]: pgmap v523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:34 smithi079 ceph-mon[23158]: pgmap v523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:35.978 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:35 smithi016 conmon[32727]: 2022-03-21T17:29:35.674+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:36 smithi016 conmon[32727]: 2022-03-21T17:29:36.641+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:36 smithi016 ceph-mon[22010]: pgmap v524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:36 smithi079 ceph-mon[23158]: pgmap v524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:37 smithi016 conmon[32727]: 2022-03-21T17:29:37.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:37 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:37 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:38 smithi016 conmon[32727]: 2022-03-21T17:29:38.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:38 smithi016 ceph-mon[22010]: pgmap v525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:38 smithi079 ceph-mon[23158]: pgmap v525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:39 smithi016 conmon[32727]: 2022-03-21T17:29:39.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:40 smithi016 conmon[32727]: 2022-03-21T17:29:40.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:41 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:41 smithi016 ceph-mon[22010]: pgmap v526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:41 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:41.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:41 smithi079 ceph-mon[23158]: pgmap v526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:41.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:41 smithi016 conmon[32727]: 2022-03-21T17:29:41.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:43.010 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:42 smithi016 conmon[32727]: 2022-03-21T17:29:42.676+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:43 smithi016 ceph-mon[22010]: pgmap v527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:43 smithi079 ceph-mon[23158]: pgmap v527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:44.011 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:43 smithi016 conmon[32727]: 2022-03-21T17:29:43.715+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:45.020 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:44 smithi016 conmon[32727]: 2022-03-21T17:29:44.721+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:45 smithi016 ceph-mon[22010]: pgmap v528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:45 smithi079 ceph-mon[23158]: pgmap v528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:46.021 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:45 smithi016 conmon[32727]: 2022-03-21T17:29:45.676+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:47.026 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:46 smithi016 conmon[32727]: 2022-03-21T17:29:46.683+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:47 smithi016 ceph-mon[22010]: pgmap v529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:47 smithi079 ceph-mon[23158]: pgmap v529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:47 smithi016 conmon[32727]: 2022-03-21T17:29:47.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:48 smithi016 conmon[32727]: 2022-03-21T17:29:48.640+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:49 smithi016 ceph-mon[22010]: pgmap v530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:49 smithi079 ceph-mon[23158]: pgmap v530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:50.041 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:49 smithi016 conmon[32727]: 2022-03-21T17:29:49.665+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:50 smithi016 conmon[32727]: 2022-03-21T17:29:50.634+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:51 smithi016 ceph-mon[22010]: pgmap v531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:51 smithi079 ceph-mon[23158]: pgmap v531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:51 smithi016 conmon[32727]: 2022-03-21T17:29:51.645+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:53.054 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:52 smithi016 conmon[32727]: 2022-03-21T17:29:52.690+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:53.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:53 smithi016 ceph-mon[22010]: pgmap v532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:53.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:53.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:53 smithi079 ceph-mon[23158]: pgmap v532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:54.055 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:53 smithi016 conmon[32727]: 2022-03-21T17:29:53.692+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:55.060 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:54 smithi016 conmon[32727]: 2022-03-21T17:29:54.717+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:55 smithi016 ceph-mon[22010]: pgmap v533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:55 smithi079 ceph-mon[23158]: pgmap v533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:29:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:56.065 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:55 smithi016 conmon[32727]: 2022-03-21T17:29:55.712+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:57.067 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:56 smithi016 conmon[32727]: 2022-03-21T17:29:56.756+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:57 smithi016 ceph-mon[22010]: pgmap v534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:29:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:29:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:57 smithi079 ceph-mon[23158]: pgmap v534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:58.074 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:57 smithi016 conmon[32727]: 2022-03-21T17:29:57.806+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:29:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:29:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:29:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:29:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:29:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:29:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:29:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:29:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:29:58.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:58 smithi016 conmon[32727]: 2022-03-21T17:29:58.802+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:29:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:59 smithi016 ceph-mon[22010]: pgmap v535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:29:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:29:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:29:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:29:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:29:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:59 smithi079 ceph-mon[23158]: pgmap v535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:29:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:29:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:29:59.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:29:59.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:29:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:29:59 smithi016 conmon[32727]: 2022-03-21T17:29:59.851+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops 2022-03-21T17:30:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-03-21T17:30:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: mds.l(mds.0): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 744 secs 2022-03-21T17:30:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:30:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:30:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:30:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:30:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:30:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops 2022-03-21T17:30:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops 2022-03-21T17:30:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-03-21T17:30:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: mds.l(mds.0): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 744 secs 2022-03-21T17:30:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:30:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:30:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:30:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:30:00.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:30:00.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 440 sec, osd.4 has slow ops 2022-03-21T17:30:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:00 smithi016 conmon[32727]: 2022-03-21T17:30:00.880+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:01.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:01 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:01.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:01 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:01.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:01 smithi016 ceph-mon[22010]: pgmap v536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:01 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:01 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:01 smithi079 ceph-mon[23158]: pgmap v536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:01.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: 2022-03-21T17:30:01.293+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:01.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:30:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: yield 2022-03-21T17:30:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:30:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:30:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:30:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:30:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:30:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:01.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:01 smithi016 conmon[32727]: 2022-03-21T17:30:01.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:03.094 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:02 smithi016 conmon[32727]: 2022-03-21T17:30:02.828+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:30:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:30:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:30:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:30:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:30:03.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:03.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:03.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:03.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:30:03.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:03.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:03.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:03.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:03 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:30:03.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:03.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:03 smithi016 ceph-mon[22010]: pgmap v537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:03.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:30:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:30:03.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:03.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:03.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:03.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:30:03.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:03.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:30:03.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:03.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:30:03.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:03.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:03.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:03.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:30:03.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:03.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:03.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:03.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:03 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:30:03.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:03.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:03 smithi079 ceph-mon[23158]: pgmap v537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:03.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:04.099 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:03 smithi016 conmon[32727]: 2022-03-21T17:30:03.813+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:04.367 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:04.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:04.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 2022-03-21T17:30:04.365+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:30:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: yield 2022-03-21T17:30:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:30:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:30:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:30:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:30:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:30:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 2022-03-21T17:30:04.366+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:30:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:30:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: yield 2022-03-21T17:30:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:30:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:30:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:30:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:30:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:30:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 2022-03-21T17:30:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:30:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: 2022-03-21T17:30:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:30:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:30:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:30:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:30:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:30:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:30:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:30:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:30:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:30:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:30:04.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:30:04.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:30:04.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:30:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:30:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:30:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:30:04.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:30:04.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:30:04.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:30:04.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:30:04.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:30:04.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:30:04.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise 2022-03-21T17:30:04.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:30:04.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:30:04.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:30:04.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:30:04.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:05.104 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:04 smithi016 conmon[32727]: 2022-03-21T17:30:04.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:05 smithi016 ceph-mon[22010]: pgmap v538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:05.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:05 smithi079 ceph-mon[23158]: pgmap v538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:06.106 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:05 smithi016 conmon[32727]: 2022-03-21T17:30:05.774+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:06 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:30:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:30:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:30:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:30:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:30:06.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:06 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:30:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:06.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:30:06.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:06.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:06.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:06.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:30:06.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:06.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:30:06.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:06.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:30:06.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:06.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:06.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:06.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:30:06.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:06.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:06.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:30:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:30:06.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:30:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:30:06.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:30:06.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:30:06.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:30:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:30:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:30:06.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:30:06.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:30:06.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:30:06.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:30:06.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:30:06.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:30:06.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:30:06.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:30:06.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:30:06.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:30:06.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:30:06.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:30:06.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:30:06.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:30:06.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:30:06.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:30:06.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:30:06.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:30:06.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:30:06.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:30:06.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:06 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:30:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:06.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:06.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:06.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:30:06.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:06.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:30:06.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:06.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:30:06.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:06.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:06.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:06.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:30:06.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:06.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:06.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:06 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:30:06.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:30:06.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:30:06.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:30:06.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:30:06.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:30:06.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:30:06.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:30:06.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:30:06.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:30:06.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:30:06.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:30:06.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:30:06.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:30:06.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:30:06.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:30:06.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:30:06.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:30:06.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:30:06.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:30:06.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:30:06.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:30:06.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:30:06.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:30:06.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:30:06.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:30:06.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:30:06.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:30:06.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:30:06.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:30:06.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:30:06.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:30:06.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:30:06.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:30:06.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:30:06.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:30:06.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:30:06.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:30:06.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:30:06.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:30:06.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:30:06.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:30:06.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:30:06.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:30:06.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:30:06.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:30:06.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:30:06.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:30:06.503 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:07.111 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:06 smithi016 conmon[32727]: 2022-03-21T17:30:06.769+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:07 smithi016 ceph-mon[22010]: pgmap v539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:07 smithi079 ceph-mon[23158]: pgmap v539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:08.116 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:07 smithi016 conmon[32727]: 2022-03-21T17:30:07.722+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:09.123 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:08 smithi016 conmon[32727]: 2022-03-21T17:30:08.759+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:09.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:09.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:09.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:09 smithi016 ceph-mon[22010]: pgmap v540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:09.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:09 smithi079 ceph-mon[23158]: pgmap v540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:10.122 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:09 smithi016 conmon[32727]: 2022-03-21T17:30:09.757+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:10.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:11.127 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:10 smithi016 conmon[32727]: 2022-03-21T17:30:10.727+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:11 smithi016 ceph-mon[22010]: pgmap v541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:11 smithi079 ceph-mon[23158]: pgmap v541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:12.129 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:11 smithi016 conmon[32727]: 2022-03-21T17:30:11.703+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:13.133 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:12 smithi016 conmon[32727]: 2022-03-21T17:30:12.716+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:13 smithi016 ceph-mon[22010]: pgmap v542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:13 smithi079 ceph-mon[23158]: pgmap v542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:14.134 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:13 smithi016 conmon[32727]: 2022-03-21T17:30:13.750+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:15.140 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:14 smithi016 conmon[32727]: 2022-03-21T17:30:14.710+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:15 smithi016 ceph-mon[22010]: pgmap v543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:15 smithi079 ceph-mon[23158]: pgmap v543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:16.141 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:15 smithi016 conmon[32727]: 2022-03-21T17:30:15.683+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:17.149 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:16 smithi016 conmon[32727]: 2022-03-21T17:30:16.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:17 smithi016 ceph-mon[22010]: pgmap v544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:17 smithi079 ceph-mon[23158]: pgmap v544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:17 smithi016 conmon[32727]: 2022-03-21T17:30:17.613+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:19.205 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:19.206 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:18 smithi016 conmon[32727]: 2022-03-21T17:30:18.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:19 smithi016 ceph-mon[22010]: pgmap v545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:19 smithi079 ceph-mon[23158]: pgmap v545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:19 smithi016 conmon[32727]: 2022-03-21T17:30:19.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:21.161 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:20 smithi016 conmon[32727]: 2022-03-21T17:30:20.710+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:21 smithi016 ceph-mon[22010]: pgmap v546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:21 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:21 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:21 smithi079 ceph-mon[23158]: pgmap v546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:21 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:21 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:21 smithi016 conmon[32727]: 2022-03-21T17:30:21.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:22.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:22 smithi016 conmon[32727]: 2022-03-21T17:30:22.658+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:23 smithi079 ceph-mon[23158]: pgmap v547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:23.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:23 smithi016 ceph-mon[22010]: pgmap v547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:23.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:23.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:23 smithi016 conmon[32727]: 2022-03-21T17:30:23.636+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:24 smithi079 ceph-mon[23158]: pgmap v548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:24.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:24.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:24 smithi016 ceph-mon[22010]: pgmap v548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:24 smithi016 conmon[32727]: 2022-03-21T17:30:24.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:25.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:25.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:25.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:25 smithi016 conmon[32727]: 2022-03-21T17:30:25.588+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:26 smithi079 ceph-mon[23158]: pgmap v549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:26.561 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:26.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:26 smithi016 ceph-mon[22010]: pgmap v549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:26 smithi016 conmon[32727]: 2022-03-21T17:30:26.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:27.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:27 smithi016 conmon[32727]: 2022-03-21T17:30:27.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:28 smithi079 ceph-mon[23158]: pgmap v550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:28.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:28.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:28.589 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:28 smithi016 ceph-mon[22010]: pgmap v550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:30:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:28 smithi016 conmon[32727]: 2022-03-21T17:30:28.586+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:29 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:29.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:29.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:29 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:29.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:29 smithi016 conmon[32727]: 2022-03-21T17:30:29.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:30 smithi079 ceph-mon[23158]: pgmap v551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:30.656 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:30.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:30 smithi016 ceph-mon[22010]: pgmap v551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:30 smithi016 conmon[32727]: 2022-03-21T17:30:30.654+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:31.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:31 smithi016 conmon[32727]: 2022-03-21T17:30:31.619+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:32 smithi079 ceph-mon[23158]: pgmap v552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:32 smithi016 ceph-mon[22010]: pgmap v552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:32 smithi016 conmon[32727]: 2022-03-21T17:30:32.666+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:33.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:33 smithi016 conmon[32727]: 2022-03-21T17:30:33.678+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:34 smithi079 ceph-mon[23158]: pgmap v553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:34.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:34.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:34 smithi016 ceph-mon[22010]: pgmap v553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:34 smithi016 conmon[32727]: 2022-03-21T17:30:34.650+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:35.603 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:35 smithi016 conmon[32727]: 2022-03-21T17:30:35.601+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:36 smithi079 ceph-mon[23158]: pgmap v554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:36.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:36.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:36 smithi016 ceph-mon[22010]: pgmap v554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:36 smithi016 conmon[32727]: 2022-03-21T17:30:36.571+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:37.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:37.566 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:37.567 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:37.567 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:37 smithi016 conmon[32727]: 2022-03-21T17:30:37.564+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:38 smithi079 ceph-mon[23158]: pgmap v555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:38.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:38.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:38.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:38 smithi016 ceph-mon[22010]: pgmap v555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:38 smithi016 conmon[32727]: 2022-03-21T17:30:38.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:39.569 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:39.921 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:39 smithi016 conmon[32727]: 2022-03-21T17:30:39.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:40.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:40.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:40 smithi016 ceph-mon[22010]: pgmap v556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:40 smithi079 ceph-mon[23158]: pgmap v556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:40 smithi016 conmon[32727]: 2022-03-21T17:30:40.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:41.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:41.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:41 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:41.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:41 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:41 smithi016 conmon[32727]: 2022-03-21T17:30:41.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:42.594 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:42.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:42 smithi016 ceph-mon[22010]: pgmap v557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:42 smithi079 ceph-mon[23158]: pgmap v557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:42 smithi016 conmon[32727]: 2022-03-21T17:30:42.592+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:43.547 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:43.548 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:43.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:43.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:43 smithi016 conmon[32727]: 2022-03-21T17:30:43.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:44.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:44.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:44 smithi016 ceph-mon[22010]: pgmap v558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:44 smithi079 ceph-mon[23158]: pgmap v558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:44 smithi016 conmon[32727]: 2022-03-21T17:30:44.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:45.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:45.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:45.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:45.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:45 smithi016 conmon[32727]: 2022-03-21T17:30:45.584+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:46.562 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:46.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:46 smithi016 ceph-mon[22010]: pgmap v559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:46 smithi079 ceph-mon[23158]: pgmap v559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:46 smithi016 conmon[32727]: 2022-03-21T17:30:46.560+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:47.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:47 smithi016 conmon[32727]: 2022-03-21T17:30:47.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:48.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:48.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:48.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:48 smithi016 ceph-mon[22010]: pgmap v560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:48 smithi079 ceph-mon[23158]: pgmap v560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:48 smithi016 conmon[32727]: 2022-03-21T17:30:48.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:49.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:49.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:49.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:49.844 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:49.844 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:49 smithi016 conmon[32727]: 2022-03-21T17:30:49.652+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:50.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:50.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:50 smithi016 ceph-mon[22010]: pgmap v561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:50 smithi079 ceph-mon[23158]: pgmap v561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:50 smithi016 conmon[32727]: 2022-03-21T17:30:50.645+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:51.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:51 smithi016 conmon[32727]: 2022-03-21T17:30:51.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:52.611 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:52.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:52 smithi016 ceph-mon[22010]: pgmap v562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:52 smithi079 ceph-mon[23158]: pgmap v562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:52 smithi016 conmon[32727]: 2022-03-21T17:30:52.609+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:53.572 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:53.572 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:53 smithi016 conmon[32727]: 2022-03-21T17:30:53.570+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:54.529 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:54.530 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:54 smithi016 ceph-mon[22010]: pgmap v563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:54 smithi079 ceph-mon[23158]: pgmap v563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:30:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:54 smithi016 conmon[32727]: 2022-03-21T17:30:54.528+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:55 smithi016 conmon[32727]: 2022-03-21T17:30:55.493+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:56 smithi016 ceph-mon[22010]: pgmap v564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:56 smithi016 conmon[32727]: 2022-03-21T17:30:56.475+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:56 smithi079 ceph-mon[23158]: pgmap v564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:57 smithi016 conmon[32727]: 2022-03-21T17:30:57.447+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:30:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:30:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:58 smithi016 conmon[32727]: 2022-03-21T17:30:58.472+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:30:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:30:58.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:30:58.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:30:58.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:58.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:58 smithi016 ceph-mon[22010]: pgmap v565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:30:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:30:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:30:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:30:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:30:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:58 smithi079 ceph-mon[23158]: pgmap v565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:30:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:30:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:30:59.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:30:59 smithi016 conmon[32727]: 2022-03-21T17:30:59.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:30:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:30:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:00 smithi016 ceph-mon[22010]: pgmap v566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:00 smithi016 conmon[32727]: 2022-03-21T17:31:00.473+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:00 smithi079 ceph-mon[23158]: pgmap v566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:01 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:01 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:01 smithi016 conmon[32727]: 2022-03-21T17:31:01.432+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:01 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:01 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:01.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:02 smithi016 ceph-mon[22010]: pgmap v567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:02 smithi016 conmon[32727]: 2022-03-21T17:31:02.447+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:02 smithi079 ceph-mon[23158]: pgmap v567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:03 smithi016 conmon[32727]: 2022-03-21T17:31:03.460+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[32727]: 2022-03-21T17:31:04.442+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:04 smithi016 ceph-mon[22010]: pgmap v568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:04 smithi079 ceph-mon[23158]: pgmap v568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:05.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: 2022-03-21T17:31:04.845+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:31:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: yield 2022-03-21T17:31:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:31:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:31:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:31:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:31:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:31:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:05 smithi016 conmon[32727]: 2022-03-21T17:31:05.436+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:05.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:31:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:31:05.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:31:05.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:05.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:31:05.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:05.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:31:05.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:05.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:05.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:05.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:31:05.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:05.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:31:05.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:05.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:31:05.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:05.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:05.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:05.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:31:05.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:05.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:05.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.679 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:05 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:31:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:31:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:31:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:31:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:31:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:31:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:31:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:31:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:31:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:31:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:05 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:31:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:06 smithi016 conmon[32727]: 2022-03-21T17:31:06.414+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:06.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:06.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:06 smithi016 ceph-mon[22010]: pgmap v569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:06 smithi079 ceph-mon[23158]: pgmap v569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[32727]: 2022-03-21T17:31:07.436+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:08.313 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 2022-03-21T17:31:07.916+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.314 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:31:08.314 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.314 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: yield 2022-03-21T17:31:08.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.315 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:31:08.316 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:31:08.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.317 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:31:08.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:31:08.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 2022-03-21T17:31:07.917+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:31:08.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:31:08.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: yield 2022-03-21T17:31:08.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.323 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.323 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.324 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:31:08.324 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:31:08.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:31:08.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:31:08.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 2022-03-21T17:31:08.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:31:08.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: 2022-03-21T17:31:08.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:31:08.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:31:08.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:31:08.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:31:08.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:31:08.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:31:08.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:31:08.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:31:08.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:31:08.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:31:08.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:31:08.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:31:08.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:31:08.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:31:08.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:31:08.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:31:08.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:31:08.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:31:08.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:31:08.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:31:08.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:31:08.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:31:08.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise 2022-03-21T17:31:08.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:31:08.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:31:08.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:31:08.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:31:08.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:31:07 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:08 smithi016 conmon[32727]: 2022-03-21T17:31:08.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:31:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:31:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:31:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:31:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:31:08.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:08 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:31:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:31:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:31:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:31:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:31:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:31:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:31:08.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:31:08.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:31:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:31:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:31:08.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:31:08.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:31:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:31:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:31:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:31:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:31:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:31:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:31:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:31:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:31:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:31:08.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:31:08.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:31:08.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:31:08.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:31:08.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:31:08.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:31:08.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:31:08.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:31:08.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:31:08.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:31:08.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:31:08.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:31:08.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:31:08.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.692 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:08.692 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:08 smithi016 ceph-mon[22010]: pgmap v570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:31:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:31:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:31:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:31:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:31:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:08 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:31:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:31:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:31:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:31:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:31:08.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:31:08.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:31:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:31:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:31:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:31:08.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:31:08.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:31:08.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:31:08.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:31:08.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:31:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:31:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:31:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:31:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:31:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:31:08.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:31:08.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:31:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:31:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:31:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:31:08.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:31:08.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:31:08.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:31:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:31:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:31:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:31:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:31:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:31:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:31:08.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:31:08.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:31:08.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:31:08.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:31:08.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:31:08.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:31:08.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:31:08.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:31:08.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:31:08.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:31:08.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:31:08.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:31:08.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:31:08.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:31:08.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:08.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:08 smithi079 ceph-mon[23158]: pgmap v570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:09 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:09 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:09 smithi016 conmon[32727]: 2022-03-21T17:31:09.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:09 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:09 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:10 smithi016 conmon[32727]: 2022-03-21T17:31:10.553+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:10 smithi016 ceph-mon[22010]: pgmap v571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:10 smithi079 ceph-mon[23158]: pgmap v571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:11.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:11 smithi016 conmon[32727]: 2022-03-21T17:31:11.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:12.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:12.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:12 smithi016 ceph-mon[22010]: pgmap v572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:12 smithi079 ceph-mon[23158]: pgmap v572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:12 smithi016 conmon[32727]: 2022-03-21T17:31:12.641+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:13.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:13.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:13 smithi016 conmon[32727]: 2022-03-21T17:31:13.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:14.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:14.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:14 smithi016 ceph-mon[22010]: pgmap v573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:14 smithi079 ceph-mon[23158]: pgmap v573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:14 smithi016 conmon[32727]: 2022-03-21T17:31:14.626+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:15.599 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:15 smithi016 conmon[32727]: 2022-03-21T17:31:15.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:16.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:16.614 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:16 smithi016 ceph-mon[22010]: pgmap v574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:16 smithi079 ceph-mon[23158]: pgmap v574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:16 smithi016 conmon[32727]: 2022-03-21T17:31:16.610+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:17 smithi016 conmon[32727]: 2022-03-21T17:31:17.582+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:17 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:17.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:17 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:17.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:17 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:17 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:17.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:18.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:18.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:18.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:18 smithi016 ceph-mon[22010]: pgmap v575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:18 smithi079 ceph-mon[23158]: pgmap v575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:18 smithi016 conmon[32727]: 2022-03-21T17:31:18.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:19.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:19 smithi016 conmon[32727]: 2022-03-21T17:31:19.619+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:20 smithi016 conmon[32727]: 2022-03-21T17:31:20.595+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:20 smithi016 ceph-mon[22010]: pgmap v576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:20 smithi079 ceph-mon[23158]: pgmap v576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:21.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:21 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:21.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:21 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:21.631 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:21 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:21 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:21 smithi016 conmon[32727]: 2022-03-21T17:31:21.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:22 smithi016 ceph-mon[22010]: pgmap v577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:22 smithi079 ceph-mon[23158]: pgmap v577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:22 smithi016 conmon[32727]: 2022-03-21T17:31:22.673+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:23 smithi016 conmon[32727]: 2022-03-21T17:31:23.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:24 smithi016 ceph-mon[22010]: pgmap v578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:24 smithi079 ceph-mon[23158]: pgmap v578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:24 smithi016 conmon[32727]: 2022-03-21T17:31:24.671+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:25.659 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:25 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:25.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:25 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:25.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:25 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:25 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:25 smithi016 conmon[32727]: 2022-03-21T17:31:25.655+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:26.649 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:26.649 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:26 smithi016 ceph-mon[22010]: pgmap v579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:26 smithi079 ceph-mon[23158]: pgmap v579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:26 smithi016 conmon[32727]: 2022-03-21T17:31:26.646+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:27 smithi016 conmon[32727]: 2022-03-21T17:31:27.684+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:28.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:28.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:28.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:28 smithi016 ceph-mon[22010]: pgmap v580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:28.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:28 smithi079 ceph-mon[23158]: pgmap v580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:28 smithi016 conmon[32727]: 2022-03-21T17:31:28.697+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:29 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:29 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:29 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:29 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:29 smithi016 conmon[32727]: 2022-03-21T17:31:29.708+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:30 smithi016 ceph-mon[22010]: pgmap v581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:30 smithi079 ceph-mon[23158]: pgmap v581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:30 smithi016 conmon[32727]: 2022-03-21T17:31:30.708+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:31 smithi016 conmon[32727]: 2022-03-21T17:31:31.694+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:32 smithi016 ceph-mon[22010]: pgmap v582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:32 smithi079 ceph-mon[23158]: pgmap v582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:32 smithi016 conmon[32727]: 2022-03-21T17:31:32.737+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:33 smithi016 conmon[32727]: 2022-03-21T17:31:33.718+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:34 smithi016 ceph-mon[22010]: pgmap v583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:34 smithi079 ceph-mon[23158]: pgmap v583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:34 smithi016 conmon[32727]: 2022-03-21T17:31:34.712+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:35 smithi016 conmon[32727]: 2022-03-21T17:31:35.679+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:36 smithi016 ceph-mon[22010]: pgmap v584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:36 smithi016 conmon[32727]: 2022-03-21T17:31:36.651+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:36 smithi079 ceph-mon[23158]: pgmap v584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:37 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:37 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:37 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:37 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:37 smithi016 conmon[32727]: 2022-03-21T17:31:37.680+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:38 smithi016 ceph-mon[22010]: pgmap v585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:38.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:38 smithi016 conmon[32727]: 2022-03-21T17:31:38.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:38 smithi079 ceph-mon[23158]: pgmap v585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:39 smithi016 conmon[32727]: 2022-03-21T17:31:39.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:40 smithi079 ceph-mon[23158]: pgmap v586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:40 smithi016 ceph-mon[22010]: pgmap v586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:40.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:40 smithi016 conmon[32727]: 2022-03-21T17:31:40.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:41 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:41 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:41 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:41 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:41.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:41 smithi016 conmon[32727]: 2022-03-21T17:31:41.639+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:42 smithi079 ceph-mon[23158]: pgmap v587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:42 smithi016 ceph-mon[22010]: pgmap v587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:42 smithi016 conmon[32727]: 2022-03-21T17:31:42.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:43 smithi016 conmon[32727]: 2022-03-21T17:31:43.674+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:44.707 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:44.707 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:44 smithi016 ceph-mon[22010]: pgmap v588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:44 smithi079 ceph-mon[23158]: pgmap v588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:44 smithi016 conmon[32727]: 2022-03-21T17:31:44.704+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:45 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:45 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:45.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:45.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:45 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:45.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:45 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:45.751 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:45 smithi016 conmon[32727]: 2022-03-21T17:31:45.748+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:46 smithi079 ceph-mon[23158]: pgmap v589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:46.771 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:46.771 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:46 smithi016 ceph-mon[22010]: pgmap v589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:46 smithi016 conmon[32727]: 2022-03-21T17:31:46.769+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:47.798 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:47 smithi016 conmon[32727]: 2022-03-21T17:31:47.796+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:48 smithi079 ceph-mon[23158]: pgmap v590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:48.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:48.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:48.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:48 smithi016 ceph-mon[22010]: pgmap v590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:48 smithi016 conmon[32727]: 2022-03-21T17:31:48.821+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:49 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:49 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:49.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:49 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:49.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:49 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:49.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:49 smithi016 conmon[32727]: 2022-03-21T17:31:49.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:50 smithi079 ceph-mon[23158]: pgmap v591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:50.874 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:50 smithi016 ceph-mon[22010]: pgmap v591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:50 smithi016 conmon[32727]: 2022-03-21T17:31:50.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:51.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:51 smithi016 conmon[32727]: 2022-03-21T17:31:51.862+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:52 smithi079 ceph-mon[23158]: pgmap v592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:52.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:52.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:52 smithi016 ceph-mon[22010]: pgmap v592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:52 smithi016 conmon[32727]: 2022-03-21T17:31:52.908+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:53.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:53.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:53 smithi016 conmon[32727]: 2022-03-21T17:31:53.870+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:54 smithi079 ceph-mon[23158]: pgmap v593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:54.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:54.861 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:54 smithi016 ceph-mon[22010]: pgmap v593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:31:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:54 smithi016 conmon[32727]: 2022-03-21T17:31:54.858+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:55.867 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:55 smithi016 conmon[32727]: 2022-03-21T17:31:55.865+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:56.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:56.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:56 smithi016 ceph-mon[22010]: pgmap v594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:56 smithi079 ceph-mon[23158]: pgmap v594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:56 smithi016 conmon[32727]: 2022-03-21T17:31:56.908+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:57 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:57 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:57 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:31:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:57 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:31:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:57 smithi016 conmon[32727]: 2022-03-21T17:31:57.938+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:31:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:31:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:31:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:31:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 561 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:58 smithi016 ceph-mon[22010]: pgmap v595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:31:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:31:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:31:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:31:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 561 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:31:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:58 smithi079 ceph-mon[23158]: pgmap v595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:31:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:58 smithi016 conmon[32727]: 2022-03-21T17:31:58.985+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:31:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:31:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:31:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:31:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:31:59 smithi016 conmon[32727]: 2022-03-21T17:31:59.969+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:00 smithi016 ceph-mon[22010]: pgmap v596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:00 smithi079 ceph-mon[23158]: pgmap v596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:00 smithi016 conmon[32727]: 2022-03-21T17:32:00.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:01 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:01 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:01 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:01 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:01 smithi016 conmon[32727]: 2022-03-21T17:32:01.934+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:02 smithi016 ceph-mon[22010]: pgmap v597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:02 smithi079 ceph-mon[23158]: pgmap v597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:02 smithi016 conmon[32727]: 2022-03-21T17:32:02.926+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:03.885 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:03.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:03 smithi016 conmon[32727]: 2022-03-21T17:32:03.883+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:04 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:04 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:04 smithi016 ceph-mon[22010]: pgmap v598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:04 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:04 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:04 smithi079 ceph-mon[23158]: pgmap v598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:04 smithi016 conmon[32727]: 2022-03-21T17:32:04.915+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:05.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:05 smithi016 conmon[32727]: 2022-03-21T17:32:05.932+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:06 smithi016 ceph-mon[22010]: pgmap v599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:06 smithi079 ceph-mon[23158]: pgmap v599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:06 smithi016 conmon[32727]: 2022-03-21T17:32:06.959+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:07.966 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:08.383 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:07 smithi016 conmon[32727]: 2022-03-21T17:32:07.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:32:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:32:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:32:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 571 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:08 smithi016 ceph-mon[22010]: pgmap v600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:32:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:32:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:32:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 571 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:08 smithi079 ceph-mon[23158]: pgmap v600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:08 smithi016 conmon[32727]: 2022-03-21T17:32:08.913+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:09 smithi016 conmon[32727]: 2022-03-21T17:32:09.961+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:10 smithi016 ceph-mon[22010]: pgmap v601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:10 smithi079 ceph-mon[23158]: pgmap v601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:10 smithi016 conmon[32727]: 2022-03-21T17:32:10.934+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: 2022-03-21T17:32:11.020+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:32:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: yield 2022-03-21T17:32:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:32:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:32:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:32:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:32:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:32:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:32:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:32:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:32:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:32:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:32:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:32:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:11.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:11.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:11 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:32:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:32:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:32:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:32:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:32:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:32:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:11.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:11.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:32:11.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:11.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:11.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:11.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:11 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:32:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:11 smithi016 conmon[32727]: 2022-03-21T17:32:11.928+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:12.905 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:12.906 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:12 smithi016 ceph-mon[22010]: pgmap v602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:12 smithi079 ceph-mon[23158]: pgmap v602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:12 smithi016 conmon[32727]: 2022-03-21T17:32:12.902+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:13.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:13.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:13 smithi016 conmon[32727]: 2022-03-21T17:32:13.907+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:14.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 2022-03-21T17:32:14.092+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:32:14.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: yield 2022-03-21T17:32:14.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:32:14.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:32:14.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:32:14.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:32:14.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 2022-03-21T17:32:14.093+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:32:14.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:32:14.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: yield 2022-03-21T17:32:14.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:32:14.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:32:14.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:32:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:32:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 2022-03-21T17:32:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:32:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: 2022-03-21T17:32:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:32:14.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:32:14.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:32:14.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:32:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:32:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:32:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:32:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:32:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:32:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:32:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:32:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:32:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:32:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:32:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:32:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:32:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:32:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:32:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:32:14.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:32:14.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:32:14.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:32:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise 2022-03-21T17:32:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:32:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:32:14.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:32:14.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:32:14.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:14.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:14 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:32:14.902 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.902 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.903 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.903 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:32:14.904 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:32:14.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:32:14.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:32:14.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:14 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:32:14.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:32:14.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.912 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.912 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:32:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:32:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:32:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:32:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:32:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:32:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:32:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:32:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:32:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:32:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:32:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:32:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:32:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:32:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:32:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:32:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:32:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:32:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:32:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:32:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:32:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:32:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:32:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:32:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:32:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:32:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:32:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:32:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:32:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:32:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:32:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:32:14.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:32:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:14 smithi016 ceph-mon[22010]: pgmap v603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:14 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:32:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:32:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:32:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:32:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:32:14.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:14 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:32:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:32:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:32:14.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:32:14.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:32:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:32:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:32:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:32:14.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:32:14.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:32:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:32:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:32:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:32:14.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:32:14.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:32:14.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:32:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:32:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:32:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:32:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:32:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:32:14.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:32:14.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:32:14.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:32:14.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:32:14.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:32:14.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:32:15.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:32:15.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:32:15.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:32:15.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:32:15.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:32:15.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:32:15.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:32:15.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:32:15.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:32:15.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:32:15.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:32:15.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:32:15.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:32:15.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:32:15.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:32:15.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:32:15.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:32:15.007 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:32:15.007 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:32:15.007 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:32:15.008 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:32:15.008 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:14 smithi079 ceph-mon[23158]: pgmap v603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:14 smithi016 conmon[32727]: 2022-03-21T17:32:14.896+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:15 smithi016 conmon[32727]: 2022-03-21T17:32:15.921+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:16 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:16 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:16 smithi016 ceph-mon[22010]: pgmap v604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:16 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:16 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:16 smithi079 ceph-mon[23158]: pgmap v604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:16 smithi016 conmon[32727]: 2022-03-21T17:32:16.922+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:17 smithi016 conmon[32727]: 2022-03-21T17:32:17.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:18 smithi016 ceph-mon[22010]: pgmap v605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:18 smithi079 ceph-mon[23158]: pgmap v605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:18 smithi016 conmon[32727]: 2022-03-21T17:32:18.968+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:19 smithi016 conmon[32727]: 2022-03-21T17:32:19.939+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:20.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:20.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:20 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:20.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:20 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:20.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:20 smithi016 ceph-mon[22010]: pgmap v606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:20 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:20 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:20.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:20 smithi079 ceph-mon[23158]: pgmap v606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:20 smithi016 conmon[32727]: 2022-03-21T17:32:20.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:22.048 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:21 smithi016 conmon[32727]: 2022-03-21T17:32:21.917+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:22.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:22.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:22 smithi016 ceph-mon[22010]: pgmap v607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:22 smithi079 ceph-mon[23158]: pgmap v607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:22 smithi016 conmon[32727]: 2022-03-21T17:32:22.962+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:24 smithi016 conmon[32727]: 2022-03-21T17:32:24.001+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:24 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:24 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:24 smithi016 ceph-mon[22010]: pgmap v608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:24 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:24 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:24 smithi079 ceph-mon[23158]: pgmap v608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:24 smithi016 conmon[32727]: 2022-03-21T17:32:24.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:26 smithi016 conmon[32727]: 2022-03-21T17:32:26.020+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:26 smithi016 ceph-mon[22010]: pgmap v609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:26 smithi079 ceph-mon[23158]: pgmap v609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:26 smithi016 conmon[32727]: 2022-03-21T17:32:26.992+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:28.424 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:28 smithi016 conmon[32727]: 2022-03-21T17:32:28.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:28 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:28 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:28 smithi016 ceph-mon[22010]: pgmap v610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:28 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:28 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:28 smithi079 ceph-mon[23158]: pgmap v610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:29 smithi016 conmon[32727]: 2022-03-21T17:32:29.062+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:30 smithi016 conmon[32727]: 2022-03-21T17:32:30.047+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:30.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:30.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:30 smithi016 ceph-mon[22010]: pgmap v611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:30 smithi079 ceph-mon[23158]: pgmap v611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:31 smithi016 conmon[32727]: 2022-03-21T17:32:31.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:31 smithi016 conmon[32727]: 2022-03-21T17:32:31.988+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:32 smithi016 ceph-mon[22010]: pgmap v612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:32 smithi079 ceph-mon[23158]: pgmap v612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:32 smithi016 conmon[32727]: 2022-03-21T17:32:32.988+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:34 smithi016 conmon[32727]: 2022-03-21T17:32:34.029+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:34 smithi016 ceph-mon[22010]: pgmap v613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:34 smithi079 ceph-mon[23158]: pgmap v613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:35 smithi016 conmon[32727]: 2022-03-21T17:32:35.052+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:35 smithi016 ceph-mon[22010]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 899 secs 2022-03-21T17:32:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:35 smithi079 ceph-mon[23158]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 899 secs 2022-03-21T17:32:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:36 smithi016 conmon[32727]: 2022-03-21T17:32:36.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:36 smithi016 ceph-mon[22010]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2022-03-21T17:32:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:36 smithi016 ceph-mon[22010]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:36 smithi016 ceph-mon[22010]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:36 smithi016 ceph-mon[22010]: pgmap v614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:36 smithi079 ceph-mon[23158]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2022-03-21T17:32:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:36 smithi079 ceph-mon[23158]: mds.? [v2:172.21.15.79:6838/1907413140,v1:172.21.15.79:6839/1907413140] up:active 2022-03-21T17:32:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:36 smithi079 ceph-mon[23158]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-03-21T17:32:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:36 smithi079 ceph-mon[23158]: pgmap v614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:37 smithi016 conmon[32727]: 2022-03-21T17:32:37.107+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:37.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:38 smithi016 conmon[32727]: 2022-03-21T17:32:38.078+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:38 smithi016 ceph-mon[22010]: pgmap v615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:38 smithi079 ceph-mon[23158]: pgmap v615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-03-21T17:32:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:39 smithi016 conmon[32727]: 2022-03-21T17:32:39.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:40 smithi016 conmon[32727]: 2022-03-21T17:32:40.028+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:40 smithi016 ceph-mon[22010]: pgmap v616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:40 smithi079 ceph-mon[23158]: pgmap v616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-03-21T17:32:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:41 smithi016 conmon[32727]: 2022-03-21T17:32:41.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:42 smithi016 conmon[32727]: 2022-03-21T17:32:42.053+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:42 smithi016 ceph-mon[22010]: pgmap v617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:42 smithi079 ceph-mon[23158]: pgmap v617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:43 smithi016 conmon[32727]: 2022-03-21T17:32:43.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:44 smithi016 conmon[32727]: 2022-03-21T17:32:44.090+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:44 smithi016 ceph-mon[22010]: pgmap v618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:44 smithi079 ceph-mon[23158]: pgmap v618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:45 smithi016 conmon[32727]: 2022-03-21T17:32:45.079+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:46 smithi016 conmon[32727]: 2022-03-21T17:32:46.121+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:46 smithi016 ceph-mon[22010]: pgmap v619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:46 smithi079 ceph-mon[23158]: pgmap v619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:47 smithi016 conmon[32727]: 2022-03-21T17:32:47.155+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:48 smithi016 conmon[32727]: 2022-03-21T17:32:48.125+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:48 smithi016 ceph-mon[22010]: pgmap v620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:48.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:48 smithi079 ceph-mon[23158]: pgmap v620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:49 smithi016 conmon[32727]: 2022-03-21T17:32:49.148+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:50.640 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:50 smithi016 conmon[32727]: 2022-03-21T17:32:50.197+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:50 smithi016 ceph-mon[22010]: pgmap v621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:50.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:50 smithi079 ceph-mon[23158]: pgmap v621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:51.642 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:51 smithi016 conmon[32727]: 2022-03-21T17:32:51.180+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:52.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:52 smithi016 conmon[32727]: 2022-03-21T17:32:52.224+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:52 smithi016 ceph-mon[22010]: pgmap v622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:52 smithi079 ceph-mon[23158]: pgmap v622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:53.647 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:53 smithi016 conmon[32727]: 2022-03-21T17:32:53.264+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:54.652 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:54 smithi016 conmon[32727]: 2022-03-21T17:32:54.281+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:54 smithi016 ceph-mon[22010]: pgmap v623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:54 smithi079 ceph-mon[23158]: pgmap v623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:32:55.654 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:55 smithi016 conmon[32727]: 2022-03-21T17:32:55.303+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:56.659 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:56 smithi016 conmon[32727]: 2022-03-21T17:32:56.261+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:56 smithi016 ceph-mon[22010]: pgmap v624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:56 smithi079 ceph-mon[23158]: pgmap v624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:57.660 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:57 smithi016 conmon[32727]: 2022-03-21T17:32:57.273+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:58.661 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:58 smithi016 conmon[32727]: 2022-03-21T17:32:58.306+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:32:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:32:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:32:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:32:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:58 smithi016 ceph-mon[22010]: pgmap v625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:32:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:32:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:32:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:32:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:32:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:58 smithi079 ceph-mon[23158]: pgmap v625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:32:59.663 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:32:59 smithi016 conmon[32727]: 2022-03-21T17:32:59.310+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:32:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:32:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:32:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:32:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:00 smithi016 conmon[32727]: 2022-03-21T17:33:00.333+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:00 smithi079 ceph-mon[23158]: pgmap v626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:00 smithi016 ceph-mon[22010]: pgmap v626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:01 smithi016 conmon[32727]: 2022-03-21T17:33:01.304+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:02 smithi016 conmon[32727]: 2022-03-21T17:33:02.353+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:02.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:02 smithi079 ceph-mon[23158]: pgmap v627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:02 smithi016 ceph-mon[22010]: pgmap v627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:03 smithi016 conmon[32727]: 2022-03-21T17:33:03.327+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:04 smithi016 conmon[32727]: 2022-03-21T17:33:04.329+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:04 smithi079 ceph-mon[23158]: pgmap v628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:04 smithi016 ceph-mon[22010]: pgmap v628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:05 smithi016 conmon[32727]: 2022-03-21T17:33:05.314+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:06 smithi016 conmon[32727]: 2022-03-21T17:33:06.316+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:06 smithi079 ceph-mon[23158]: pgmap v629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:06 smithi016 ceph-mon[22010]: pgmap v629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:07 smithi016 conmon[32727]: 2022-03-21T17:33:07.328+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:08 smithi016 conmon[32727]: 2022-03-21T17:33:08.352+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:08 smithi079 ceph-mon[23158]: pgmap v630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:09.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:08 smithi016 ceph-mon[22010]: pgmap v630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:09.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:09 smithi016 conmon[32727]: 2022-03-21T17:33:09.375+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:09.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:10 smithi016 conmon[32727]: 2022-03-21T17:33:10.337+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:10 smithi079 ceph-mon[23158]: pgmap v631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:10 smithi016 ceph-mon[22010]: pgmap v631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:11.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:11 smithi016 conmon[32727]: 2022-03-21T17:33:11.290+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:12 smithi016 conmon[32727]: 2022-03-21T17:33:12.289+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:12 smithi079 ceph-mon[23158]: pgmap v632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:12 smithi016 ceph-mon[22010]: pgmap v632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:13 smithi016 conmon[32727]: 2022-03-21T17:33:13.304+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:14.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:14.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:14 smithi016 conmon[32727]: 2022-03-21T17:33:14.274+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:33:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:33:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:33:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:14 smithi016 ceph-mon[22010]: pgmap v633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:33:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:33:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:33:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:14 smithi079 ceph-mon[23158]: pgmap v633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:15.536 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[32727]: 2022-03-21T17:33:15.248+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:15.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: 2022-03-21T17:33:15.532+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:15.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:33:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: yield 2022-03-21T17:33:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:33:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:33:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:15.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:33:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:33:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:33:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:15.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:16 smithi016 conmon[32727]: 2022-03-21T17:33:16.218+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:16 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:33:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:33:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:33:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:33:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:33:16.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:16.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:16.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:16.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:33:16.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:16.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:16.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:16.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:16 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:33:16.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:16 smithi079 ceph-mon[23158]: pgmap v634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:16 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:33:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:17.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:33:17.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:17.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:17.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:17.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:33:17.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:17.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:33:17.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:17.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:33:17.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:17.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:17.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:17.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:33:17.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:17.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:17.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:17.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:16 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:33:17.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:16 smithi016 ceph-mon[22010]: pgmap v634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:17 smithi016 conmon[32727]: 2022-03-21T17:33:17.209+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:18.607 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[32727]: 2022-03-21T17:33:18.193+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:18 smithi016 ceph-mon[22010]: pgmap v635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:18.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 2022-03-21T17:33:18.604+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:18.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:33:18.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:18.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: yield 2022-03-21T17:33:18.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:18.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:18.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:18.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:33:18.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:18.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:33:18.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:18.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:33:18.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:18.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:33:18.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:18.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:33:18.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:18.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:18.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:18.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 2022-03-21T17:33:18.605+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:33:18.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:33:18.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:18.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: yield 2022-03-21T17:33:18.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:18.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:18.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:18.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:33:18.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:18.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:33:18.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:18.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:33:18.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:18.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:33:18.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:18.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:33:18.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:18.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:18.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:18.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 2022-03-21T17:33:18.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:33:18.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: 2022-03-21T17:33:18.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:33:18.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:33:18.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:33:18.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:33:18.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:33:18.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:33:18.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:33:18.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:33:18.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:33:18.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:33:18.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:33:18.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:33:18.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:33:18.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:33:18.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:33:18.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:33:18.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:33:18.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:33:18.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:33:18.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:33:18.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:33:18.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:33:18.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise 2022-03-21T17:33:18.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:33:18.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:33:18.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:33:18.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:33:18.960 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:33:18 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:18 smithi079 ceph-mon[23158]: pgmap v635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:19 smithi016 conmon[32727]: 2022-03-21T17:33:19.234+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:19 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:33:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:33:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:33:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:33:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:33:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:19.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:33:19.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:19 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:33:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:33:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:33:19.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:19.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:19.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:19.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:33:19.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:19.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:33:19.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:33:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:33:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:19.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:33:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:33:19.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:33:19.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:33:19.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:33:19.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:33:19.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:33:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:33:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:33:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:33:19.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:33:19.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:33:19.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:33:19.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:33:19.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:33:20.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:33:20.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:33:20.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:33:20.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:33:20.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:33:20.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:33:20.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:33:20.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:33:20.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:33:20.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:33:20.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:33:20.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:33:20.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:33:20.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:33:20.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:33:20.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:19 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:33:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:33:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:20.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:20.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:20.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:33:20.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:20.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:33:20.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:20.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:33:20.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:20.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:20.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:20.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:33:20.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:20.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:20.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:20.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:19 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:33:20.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:33:20.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:33:20.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:33:20.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:33:20.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:33:20.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:33:20.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:33:20.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:33:20.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:33:20.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:33:20.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:33:20.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:33:20.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:33:20.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:33:20.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:33:20.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:33:20.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:33:20.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:20.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:33:20.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:33:20.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:33:20.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:33:20.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:33:20.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:33:20.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:33:20.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:33:20.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:33:20.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:33:20.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:33:20.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:33:20.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:33:20.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:33:20.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:33:20.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:33:20.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:33:20.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:33:20.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:33:20.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:33:20.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:33:20.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:33:20.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:33:20.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:33:20.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:33:20.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:33:20.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:33:20.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:33:20.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:33:20.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:33:20.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:33:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:20 smithi016 conmon[32727]: 2022-03-21T17:33:20.249+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:21.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:21.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:20 smithi016 ceph-mon[22010]: pgmap v636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:20 smithi079 ceph-mon[23158]: pgmap v636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:21 smithi016 conmon[32727]: 2022-03-21T17:33:21.278+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:22 smithi016 conmon[32727]: 2022-03-21T17:33:22.329+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:23 smithi016 conmon[32727]: 2022-03-21T17:33:23.342+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:23 smithi016 ceph-mon[22010]: pgmap v637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:23 smithi079 ceph-mon[23158]: pgmap v637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:24 smithi016 conmon[32727]: 2022-03-21T17:33:24.377+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:25.739 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:25 smithi016 conmon[32727]: 2022-03-21T17:33:25.422+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:25 smithi016 ceph-mon[22010]: pgmap v638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:25 smithi079 ceph-mon[23158]: pgmap v638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:26 smithi016 conmon[32727]: 2022-03-21T17:33:26.399+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:27 smithi016 conmon[32727]: 2022-03-21T17:33:27.384+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:27 smithi016 ceph-mon[22010]: pgmap v639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:27 smithi079 ceph-mon[23158]: pgmap v639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:28.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:28.747 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:28 smithi016 conmon[32727]: 2022-03-21T17:33:28.421+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:29.236 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:29.236 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:29.747 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:29 smithi016 conmon[32727]: 2022-03-21T17:33:29.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:29 smithi016 ceph-mon[22010]: pgmap v640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:29 smithi079 ceph-mon[23158]: pgmap v640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:30.752 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:30 smithi016 conmon[32727]: 2022-03-21T17:33:30.453+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:31.753 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:31 smithi016 conmon[32727]: 2022-03-21T17:33:31.421+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:31 smithi016 ceph-mon[22010]: pgmap v641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:31 smithi079 ceph-mon[23158]: pgmap v641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:32 smithi016 conmon[32727]: 2022-03-21T17:33:32.380+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:33 smithi016 conmon[32727]: 2022-03-21T17:33:33.370+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:33 smithi016 ceph-mon[22010]: pgmap v642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:33 smithi079 ceph-mon[23158]: pgmap v642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:34 smithi016 conmon[32727]: 2022-03-21T17:33:34.390+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:35 smithi016 conmon[32727]: 2022-03-21T17:33:35.367+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:35 smithi016 ceph-mon[22010]: pgmap v643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:35 smithi079 ceph-mon[23158]: pgmap v643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:36 smithi016 conmon[32727]: 2022-03-21T17:33:36.346+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:37 smithi016 conmon[32727]: 2022-03-21T17:33:37.336+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:37 smithi016 ceph-mon[22010]: pgmap v644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:37 smithi079 ceph-mon[23158]: pgmap v644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:38 smithi016 conmon[32727]: 2022-03-21T17:33:38.374+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:39.776 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:39 smithi016 conmon[32727]: 2022-03-21T17:33:39.414+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:39 smithi016 ceph-mon[22010]: pgmap v645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:39 smithi079 ceph-mon[23158]: pgmap v645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:40.781 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:40 smithi016 conmon[32727]: 2022-03-21T17:33:40.430+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:41.783 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:41 smithi016 conmon[32727]: 2022-03-21T17:33:41.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:41 smithi016 ceph-mon[22010]: pgmap v646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:41 smithi079 ceph-mon[23158]: pgmap v646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:42.789 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:42 smithi016 conmon[32727]: 2022-03-21T17:33:42.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:43.790 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:43 smithi016 conmon[32727]: 2022-03-21T17:33:43.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:43 smithi016 ceph-mon[22010]: pgmap v647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:43 smithi079 ceph-mon[23158]: pgmap v647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:44.795 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:44 smithi016 conmon[32727]: 2022-03-21T17:33:44.504+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:45.797 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:45 smithi016 conmon[32727]: 2022-03-21T17:33:45.481+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:45 smithi016 ceph-mon[22010]: pgmap v648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:45 smithi079 ceph-mon[23158]: pgmap v648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:46.802 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:46 smithi016 conmon[32727]: 2022-03-21T17:33:46.530+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:47.803 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:47 smithi016 conmon[32727]: 2022-03-21T17:33:47.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:47 smithi016 ceph-mon[22010]: pgmap v649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:47 smithi079 ceph-mon[23158]: pgmap v649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:48.805 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:48 smithi016 conmon[32727]: 2022-03-21T17:33:48.492+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:49.806 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:49 smithi016 conmon[32727]: 2022-03-21T17:33:49.465+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:49 smithi016 ceph-mon[22010]: pgmap v650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:49 smithi079 ceph-mon[23158]: pgmap v650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:50.811 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:50 smithi016 conmon[32727]: 2022-03-21T17:33:50.494+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:51.812 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:51 smithi016 conmon[32727]: 2022-03-21T17:33:51.490+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:51 smithi016 ceph-mon[22010]: pgmap v651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:51 smithi079 ceph-mon[23158]: pgmap v651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:52.817 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:52 smithi016 conmon[32727]: 2022-03-21T17:33:52.446+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:53.819 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:53 smithi016 conmon[32727]: 2022-03-21T17:33:53.460+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:53 smithi016 ceph-mon[22010]: pgmap v652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:53 smithi079 ceph-mon[23158]: pgmap v652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:54.824 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:54 smithi016 conmon[32727]: 2022-03-21T17:33:54.506+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:55.825 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:55 smithi016 conmon[32727]: 2022-03-21T17:33:55.463+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:55 smithi016 ceph-mon[22010]: pgmap v653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:55 smithi079 ceph-mon[23158]: pgmap v653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:33:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:56.830 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:56 smithi016 conmon[32727]: 2022-03-21T17:33:56.474+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:57.832 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:57 smithi016 conmon[32727]: 2022-03-21T17:33:57.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:57 smithi016 ceph-mon[22010]: pgmap v654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:57 smithi079 ceph-mon[23158]: pgmap v654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:33:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:58.834 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:58 smithi016 conmon[32727]: 2022-03-21T17:33:58.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:33:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:33:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:33:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:33:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:33:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:33:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:33:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:33:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:33:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:33:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:33:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:33:59 smithi016 conmon[32727]: 2022-03-21T17:33:59.405+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:59 smithi016 ceph-mon[22010]: pgmap v655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:33:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:59 smithi079 ceph-mon[23158]: pgmap v655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:33:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:00 smithi016 conmon[32727]: 2022-03-21T17:34:00.398+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:01.841 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:01 smithi016 conmon[32727]: 2022-03-21T17:34:01.426+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:01 smithi016 ceph-mon[22010]: pgmap v656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:01 smithi079 ceph-mon[23158]: pgmap v656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:02.846 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:02 smithi016 conmon[32727]: 2022-03-21T17:34:02.443+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:03.847 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:03 smithi016 conmon[32727]: 2022-03-21T17:34:03.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:03 smithi016 ceph-mon[22010]: pgmap v657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:03 smithi079 ceph-mon[23158]: pgmap v657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:04.852 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:04 smithi016 conmon[32727]: 2022-03-21T17:34:04.462+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:05.859 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:05 smithi016 conmon[32727]: 2022-03-21T17:34:05.419+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:05 smithi016 ceph-mon[22010]: pgmap v658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:05 smithi079 ceph-mon[23158]: pgmap v658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:06.864 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:06 smithi016 conmon[32727]: 2022-03-21T17:34:06.446+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:07 smithi016 conmon[32727]: 2022-03-21T17:34:07.405+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:07 smithi016 ceph-mon[22010]: pgmap v659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:07 smithi079 ceph-mon[23158]: pgmap v659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:08.867 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:08 smithi016 conmon[32727]: 2022-03-21T17:34:08.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:09.868 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:09 smithi016 conmon[32727]: 2022-03-21T17:34:09.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:09 smithi016 ceph-mon[22010]: pgmap v660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:09 smithi079 ceph-mon[23158]: pgmap v660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:10.873 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:10 smithi016 conmon[32727]: 2022-03-21T17:34:10.503+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:11.874 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:11 smithi016 conmon[32727]: 2022-03-21T17:34:11.484+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:11 smithi016 ceph-mon[22010]: pgmap v661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:11 smithi079 ceph-mon[23158]: pgmap v661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:12.879 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:12 smithi016 conmon[32727]: 2022-03-21T17:34:12.494+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:13.881 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:13 smithi016 conmon[32727]: 2022-03-21T17:34:13.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:13 smithi016 ceph-mon[22010]: pgmap v662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:13 smithi079 ceph-mon[23158]: pgmap v662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:14.890 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:14 smithi016 conmon[32727]: 2022-03-21T17:34:14.557+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:15.888 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:15 smithi016 conmon[32727]: 2022-03-21T17:34:15.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:15 smithi016 ceph-mon[22010]: pgmap v663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:15 smithi079 ceph-mon[23158]: pgmap v663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:16.892 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:16 smithi016 conmon[32727]: 2022-03-21T17:34:16.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:17.894 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:17 smithi016 conmon[32727]: 2022-03-21T17:34:17.593+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:17 smithi016 ceph-mon[22010]: pgmap v664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:17 smithi079 ceph-mon[23158]: pgmap v664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:18.895 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:18 smithi016 conmon[32727]: 2022-03-21T17:34:18.614+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:34:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:34:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:18 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:34:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:34:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:34:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:18 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:34:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:19 smithi016 conmon[32727]: 2022-03-21T17:34:19.624+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:20 smithi016 ceph-mon[22010]: pgmap v665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:20.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: 2022-03-21T17:34:20.044+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:20.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:34:20.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:20.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: yield 2022-03-21T17:34:20.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:20.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:20.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:20.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:34:20.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:20.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:34:20.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:20.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:34:20.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:20.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:34:20.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:20.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:34:20.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:20.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:20.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:20 smithi079 ceph-mon[23158]: pgmap v665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:20 smithi016 conmon[32727]: 2022-03-21T17:34:20.617+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:34:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:34:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:34:21.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:21.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:34:21.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:21.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:34:21.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:21.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:21.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:21.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:34:21.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:21.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:21.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:21.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:34:21.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:34:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:34:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:34:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:34:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:34:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:34:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:34:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:21 smithi016 conmon[32727]: 2022-03-21T17:34:21.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: pgmap v666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:34:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:34:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: pgmap v666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:22.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:34:22.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:34:22.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:22.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:34:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:22 smithi016 conmon[32727]: 2022-03-21T17:34:22.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:34:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:34:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:34:23.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:34:23.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:23.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 2022-03-21T17:34:23.116+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:34:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: yield 2022-03-21T17:34:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:34:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:34:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:34:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:34:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:34:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 2022-03-21T17:34:23.117+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:34:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:34:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: yield 2022-03-21T17:34:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:34:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:34:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:34:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:34:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:34:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 2022-03-21T17:34:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:34:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: 2022-03-21T17:34:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:34:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:34:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:34:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:34:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:34:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:34:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:34:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:34:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:34:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:34:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:34:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:34:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:34:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:34:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:34:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:34:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:34:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:34:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:34:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:34:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:34:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:34:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise 2022-03-21T17:34:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:34:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:34:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:34:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:34:23.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:34:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:34:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:34:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:34:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:23 smithi016 conmon[32727]: 2022-03-21T17:34:23.615+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:24 smithi016 ceph-mon[22010]: pgmap v667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:34:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:24.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:24.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:34:24.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:24.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:34:24.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:24.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:34:24.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:24.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:24.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:24.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:34:24.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:24.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:24.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:34:24.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:24.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:34:24.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:24.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:24.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:24.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:34:24.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:24.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:34:24.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:24.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:34:24.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:24.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:24.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:24.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:34:24.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:24.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:24.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:34:24.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:34:24.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:34:24.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:34:24.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:34:24.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:34:24.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:34:24.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:34:24.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:34:24.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:34:24.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:34:24.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:34:24.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:34:24.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:34:24.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:34:24.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:34:24.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:34:24.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:34:24.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:34:24.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:34:24.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:34:24.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:34:24.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:34:24.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:34:24.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:34:24.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:34:24.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:34:24.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:34:24.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:34:24.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:24 smithi079 ceph-mon[23158]: pgmap v667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:34:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:34:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:34:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:34:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:34:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:34:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:34:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:34:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:34:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:34:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:34:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:34:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:34:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:34:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:34:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:34:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:34:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:34:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:34:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:34:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:34:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:34:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:34:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:34:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:34:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:34:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:34:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:34:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:34:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:34:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:34:24.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:34:24.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:34:24.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:34:24.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:34:24.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:34:24.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:34:24.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:34:24.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:34:24.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:34:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:34:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:34:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:34:24.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:34:24.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:34:24.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:34:24.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:34:24.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:34:24.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:34:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:34:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:34:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:34:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:34:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:24 smithi016 conmon[32727]: 2022-03-21T17:34:24.587+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:25 smithi016 conmon[32727]: 2022-03-21T17:34:25.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:26 smithi079 ceph-mon[23158]: pgmap v668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:26.594 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:26 smithi016 ceph-mon[22010]: pgmap v668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:26.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:26 smithi016 conmon[32727]: 2022-03-21T17:34:26.592+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:27.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:27 smithi016 conmon[32727]: 2022-03-21T17:34:27.632+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:28 smithi079 ceph-mon[23158]: pgmap v669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:28 smithi016 ceph-mon[22010]: pgmap v669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:28 smithi016 conmon[32727]: 2022-03-21T17:34:28.664+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:29.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:29 smithi016 conmon[32727]: 2022-03-21T17:34:29.652+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:30 smithi079 ceph-mon[23158]: pgmap v670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:30.616 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:30 smithi016 ceph-mon[22010]: pgmap v670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:30.617 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:30 smithi016 conmon[32727]: 2022-03-21T17:34:30.614+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:31.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:31 smithi016 conmon[32727]: 2022-03-21T17:34:31.589+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:32 smithi079 ceph-mon[23158]: pgmap v671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:32.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:32 smithi016 ceph-mon[22010]: pgmap v671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:32.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:32 smithi016 conmon[32727]: 2022-03-21T17:34:32.576+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:33.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:33.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:33 smithi016 conmon[32727]: 2022-03-21T17:34:33.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:34 smithi079 ceph-mon[23158]: pgmap v672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:34.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:34 smithi016 ceph-mon[22010]: pgmap v672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:34.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:34 smithi016 conmon[32727]: 2022-03-21T17:34:34.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:35.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:35 smithi016 conmon[32727]: 2022-03-21T17:34:35.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:36 smithi079 ceph-mon[23158]: pgmap v673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:36.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:36 smithi016 ceph-mon[22010]: pgmap v673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:36.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:36 smithi016 conmon[32727]: 2022-03-21T17:34:36.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:37.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:37 smithi016 conmon[32727]: 2022-03-21T17:34:37.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:38 smithi079 ceph-mon[23158]: pgmap v674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:38.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:38 smithi016 ceph-mon[22010]: pgmap v674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:38.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:38.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:38 smithi016 conmon[32727]: 2022-03-21T17:34:38.626+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:39 smithi079 ceph-mon[23158]: pgmap v675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:39 smithi016 ceph-mon[22010]: pgmap v675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:39 smithi016 conmon[32727]: 2022-03-21T17:34:39.672+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:40.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:40.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:40 smithi016 conmon[32727]: 2022-03-21T17:34:40.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:41 smithi079 ceph-mon[23158]: pgmap v676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:41.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:41 smithi016 ceph-mon[22010]: pgmap v676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:41.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:41 smithi016 conmon[32727]: 2022-03-21T17:34:41.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:42.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:42 smithi016 conmon[32727]: 2022-03-21T17:34:42.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:43.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:43 smithi016 ceph-mon[22010]: pgmap v677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:43.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:43 smithi079 ceph-mon[23158]: pgmap v677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:43 smithi016 conmon[32727]: 2022-03-21T17:34:43.655+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:44.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:44 smithi016 conmon[32727]: 2022-03-21T17:34:44.655+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:45.659 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:45 smithi016 ceph-mon[22010]: pgmap v678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:45.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:45 smithi079 ceph-mon[23158]: pgmap v678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:45.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:45 smithi016 conmon[32727]: 2022-03-21T17:34:45.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:46 smithi016 conmon[32727]: 2022-03-21T17:34:46.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:47 smithi016 ceph-mon[22010]: pgmap v679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:47.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:47 smithi079 ceph-mon[23158]: pgmap v679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:47 smithi016 conmon[32727]: 2022-03-21T17:34:47.669+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:49.170 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:48 smithi016 conmon[32727]: 2022-03-21T17:34:48.680+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:49 smithi016 ceph-mon[22010]: pgmap v680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:49 smithi079 ceph-mon[23158]: pgmap v680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:49 smithi016 conmon[32727]: 2022-03-21T17:34:49.668+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:50 smithi016 conmon[32727]: 2022-03-21T17:34:50.713+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:51 smithi016 ceph-mon[22010]: pgmap v681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:51 smithi079 ceph-mon[23158]: pgmap v681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:51 smithi016 conmon[32727]: 2022-03-21T17:34:51.726+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:52.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:53.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:52 smithi016 conmon[32727]: 2022-03-21T17:34:52.768+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:53.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:53 smithi016 ceph-mon[22010]: pgmap v682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:53.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:53 smithi079 ceph-mon[23158]: pgmap v682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:53.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:53.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:54.171 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:53 smithi016 conmon[32727]: 2022-03-21T17:34:53.739+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:54 smithi016 conmon[32727]: 2022-03-21T17:34:54.734+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:55 smithi016 ceph-mon[22010]: pgmap v683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:55 smithi079 ceph-mon[23158]: pgmap v683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:34:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:55 smithi016 conmon[32727]: 2022-03-21T17:34:55.717+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:56 smithi016 conmon[32727]: 2022-03-21T17:34:56.685+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:57 smithi016 ceph-mon[22010]: pgmap v684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:57 smithi079 ceph-mon[23158]: pgmap v684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:57 smithi016 conmon[32727]: 2022-03-21T17:34:57.705+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:34:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:34:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:34:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:34:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:34:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:34:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:34:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:34:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:34:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:58 smithi016 conmon[32727]: 2022-03-21T17:34:58.718+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:34:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:59 smithi016 ceph-mon[22010]: pgmap v685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:34:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:34:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:59 smithi079 ceph-mon[23158]: pgmap v685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:34:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:34:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:34:59 smithi016 conmon[32727]: 2022-03-21T17:34:59.754+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:00 smithi016 conmon[32727]: 2022-03-21T17:35:00.801+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:01.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:01 smithi016 ceph-mon[22010]: pgmap v686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:01.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:01.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:01 smithi079 ceph-mon[23158]: pgmap v686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:01.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:01 smithi016 conmon[32727]: 2022-03-21T17:35:01.802+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:02 smithi016 conmon[32727]: 2022-03-21T17:35:02.774+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:03 smithi016 ceph-mon[22010]: pgmap v687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:03 smithi079 ceph-mon[23158]: pgmap v687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:03 smithi016 conmon[32727]: 2022-03-21T17:35:03.791+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:04 smithi016 conmon[32727]: 2022-03-21T17:35:04.753+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:05 smithi016 ceph-mon[22010]: pgmap v688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:05 smithi079 ceph-mon[23158]: pgmap v688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:05 smithi016 conmon[32727]: 2022-03-21T17:35:05.782+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:06 smithi016 conmon[32727]: 2022-03-21T17:35:06.734+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:07 smithi016 ceph-mon[22010]: pgmap v689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:07 smithi079 ceph-mon[23158]: pgmap v689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:07 smithi016 conmon[32727]: 2022-03-21T17:35:07.726+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:08 smithi016 conmon[32727]: 2022-03-21T17:35:08.740+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:09.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:09 smithi016 ceph-mon[22010]: pgmap v690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:09.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:09 smithi079 ceph-mon[23158]: pgmap v690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:09 smithi016 conmon[32727]: 2022-03-21T17:35:09.766+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:10 smithi016 conmon[32727]: 2022-03-21T17:35:10.785+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:11 smithi016 ceph-mon[22010]: pgmap v691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:11 smithi079 ceph-mon[23158]: pgmap v691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:11 smithi016 conmon[32727]: 2022-03-21T17:35:11.736+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:12 smithi016 conmon[32727]: 2022-03-21T17:35:12.722+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:13 smithi016 ceph-mon[22010]: pgmap v692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:13 smithi079 ceph-mon[23158]: pgmap v692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:13 smithi016 conmon[32727]: 2022-03-21T17:35:13.677+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:14.656 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:14.920 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:14 smithi016 conmon[32727]: 2022-03-21T17:35:14.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:15.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:15 smithi016 ceph-mon[22010]: pgmap v693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:15.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:15 smithi079 ceph-mon[23158]: pgmap v693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:15 smithi016 conmon[32727]: 2022-03-21T17:35:15.642+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:16.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:16.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:16 smithi016 conmon[32727]: 2022-03-21T17:35:16.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:17.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:17 smithi016 ceph-mon[22010]: pgmap v694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:17.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:17 smithi079 ceph-mon[23158]: pgmap v694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:17 smithi016 conmon[32727]: 2022-03-21T17:35:17.604+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:18.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:18.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:18.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:18.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:18 smithi016 conmon[32727]: 2022-03-21T17:35:18.650+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:19.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:19 smithi016 ceph-mon[22010]: pgmap v695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:19.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:19 smithi079 ceph-mon[23158]: pgmap v695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:19 smithi016 conmon[32727]: 2022-03-21T17:35:19.651+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:20.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:20 smithi016 conmon[32727]: 2022-03-21T17:35:20.652+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:21.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:21 smithi016 ceph-mon[22010]: pgmap v696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:21.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:21.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:21 smithi079 ceph-mon[23158]: pgmap v696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:21.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:21 smithi016 conmon[32727]: 2022-03-21T17:35:21.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:22 smithi016 conmon[32727]: 2022-03-21T17:35:22.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: pgmap v697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:35:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:35:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:35:23.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: pgmap v697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:23.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:23.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:23.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:35:23.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:35:23.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:35:23.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:23 smithi016 conmon[32727]: 2022-03-21T17:35:23.601+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[32727]: 2022-03-21T17:35:24.556+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: 2022-03-21T17:35:24.556+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:35:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: yield 2022-03-21T17:35:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:35:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:35:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:35:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:35:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:35:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:24.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:24.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:25 smithi016 conmon[32727]: 2022-03-21T17:35:25.591+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:25 smithi016 ceph-mon[22010]: pgmap v698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:25.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:25 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:25.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:35:25.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:25.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:35:25.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:25.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:25.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:25.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:35:25.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:25.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:35:25.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:25.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:35:25.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:25.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:25.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:25.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:35:25.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:25.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:25.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:25.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:25 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:35:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:25 smithi079 ceph-mon[23158]: pgmap v698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:25 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:35:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:35:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:35:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:35:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:35:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:35:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:25.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:25 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:35:26.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:26 smithi016 conmon[32727]: 2022-03-21T17:35:26.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:27.632 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:27 smithi016 ceph-mon[22010]: pgmap v699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:27.633 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:27 smithi079 ceph-mon[23158]: pgmap v699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[32727]: 2022-03-21T17:35:27.659+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:27.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 2022-03-21T17:35:27.628+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:27.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:35:27.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:27.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: yield 2022-03-21T17:35:27.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:27.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:27.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:27.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:35:27.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:27.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:35:27.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:27.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:35:27.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:27.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:35:27.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:27.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:35:27.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:27.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:27.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:27.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 2022-03-21T17:35:27.629+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:35:27.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:35:27.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:27.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: yield 2022-03-21T17:35:27.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:27.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:27.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:27.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:35:27.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:27.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:35:27.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:27.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:35:27.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:27.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:35:27.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:27.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:35:27.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:27.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:27.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:27.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 2022-03-21T17:35:27.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:35:27.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: 2022-03-21T17:35:27.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:35:27.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:35:27.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:35:27.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:35:27.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:35:27.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:35:27.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:35:27.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:35:27.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:35:27.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:35:27.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:35:27.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:35:27.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:35:27.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:35:27.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:35:27.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:35:27.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:35:27.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:35:27.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:35:27.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:35:27.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:35:27.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:35:27.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise 2022-03-21T17:35:27.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:35:27.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:35:27.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:35:27.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:35:27.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:35:27 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:28.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:35:28.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:28.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:28.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:28.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:35:28.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:28.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:35:28.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:28.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:35:28.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:28.635 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:28.635 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:28.635 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:35:28.636 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:28.636 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:28.636 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:28 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:35:28.637 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.637 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:28.638 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:35:28.638 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:28.638 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:28.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:28.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:35:28.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:28.640 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:35:28.640 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:28.640 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:35:28.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:28.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:28.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:28.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:35:28.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:28.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:28.643 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.643 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:35:28.644 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:35:28.644 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:35:28.644 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:35:28.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:35:28.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:35:28.646 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:35:28.646 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:35:28.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:35:28.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:35:28.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:35:28.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:35:28.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:35:28.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:35:28.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:35:28.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:35:28.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:35:28.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:35:28.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:35:28.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:35:28.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:35:28.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:35:28.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:35:28.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:35:28.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:35:28.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:35:28.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:35:28.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:35:28.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:35:28.654 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:28.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:35:28.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:28.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:28.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:28.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:35:28.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:28.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:35:28.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:28.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:35:28.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:28.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:28.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:28.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:35:28.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:28.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:28.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:28 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:35:28.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:35:28.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:35:28.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:35:28.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:35:28.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:35:28.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:35:28.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:35:28.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:35:28.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:35:28.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:35:28.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:35:28.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:35:28.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:35:28.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:35:28.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:35:28.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:35:28.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:35:28.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:35:28.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:35:28.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:35:28.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:35:28.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:35:28.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:35:28.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:35:28.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:35:28.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:35:28.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:35:28.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:35:28.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:35:28.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:35:28.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:35:28.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:35:28.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:35:28.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:35:28.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:35:28.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:35:28.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:35:28.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:35:28.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:35:28.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:35:28.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:35:28.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:35:28.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:35:28.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:35:28.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:35:28.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:35:28.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:35:28.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:28.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:28 smithi016 conmon[32727]: 2022-03-21T17:35:28.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:29.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:29 smithi016 ceph-mon[22010]: pgmap v700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:29.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:29 smithi079 ceph-mon[23158]: pgmap v700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:29 smithi016 conmon[32727]: 2022-03-21T17:35:29.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:30.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:30 smithi016 conmon[32727]: 2022-03-21T17:35:30.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:31 smithi016 ceph-mon[22010]: pgmap v701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:31.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:31 smithi016 conmon[32727]: 2022-03-21T17:35:31.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:31 smithi079 ceph-mon[23158]: pgmap v701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:32.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:32 smithi016 conmon[32727]: 2022-03-21T17:35:32.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:33 smithi016 ceph-mon[22010]: pgmap v702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:33.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:33 smithi016 conmon[32727]: 2022-03-21T17:35:33.564+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:33 smithi079 ceph-mon[23158]: pgmap v702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:34 smithi016 conmon[32727]: 2022-03-21T17:35:34.565+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:35 smithi016 ceph-mon[22010]: pgmap v703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:35 smithi016 conmon[32727]: 2022-03-21T17:35:35.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:35 smithi079 ceph-mon[23158]: pgmap v703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:36 smithi016 conmon[32727]: 2022-03-21T17:35:36.499+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:37 smithi016 ceph-mon[22010]: pgmap v704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:37 smithi016 conmon[32727]: 2022-03-21T17:35:37.472+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:37 smithi079 ceph-mon[23158]: pgmap v704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:38 smithi016 conmon[32727]: 2022-03-21T17:35:38.455+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:39 smithi016 ceph-mon[22010]: pgmap v705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:39 smithi016 conmon[32727]: 2022-03-21T17:35:39.450+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:39 smithi079 ceph-mon[23158]: pgmap v705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:39.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:40 smithi016 conmon[32727]: 2022-03-21T17:35:40.491+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:41 smithi016 ceph-mon[22010]: pgmap v706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:41.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:41 smithi016 conmon[32727]: 2022-03-21T17:35:41.469+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:41 smithi079 ceph-mon[23158]: pgmap v706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:42 smithi016 conmon[32727]: 2022-03-21T17:35:42.440+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:43 smithi016 ceph-mon[22010]: pgmap v707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:43.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:43 smithi016 conmon[32727]: 2022-03-21T17:35:43.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:43 smithi079 ceph-mon[23158]: pgmap v707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:43.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:44 smithi016 conmon[32727]: 2022-03-21T17:35:44.474+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:45 smithi016 ceph-mon[22010]: pgmap v708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:45 smithi016 conmon[32727]: 2022-03-21T17:35:45.427+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:45 smithi079 ceph-mon[23158]: pgmap v708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:46 smithi016 conmon[32727]: 2022-03-21T17:35:46.406+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:47 smithi016 ceph-mon[22010]: pgmap v709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:47.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:47 smithi016 conmon[32727]: 2022-03-21T17:35:47.390+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:47 smithi079 ceph-mon[23158]: pgmap v709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:48.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:48 smithi016 conmon[32727]: 2022-03-21T17:35:48.412+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:49 smithi016 ceph-mon[22010]: pgmap v710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:49.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:49 smithi016 conmon[32727]: 2022-03-21T17:35:49.384+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:49 smithi079 ceph-mon[23158]: pgmap v710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:50.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:50 smithi016 conmon[32727]: 2022-03-21T17:35:50.369+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:51 smithi016 ceph-mon[22010]: pgmap v711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:51 smithi016 conmon[32727]: 2022-03-21T17:35:51.332+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:51 smithi079 ceph-mon[23158]: pgmap v711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:52.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:52 smithi016 conmon[32727]: 2022-03-21T17:35:52.351+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:53 smithi016 ceph-mon[22010]: pgmap v712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:53.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:53 smithi016 conmon[32727]: 2022-03-21T17:35:53.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:53 smithi079 ceph-mon[23158]: pgmap v712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:54 smithi016 conmon[32727]: 2022-03-21T17:35:54.431+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:55 smithi016 ceph-mon[22010]: pgmap v713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:55 smithi016 conmon[32727]: 2022-03-21T17:35:55.387+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:55 smithi079 ceph-mon[23158]: pgmap v713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:35:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:56 smithi016 conmon[32727]: 2022-03-21T17:35:56.391+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:57 smithi016 ceph-mon[22010]: pgmap v714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:57.760 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:57.761 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:57 smithi016 conmon[32727]: 2022-03-21T17:35:57.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:57.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:57 smithi079 ceph-mon[23158]: pgmap v714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:57.762 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:58 smithi016 conmon[32727]: 2022-03-21T17:35:58.317+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:35:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:35:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:35:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:35:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:35:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:35:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:35:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:35:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:35:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:59 smithi016 ceph-mon[22010]: pgmap v715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:35:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:35:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:35:59 smithi016 conmon[32727]: 2022-03-21T17:35:59.305+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:35:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:59 smithi079 ceph-mon[23158]: pgmap v715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:35:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:35:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:00 smithi016 conmon[32727]: 2022-03-21T17:36:00.298+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:01 smithi016 ceph-mon[22010]: pgmap v716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:01 smithi016 conmon[32727]: 2022-03-21T17:36:01.285+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:01 smithi079 ceph-mon[23158]: pgmap v716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:02.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:02 smithi016 conmon[32727]: 2022-03-21T17:36:02.312+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:03 smithi016 ceph-mon[22010]: pgmap v717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:03.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:03 smithi016 conmon[32727]: 2022-03-21T17:36:03.348+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:03 smithi079 ceph-mon[23158]: pgmap v717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:04 smithi016 conmon[32727]: 2022-03-21T17:36:04.302+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:04.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:05 smithi016 conmon[32727]: 2022-03-21T17:36:05.270+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:05 smithi016 ceph-mon[22010]: pgmap v718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:05 smithi079 ceph-mon[23158]: pgmap v718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:06.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:06 smithi016 conmon[32727]: 2022-03-21T17:36:06.301+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:07 smithi016 ceph-mon[22010]: pgmap v719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:07 smithi016 conmon[32727]: 2022-03-21T17:36:07.268+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:07 smithi079 ceph-mon[23158]: pgmap v719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:08 smithi016 conmon[32727]: 2022-03-21T17:36:08.305+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:09 smithi016 ceph-mon[22010]: pgmap v720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:09 smithi016 conmon[32727]: 2022-03-21T17:36:09.330+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:09 smithi079 ceph-mon[23158]: pgmap v720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:10 smithi016 conmon[32727]: 2022-03-21T17:36:10.292+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:11 smithi016 ceph-mon[22010]: pgmap v721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:11.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:11 smithi016 conmon[32727]: 2022-03-21T17:36:11.296+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:11 smithi079 ceph-mon[23158]: pgmap v721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:12 smithi016 conmon[32727]: 2022-03-21T17:36:12.254+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:13 smithi016 ceph-mon[22010]: pgmap v722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:13 smithi016 conmon[32727]: 2022-03-21T17:36:13.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:13 smithi079 ceph-mon[23158]: pgmap v722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:14 smithi016 conmon[32727]: 2022-03-21T17:36:14.248+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:15 smithi016 ceph-mon[22010]: pgmap v723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:15 smithi016 conmon[32727]: 2022-03-21T17:36:15.273+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:15 smithi079 ceph-mon[23158]: pgmap v723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:16.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:16 smithi016 conmon[32727]: 2022-03-21T17:36:16.294+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:17 smithi016 ceph-mon[22010]: pgmap v724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:17 smithi016 conmon[32727]: 2022-03-21T17:36:17.274+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:17 smithi079 ceph-mon[23158]: pgmap v724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:18 smithi016 conmon[32727]: 2022-03-21T17:36:18.297+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:19 smithi016 conmon[32727]: 2022-03-21T17:36:19.306+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:19 smithi016 ceph-mon[22010]: pgmap v725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:19 smithi079 ceph-mon[23158]: pgmap v725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:20 smithi016 conmon[32727]: 2022-03-21T17:36:20.264+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:21 smithi016 ceph-mon[22010]: pgmap v726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:21 smithi016 conmon[32727]: 2022-03-21T17:36:21.293+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:21 smithi079 ceph-mon[23158]: pgmap v726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:22.519 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:22 smithi016 conmon[32727]: 2022-03-21T17:36:22.249+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:23 smithi016 conmon[32727]: 2022-03-21T17:36:23.271+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:23 smithi016 ceph-mon[22010]: pgmap v727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:23 smithi079 ceph-mon[23158]: pgmap v727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:24.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:24 smithi016 conmon[32727]: 2022-03-21T17:36:24.286+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:25.526 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:25 smithi016 conmon[32727]: 2022-03-21T17:36:25.242+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:25 smithi016 ceph-mon[22010]: pgmap v728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:25 smithi079 ceph-mon[23158]: pgmap v728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:26.531 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:26 smithi016 conmon[32727]: 2022-03-21T17:36:26.233+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:27.532 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:27 smithi016 conmon[32727]: 2022-03-21T17:36:27.269+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:27 smithi016 ceph-mon[22010]: pgmap v729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:27 smithi079 ceph-mon[23158]: pgmap v729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:28 smithi016 conmon[32727]: 2022-03-21T17:36:28.252+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:36:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:36:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:36:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:36:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:36:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:36:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[32727]: 2022-03-21T17:36:29.212+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:29.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: 2022-03-21T17:36:29.068+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:36:29.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: yield 2022-03-21T17:36:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:29.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:29.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:36:29.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:29.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:36:29.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:36:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:36:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:36:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:29.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:29.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:29 smithi016 ceph-mon[22010]: pgmap v730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:36:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:36:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:36:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:36:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:36:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:36:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:29 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:36:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:29 smithi079 ceph-mon[23158]: pgmap v730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:36:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:36:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:36:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:36:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:29.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:36:29.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:29.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:36:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:29 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:36:30.543 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:30 smithi016 conmon[32727]: 2022-03-21T17:36:30.255+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:31.544 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:31 smithi016 conmon[32727]: 2022-03-21T17:36:31.253+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:31 smithi016 ceph-mon[22010]: pgmap v731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:31 smithi079 ceph-mon[23158]: pgmap v731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[32727]: 2022-03-21T17:36:32.288+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:32.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 2022-03-21T17:36:32.140+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:36:32.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: yield 2022-03-21T17:36:32.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:36:32.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:36:32.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:36:32.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:36:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 2022-03-21T17:36:32.141+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:36:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:36:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: yield 2022-03-21T17:36:32.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:36:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:36:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:36:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:36:32.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 2022-03-21T17:36:32.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:36:32.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: 2022-03-21T17:36:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:36:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:36:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:36:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:36:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:36:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:36:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:36:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:36:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:36:32.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:36:32.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:36:32.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:36:32.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:36:32.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:36:32.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:36:32.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:36:32.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:36:32.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:36:32.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:36:32.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:36:32.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:36:32.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:36:32.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise 2022-03-21T17:36:32.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:36:32.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:36:32.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:36:32.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:36:32.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:36:32 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:32.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:32 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:36:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:36:32.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:36:32.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:36:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:36:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.927 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:32 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:36:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:36:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:36:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:36:32.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:36:32.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:36:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:36:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:36:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:36:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:36:32.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:36:32.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:36:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:36:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:36:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:36:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:36:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:36:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:36:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:36:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:36:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:36:32.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:36:32.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:36:32.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:36:32.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:36:32.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:36:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:36:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:36:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:36:32.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:36:32.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:36:32.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:36:32.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:36:32.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:36:32.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:36:32.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:32 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:36:32.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:36:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:36:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:36:32.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:36:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:32 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:36:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:36:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:36:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:36:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:36:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:36:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:36:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:36:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:36:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:36:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:36:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:36:32.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:36:32.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:36:32.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:36:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:36:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:36:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:36:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:36:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:36:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:36:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:36:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:36:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:36:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:36:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:36:32.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:36:32.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:36:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:36:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:36:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:36:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:36:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:36:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:36:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:36:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:36:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:36:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:36:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:36:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:36:32.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:36:32.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:36:32.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:36:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:36:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:36:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:36:32.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:36:32.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:36:33.552 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:33 smithi016 conmon[32727]: 2022-03-21T17:36:33.275+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:33 smithi016 ceph-mon[22010]: pgmap v732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:33 smithi079 ceph-mon[23158]: pgmap v732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:34.556 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:34 smithi016 conmon[32727]: 2022-03-21T17:36:34.254+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:35.559 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:35 smithi016 conmon[32727]: 2022-03-21T17:36:35.297+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:35 smithi016 ceph-mon[22010]: pgmap v733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:35 smithi079 ceph-mon[23158]: pgmap v733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:36.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:36 smithi016 conmon[32727]: 2022-03-21T17:36:36.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:37 smithi016 conmon[32727]: 2022-03-21T17:36:37.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:37 smithi016 ceph-mon[22010]: pgmap v734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:37 smithi079 ceph-mon[23158]: pgmap v734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:38 smithi016 conmon[32727]: 2022-03-21T17:36:38.375+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:39 smithi016 ceph-mon[22010]: pgmap v735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:39 smithi016 conmon[32727]: 2022-03-21T17:36:39.385+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:39 smithi079 ceph-mon[23158]: pgmap v735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:40 smithi016 conmon[32727]: 2022-03-21T17:36:40.424+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:41 smithi016 ceph-mon[22010]: pgmap v736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:41.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:41 smithi016 conmon[32727]: 2022-03-21T17:36:41.466+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:41 smithi079 ceph-mon[23158]: pgmap v736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:42 smithi016 conmon[32727]: 2022-03-21T17:36:42.477+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:43 smithi016 ceph-mon[22010]: pgmap v737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:43 smithi016 conmon[32727]: 2022-03-21T17:36:43.484+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:43 smithi079 ceph-mon[23158]: pgmap v737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:44 smithi016 conmon[32727]: 2022-03-21T17:36:44.485+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:45 smithi016 conmon[32727]: 2022-03-21T17:36:45.502+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:45 smithi016 ceph-mon[22010]: pgmap v738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:45 smithi079 ceph-mon[23158]: pgmap v738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:46 smithi016 conmon[32727]: 2022-03-21T17:36:46.552+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:47 smithi016 ceph-mon[22010]: pgmap v739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:47 smithi016 conmon[32727]: 2022-03-21T17:36:47.560+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:47 smithi079 ceph-mon[23158]: pgmap v739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:48 smithi016 conmon[32727]: 2022-03-21T17:36:48.552+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:49 smithi016 ceph-mon[22010]: pgmap v740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:49 smithi016 conmon[32727]: 2022-03-21T17:36:49.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:49 smithi079 ceph-mon[23158]: pgmap v740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:50 smithi016 conmon[32727]: 2022-03-21T17:36:50.549+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:51 smithi016 ceph-mon[22010]: pgmap v741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:51.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:51 smithi016 conmon[32727]: 2022-03-21T17:36:51.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:51 smithi079 ceph-mon[23158]: pgmap v741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:52.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:52 smithi016 conmon[32727]: 2022-03-21T17:36:52.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:53 smithi016 conmon[32727]: 2022-03-21T17:36:53.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:53 smithi016 ceph-mon[22010]: pgmap v742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:53 smithi079 ceph-mon[23158]: pgmap v742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:54 smithi016 conmon[32727]: 2022-03-21T17:36:54.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:55 smithi016 conmon[32727]: 2022-03-21T17:36:55.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:55 smithi016 ceph-mon[22010]: pgmap v743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:55 smithi079 ceph-mon[23158]: pgmap v743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:36:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:56 smithi016 conmon[32727]: 2022-03-21T17:36:56.687+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:57 smithi016 conmon[32727]: 2022-03-21T17:36:57.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:57 smithi016 ceph-mon[22010]: pgmap v744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:57 smithi079 ceph-mon[23158]: pgmap v744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:58 smithi016 conmon[32727]: 2022-03-21T17:36:58.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:36:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:36:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:36:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:36:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:36:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:36:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:36:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:36:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:36:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:36:59 smithi016 conmon[32727]: 2022-03-21T17:36:59.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:36:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:59 smithi016 ceph-mon[22010]: pgmap v745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:36:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:36:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:59 smithi079 ceph-mon[23158]: pgmap v745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:36:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:36:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:00 smithi016 conmon[32727]: 2022-03-21T17:37:00.680+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:01 smithi016 conmon[32727]: 2022-03-21T17:37:01.685+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:01 smithi016 ceph-mon[22010]: pgmap v746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:01 smithi079 ceph-mon[23158]: pgmap v746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:02 smithi016 conmon[32727]: 2022-03-21T17:37:02.636+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:03 smithi016 conmon[32727]: 2022-03-21T17:37:03.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:03 smithi016 ceph-mon[22010]: pgmap v747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:03 smithi079 ceph-mon[23158]: pgmap v747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:04 smithi016 conmon[32727]: 2022-03-21T17:37:04.704+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:05 smithi016 ceph-mon[22010]: pgmap v748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:05 smithi016 conmon[32727]: 2022-03-21T17:37:05.720+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:05 smithi079 ceph-mon[23158]: pgmap v748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:06.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:06 smithi016 conmon[32727]: 2022-03-21T17:37:06.715+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:07 smithi016 ceph-mon[22010]: pgmap v749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:07 smithi016 conmon[32727]: 2022-03-21T17:37:07.686+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:07 smithi079 ceph-mon[23158]: pgmap v749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:08 smithi016 conmon[32727]: 2022-03-21T17:37:08.668+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:09 smithi016 ceph-mon[22010]: pgmap v750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:09.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:09 smithi016 conmon[32727]: 2022-03-21T17:37:09.634+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:09 smithi079 ceph-mon[23158]: pgmap v750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:10 smithi016 conmon[32727]: 2022-03-21T17:37:10.665+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:11 smithi016 ceph-mon[22010]: pgmap v751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:11 smithi016 conmon[32727]: 2022-03-21T17:37:11.715+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:11 smithi079 ceph-mon[23158]: pgmap v751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:12 smithi016 conmon[32727]: 2022-03-21T17:37:12.709+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:13 smithi079 ceph-mon[23158]: pgmap v752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:13.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:14.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:13 smithi016 conmon[32727]: 2022-03-21T17:37:13.714+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:13 smithi016 ceph-mon[22010]: pgmap v752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:14 smithi016 conmon[32727]: 2022-03-21T17:37:14.754+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:15 smithi079 ceph-mon[23158]: pgmap v753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:15 smithi016 ceph-mon[22010]: pgmap v753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:15 smithi016 conmon[32727]: 2022-03-21T17:37:15.755+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:16 smithi016 conmon[32727]: 2022-03-21T17:37:16.792+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:17 smithi079 ceph-mon[23158]: pgmap v754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:17 smithi016 ceph-mon[22010]: pgmap v754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:17 smithi016 conmon[32727]: 2022-03-21T17:37:17.774+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:19.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:18 smithi016 conmon[32727]: 2022-03-21T17:37:18.806+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:19 smithi079 ceph-mon[23158]: pgmap v755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:19 smithi016 ceph-mon[22010]: pgmap v755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:19 smithi016 conmon[32727]: 2022-03-21T17:37:19.835+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:20 smithi016 conmon[32727]: 2022-03-21T17:37:20.882+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:21 smithi079 ceph-mon[23158]: pgmap v756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:21 smithi016 ceph-mon[22010]: pgmap v756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:21 smithi016 conmon[32727]: 2022-03-21T17:37:21.933+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:22 smithi016 conmon[32727]: 2022-03-21T17:37:22.926+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:23 smithi079 ceph-mon[23158]: pgmap v757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:23 smithi016 ceph-mon[22010]: pgmap v757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:23 smithi016 conmon[32727]: 2022-03-21T17:37:23.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:24 smithi016 conmon[32727]: 2022-03-21T17:37:24.939+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:25 smithi079 ceph-mon[23158]: pgmap v758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:25 smithi016 ceph-mon[22010]: pgmap v758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:25 smithi016 conmon[32727]: 2022-03-21T17:37:25.933+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:26.960 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:26 smithi016 conmon[32727]: 2022-03-21T17:37:26.957+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:27 smithi079 ceph-mon[23158]: pgmap v759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:27.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:27.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:27 smithi016 ceph-mon[22010]: pgmap v759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:27 smithi016 conmon[32727]: 2022-03-21T17:37:27.987+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 891 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:28.988 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:28.989 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 891 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:28 smithi016 conmon[32727]: 2022-03-21T17:37:28.986+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:29 smithi079 ceph-mon[23158]: pgmap v760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:29.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:29.991 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:29 smithi016 ceph-mon[22010]: pgmap v760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:29 smithi016 conmon[32727]: 2022-03-21T17:37:29.987+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:30 smithi016 conmon[32727]: 2022-03-21T17:37:30.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:31 smithi079 ceph-mon[23158]: pgmap v761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:31 smithi016 ceph-mon[22010]: pgmap v761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:31 smithi016 conmon[32727]: 2022-03-21T17:37:31.947+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:37:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:37:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:37:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:37:32.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:32 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:37:32.965 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:32.965 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:37:32.965 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:37:32.966 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:37:32.966 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:37:32.966 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:32 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:37:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:32 smithi016 conmon[32727]: 2022-03-21T17:37:32.919+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:33 smithi079 ceph-mon[23158]: pgmap v762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:33 smithi016 ceph-mon[22010]: pgmap v762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:34.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:33 smithi016 conmon[32727]: 2022-03-21T17:37:33.935+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:34 smithi016 conmon[32727]: 2022-03-21T17:37:34.894+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:35.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: 2022-03-21T17:37:35.244+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:35.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:37:35.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: yield 2022-03-21T17:37:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:35.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:35.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:37:35.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:35.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:37:35.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:37:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:37:35.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:35.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:37:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:35 smithi016 ceph-mon[22010]: pgmap v763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:36.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:35 smithi016 conmon[32727]: 2022-03-21T17:37:35.942+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:35 smithi079 ceph-mon[23158]: pgmap v763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:36 smithi016 conmon[32727]: 2022-03-21T17:37:36.933+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:37:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:37:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:37:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:37:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:37:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:37:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:37:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:37:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:37:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:37.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:37.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:37:37.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:37.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:37:37.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:37.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:37:37.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:37.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:37.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:37.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:37:37.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:37.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:37.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:37.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:37:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:37 smithi016 ceph-mon[22010]: pgmap v764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:37 smithi016 conmon[32727]: 2022-03-21T17:37:37.970+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:37 smithi079 ceph-mon[23158]: pgmap v764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:38.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 2022-03-21T17:37:38.317+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:38.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:37:38.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:38.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: yield 2022-03-21T17:37:38.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:38.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:38.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:38.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:37:38.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:38.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:37:38.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:38.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:37:38.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:38.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:37:38.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:38.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:37:38.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:38.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:38.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:38.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 2022-03-21T17:37:38.318+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:37:38.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:37:38.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:38.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: yield 2022-03-21T17:37:38.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:38.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:38.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:38.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:37:38.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:38.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:37:38.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:38.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:37:38.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:38.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:37:38.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:38.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:37:38.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:38.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:38.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:38.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 2022-03-21T17:37:38.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:37:38.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: 2022-03-21T17:37:38.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:37:38.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:37:38.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:37:38.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:37:38.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:37:38.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:37:38.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:37:38.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:37:38.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:37:38.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:37:38.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:37:38.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:37:38.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:37:38.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:37:38.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:37:38.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:37:38.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:37:38.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:37:38.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:37:38.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:37:38.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:37:38.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:37:38.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise 2022-03-21T17:37:38.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:37:38.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:37:38.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:37:38.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:37:38.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:37:38 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:39.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:39.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:39 smithi016 conmon[32727]: 2022-03-21T17:37:39.019+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:40.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:40.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:39 smithi016 ceph-mon[22010]: pgmap v765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:40.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:40.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:37:40.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:40.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:40.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:40.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:37:40.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:40.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:37:40.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:40.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:37:40.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:40.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:40.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:40.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:37:40.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:40.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:40.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.053 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:39 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:37:40.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:40.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:37:40.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:40.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:40.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:40.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:37:40.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:40.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:37:40.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:40.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:37:40.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:40.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:40.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:40.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:37:40.059 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:40.060 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:40.060 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.061 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:37:40.063 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:37:40.063 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:37:40.063 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.064 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:37:40.064 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:37:40.064 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:37:40.065 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:37:40.065 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:37:40.066 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:37:40.066 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:37:40.066 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:37:40.067 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:37:40.067 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:37:40.067 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:37:40.068 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:37:40.068 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:37:40.068 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:37:40.069 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:37:40.069 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:37:40.070 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:37:40.070 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:37:40.071 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:37:40.071 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:37:40.071 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:37:40.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:37:40.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:37:40.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:37:40.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:37:40.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:37:40.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:39 smithi079 ceph-mon[23158]: pgmap v765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:40.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:37:40.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:40.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:37:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:37:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:37:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:40.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:37:40.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:40.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:40.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:39 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:37:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:37:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:37:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:37:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:37:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:37:40.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:37:40.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:37:40.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:37:40.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:37:40.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:37:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:37:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:37:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:37:40.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:37:40.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:37:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:37:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:37:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:37:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:37:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:37:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:37:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:37:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:37:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:37:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:37:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:37:40.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:37:40.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:37:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:37:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:37:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:37:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:37:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:37:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:37:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:37:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:37:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:37:40.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:37:40.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:37:40.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:37:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:37:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:37:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:37:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:37:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:37:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:37:40.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:37:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:40 smithi016 conmon[32727]: 2022-03-21T17:37:40.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:41.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:41 smithi016 conmon[32727]: 2022-03-21T17:37:41.068+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:42.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:42.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:41 smithi016 ceph-mon[22010]: pgmap v766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:41 smithi079 ceph-mon[23158]: pgmap v766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:42 smithi016 conmon[32727]: 2022-03-21T17:37:42.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:43.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:43 smithi016 conmon[32727]: 2022-03-21T17:37:43.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:44.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:44.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:43 smithi016 ceph-mon[22010]: pgmap v767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:44.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:43 smithi079 ceph-mon[23158]: pgmap v767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:44 smithi016 conmon[32727]: 2022-03-21T17:37:44.058+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:45.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:45 smithi016 conmon[32727]: 2022-03-21T17:37:45.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:45 smithi016 ceph-mon[22010]: pgmap v768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:45 smithi016 conmon[32727]: 2022-03-21T17:37:45.982+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:45 smithi079 ceph-mon[23158]: pgmap v768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:46 smithi016 conmon[32727]: 2022-03-21T17:37:46.972+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:47 smithi016 ceph-mon[22010]: pgmap v769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:47 smithi016 conmon[32727]: 2022-03-21T17:37:47.972+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:47 smithi079 ceph-mon[23158]: pgmap v769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:49.021 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:49.021 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:49 smithi016 conmon[32727]: 2022-03-21T17:37:49.018+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:50.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:50.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:49 smithi016 ceph-mon[22010]: pgmap v770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:49 smithi079 ceph-mon[23158]: pgmap v770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:50 smithi016 conmon[32727]: 2022-03-21T17:37:50.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:51.073 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:51 smithi016 conmon[32727]: 2022-03-21T17:37:51.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:52.047 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:52.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:51 smithi016 ceph-mon[22010]: pgmap v771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:51 smithi079 ceph-mon[23158]: pgmap v771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:52 smithi016 conmon[32727]: 2022-03-21T17:37:52.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:53.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:53 smithi016 conmon[32727]: 2022-03-21T17:37:53.058+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:54.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:54.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:53 smithi016 ceph-mon[22010]: pgmap v772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:54.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:53 smithi079 ceph-mon[23158]: pgmap v772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:54.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:54 smithi016 conmon[32727]: 2022-03-21T17:37:54.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:55.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:55 smithi016 conmon[32727]: 2022-03-21T17:37:55.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:56.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:56.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:55 smithi016 ceph-mon[22010]: pgmap v773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:55 smithi079 ceph-mon[23158]: pgmap v773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:37:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:56 smithi016 conmon[32727]: 2022-03-21T17:37:56.106+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:57.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:57 smithi016 conmon[32727]: 2022-03-21T17:37:57.122+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:58.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:58.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:57 smithi016 ceph-mon[22010]: pgmap v774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:57 smithi079 ceph-mon[23158]: pgmap v774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:37:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:58 smithi016 conmon[32727]: 2022-03-21T17:37:58.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:37:59.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:59.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:37:59.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:37:59.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:37:59.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:37:59.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:37:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:37:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:37:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:37:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:37:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:37:59.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:37:59 smithi016 conmon[32727]: 2022-03-21T17:37:59.131+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:00.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:00.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:37:59 smithi016 ceph-mon[22010]: pgmap v775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:37:59 smithi079 ceph-mon[23158]: pgmap v775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:00 smithi016 conmon[32727]: 2022-03-21T17:38:00.134+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:01.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:01 smithi016 conmon[32727]: 2022-03-21T17:38:01.151+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:02.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:02.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:01 smithi016 ceph-mon[22010]: pgmap v776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:01 smithi079 ceph-mon[23158]: pgmap v776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:02 smithi016 conmon[32727]: 2022-03-21T17:38:02.147+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:03 smithi016 conmon[32727]: 2022-03-21T17:38:03.177+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:04.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:04.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:03 smithi016 ceph-mon[22010]: pgmap v777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:04.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:03 smithi079 ceph-mon[23158]: pgmap v777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:04 smithi016 conmon[32727]: 2022-03-21T17:38:04.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:05.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:05 smithi016 conmon[32727]: 2022-03-21T17:38:05.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:06.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:06.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:05 smithi016 ceph-mon[22010]: pgmap v778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:05 smithi079 ceph-mon[23158]: pgmap v778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:06 smithi016 conmon[32727]: 2022-03-21T17:38:06.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:07.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:07 smithi016 conmon[32727]: 2022-03-21T17:38:07.128+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:07 smithi016 ceph-mon[22010]: pgmap v779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:07 smithi079 ceph-mon[23158]: pgmap v779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:08 smithi016 conmon[32727]: 2022-03-21T17:38:08.166+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:09.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:09.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:09 smithi016 conmon[32727]: 2022-03-21T17:38:09.137+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:10.106 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:10.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:09 smithi016 ceph-mon[22010]: pgmap v780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:09 smithi079 ceph-mon[23158]: pgmap v780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:10 smithi016 conmon[32727]: 2022-03-21T17:38:10.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:11.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:11 smithi016 conmon[32727]: 2022-03-21T17:38:11.122+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:12.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:12.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:11 smithi016 ceph-mon[22010]: pgmap v781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:11 smithi079 ceph-mon[23158]: pgmap v781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:12 smithi016 conmon[32727]: 2022-03-21T17:38:12.130+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:13.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:13 smithi016 conmon[32727]: 2022-03-21T17:38:13.146+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:13 smithi016 ceph-mon[22010]: pgmap v782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:13 smithi079 ceph-mon[23158]: pgmap v782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:14 smithi016 conmon[32727]: 2022-03-21T17:38:14.171+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:15.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:15 smithi016 conmon[32727]: 2022-03-21T17:38:15.140+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:16.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:16.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:15 smithi016 ceph-mon[22010]: pgmap v783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:15 smithi079 ceph-mon[23158]: pgmap v783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:16 smithi016 conmon[32727]: 2022-03-21T17:38:16.138+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:17.111 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:17 smithi016 conmon[32727]: 2022-03-21T17:38:17.108+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:18.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:18.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:17 smithi016 ceph-mon[22010]: pgmap v784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:17 smithi079 ceph-mon[23158]: pgmap v784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:18 smithi016 conmon[32727]: 2022-03-21T17:38:18.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:19 smithi016 conmon[32727]: 2022-03-21T17:38:19.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:19 smithi016 ceph-mon[22010]: pgmap v785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:20 smithi016 conmon[32727]: 2022-03-21T17:38:20.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:19 smithi079 ceph-mon[23158]: pgmap v785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:21.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:21 smithi016 conmon[32727]: 2022-03-21T17:38:21.112+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:22 smithi016 conmon[32727]: 2022-03-21T17:38:22.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:21 smithi016 ceph-mon[22010]: pgmap v786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:21 smithi079 ceph-mon[23158]: pgmap v786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:23 smithi016 conmon[32727]: 2022-03-21T17:38:23.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:23 smithi016 ceph-mon[22010]: pgmap v787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:24 smithi016 conmon[32727]: 2022-03-21T17:38:24.022+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:23 smithi079 ceph-mon[23158]: pgmap v787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:24 smithi016 conmon[32727]: 2022-03-21T17:38:24.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:25 smithi016 ceph-mon[22010]: pgmap v788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:25 smithi016 conmon[32727]: 2022-03-21T17:38:25.960+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:25 smithi079 ceph-mon[23158]: pgmap v788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:26 smithi016 conmon[32727]: 2022-03-21T17:38:26.978+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:27 smithi016 ceph-mon[22010]: pgmap v789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:27 smithi016 conmon[32727]: 2022-03-21T17:38:27.957+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:27 smithi079 ceph-mon[23158]: pgmap v789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:28 smithi016 conmon[32727]: 2022-03-21T17:38:28.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:29 smithi016 ceph-mon[22010]: pgmap v790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:29 smithi016 conmon[32727]: 2022-03-21T17:38:29.962+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:29 smithi079 ceph-mon[23158]: pgmap v790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:31 smithi016 conmon[32727]: 2022-03-21T17:38:31.009+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:31 smithi016 ceph-mon[22010]: pgmap v791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:32 smithi016 conmon[32727]: 2022-03-21T17:38:32.011+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:31 smithi079 ceph-mon[23158]: pgmap v791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:33 smithi016 conmon[32727]: 2022-03-21T17:38:33.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:33 smithi016 ceph-mon[22010]: pgmap v792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:34 smithi016 conmon[32727]: 2022-03-21T17:38:34.069+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:33 smithi079 ceph-mon[23158]: pgmap v792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:35 smithi016 conmon[32727]: 2022-03-21T17:38:35.072+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:35 smithi016 ceph-mon[22010]: pgmap v793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:36.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:36 smithi016 conmon[32727]: 2022-03-21T17:38:36.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:35 smithi079 ceph-mon[23158]: pgmap v793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:37 smithi016 conmon[32727]: 2022-03-21T17:38:37.025+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:37 smithi016 ceph-mon[22010]: pgmap v794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:37 smithi016 conmon[32727]: 2022-03-21T17:38:37.998+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:37 smithi079 ceph-mon[23158]: pgmap v794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:38 smithi016 conmon[32727]: 2022-03-21T17:38:38.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:38:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:38:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:38:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:38:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:38:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:38:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:39 smithi016 ceph-mon[22010]: pgmap v795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:39 smithi016 conmon[32727]: 2022-03-21T17:38:39.979+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:39 smithi079 ceph-mon[23158]: pgmap v795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[32727]: 2022-03-21T17:38:41.004+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:41.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: 2022-03-21T17:38:41.421+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:41.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:38:41.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:41.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: yield 2022-03-21T17:38:41.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:41.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:41.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:41.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:38:41.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:41.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:38:41.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:41.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:38:41.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:41.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:38:41.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:41.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:38:41.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:41.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:41.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:41 smithi016 ceph-mon[22010]: pgmap v796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:42 smithi016 conmon[32727]: 2022-03-21T17:38:42.035+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:41 smithi079 ceph-mon[23158]: pgmap v796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:42 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:38:43.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:43.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:38:43.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:43.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:43.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:43.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:38:43.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:43.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:38:43.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:43.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:38:43.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:43.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:43.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:43.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:38:43.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:43.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:43.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:43.237 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:42 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:38:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:42 smithi016 conmon[32727]: 2022-03-21T17:38:42.993+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:42 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:38:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:38:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:38:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:38:43.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:43.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:38:43.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:43.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:43.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:43.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:38:43.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:43.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:43.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:43.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:42 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:38:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:43 smithi079 ceph-mon[23158]: pgmap v797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:43 smithi016 ceph-mon[22010]: pgmap v797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:43 smithi016 conmon[32727]: 2022-03-21T17:38:43.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:44.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 2022-03-21T17:38:44.493+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:44.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:38:44.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:44.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: yield 2022-03-21T17:38:44.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:44.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:44.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:44.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:38:44.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:44.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:38:44.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:44.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:38:44.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:44.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:38:44.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:44.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:38:44.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:44.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:44.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:44.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 2022-03-21T17:38:44.494+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:38:44.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:38:44.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:44.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: yield 2022-03-21T17:38:44.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:44.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:44.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:44.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:38:44.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:44.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:38:44.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:44.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:38:44.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:44.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:38:44.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:44.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:38:44.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:44.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:44.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:44.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 2022-03-21T17:38:44.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:38:44.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: 2022-03-21T17:38:44.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:38:44.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:38:44.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:38:44.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:38:44.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:38:44.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:38:44.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:38:44.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:38:44.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:38:44.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:38:44.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:38:44.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:38:44.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:38:44.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:38:44.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:38:44.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:38:44.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:38:44.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:38:44.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:38:44.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:38:44.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:38:44.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:38:44.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise 2022-03-21T17:38:44.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:38:44.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:38:44.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:38:44.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:38:44.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:44 smithi016 conmon[32727]: 2022-03-21T17:38:44.916+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:45 smithi079 ceph-mon[23158]: pgmap v798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:45 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:38:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:38:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:38:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:38:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:46.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:38:46.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:46.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:46.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:45 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:38:46.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:46.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:38:46.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:46.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:46.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:46.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:38:46.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:46.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:38:46.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:46.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:38:46.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:46.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:46.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:46.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:38:46.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:46.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:46.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:38:46.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:38:46.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:38:46.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:38:46.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:38:46.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:38:46.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:38:46.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:38:46.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:38:46.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:38:46.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:38:46.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:38:46.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:38:46.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:38:46.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:38:46.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:38:46.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:38:46.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:38:46.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:38:46.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:38:46.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:38:46.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:38:46.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:38:46.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:38:46.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:38:46.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:38:46.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:38:46.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:38:46.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:38:46.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:45 smithi016 conmon[32727]: 2022-03-21T17:38:45.930+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:45 smithi016 ceph-mon[22010]: pgmap v798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:45 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:38:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:38:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:38:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:38:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:46.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:46.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:38:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:45 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:38:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:38:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:38:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:38:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:38:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:38:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:38:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:38:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:38:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:38:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:38:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:38:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:38:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:38:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:38:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:38:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:38:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:38:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:38:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:38:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:38:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:38:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:38:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:38:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:38:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:38:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:38:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:38:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:38:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:38:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:38:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:38:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:38:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:38:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:38:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:38:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:38:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:38:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:38:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:38:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:38:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:38:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:38:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:38:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:38:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:38:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:38:46.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:38:46.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:46 smithi016 conmon[32727]: 2022-03-21T17:38:46.922+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:47 smithi079 ceph-mon[23158]: pgmap v799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:47 smithi016 conmon[32727]: 2022-03-21T17:38:47.938+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:47 smithi016 ceph-mon[22010]: pgmap v799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:48 smithi016 conmon[32727]: 2022-03-21T17:38:48.890+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:50 smithi016 ceph-mon[22010]: pgmap v800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:49 smithi016 conmon[32727]: 2022-03-21T17:38:49.899+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:50 smithi079 ceph-mon[23158]: pgmap v800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:50 smithi016 conmon[32727]: 2022-03-21T17:38:50.914+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:52 smithi016 ceph-mon[22010]: pgmap v801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:51 smithi016 conmon[32727]: 2022-03-21T17:38:51.941+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:52 smithi079 ceph-mon[23158]: pgmap v801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 976 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:52 smithi016 conmon[32727]: 2022-03-21T17:38:52.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 976 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:54 smithi016 ceph-mon[22010]: pgmap v802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:53 smithi016 conmon[32727]: 2022-03-21T17:38:53.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:54 smithi079 ceph-mon[23158]: pgmap v802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:54 smithi016 conmon[32727]: 2022-03-21T17:38:54.989+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:56 smithi016 ceph-mon[22010]: pgmap v803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:55 smithi016 conmon[32727]: 2022-03-21T17:38:55.965+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:56 smithi079 ceph-mon[23158]: pgmap v803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:38:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:56 smithi016 conmon[32727]: 2022-03-21T17:38:56.962+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:57 smithi016 conmon[32727]: 2022-03-21T17:38:57.981+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: pgmap v804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:38:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:38:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:38:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:38:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: pgmap v804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:38:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:38:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:38:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:38:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:38:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:38:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:38:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:38:59.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:38:59 smithi016 conmon[32727]: 2022-03-21T17:38:59.020+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:38:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:38:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:00 smithi016 ceph-mon[22010]: pgmap v805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:00 smithi016 conmon[32727]: 2022-03-21T17:39:00.025+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:00 smithi079 ceph-mon[23158]: pgmap v805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:01 smithi016 conmon[32727]: 2022-03-21T17:39:01.055+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:02 smithi016 ceph-mon[22010]: pgmap v806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:02 smithi016 conmon[32727]: 2022-03-21T17:39:02.078+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:02 smithi079 ceph-mon[23158]: pgmap v806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:03 smithi016 conmon[32727]: 2022-03-21T17:39:03.125+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:04 smithi016 ceph-mon[22010]: pgmap v807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:04 smithi016 conmon[32727]: 2022-03-21T17:39:04.147+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:04 smithi079 ceph-mon[23158]: pgmap v807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:05 smithi016 conmon[32727]: 2022-03-21T17:39:05.111+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:06 smithi016 ceph-mon[22010]: pgmap v808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:06 smithi016 conmon[32727]: 2022-03-21T17:39:06.142+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:06 smithi079 ceph-mon[23158]: pgmap v808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:07 smithi016 conmon[32727]: 2022-03-21T17:39:07.110+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:08 smithi016 conmon[32727]: 2022-03-21T17:39:08.129+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:08 smithi016 ceph-mon[22010]: pgmap v809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:08 smithi079 ceph-mon[23158]: pgmap v809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:09 smithi016 conmon[32727]: 2022-03-21T17:39:09.124+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:10 smithi016 conmon[32727]: 2022-03-21T17:39:10.146+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:10 smithi016 ceph-mon[22010]: pgmap v810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:10 smithi079 ceph-mon[23158]: pgmap v810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:11 smithi016 conmon[32727]: 2022-03-21T17:39:11.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:12 smithi016 conmon[32727]: 2022-03-21T17:39:12.077+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:12 smithi016 ceph-mon[22010]: pgmap v811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:12 smithi079 ceph-mon[23158]: pgmap v811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:13 smithi016 conmon[32727]: 2022-03-21T17:39:13.070+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:14 smithi016 conmon[32727]: 2022-03-21T17:39:14.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:14 smithi016 ceph-mon[22010]: pgmap v812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:14 smithi079 ceph-mon[23158]: pgmap v812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:15 smithi016 conmon[32727]: 2022-03-21T17:39:15.033+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:16 smithi016 conmon[32727]: 2022-03-21T17:39:16.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:16 smithi016 ceph-mon[22010]: pgmap v813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:16 smithi079 ceph-mon[23158]: pgmap v813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:17 smithi016 conmon[32727]: 2022-03-21T17:39:17.049+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:18 smithi016 ceph-mon[22010]: pgmap v814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:18.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:18.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:18 smithi016 conmon[32727]: 2022-03-21T17:39:18.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:18 smithi079 ceph-mon[23158]: pgmap v814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:19 smithi016 conmon[32727]: 2022-03-21T17:39:19.065+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:20 smithi016 ceph-mon[22010]: pgmap v815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:20 smithi016 conmon[32727]: 2022-03-21T17:39:20.027+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:20 smithi079 ceph-mon[23158]: pgmap v815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:20 smithi016 conmon[32727]: 2022-03-21T17:39:20.992+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:22 smithi016 ceph-mon[22010]: pgmap v816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:21 smithi016 conmon[32727]: 2022-03-21T17:39:21.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:22 smithi079 ceph-mon[23158]: pgmap v816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:22 smithi016 conmon[32727]: 2022-03-21T17:39:22.966+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:24 smithi016 ceph-mon[22010]: pgmap v817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:23 smithi016 conmon[32727]: 2022-03-21T17:39:23.969+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:24 smithi079 ceph-mon[23158]: pgmap v817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:24 smithi016 conmon[32727]: 2022-03-21T17:39:24.993+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:26 smithi016 conmon[32727]: 2022-03-21T17:39:26.038+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:26 smithi016 ceph-mon[22010]: pgmap v818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:26 smithi079 ceph-mon[23158]: pgmap v818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:27 smithi016 conmon[32727]: 2022-03-21T17:39:27.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:28 smithi016 conmon[32727]: 2022-03-21T17:39:28.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:28 smithi016 ceph-mon[22010]: pgmap v819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:28 smithi079 ceph-mon[23158]: pgmap v819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:29 smithi016 conmon[32727]: 2022-03-21T17:39:29.111+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:30 smithi016 ceph-mon[22010]: pgmap v820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:30 smithi016 conmon[32727]: 2022-03-21T17:39:30.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:30 smithi079 ceph-mon[23158]: pgmap v820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:31 smithi016 conmon[32727]: 2022-03-21T17:39:31.108+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:32 smithi016 ceph-mon[22010]: pgmap v821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:32 smithi016 conmon[32727]: 2022-03-21T17:39:32.141+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:32 smithi079 ceph-mon[23158]: pgmap v821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:33 smithi016 conmon[32727]: 2022-03-21T17:39:33.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:34 smithi016 ceph-mon[22010]: pgmap v822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:34 smithi016 conmon[32727]: 2022-03-21T17:39:34.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:34 smithi079 ceph-mon[23158]: pgmap v822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:35 smithi016 conmon[32727]: 2022-03-21T17:39:35.151+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:36 smithi079 ceph-mon[23158]: pgmap v823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:36 smithi016 conmon[32727]: 2022-03-21T17:39:36.187+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:36 smithi016 ceph-mon[22010]: pgmap v823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:36.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:37 smithi016 conmon[32727]: 2022-03-21T17:39:37.176+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:38 smithi016 conmon[32727]: 2022-03-21T17:39:38.139+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:38 smithi016 ceph-mon[22010]: pgmap v824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:38 smithi079 ceph-mon[23158]: pgmap v824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:39 smithi016 conmon[32727]: 2022-03-21T17:39:39.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:40 smithi016 ceph-mon[22010]: pgmap v825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:40 smithi016 conmon[32727]: 2022-03-21T17:39:40.106+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:40 smithi079 ceph-mon[23158]: pgmap v825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:41 smithi016 conmon[32727]: 2022-03-21T17:39:41.128+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:42 smithi016 ceph-mon[22010]: pgmap v826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:42 smithi016 conmon[32727]: 2022-03-21T17:39:42.153+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:42 smithi079 ceph-mon[23158]: pgmap v826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:43 smithi016 conmon[32727]: 2022-03-21T17:39:43.164+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:44 smithi016 ceph-mon[22010]: pgmap v827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:44 smithi016 conmon[32727]: 2022-03-21T17:39:44.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:44.541 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:44 smithi079 ceph-mon[23158]: pgmap v827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:44.542 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:45.250 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[32727]: 2022-03-21T17:39:45.155+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:45.251 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: 2022-03-21T17:39:45.101+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:45.252 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:39:45.252 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:45.253 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: yield 2022-03-21T17:39:45.253 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:45.254 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:45.254 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:45.255 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:39:45.255 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:45.256 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:39:45.256 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:45.257 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:39:45.257 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:45.258 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:39:45.258 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:45.259 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:39:45.259 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:45.259 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:45.260 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:45.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:45.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:39:45.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:39:45.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:39:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:39:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:39:45.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:39:46.252 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:46 smithi016 conmon[32727]: 2022-03-21T17:39:46.192+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:46 smithi016 ceph-mon[22010]: pgmap v828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:39:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:39:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:39:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:39:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:39:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:39:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:39:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:46 smithi079 ceph-mon[23158]: pgmap v828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:39:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:39:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:39:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:39:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:39:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:39:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:39:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:47.514 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:47.515 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:47 smithi016 conmon[32727]: 2022-03-21T17:39:47.185+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[32727]: 2022-03-21T17:39:48.221+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: pgmap v829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T17:39:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:39:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:48.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:39:48.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:39:48.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:39:48.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:39:48.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:39:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:48 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:39:48.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:39:48.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:39:48.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:39:48.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:39:48.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:39:48.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:39:48.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:39:48.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:39:48.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:39:48.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:39:48.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:39:48.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:39:48.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:39:48.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:39:48.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:39:48.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:39:48.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:39:48.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:39:48.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:39:48.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:39:48.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:39:48.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:39:48.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:39:48.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:39:48.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:39:48.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:39:48.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:39:48.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:39:48.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:39:48.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:39:48.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:39:48.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:39:48.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:39:48.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:39:48.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 2022-03-21T17:39:48.173+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:39:48.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: yield 2022-03-21T17:39:48.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:39:48.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.697 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:39:48.697 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.697 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:39:48.698 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.698 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.698 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.699 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:39:48.699 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.700 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.700 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.700 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 2022-03-21T17:39:48.174+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:39:48.701 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:39:48.701 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.701 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: yield 2022-03-21T17:39:48.702 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.702 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.702 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:39:48.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:39:48.704 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.704 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:39:48.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.706 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:39:48.706 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 2022-03-21T17:39:48.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:39:48.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: 2022-03-21T17:39:48.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:39:48.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:39:48.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:39:48.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:39:48.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:39:48.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:39:48.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:39:48.711 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:39:48.711 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:39:48.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:39:48.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:39:48.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:39:48.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:39:48.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:39:48.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:39:48.714 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:39:48.714 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:39:48.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:39:48.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:39:48.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:39:48.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:39:48.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:39:48.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise 2022-03-21T17:39:48.717 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:39:48.717 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:39:48.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:39:48.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:39:48.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:39:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: pgmap v829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T17:39:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:39:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:39:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:39:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:39:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:39:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:39:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:48 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:39:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:39:48.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:39:48.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:39:48.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:39:48.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:39:48.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:39:48.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:39:48.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:39:48.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:39:48.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:39:48.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:39:48.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:39:48.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:39:48.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:39:48.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:39:48.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:39:48.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:48.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:39:48.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:39:48.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:39:48.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:39:48.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:39:48.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:39:48.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:39:48.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:39:48.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:39:48.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:39:48.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:39:48.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:39:48.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:39:48.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:39:48.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:39:48.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:39:48.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:39:48.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:39:48.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:39:48.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:39:48.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:39:48.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:39:48.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:39:48.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:39:48.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:39:48.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:39:48.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:39:48.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:39:48.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:39:48.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:39:48.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:39:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:49 smithi016 ceph-mon[22010]: pgmap v830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:49.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:49 smithi016 conmon[32727]: 2022-03-21T17:39:49.244+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:49 smithi079 ceph-mon[23158]: pgmap v830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:50.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:50 smithi016 conmon[32727]: 2022-03-21T17:39:50.225+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:51 smithi016 ceph-mon[22010]: pgmap v831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:51 smithi016 conmon[32727]: 2022-03-21T17:39:51.269+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:51 smithi079 ceph-mon[23158]: pgmap v831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:52.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:52 smithi016 conmon[32727]: 2022-03-21T17:39:52.252+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:53 smithi016 ceph-mon[22010]: pgmap v832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:53 smithi016 conmon[32727]: 2022-03-21T17:39:53.296+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:53 smithi079 ceph-mon[23158]: pgmap v832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:54 smithi016 conmon[32727]: 2022-03-21T17:39:54.315+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:55 smithi016 ceph-mon[22010]: pgmap v833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:55 smithi016 conmon[32727]: 2022-03-21T17:39:55.351+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:55 smithi079 ceph-mon[23158]: pgmap v833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:39:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:56 smithi016 conmon[32727]: 2022-03-21T17:39:56.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:57 smithi016 conmon[32727]: 2022-03-21T17:39:57.410+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:57 smithi016 ceph-mon[22010]: pgmap v834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:57 smithi079 ceph-mon[23158]: pgmap v834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:58 smithi016 conmon[32727]: 2022-03-21T17:39:58.440+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:39:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:39:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:39:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:39:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:39:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:39:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:39:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:39:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:39:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:59 smithi016 ceph-mon[22010]: pgmap v835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:39:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:39:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:39:59 smithi016 conmon[32727]: 2022-03-21T17:39:59.473+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:39:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:59 smithi079 ceph-mon[23158]: pgmap v835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:39:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:39:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:00 smithi016 conmon[32727]: 2022-03-21T17:40:00.515+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops 2022-03-21T17:40:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:40:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:40:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:40:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:40:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:40:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops 2022-03-21T17:40:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops 2022-03-21T17:40:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:40:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:40:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:40:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:40:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:40:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1040 sec, osd.4 has slow ops 2022-03-21T17:40:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:01 smithi016 conmon[32727]: 2022-03-21T17:40:01.521+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:01 smithi016 ceph-mon[22010]: pgmap v836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:01.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:01 smithi079 ceph-mon[23158]: pgmap v836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:01.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:02.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:02 smithi016 conmon[32727]: 2022-03-21T17:40:02.554+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:03 smithi016 conmon[32727]: 2022-03-21T17:40:03.517+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:03 smithi016 ceph-mon[22010]: pgmap v837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:03 smithi079 ceph-mon[23158]: pgmap v837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:04 smithi016 conmon[32727]: 2022-03-21T17:40:04.482+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:05 smithi016 ceph-mon[22010]: pgmap v838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:05 smithi016 conmon[32727]: 2022-03-21T17:40:05.500+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:05 smithi079 ceph-mon[23158]: pgmap v838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:06 smithi016 conmon[32727]: 2022-03-21T17:40:06.527+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:07 smithi016 ceph-mon[22010]: pgmap v839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:07 smithi016 conmon[32727]: 2022-03-21T17:40:07.506+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:07 smithi079 ceph-mon[23158]: pgmap v839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:08 smithi016 conmon[32727]: 2022-03-21T17:40:08.543+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:09 smithi016 ceph-mon[22010]: pgmap v840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:09 smithi016 conmon[32727]: 2022-03-21T17:40:09.500+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:09 smithi079 ceph-mon[23158]: pgmap v840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:10 smithi016 conmon[32727]: 2022-03-21T17:40:10.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:11 smithi016 ceph-mon[22010]: pgmap v841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:11.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:11 smithi016 conmon[32727]: 2022-03-21T17:40:11.547+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:11 smithi079 ceph-mon[23158]: pgmap v841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:12 smithi016 conmon[32727]: 2022-03-21T17:40:12.499+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:13 smithi016 ceph-mon[22010]: pgmap v842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:13 smithi016 conmon[32727]: 2022-03-21T17:40:13.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:13 smithi079 ceph-mon[23158]: pgmap v842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:14.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:14 smithi016 conmon[32727]: 2022-03-21T17:40:14.536+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:15 smithi016 ceph-mon[22010]: pgmap v843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:15 smithi016 conmon[32727]: 2022-03-21T17:40:15.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:15 smithi079 ceph-mon[23158]: pgmap v843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:16 smithi016 conmon[32727]: 2022-03-21T17:40:16.546+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:17 smithi016 ceph-mon[22010]: pgmap v844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:17 smithi016 conmon[32727]: 2022-03-21T17:40:17.562+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:17 smithi079 ceph-mon[23158]: pgmap v844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:18 smithi016 conmon[32727]: 2022-03-21T17:40:18.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:19 smithi016 ceph-mon[22010]: pgmap v845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:19 smithi016 conmon[32727]: 2022-03-21T17:40:19.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:19 smithi079 ceph-mon[23158]: pgmap v845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:20 smithi016 conmon[32727]: 2022-03-21T17:40:20.546+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:21 smithi016 ceph-mon[22010]: pgmap v846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:21.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:21.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:21 smithi016 conmon[32727]: 2022-03-21T17:40:21.538+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:21 smithi079 ceph-mon[23158]: pgmap v846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:22 smithi016 conmon[32727]: 2022-03-21T17:40:22.541+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:23 smithi016 conmon[32727]: 2022-03-21T17:40:23.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:23 smithi016 ceph-mon[22010]: pgmap v847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:23.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:23 smithi079 ceph-mon[23158]: pgmap v847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:24 smithi016 conmon[32727]: 2022-03-21T17:40:24.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:25 smithi016 ceph-mon[22010]: pgmap v848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:25.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:25 smithi016 conmon[32727]: 2022-03-21T17:40:25.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:25 smithi079 ceph-mon[23158]: pgmap v848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:26.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:26 smithi016 conmon[32727]: 2022-03-21T17:40:26.650+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:27 smithi016 ceph-mon[22010]: pgmap v849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:27 smithi079 ceph-mon[23158]: pgmap v849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:27 smithi016 conmon[32727]: 2022-03-21T17:40:27.691+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:28 smithi016 conmon[32727]: 2022-03-21T17:40:28.683+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:29 smithi016 ceph-mon[22010]: pgmap v850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:29 smithi079 ceph-mon[23158]: pgmap v850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:29 smithi016 conmon[32727]: 2022-03-21T17:40:29.687+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:30 smithi016 conmon[32727]: 2022-03-21T17:40:30.711+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:31 smithi016 ceph-mon[22010]: pgmap v851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:31 smithi079 ceph-mon[23158]: pgmap v851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:31 smithi016 conmon[32727]: 2022-03-21T17:40:31.754+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:32 smithi016 conmon[32727]: 2022-03-21T17:40:32.791+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:33 smithi016 ceph-mon[22010]: pgmap v852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:33 smithi079 ceph-mon[23158]: pgmap v852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:33 smithi016 conmon[32727]: 2022-03-21T17:40:33.823+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:34 smithi016 conmon[32727]: 2022-03-21T17:40:34.818+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:35 smithi016 ceph-mon[22010]: pgmap v853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:35 smithi079 ceph-mon[23158]: pgmap v853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:35.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:35 smithi016 conmon[32727]: 2022-03-21T17:40:35.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:36 smithi016 conmon[32727]: 2022-03-21T17:40:36.789+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:37 smithi016 ceph-mon[22010]: pgmap v854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:37 smithi079 ceph-mon[23158]: pgmap v854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:37 smithi016 conmon[32727]: 2022-03-21T17:40:37.830+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:38 smithi016 conmon[32727]: 2022-03-21T17:40:38.877+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:39 smithi016 ceph-mon[22010]: pgmap v855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:39 smithi079 ceph-mon[23158]: pgmap v855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:39 smithi016 conmon[32727]: 2022-03-21T17:40:39.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:40.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:40 smithi016 conmon[32727]: 2022-03-21T17:40:40.866+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:41 smithi079 ceph-mon[23158]: pgmap v856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:41.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:41 smithi016 ceph-mon[22010]: pgmap v856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:41.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:41 smithi016 conmon[32727]: 2022-03-21T17:40:41.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:42.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:42 smithi016 conmon[32727]: 2022-03-21T17:40:42.822+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:43 smithi079 ceph-mon[23158]: pgmap v857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1086 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:43.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:43 smithi016 ceph-mon[22010]: pgmap v857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:43.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:43.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1086 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:43 smithi016 conmon[32727]: 2022-03-21T17:40:43.851+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:44.704 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:44 smithi016 conmon[32727]: 2022-03-21T17:40:44.847+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:45 smithi079 ceph-mon[23158]: pgmap v858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:45.819 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:45 smithi016 ceph-mon[22010]: pgmap v858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:45.819 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:45 smithi016 conmon[32727]: 2022-03-21T17:40:45.817+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:46.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:46 smithi016 conmon[32727]: 2022-03-21T17:40:46.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:47 smithi079 ceph-mon[23158]: pgmap v859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:47.857 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:47 smithi016 ceph-mon[22010]: pgmap v859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:47.858 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:47 smithi016 conmon[32727]: 2022-03-21T17:40:47.855+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:48.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:48.622 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1091 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:48.623 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:40:48.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:40:48.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:40:48.650 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:48.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1091 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:48.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:40:48.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:40:48.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:40:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:48 smithi016 conmon[32727]: 2022-03-21T17:40:48.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:49.625 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:49 smithi079 ceph-mon[23158]: pgmap v860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:49.626 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:49.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:49 smithi016 ceph-mon[22010]: pgmap v860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:49.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:49 smithi016 conmon[32727]: 2022-03-21T17:40:49.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:51.279 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:50 smithi016 conmon[32727]: 2022-03-21T17:40:50.924+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:51.495 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:51 smithi079 ceph-mon[23158]: pgmap v861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:51.495 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:51 smithi016 ceph-mon[22010]: pgmap v861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:51.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: 2022-03-21T17:40:51.277+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:40:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: yield 2022-03-21T17:40:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:51.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:40:51.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:51.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:40:51.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:51.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:40:51.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:40:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:40:51.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:51.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:51.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:51 smithi016 conmon[32727]: 2022-03-21T17:40:51.925+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:52 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:40:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:40:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:40:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:40:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:40:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:40:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:52 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:40:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:52 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T17:40:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:52 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:40:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:52 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:40:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:40:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:40:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:40:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:40:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:40:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:52 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:40:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:52 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T17:40:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:52 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:40:52.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:52 smithi016 conmon[32727]: 2022-03-21T17:40:52.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:53 smithi016 ceph-mon[22010]: pgmap v862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1096 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:53 smithi079 ceph-mon[23158]: pgmap v862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:40:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1096 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:54.351 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:53 smithi016 conmon[32727]: 2022-03-21T17:40:53.943+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:54.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 2022-03-21T17:40:54.349+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:54.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:40:54.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:54.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: yield 2022-03-21T17:40:54.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:54.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:54.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:54.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:40:54.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:54.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:40:54.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:54.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:40:54.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:54.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:40:54.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:54.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:40:54.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:54.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:54.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:54.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 2022-03-21T17:40:54.350+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:40:54.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:40:54.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:54.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: yield 2022-03-21T17:40:54.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:54.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:54.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:54.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:40:54.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:54.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:40:54.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:54.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:40:54.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:54.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:40:54.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:54.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:40:54.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:54.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:54.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:54.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 2022-03-21T17:40:54.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:40:54.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: 2022-03-21T17:40:54.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:40:54.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:40:54.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:40:54.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:40:54.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:40:54.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:40:54.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:40:54.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:40:54.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:40:54.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:40:54.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:40:54.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:40:54.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:40:54.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:40:54.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:40:54.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:40:54.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:40:54.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:40:54.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:40:54.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:40:54.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:40:54.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:40:54.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise 2022-03-21T17:40:54.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:40:54.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:40:54.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:40:54.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:40:54.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:54 smithi016 conmon[32727]: 2022-03-21T17:40:54.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:55.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:55 smithi016 ceph-mon[22010]: pgmap v863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:55.905 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:55.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:40:55.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:55.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:55.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:55.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:40:55.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:55.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:40:55.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:55.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:40:55.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:55.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:55.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:55.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:40:55.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:55.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:55.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.911 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:55 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:40:55.912 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.912 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:55.912 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:40:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:55.913 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:40:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:40:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:40:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:40:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:40:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:40:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:40:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:40:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:40:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:40:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:40:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:40:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:40:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:40:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:40:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:40:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:40:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:40:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:40:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:40:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:40:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:40:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:40:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:40:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:40:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:40:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:40:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:40:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:40:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:40:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:40:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:40:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:40:55.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:55 smithi079 ceph-mon[23158]: pgmap v863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:40:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:40:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:40:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:40:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:40:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:40:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:55 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:40:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:40:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:40:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:40:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:40:55.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:40:55.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:40:55.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:40:55.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:40:55.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:40:55.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:40:55.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:40:55.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:40:55.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:40:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:40:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:40:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:40:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:40:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:40:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:40:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:40:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:40:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:40:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:40:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:40:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:40:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:40:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:40:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:40:55.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:40:55.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:40:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:40:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:40:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:40:55.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:40:55.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:40:55.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:40:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:40:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:40:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:40:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:40:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:40:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:40:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:40:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:40:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:40:55.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:40:55.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:40:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:55 smithi016 conmon[32727]: 2022-03-21T17:40:55.902+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:56 smithi016 conmon[32727]: 2022-03-21T17:40:56.913+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:57 smithi016 ceph-mon[22010]: pgmap v864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:40:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:57 smithi079 ceph-mon[23158]: pgmap v864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:40:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:57 smithi016 conmon[32727]: 2022-03-21T17:40:57.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:40:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:40:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:40:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:40:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1101 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:40:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:40:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:40:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:40:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1101 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:40:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:40:58 smithi016 conmon[32727]: 2022-03-21T17:40:58.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:40:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:40:59 smithi016 ceph-mon[22010]: pgmap v865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:40:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:40:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:40:59 smithi079 ceph-mon[23158]: pgmap v865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:41:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:00 smithi016 conmon[32727]: 2022-03-21T17:41:00.009+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:01 smithi016 conmon[32727]: 2022-03-21T17:41:01.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:01 smithi016 ceph-mon[22010]: pgmap v866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:01 smithi079 ceph-mon[23158]: pgmap v866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:02 smithi016 conmon[32727]: 2022-03-21T17:41:02.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:03 smithi016 conmon[32727]: 2022-03-21T17:41:03.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:03 smithi016 ceph-mon[22010]: pgmap v867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:41:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:03 smithi079 ceph-mon[23158]: pgmap v867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:41:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:03 smithi016 conmon[32727]: 2022-03-21T17:41:03.979+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:05.029 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:04 smithi016 conmon[32727]: 2022-03-21T17:41:04.962+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:05 smithi016 ceph-mon[22010]: pgmap v868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:05 smithi079 ceph-mon[23158]: pgmap v868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:05 smithi016 conmon[32727]: 2022-03-21T17:41:05.931+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:06 smithi016 conmon[32727]: 2022-03-21T17:41:06.972+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:07 smithi016 ceph-mon[22010]: pgmap v869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:07 smithi079 ceph-mon[23158]: pgmap v869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:08 smithi016 conmon[32727]: 2022-03-21T17:41:08.009+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:09 smithi016 conmon[32727]: 2022-03-21T17:41:09.031+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:09 smithi016 ceph-mon[22010]: pgmap v870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:09 smithi079 ceph-mon[23158]: pgmap v870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:10 smithi016 conmon[32727]: 2022-03-21T17:41:10.072+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:11 smithi016 conmon[32727]: 2022-03-21T17:41:11.044+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:11 smithi016 ceph-mon[22010]: pgmap v871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:11 smithi079 ceph-mon[23158]: pgmap v871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:12 smithi016 conmon[32727]: 2022-03-21T17:41:12.027+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:13 smithi016 conmon[32727]: 2022-03-21T17:41:13.062+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:13 smithi016 ceph-mon[22010]: pgmap v872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:13 smithi079 ceph-mon[23158]: pgmap v872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:14 smithi016 conmon[32727]: 2022-03-21T17:41:14.107+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:15 smithi016 conmon[32727]: 2022-03-21T17:41:15.068+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:15 smithi016 ceph-mon[22010]: pgmap v873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:15 smithi079 ceph-mon[23158]: pgmap v873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:16 smithi016 conmon[32727]: 2022-03-21T17:41:16.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:17 smithi016 conmon[32727]: 2022-03-21T17:41:17.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:17 smithi016 ceph-mon[22010]: pgmap v874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:17.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:17 smithi079 ceph-mon[23158]: pgmap v874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:18 smithi016 conmon[32727]: 2022-03-21T17:41:18.060+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:19 smithi016 conmon[32727]: 2022-03-21T17:41:19.020+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:19 smithi016 ceph-mon[22010]: pgmap v875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:19 smithi079 ceph-mon[23158]: pgmap v875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:20 smithi016 conmon[32727]: 2022-03-21T17:41:20.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:20 smithi016 conmon[32727]: 2022-03-21T17:41:20.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:21 smithi016 ceph-mon[22010]: pgmap v876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:21 smithi079 ceph-mon[23158]: pgmap v876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:21 smithi016 conmon[32727]: 2022-03-21T17:41:21.932+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:22 smithi016 conmon[32727]: 2022-03-21T17:41:22.968+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:23 smithi016 ceph-mon[22010]: pgmap v877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:23 smithi079 ceph-mon[23158]: pgmap v877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:23 smithi016 conmon[32727]: 2022-03-21T17:41:23.965+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:25 smithi016 conmon[32727]: 2022-03-21T17:41:25.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:25 smithi016 ceph-mon[22010]: pgmap v878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:25 smithi079 ceph-mon[23158]: pgmap v878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:25 smithi016 conmon[32727]: 2022-03-21T17:41:25.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:26 smithi016 conmon[32727]: 2022-03-21T17:41:26.972+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:27 smithi016 ceph-mon[22010]: pgmap v879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:27 smithi079 ceph-mon[23158]: pgmap v879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:28 smithi016 conmon[32727]: 2022-03-21T17:41:28.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:29 smithi016 conmon[32727]: 2022-03-21T17:41:29.021+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:29 smithi016 ceph-mon[22010]: pgmap v880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:29 smithi079 ceph-mon[23158]: pgmap v880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:30 smithi016 conmon[32727]: 2022-03-21T17:41:30.068+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:31 smithi016 conmon[32727]: 2022-03-21T17:41:31.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:31 smithi016 ceph-mon[22010]: pgmap v881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:31 smithi079 ceph-mon[23158]: pgmap v881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:32 smithi016 conmon[32727]: 2022-03-21T17:41:32.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:33 smithi016 conmon[32727]: 2022-03-21T17:41:33.088+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:33 smithi016 ceph-mon[22010]: pgmap v882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:33 smithi079 ceph-mon[23158]: pgmap v882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:34 smithi016 conmon[32727]: 2022-03-21T17:41:34.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:35 smithi016 conmon[32727]: 2022-03-21T17:41:35.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:35 smithi016 ceph-mon[22010]: pgmap v883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:35 smithi079 ceph-mon[23158]: pgmap v883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:36 smithi016 conmon[32727]: 2022-03-21T17:41:36.109+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:37 smithi016 conmon[32727]: 2022-03-21T17:41:37.118+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:37 smithi016 ceph-mon[22010]: pgmap v884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:37 smithi079 ceph-mon[23158]: pgmap v884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:38 smithi016 conmon[32727]: 2022-03-21T17:41:38.135+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:39 smithi016 conmon[32727]: 2022-03-21T17:41:39.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:39 smithi016 ceph-mon[22010]: pgmap v885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:39 smithi079 ceph-mon[23158]: pgmap v885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:40 smithi016 conmon[32727]: 2022-03-21T17:41:40.142+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:41 smithi016 conmon[32727]: 2022-03-21T17:41:41.141+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:41 smithi016 ceph-mon[22010]: pgmap v886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:41 smithi079 ceph-mon[23158]: pgmap v886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:42 smithi016 conmon[32727]: 2022-03-21T17:41:42.159+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:43.655 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:43 smithi016 conmon[32727]: 2022-03-21T17:41:43.177+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:43 smithi016 ceph-mon[22010]: pgmap v887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:43 smithi079 ceph-mon[23158]: pgmap v887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:44 smithi016 conmon[32727]: 2022-03-21T17:41:44.132+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:45 smithi016 conmon[32727]: 2022-03-21T17:41:45.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:45 smithi016 ceph-mon[22010]: pgmap v888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:45 smithi079 ceph-mon[23158]: pgmap v888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:46 smithi016 conmon[32727]: 2022-03-21T17:41:46.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:47.103 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:47 smithi016 conmon[32727]: 2022-03-21T17:41:47.101+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:47 smithi079 ceph-mon[23158]: pgmap v889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:48.125 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:48.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:47 smithi016 ceph-mon[22010]: pgmap v889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:48 smithi016 conmon[32727]: 2022-03-21T17:41:48.123+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:49 smithi016 conmon[32727]: 2022-03-21T17:41:49.171+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:49 smithi079 ceph-mon[23158]: pgmap v890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:49 smithi016 ceph-mon[22010]: pgmap v890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:50 smithi016 conmon[32727]: 2022-03-21T17:41:50.211+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:51 smithi016 conmon[32727]: 2022-03-21T17:41:51.221+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:51 smithi079 ceph-mon[23158]: pgmap v891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:51 smithi016 ceph-mon[22010]: pgmap v891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:52 smithi016 conmon[32727]: 2022-03-21T17:41:52.267+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:53 smithi016 conmon[32727]: 2022-03-21T17:41:53.314+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:53 smithi079 ceph-mon[23158]: pgmap v892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:53 smithi016 ceph-mon[22010]: pgmap v892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[32727]: 2022-03-21T17:41:54.347+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:41:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:41:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:41:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:41:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:41:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:41:55.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: 2022-03-21T17:41:54.830+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:55.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:41:55.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: yield 2022-03-21T17:41:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:55.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:55.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:41:55.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:55.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:41:55.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:55.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:41:55.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:55.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:41:55.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:55.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:41:55.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:55.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:55.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:55 smithi016 conmon[32727]: 2022-03-21T17:41:55.319+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:55 smithi079 ceph-mon[23158]: pgmap v893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:41:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:41:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:41:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:41:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:41:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:41:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:55 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:41:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:55 smithi016 ceph-mon[22010]: pgmap v893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:41:56.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:56.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:41:56.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:56.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:41:56.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:56.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:41:56.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:41:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:56.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:41:56.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:56.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:41:56.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:56.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:55 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:41:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:56 smithi016 conmon[32727]: 2022-03-21T17:41:56.347+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[32727]: 2022-03-21T17:41:57.382+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:57 smithi079 ceph-mon[23158]: pgmap v894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:57 smithi016 ceph-mon[22010]: pgmap v894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:41:58.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 2022-03-21T17:41:57.901+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:41:58.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:58.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: yield 2022-03-21T17:41:58.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:58.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:58.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:58.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:41:58.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:58.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:41:58.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:58.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:41:58.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:58.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:41:58.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:58.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:41:58.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:58.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:58.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 2022-03-21T17:41:57.902+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:41:58.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:41:58.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:58.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: yield 2022-03-21T17:41:58.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:58.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:58.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:58.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:41:58.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:58.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:41:58.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:58.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:41:58.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:58.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:41:58.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:58.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:41:58.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:58.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:58.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 2022-03-21T17:41:58.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:41:58.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: 2022-03-21T17:41:58.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:41:58.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:41:58.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:41:58.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:41:58.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:41:58.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:41:58.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:41:58.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:41:58.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:41:58.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:41:58.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:41:58.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:41:58.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:41:58.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:41:58.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:41:58.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:41:58.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:41:58.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:41:58.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:41:58.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:41:58.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:41:58.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:41:58.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise 2022-03-21T17:41:58.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:41:58.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:41:58.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:41:58.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:41:58.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:41:57 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:58 smithi016 conmon[32727]: 2022-03-21T17:41:58.347+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:41:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:41:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:41:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:41:58.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:58.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:41:58.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:58.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:58.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:41:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:41:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:41:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:41:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:41:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:41:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:41:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:41:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:58.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:58.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:41:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:41:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:41:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:41:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:41:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:41:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:41:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:41:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:41:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:41:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:41:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:41:58.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:41:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:41:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:41:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:41:58.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:41:58.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:41:58.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:41:58.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:41:58.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:41:59.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:41:59.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:41:59.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:41:59.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:41:59.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:41:59.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:41:59.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:41:59.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:41:59.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:41:59.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:59.003 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:41:59.004 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:41:59.004 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:41:59.004 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:41:59.005 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:41:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:41:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:41:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:41:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:41:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:41:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:41:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:41:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:41:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:41:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:41:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:41:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:41:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:41:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:41:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:41:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:41:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:41:59.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:41:59.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:41:59.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:41:59.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:41:59.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:41:59.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:41:59.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:59.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:41:59.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:41:59.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:41:59.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:41:59.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:41:59.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:41:59.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:41:59.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:41:59.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:41:59.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:41:59.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:41:59.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:41:59.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:41:59.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:41:59.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:41:59.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:41:59.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:41:59.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:41:59.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:41:59.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:41:59.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:41:59.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:41:59.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:41:59.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:41:59.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:41:59.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:41:59.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:41:59.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:41:59.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:41:59.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:41:59.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:41:59.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:41:59.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:41:59.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:41:59.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:41:59.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:41:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:41:59 smithi016 conmon[32727]: 2022-03-21T17:41:59.364+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:41:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:41:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:41:59 smithi079 ceph-mon[23158]: pgmap v895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:41:59 smithi016 ceph-mon[22010]: pgmap v895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:00 smithi016 conmon[32727]: 2022-03-21T17:42:00.358+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:01 smithi016 conmon[32727]: 2022-03-21T17:42:01.319+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:01 smithi079 ceph-mon[23158]: pgmap v896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:01 smithi016 ceph-mon[22010]: pgmap v896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:02 smithi016 conmon[32727]: 2022-03-21T17:42:02.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:03 smithi016 conmon[32727]: 2022-03-21T17:42:03.400+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:03 smithi079 ceph-mon[23158]: pgmap v897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:03 smithi016 ceph-mon[22010]: pgmap v897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:04.716 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:04 smithi016 conmon[32727]: 2022-03-21T17:42:04.427+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:05 smithi016 ceph-mon[22010]: pgmap v898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:05 smithi016 conmon[32727]: 2022-03-21T17:42:05.477+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:05 smithi079 ceph-mon[23158]: pgmap v898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:06 smithi016 conmon[32727]: 2022-03-21T17:42:06.470+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:07 smithi016 conmon[32727]: 2022-03-21T17:42:07.499+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:07 smithi016 ceph-mon[22010]: pgmap v899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:07 smithi079 ceph-mon[23158]: pgmap v899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:08 smithi016 conmon[32727]: 2022-03-21T17:42:08.500+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:09 smithi016 ceph-mon[22010]: pgmap v900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:09 smithi016 conmon[32727]: 2022-03-21T17:42:09.542+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:09 smithi079 ceph-mon[23158]: pgmap v900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:10 smithi016 conmon[32727]: 2022-03-21T17:42:10.564+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:11 smithi016 ceph-mon[22010]: pgmap v901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:11 smithi016 conmon[32727]: 2022-03-21T17:42:11.565+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:11 smithi079 ceph-mon[23158]: pgmap v901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:12 smithi016 conmon[32727]: 2022-03-21T17:42:12.520+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:13 smithi016 conmon[32727]: 2022-03-21T17:42:13.566+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:13 smithi016 ceph-mon[22010]: pgmap v902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:13 smithi079 ceph-mon[23158]: pgmap v902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:14 smithi016 conmon[32727]: 2022-03-21T17:42:14.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:15 smithi016 conmon[32727]: 2022-03-21T17:42:15.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:15 smithi016 ceph-mon[22010]: pgmap v903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:15 smithi079 ceph-mon[23158]: pgmap v903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:16.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:16.921 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:16 smithi016 conmon[32727]: 2022-03-21T17:42:16.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:17 smithi016 ceph-mon[22010]: pgmap v904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:17 smithi016 conmon[32727]: 2022-03-21T17:42:17.601+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:17 smithi079 ceph-mon[23158]: pgmap v904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:18 smithi016 conmon[32727]: 2022-03-21T17:42:18.599+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:19 smithi016 ceph-mon[22010]: pgmap v905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:19 smithi016 conmon[32727]: 2022-03-21T17:42:19.605+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:19 smithi079 ceph-mon[23158]: pgmap v905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:20 smithi016 conmon[32727]: 2022-03-21T17:42:20.612+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:21 smithi016 ceph-mon[22010]: pgmap v906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:21.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:21 smithi016 conmon[32727]: 2022-03-21T17:42:21.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:21 smithi079 ceph-mon[23158]: pgmap v906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:22 smithi016 conmon[32727]: 2022-03-21T17:42:22.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:23 smithi016 ceph-mon[22010]: pgmap v907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:23 smithi016 conmon[32727]: 2022-03-21T17:42:23.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:23 smithi079 ceph-mon[23158]: pgmap v907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:24 smithi016 conmon[32727]: 2022-03-21T17:42:24.534+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:25 smithi016 ceph-mon[22010]: pgmap v908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:25.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:25 smithi016 conmon[32727]: 2022-03-21T17:42:25.548+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:25 smithi079 ceph-mon[23158]: pgmap v908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:26 smithi016 conmon[32727]: 2022-03-21T17:42:26.585+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:27 smithi016 ceph-mon[22010]: pgmap v909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:27 smithi016 conmon[32727]: 2022-03-21T17:42:27.543+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:27 smithi079 ceph-mon[23158]: pgmap v909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:28 smithi016 conmon[32727]: 2022-03-21T17:42:28.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:29 smithi016 ceph-mon[22010]: pgmap v910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:29 smithi016 conmon[32727]: 2022-03-21T17:42:29.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:29 smithi079 ceph-mon[23158]: pgmap v910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:30 smithi016 conmon[32727]: 2022-03-21T17:42:30.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:31 smithi016 ceph-mon[22010]: pgmap v911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:31 smithi016 conmon[32727]: 2022-03-21T17:42:31.595+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:31 smithi079 ceph-mon[23158]: pgmap v911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:32 smithi016 conmon[32727]: 2022-03-21T17:42:32.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:33 smithi016 ceph-mon[22010]: pgmap v912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:33 smithi016 conmon[32727]: 2022-03-21T17:42:33.607+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:33 smithi079 ceph-mon[23158]: pgmap v912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:34 smithi016 conmon[32727]: 2022-03-21T17:42:34.595+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:35 smithi016 ceph-mon[22010]: pgmap v913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:35.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:35 smithi016 conmon[32727]: 2022-03-21T17:42:35.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:35 smithi079 ceph-mon[23158]: pgmap v913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:36 smithi016 conmon[32727]: 2022-03-21T17:42:36.669+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:37 smithi016 ceph-mon[22010]: pgmap v914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:37 smithi016 conmon[32727]: 2022-03-21T17:42:37.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:37 smithi079 ceph-mon[23158]: pgmap v914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:38.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:38 smithi016 conmon[32727]: 2022-03-21T17:42:38.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:39 smithi016 ceph-mon[22010]: pgmap v915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:39 smithi016 conmon[32727]: 2022-03-21T17:42:39.616+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:39 smithi079 ceph-mon[23158]: pgmap v915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:40 smithi016 conmon[32727]: 2022-03-21T17:42:40.583+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:41 smithi016 ceph-mon[22010]: pgmap v916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:41 smithi016 conmon[32727]: 2022-03-21T17:42:41.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:41 smithi079 ceph-mon[23158]: pgmap v916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:42.823 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:42 smithi016 conmon[32727]: 2022-03-21T17:42:42.562+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:43 smithi016 conmon[32727]: 2022-03-21T17:42:43.605+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:43 smithi016 ceph-mon[22010]: pgmap v917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:43.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:43.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:43 smithi079 ceph-mon[23158]: pgmap v917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:44 smithi016 conmon[32727]: 2022-03-21T17:42:44.593+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:45 smithi016 ceph-mon[22010]: pgmap v918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:45 smithi016 conmon[32727]: 2022-03-21T17:42:45.591+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:45 smithi079 ceph-mon[23158]: pgmap v918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:46.836 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:46 smithi016 conmon[32727]: 2022-03-21T17:42:46.569+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:47.838 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:47 smithi016 conmon[32727]: 2022-03-21T17:42:47.533+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:47 smithi016 ceph-mon[22010]: pgmap v919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:47 smithi079 ceph-mon[23158]: pgmap v919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:48.839 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:48 smithi016 conmon[32727]: 2022-03-21T17:42:48.502+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:49.840 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:49 smithi016 conmon[32727]: 2022-03-21T17:42:49.500+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:49 smithi016 ceph-mon[22010]: pgmap v920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:49 smithi079 ceph-mon[23158]: pgmap v920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:50.845 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:50 smithi016 conmon[32727]: 2022-03-21T17:42:50.467+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:51.847 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:51 smithi016 conmon[32727]: 2022-03-21T17:42:51.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:51 smithi016 ceph-mon[22010]: pgmap v921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:51 smithi079 ceph-mon[23158]: pgmap v921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:52.852 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:52 smithi016 conmon[32727]: 2022-03-21T17:42:52.490+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:53.854 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:53 smithi016 conmon[32727]: 2022-03-21T17:42:53.491+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:53 smithi016 ceph-mon[22010]: pgmap v922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:53 smithi079 ceph-mon[23158]: pgmap v922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:54.859 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:54 smithi016 conmon[32727]: 2022-03-21T17:42:54.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:55.860 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:55 smithi016 conmon[32727]: 2022-03-21T17:42:55.513+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:55 smithi016 ceph-mon[22010]: pgmap v923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:55 smithi079 ceph-mon[23158]: pgmap v923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:42:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:56.865 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:56 smithi016 conmon[32727]: 2022-03-21T17:42:56.481+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:57.867 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:57 smithi016 conmon[32727]: 2022-03-21T17:42:57.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:57 smithi016 ceph-mon[22010]: pgmap v924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:57 smithi079 ceph-mon[23158]: pgmap v924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:42:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:58 smithi016 conmon[32727]: 2022-03-21T17:42:58.416+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:42:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:42:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:42:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:42:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:42:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:42:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:42:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:42:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:42:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:42:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:42:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:42:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:42:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:42:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:42:59 smithi016 conmon[32727]: 2022-03-21T17:42:59.374+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:59 smithi016 ceph-mon[22010]: pgmap v925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:42:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:59 smithi079 ceph-mon[23158]: pgmap v925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:42:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:00 smithi016 conmon[32727]: 2022-03-21T17:43:00.331+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:01.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: 2022-03-21T17:43:01.005+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:01.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:43:01.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:01.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: yield 2022-03-21T17:43:01.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:01.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:01.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:01.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:43:01.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:01.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:43:01.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:01.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:43:01.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:01.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:43:01.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:01.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:43:01.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:01.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:01.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:01 smithi016 conmon[32727]: 2022-03-21T17:43:01.363+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:01 smithi016 ceph-mon[22010]: pgmap v926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:43:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:43:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:43:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:43:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:43:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:43:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:43:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:01 smithi079 ceph-mon[23158]: pgmap v926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:43:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:43:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:43:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:43:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:43:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:43:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:43:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:02 smithi016 conmon[32727]: 2022-03-21T17:43:02.360+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:03 smithi016 conmon[32727]: 2022-03-21T17:43:03.326+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:03 smithi016 ceph-mon[22010]: pgmap v927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:04.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 2022-03-21T17:43:04.077+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:04.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:43:04.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: yield 2022-03-21T17:43:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:43:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:43:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:43:04.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:04.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:43:04.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:04.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:43:04.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:04.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:04.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:04.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 2022-03-21T17:43:04.078+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:43:04.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:43:04.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:04.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: yield 2022-03-21T17:43:04.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:04.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:04.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:04.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:43:04.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:04.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:43:04.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:04.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:43:04.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:04.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:43:04.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:04.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:43:04.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:04.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:04.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:04.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 2022-03-21T17:43:04.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:43:04.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: 2022-03-21T17:43:04.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:43:04.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:43:04.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:43:04.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:43:04.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:43:04.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:43:04.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:43:04.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:43:04.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:43:04.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:43:04.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:43:04.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:43:04.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:43:04.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:43:04.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:43:04.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:43:04.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:43:04.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:43:04.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:43:04.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:43:04.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:43:04.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:43:04.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise 2022-03-21T17:43:04.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:43:04.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:43:04.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:43:04.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:43:04.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:03 smithi079 ceph-mon[23158]: pgmap v927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:04 smithi016 conmon[32727]: 2022-03-21T17:43:04.280+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:43:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:43:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:43:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:43:05.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:05.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:05.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:05.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:43:05.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:05.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:05.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:04 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:43:05.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:05.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:43:05.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:05.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:05.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:05.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:43:05.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:05.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:43:05.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:05.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:43:05.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:05.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:05.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:05.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:43:05.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:05.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:05.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:43:05.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:43:05.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:43:05.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:43:05.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:43:05.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:43:05.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:43:05.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:43:05.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:43:05.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:43:05.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:43:05.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:43:05.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:43:05.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:43:05.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:43:05.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:43:05.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:43:05.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:43:05.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:43:05.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:43:05.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:43:05.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:43:05.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:43:05.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:43:05.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:43:05.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:43:05.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:43:05.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:43:05.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:43:05.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:43:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:43:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:43:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:43:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:43:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:05.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:05.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:04 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:43:05.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:43:05.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:43:05.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:43:05.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:43:05.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:43:05.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:43:05.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:43:05.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:43:05.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:43:05.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:43:05.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:43:05.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:43:05.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:43:05.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:43:05.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:43:05.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:43:05.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:43:05.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:43:05.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:43:05.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:43:05.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:43:05.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:43:05.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:43:05.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:43:05.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:43:05.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:43:05.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:43:05.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:43:05.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:43:05.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:43:05.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:43:05.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:43:05.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:43:05.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:43:05.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:43:05.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:43:05.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:43:05.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:43:05.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:43:05.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:43:05.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:43:05.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:43:05.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:43:05.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:43:05.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:43:05.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:43:05.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:43:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:05 smithi016 conmon[32727]: 2022-03-21T17:43:05.300+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:05 smithi016 ceph-mon[22010]: pgmap v928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:05 smithi079 ceph-mon[23158]: pgmap v928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:06 smithi016 conmon[32727]: 2022-03-21T17:43:06.278+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:07 smithi016 conmon[32727]: 2022-03-21T17:43:07.296+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:07 smithi016 ceph-mon[22010]: pgmap v929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:07 smithi079 ceph-mon[23158]: pgmap v929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:08 smithi016 conmon[32727]: 2022-03-21T17:43:08.265+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:09 smithi016 conmon[32727]: 2022-03-21T17:43:09.252+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:09 smithi016 ceph-mon[22010]: pgmap v930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:09 smithi079 ceph-mon[23158]: pgmap v930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:10 smithi016 conmon[32727]: 2022-03-21T17:43:10.258+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:11 smithi016 conmon[32727]: 2022-03-21T17:43:11.273+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:11 smithi016 ceph-mon[22010]: pgmap v931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:11 smithi079 ceph-mon[23158]: pgmap v931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:12 smithi016 conmon[32727]: 2022-03-21T17:43:12.314+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:13 smithi016 conmon[32727]: 2022-03-21T17:43:13.278+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:13 smithi016 ceph-mon[22010]: pgmap v932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:13 smithi079 ceph-mon[23158]: pgmap v932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:14 smithi016 conmon[32727]: 2022-03-21T17:43:14.259+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:15.217 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:15 smithi016 conmon[32727]: 2022-03-21T17:43:15.216+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:15 smithi079 ceph-mon[23158]: pgmap v933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:16.252 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:16.253 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:15 smithi016 ceph-mon[22010]: pgmap v933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:16 smithi016 conmon[32727]: 2022-03-21T17:43:16.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:17.287 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:17 smithi016 conmon[32727]: 2022-03-21T17:43:17.285+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:17 smithi079 ceph-mon[23158]: pgmap v934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:18.301 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:18.302 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:17 smithi016 ceph-mon[22010]: pgmap v934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:18 smithi016 conmon[32727]: 2022-03-21T17:43:18.299+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:19 smithi016 conmon[32727]: 2022-03-21T17:43:19.302+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:19 smithi079 ceph-mon[23158]: pgmap v935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:20.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:20.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:19 smithi016 ceph-mon[22010]: pgmap v935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:20.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:20 smithi016 conmon[32727]: 2022-03-21T17:43:20.346+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:21.299 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:21 smithi016 conmon[32727]: 2022-03-21T17:43:21.298+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:21 smithi079 ceph-mon[23158]: pgmap v936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:22.319 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:22.320 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:21 smithi016 ceph-mon[22010]: pgmap v936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:22 smithi016 conmon[32727]: 2022-03-21T17:43:22.317+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:23.334 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:23 smithi016 conmon[32727]: 2022-03-21T17:43:23.331+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:23 smithi079 ceph-mon[23158]: pgmap v937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:24.317 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:24.318 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:23 smithi016 ceph-mon[22010]: pgmap v937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:24.318 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:24 smithi016 conmon[32727]: 2022-03-21T17:43:24.314+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:25.363 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:25 smithi016 conmon[32727]: 2022-03-21T17:43:25.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:25 smithi079 ceph-mon[23158]: pgmap v938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:26.359 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:26.359 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:25 smithi016 ceph-mon[22010]: pgmap v938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:26 smithi016 conmon[32727]: 2022-03-21T17:43:26.356+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:27.310 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:27 smithi016 conmon[32727]: 2022-03-21T17:43:27.307+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:27 smithi079 ceph-mon[23158]: pgmap v939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:28.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:28.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:27 smithi016 ceph-mon[22010]: pgmap v939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:28 smithi016 conmon[32727]: 2022-03-21T17:43:28.305+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:29.356 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:29.356 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:29 smithi016 conmon[32727]: 2022-03-21T17:43:29.353+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:29 smithi079 ceph-mon[23158]: pgmap v940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:30.310 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:30.311 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:29 smithi016 ceph-mon[22010]: pgmap v940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:30 smithi016 conmon[32727]: 2022-03-21T17:43:30.308+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:31.297 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:31 smithi016 conmon[32727]: 2022-03-21T17:43:31.294+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:31 smithi079 ceph-mon[23158]: pgmap v941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:32.266 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:32.266 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:31 smithi016 ceph-mon[22010]: pgmap v941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:32 smithi016 conmon[32727]: 2022-03-21T17:43:32.264+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:33.221 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:33 smithi016 conmon[32727]: 2022-03-21T17:43:33.218+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:33 smithi079 ceph-mon[23158]: pgmap v942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:33 smithi016 ceph-mon[22010]: pgmap v942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:34 smithi016 conmon[32727]: 2022-03-21T17:43:34.187+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:35 smithi016 conmon[32727]: 2022-03-21T17:43:35.172+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:35 smithi016 ceph-mon[22010]: pgmap v943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:36 smithi016 conmon[32727]: 2022-03-21T17:43:36.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:35 smithi079 ceph-mon[23158]: pgmap v943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:37 smithi016 conmon[32727]: 2022-03-21T17:43:37.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:38 smithi016 conmon[32727]: 2022-03-21T17:43:38.128+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:37 smithi016 ceph-mon[22010]: pgmap v944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:37 smithi079 ceph-mon[23158]: pgmap v944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:39 smithi016 conmon[32727]: 2022-03-21T17:43:39.164+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:40 smithi016 ceph-mon[22010]: pgmap v945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:40 smithi016 conmon[32727]: 2022-03-21T17:43:40.148+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:40 smithi079 ceph-mon[23158]: pgmap v945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:41 smithi016 conmon[32727]: 2022-03-21T17:43:41.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:42 smithi016 conmon[32727]: 2022-03-21T17:43:42.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:42 smithi016 ceph-mon[22010]: pgmap v946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:42 smithi079 ceph-mon[23158]: pgmap v946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:43 smithi016 conmon[32727]: 2022-03-21T17:43:43.137+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:44 smithi016 conmon[32727]: 2022-03-21T17:43:44.111+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:44 smithi016 ceph-mon[22010]: pgmap v947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:44 smithi079 ceph-mon[23158]: pgmap v947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:45 smithi016 conmon[32727]: 2022-03-21T17:43:45.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:46 smithi016 ceph-mon[22010]: pgmap v948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:46 smithi016 conmon[32727]: 2022-03-21T17:43:46.077+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:46 smithi079 ceph-mon[23158]: pgmap v948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:47 smithi016 conmon[32727]: 2022-03-21T17:43:47.033+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:47 smithi016 conmon[32727]: 2022-03-21T17:43:47.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:48 smithi016 ceph-mon[22010]: pgmap v949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:48.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:48 smithi079 ceph-mon[23158]: pgmap v949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:49 smithi016 conmon[32727]: 2022-03-21T17:43:49.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:50 smithi016 ceph-mon[22010]: pgmap v950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:49 smithi016 conmon[32727]: 2022-03-21T17:43:49.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:50 smithi079 ceph-mon[23158]: pgmap v950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:51 smithi016 conmon[32727]: 2022-03-21T17:43:51.024+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:52 smithi016 ceph-mon[22010]: pgmap v951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:52.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:52 smithi016 conmon[32727]: 2022-03-21T17:43:52.027+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:52 smithi079 ceph-mon[23158]: pgmap v951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:53 smithi016 conmon[32727]: 2022-03-21T17:43:53.007+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:54 smithi016 ceph-mon[22010]: pgmap v952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:54 smithi016 conmon[32727]: 2022-03-21T17:43:54.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:54 smithi079 ceph-mon[23158]: pgmap v952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:55 smithi016 conmon[32727]: 2022-03-21T17:43:55.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:55.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:56 smithi016 ceph-mon[22010]: pgmap v953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:56 smithi016 conmon[32727]: 2022-03-21T17:43:56.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:56 smithi079 ceph-mon[23158]: pgmap v953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:43:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:57.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:57 smithi016 conmon[32727]: 2022-03-21T17:43:57.053+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:58 smithi016 conmon[32727]: 2022-03-21T17:43:58.062+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: pgmap v954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:43:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:43:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:43:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:43:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: pgmap v954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:43:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:43:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:43:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:43:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:43:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:43:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:43:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:43:59.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:43:59 smithi016 conmon[32727]: 2022-03-21T17:43:59.018+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:43:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:43:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:00 smithi016 ceph-mon[22010]: pgmap v955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:00 smithi016 conmon[32727]: 2022-03-21T17:44:00.031+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:00 smithi079 ceph-mon[23158]: pgmap v955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:01 smithi016 conmon[32727]: 2022-03-21T17:44:01.060+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:02 smithi016 ceph-mon[22010]: pgmap v956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:02 smithi016 conmon[32727]: 2022-03-21T17:44:02.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:02 smithi079 ceph-mon[23158]: pgmap v956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:03 smithi016 conmon[32727]: 2022-03-21T17:44:03.018+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:04 smithi016 conmon[32727]: 2022-03-21T17:44:04.031+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:04 smithi016 ceph-mon[22010]: pgmap v957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:44:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:44:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:44:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:04 smithi079 ceph-mon[23158]: pgmap v957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:44:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:44:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:44:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:05 smithi016 conmon[32727]: 2022-03-21T17:44:05.033+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:06 smithi016 conmon[32727]: 2022-03-21T17:44:06.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:05 smithi016 ceph-mon[22010]: pgmap v958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:44:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:44:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:05 smithi079 ceph-mon[23158]: pgmap v958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:44:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:44:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[32727]: 2022-03-21T17:44:07.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:07.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: 2022-03-21T17:44:07.181+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:07.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:44:07.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:07.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: yield 2022-03-21T17:44:07.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:07.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:07.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:07.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:44:07.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:07.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:44:07.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:07.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:44:07.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:07.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:44:07.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:07.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:44:07.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:07.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:07.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:08 smithi016 conmon[32727]: 2022-03-21T17:44:08.034+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:07 smithi016 ceph-mon[22010]: pgmap v959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:07 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:08.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:44:08.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:08.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:44:08.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:08.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:08.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:08.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:44:08.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:08.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:44:08.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:08.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:44:08.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:08.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:08.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:08.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:44:08.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:08.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:08.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:08.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:07 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:44:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:07 smithi079 ceph-mon[23158]: pgmap v959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:07 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:44:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:44:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:08.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:08.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:44:08.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:08.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:44:08.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:08.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:44:08.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:08.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:08.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:08.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:44:08.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:08.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:08.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:08.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:07 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:44:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:09.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:09 smithi016 conmon[32727]: 2022-03-21T17:44:09.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[32727]: 2022-03-21T17:44:10.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:09 smithi016 ceph-mon[22010]: pgmap v960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:10.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:09 smithi079 ceph-mon[23158]: pgmap v960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:10.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 2022-03-21T17:44:10.253+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:10.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:44:10.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:10.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: yield 2022-03-21T17:44:10.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:10.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:10.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:10.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:44:10.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:10.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:44:10.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:10.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:44:10.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:10.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:44:10.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:10.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:44:10.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:10.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:10.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:10.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 2022-03-21T17:44:10.254+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:44:10.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:44:10.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:10.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: yield 2022-03-21T17:44:10.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:10.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:10.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:10.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:44:10.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:10.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:44:10.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:10.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:44:10.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:10.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:44:10.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:10.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:44:10.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:10.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:10.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:10.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 2022-03-21T17:44:10.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:44:10.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: 2022-03-21T17:44:10.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:44:10.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:44:10.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:44:10.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:44:10.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:44:10.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:44:10.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:44:10.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:44:10.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:44:10.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:44:10.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:44:10.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:44:10.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:44:10.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:44:10.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:44:10.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:44:10.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:44:10.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:44:10.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:44:10.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:44:10.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:44:10.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:44:10.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise 2022-03-21T17:44:10.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:44:10.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:44:10.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:44:10.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:44:10.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:10 smithi016 conmon[32727]: 2022-03-21T17:44:10.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:44:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:44:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:44:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:44:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:44:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:10 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:44:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:44:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:11.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:11.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:11.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:44:11.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:11.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:44:11.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:11.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:44:11.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:11.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:11.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:11.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:44:11.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:11.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:11.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:44:11.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:44:11.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:44:11.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:44:11.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:44:11.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:44:11.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:44:11.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:44:11.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:44:11.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:44:11.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:44:11.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:44:11.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:44:11.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:44:11.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:44:11.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:44:11.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:44:11.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:44:11.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:44:11.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:44:11.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:44:11.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:44:11.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:44:11.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:44:11.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:44:11.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:44:11.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:44:11.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:44:11.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:44:11.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:44:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:44:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:44:11.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:11.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:44:11.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:11.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:11.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:11.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:44:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:10 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:44:11.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:44:11.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:44:11.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:44:11.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:44:11.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:44:11.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:44:11.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:44:11.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:44:11.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:44:11.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:44:11.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:44:11.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:44:11.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:44:11.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:44:11.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:44:11.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:44:11.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:11.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:44:11.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:44:11.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:44:11.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:44:11.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:44:11.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:44:11.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:44:11.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:44:11.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:44:11.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:44:11.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:44:11.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:44:11.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:44:11.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:44:11.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:44:11.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:44:11.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:44:11.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:44:11.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:44:11.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:44:11.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:44:11.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:44:11.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:44:11.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:44:11.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:44:11.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:44:11.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:44:11.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:44:11.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:44:11.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:44:11.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:44:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:11 smithi016 ceph-mon[22010]: pgmap v961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:11 smithi016 conmon[32727]: 2022-03-21T17:44:11.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:11 smithi079 ceph-mon[23158]: pgmap v961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:12 smithi016 conmon[32727]: 2022-03-21T17:44:12.931+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:13 smithi016 ceph-mon[22010]: pgmap v962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:14.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:13 smithi016 conmon[32727]: 2022-03-21T17:44:13.894+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:13 smithi079 ceph-mon[23158]: pgmap v962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:14 smithi016 conmon[32727]: 2022-03-21T17:44:14.861+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:15 smithi016 ceph-mon[22010]: pgmap v963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:15 smithi016 conmon[32727]: 2022-03-21T17:44:15.838+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:15 smithi079 ceph-mon[23158]: pgmap v963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:16 smithi016 conmon[32727]: 2022-03-21T17:44:16.860+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:17 smithi016 ceph-mon[22010]: pgmap v964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:17 smithi016 conmon[32727]: 2022-03-21T17:44:17.899+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:17 smithi079 ceph-mon[23158]: pgmap v964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:18 smithi016 conmon[32727]: 2022-03-21T17:44:18.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:19 smithi079 ceph-mon[23158]: pgmap v965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:19 smithi016 conmon[32727]: 2022-03-21T17:44:19.930+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:19 smithi016 ceph-mon[22010]: pgmap v965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:20 smithi016 conmon[32727]: 2022-03-21T17:44:20.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:21 smithi079 ceph-mon[23158]: pgmap v966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:21 smithi016 conmon[32727]: 2022-03-21T17:44:21.928+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:21 smithi016 ceph-mon[22010]: pgmap v966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:22 smithi016 conmon[32727]: 2022-03-21T17:44:22.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:23 smithi079 ceph-mon[23158]: pgmap v967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:23 smithi016 ceph-mon[22010]: pgmap v967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:23 smithi016 conmon[32727]: 2022-03-21T17:44:23.976+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:24 smithi016 conmon[32727]: 2022-03-21T17:44:24.942+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:25 smithi079 ceph-mon[23158]: pgmap v968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:25 smithi016 ceph-mon[22010]: pgmap v968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:25 smithi016 conmon[32727]: 2022-03-21T17:44:25.914+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:26 smithi016 conmon[32727]: 2022-03-21T17:44:26.959+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:27 smithi079 ceph-mon[23158]: pgmap v969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:27 smithi016 ceph-mon[22010]: pgmap v969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:28 smithi016 conmon[32727]: 2022-03-21T17:44:28.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:29 smithi016 conmon[32727]: 2022-03-21T17:44:29.021+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:29 smithi079 ceph-mon[23158]: pgmap v970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:29 smithi016 ceph-mon[22010]: pgmap v970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:30 smithi016 conmon[32727]: 2022-03-21T17:44:30.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:31 smithi016 conmon[32727]: 2022-03-21T17:44:31.053+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:31 smithi079 ceph-mon[23158]: pgmap v971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:31 smithi016 ceph-mon[22010]: pgmap v971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:32 smithi016 conmon[32727]: 2022-03-21T17:44:32.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:33 smithi016 conmon[32727]: 2022-03-21T17:44:33.055+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:33 smithi079 ceph-mon[23158]: pgmap v972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:34 smithi016 conmon[32727]: 2022-03-21T17:44:34.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:33 smithi016 ceph-mon[22010]: pgmap v972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:35 smithi016 conmon[32727]: 2022-03-21T17:44:35.063+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:35 smithi079 ceph-mon[23158]: pgmap v973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:36 smithi016 conmon[32727]: 2022-03-21T17:44:36.061+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:35 smithi016 ceph-mon[22010]: pgmap v973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:37 smithi016 conmon[32727]: 2022-03-21T17:44:37.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:38 smithi016 conmon[32727]: 2022-03-21T17:44:38.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:37 smithi016 ceph-mon[22010]: pgmap v974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:37 smithi079 ceph-mon[23158]: pgmap v974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:39 smithi016 conmon[32727]: 2022-03-21T17:44:39.139+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:40 smithi016 conmon[32727]: 2022-03-21T17:44:40.160+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:39 smithi016 ceph-mon[22010]: pgmap v975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:39 smithi079 ceph-mon[23158]: pgmap v975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:41 smithi016 conmon[32727]: 2022-03-21T17:44:41.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:42 smithi016 conmon[32727]: 2022-03-21T17:44:42.076+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:41 smithi016 ceph-mon[22010]: pgmap v976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:41 smithi079 ceph-mon[23158]: pgmap v976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:43 smithi016 conmon[32727]: 2022-03-21T17:44:43.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:44 smithi016 conmon[32727]: 2022-03-21T17:44:44.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:43 smithi016 ceph-mon[22010]: pgmap v977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:43 smithi079 ceph-mon[23158]: pgmap v977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:45 smithi016 conmon[32727]: 2022-03-21T17:44:45.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:45 smithi016 ceph-mon[22010]: pgmap v978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:46 smithi016 conmon[32727]: 2022-03-21T17:44:46.143+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:45 smithi079 ceph-mon[23158]: pgmap v978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:47 smithi016 conmon[32727]: 2022-03-21T17:44:47.155+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:48 smithi016 ceph-mon[22010]: pgmap v979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:48 smithi016 conmon[32727]: 2022-03-21T17:44:48.118+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:48 smithi079 ceph-mon[23158]: pgmap v979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:49 smithi016 conmon[32727]: 2022-03-21T17:44:49.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:50 smithi016 ceph-mon[22010]: pgmap v980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:50 smithi016 conmon[32727]: 2022-03-21T17:44:50.124+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:50 smithi079 ceph-mon[23158]: pgmap v980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:51 smithi016 conmon[32727]: 2022-03-21T17:44:51.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:52 smithi016 ceph-mon[22010]: pgmap v981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:52 smithi016 conmon[32727]: 2022-03-21T17:44:52.077+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:52 smithi079 ceph-mon[23158]: pgmap v981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:53 smithi016 conmon[32727]: 2022-03-21T17:44:53.118+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:54 smithi016 conmon[32727]: 2022-03-21T17:44:54.090+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:54 smithi016 ceph-mon[22010]: pgmap v982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:54 smithi079 ceph-mon[23158]: pgmap v982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:55 smithi016 conmon[32727]: 2022-03-21T17:44:55.124+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:56 smithi016 conmon[32727]: 2022-03-21T17:44:56.114+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:56 smithi016 ceph-mon[22010]: pgmap v983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:56 smithi079 ceph-mon[23158]: pgmap v983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:44:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:57 smithi016 conmon[32727]: 2022-03-21T17:44:57.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: pgmap v984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:44:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:44:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:44:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:44:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: pgmap v984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:44:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:44:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:44:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:44:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:44:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:44:58.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:58 smithi016 conmon[32727]: 2022-03-21T17:44:58.168+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:44:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:44:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:44:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:44:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:44:59 smithi016 conmon[32727]: 2022-03-21T17:44:59.172+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:00 smithi079 ceph-mon[23158]: pgmap v985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:00 smithi016 ceph-mon[22010]: pgmap v985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:00 smithi016 conmon[32727]: 2022-03-21T17:45:00.180+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:01 smithi016 conmon[32727]: 2022-03-21T17:45:01.211+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:02 smithi079 ceph-mon[23158]: pgmap v986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:02.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:02 smithi016 ceph-mon[22010]: pgmap v986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:02 smithi016 conmon[32727]: 2022-03-21T17:45:02.215+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:03.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:03 smithi016 conmon[32727]: 2022-03-21T17:45:03.255+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:04 smithi079 ceph-mon[23158]: pgmap v987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:04.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:04 smithi016 ceph-mon[22010]: pgmap v987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:04.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:04.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:04 smithi016 conmon[32727]: 2022-03-21T17:45:04.265+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:05 smithi016 conmon[32727]: 2022-03-21T17:45:05.238+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:06 smithi079 ceph-mon[23158]: pgmap v988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:06 smithi016 ceph-mon[22010]: pgmap v988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:06 smithi016 conmon[32727]: 2022-03-21T17:45:06.240+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:07 smithi016 conmon[32727]: 2022-03-21T17:45:07.222+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:08 smithi079 ceph-mon[23158]: pgmap v989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:08 smithi016 ceph-mon[22010]: pgmap v989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:08.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:08 smithi016 conmon[32727]: 2022-03-21T17:45:08.216+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:09 smithi016 conmon[32727]: 2022-03-21T17:45:09.192+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:10 smithi079 ceph-mon[23158]: pgmap v990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:10 smithi016 ceph-mon[22010]: pgmap v990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[32727]: 2022-03-21T17:45:10.187+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: 2022-03-21T17:45:10.862+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:45:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: yield 2022-03-21T17:45:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:45:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:45:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:45:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:45:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:45:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:11 smithi016 conmon[32727]: 2022-03-21T17:45:11.224+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:45:11.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:45:11.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:45:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:45:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:45:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:45:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:12 smithi016 conmon[32727]: 2022-03-21T17:45:12.247+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:12 smithi016 ceph-mon[22010]: pgmap v991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:45:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:45:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:45:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:45:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:45:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:45:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:12 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:45:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:12 smithi079 ceph-mon[23158]: pgmap v991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:45:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:45:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:45:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:45:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:45:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:45:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:12 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:45:12.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[32727]: 2022-03-21T17:45:13.215+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: pgmap v992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:45:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:45:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:45:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:45:14.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:45:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:13 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:45:14.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 2022-03-21T17:45:13.933+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:14.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:45:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: yield 2022-03-21T17:45:14.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:14.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:45:14.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:14.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:45:14.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:45:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:14.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:45:14.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:14.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:45:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:14.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 2022-03-21T17:45:13.934+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:45:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:45:14.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: yield 2022-03-21T17:45:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:14.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:45:14.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:45:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:14.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:45:14.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:14.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:45:14.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:14.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:45:14.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:14.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 2022-03-21T17:45:14.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:45:14.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: 2022-03-21T17:45:14.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:45:14.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:45:14.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:45:14.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:45:14.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:45:14.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:45:14.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:45:14.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:45:14.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:45:14.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:45:14.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:45:14.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:45:14.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:45:14.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:45:14.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:45:14.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:45:14.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:45:14.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:45:14.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:45:14.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:45:14.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:45:14.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:45:14.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise 2022-03-21T17:45:14.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:45:14.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:45:14.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:45:14.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:45:14.200 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:45:13 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: pgmap v992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:45:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:45:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:45:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:45:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:45:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:45:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:13 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:45:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:14 smithi016 conmon[32727]: 2022-03-21T17:45:14.186+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:15.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:15.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:14 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:15.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:45:15.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:15.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:15.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:15.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:45:15.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:15.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:45:15.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:15.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:45:15.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:15.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:15.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:15.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:45:15.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:15.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:15.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:14 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:45:15.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:15.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:45:15.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:15.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:15.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:15.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:45:15.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:15.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:45:15.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:15.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:45:15.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:15.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:45:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:45:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:45:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:45:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:45:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:45:15.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:45:15.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:45:15.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:45:15.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:45:15.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:45:15.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:45:15.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:45:15.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:45:15.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:45:15.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:45:15.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:45:15.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:45:15.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:45:15.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:45:15.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:45:15.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:45:15.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:45:15.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:45:15.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:45:15.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:45:15.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:45:15.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:45:15.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:45:15.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:45:15.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:14 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:45:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:15.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:15.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:15.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:45:15.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:15.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:45:15.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:15.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:45:15.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:15.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:15.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:15.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:45:15.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:15.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:15.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:14 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:45:15.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:45:15.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:45:15.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:45:15.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:45:15.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:45:15.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:45:15.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:45:15.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:45:15.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:45:15.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:45:15.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:45:15.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:45:15.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:45:15.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:45:15.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:45:15.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:45:15.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:45:15.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:45:15.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:45:15.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:45:15.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:45:15.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:45:15.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:45:15.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:45:15.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:45:15.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:45:15.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:45:15.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:45:15.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:45:15.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:45:15.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:45:15.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:45:15.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:45:15.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:45:15.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:45:15.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:45:15.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:45:15.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:45:15.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:45:15.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:45:15.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:45:15.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:45:15.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:45:15.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:45:15.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:45:15.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:45:15.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:45:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:15 smithi016 conmon[32727]: 2022-03-21T17:45:15.147+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:16.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:16.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:15 smithi016 ceph-mon[22010]: pgmap v993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:15 smithi079 ceph-mon[23158]: pgmap v993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:16 smithi016 conmon[32727]: 2022-03-21T17:45:16.134+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:17.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:17 smithi016 conmon[32727]: 2022-03-21T17:45:17.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:18.090 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:18.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:17 smithi016 ceph-mon[22010]: pgmap v994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:17 smithi079 ceph-mon[23158]: pgmap v994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:18.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:18 smithi016 conmon[32727]: 2022-03-21T17:45:18.089+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:19.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:19.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:19 smithi016 conmon[32727]: 2022-03-21T17:45:19.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:20.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:20.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:19 smithi016 ceph-mon[22010]: pgmap v995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:19 smithi079 ceph-mon[23158]: pgmap v995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:20 smithi016 conmon[32727]: 2022-03-21T17:45:20.135+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:21.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:21 smithi016 conmon[32727]: 2022-03-21T17:45:21.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:22.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:22.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:21 smithi016 ceph-mon[22010]: pgmap v996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:21 smithi079 ceph-mon[23158]: pgmap v996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:22 smithi016 conmon[32727]: 2022-03-21T17:45:22.097+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:23.105 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:23 smithi016 conmon[32727]: 2022-03-21T17:45:23.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:24.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:24.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:23 smithi016 ceph-mon[22010]: pgmap v997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:24.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:23 smithi079 ceph-mon[23158]: pgmap v997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:24 smithi016 conmon[32727]: 2022-03-21T17:45:24.128+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:25.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:25 smithi016 conmon[32727]: 2022-03-21T17:45:25.107+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:26.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:26.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:25 smithi016 ceph-mon[22010]: pgmap v998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:25 smithi079 ceph-mon[23158]: pgmap v998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:26 smithi016 conmon[32727]: 2022-03-21T17:45:26.154+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:27 smithi016 conmon[32727]: 2022-03-21T17:45:27.196+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:27 smithi016 ceph-mon[22010]: pgmap v999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:27 smithi079 ceph-mon[23158]: pgmap v999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:28 smithi016 conmon[32727]: 2022-03-21T17:45:28.185+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:29 smithi016 conmon[32727]: 2022-03-21T17:45:29.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:29 smithi016 ceph-mon[22010]: pgmap v1000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:29 smithi079 ceph-mon[23158]: pgmap v1000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:30 smithi016 conmon[32727]: 2022-03-21T17:45:30.181+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:31 smithi016 conmon[32727]: 2022-03-21T17:45:31.172+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:32.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:32.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:31 smithi016 ceph-mon[22010]: pgmap v1001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:31 smithi079 ceph-mon[23158]: pgmap v1001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:32 smithi016 conmon[32727]: 2022-03-21T17:45:32.141+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:33.151 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:33 smithi016 conmon[32727]: 2022-03-21T17:45:33.148+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:34.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:34.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:33 smithi016 ceph-mon[22010]: pgmap v1002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:34.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:33 smithi079 ceph-mon[23158]: pgmap v1002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:34 smithi016 conmon[32727]: 2022-03-21T17:45:34.137+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:35 smithi016 conmon[32727]: 2022-03-21T17:45:35.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:35 smithi016 ceph-mon[22010]: pgmap v1003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:36 smithi016 conmon[32727]: 2022-03-21T17:45:36.059+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:35 smithi079 ceph-mon[23158]: pgmap v1003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:37 smithi016 conmon[32727]: 2022-03-21T17:45:37.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:37 smithi016 ceph-mon[22010]: pgmap v1004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:38 smithi016 conmon[32727]: 2022-03-21T17:45:38.088+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:37 smithi079 ceph-mon[23158]: pgmap v1004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:39 smithi016 conmon[32727]: 2022-03-21T17:45:39.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:39 smithi016 ceph-mon[22010]: pgmap v1005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:40.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:40 smithi016 conmon[32727]: 2022-03-21T17:45:40.112+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:39 smithi079 ceph-mon[23158]: pgmap v1005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:41.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:41 smithi016 conmon[32727]: 2022-03-21T17:45:41.152+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:41 smithi016 ceph-mon[22010]: pgmap v1006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:41 smithi079 ceph-mon[23158]: pgmap v1006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:42 smithi016 conmon[32727]: 2022-03-21T17:45:42.189+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:43 smithi016 conmon[32727]: 2022-03-21T17:45:43.193+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:44.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:43 smithi016 ceph-mon[22010]: pgmap v1007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:43 smithi079 ceph-mon[23158]: pgmap v1007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:44 smithi016 conmon[32727]: 2022-03-21T17:45:44.160+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:45 smithi016 conmon[32727]: 2022-03-21T17:45:45.145+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:46.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:46.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:45 smithi016 ceph-mon[22010]: pgmap v1008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:46.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:46 smithi016 conmon[32727]: 2022-03-21T17:45:46.139+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:45 smithi079 ceph-mon[23158]: pgmap v1008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:47.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:47 smithi016 conmon[32727]: 2022-03-21T17:45:47.099+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:47 smithi016 ceph-mon[22010]: pgmap v1009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:48 smithi016 conmon[32727]: 2022-03-21T17:45:48.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:47 smithi079 ceph-mon[23158]: pgmap v1009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:49 smithi016 conmon[32727]: 2022-03-21T17:45:49.108+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:49 smithi016 ceph-mon[22010]: pgmap v1010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:50 smithi016 conmon[32727]: 2022-03-21T17:45:50.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:49 smithi079 ceph-mon[23158]: pgmap v1010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:51 smithi016 conmon[32727]: 2022-03-21T17:45:51.142+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:51 smithi079 ceph-mon[23158]: pgmap v1011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:51 smithi016 ceph-mon[22010]: pgmap v1011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:52.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:52 smithi016 conmon[32727]: 2022-03-21T17:45:52.108+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:53 smithi016 conmon[32727]: 2022-03-21T17:45:53.078+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:53 smithi079 ceph-mon[23158]: pgmap v1012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:53 smithi016 ceph-mon[22010]: pgmap v1012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:54 smithi016 conmon[32727]: 2022-03-21T17:45:54.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:55 smithi016 conmon[32727]: 2022-03-21T17:45:55.065+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:55 smithi079 ceph-mon[23158]: pgmap v1013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:55 smithi016 ceph-mon[22010]: pgmap v1013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:45:56.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:56 smithi016 conmon[32727]: 2022-03-21T17:45:56.023+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:57 smithi016 conmon[32727]: 2022-03-21T17:45:57.044+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:57 smithi079 ceph-mon[23158]: pgmap v1014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:58 smithi016 conmon[32727]: 2022-03-21T17:45:58.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:57 smithi016 ceph-mon[22010]: pgmap v1014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:45:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:45:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:45:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:45:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:45:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:45:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:45:59 smithi016 conmon[32727]: 2022-03-21T17:45:59.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:45:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:45:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:45:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:45:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:45:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:45:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:45:59 smithi079 ceph-mon[23158]: pgmap v1015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:45:59 smithi016 ceph-mon[22010]: pgmap v1015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:00.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:00 smithi016 conmon[32727]: 2022-03-21T17:46:00.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:01.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:01 smithi016 conmon[32727]: 2022-03-21T17:46:01.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:01 smithi079 ceph-mon[23158]: pgmap v1016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:01 smithi016 ceph-mon[22010]: pgmap v1016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:01 smithi016 conmon[32727]: 2022-03-21T17:46:01.998+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:03 smithi016 conmon[32727]: 2022-03-21T17:46:03.019+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:03 smithi079 ceph-mon[23158]: pgmap v1017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:03 smithi016 ceph-mon[22010]: pgmap v1017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:04 smithi016 conmon[32727]: 2022-03-21T17:46:04.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:04 smithi016 conmon[32727]: 2022-03-21T17:46:04.979+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:05 smithi079 ceph-mon[23158]: pgmap v1018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:05 smithi016 ceph-mon[22010]: pgmap v1018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:05 smithi016 conmon[32727]: 2022-03-21T17:46:05.966+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:06 smithi016 conmon[32727]: 2022-03-21T17:46:06.916+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:07 smithi016 ceph-mon[22010]: pgmap v1019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:08.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:07 smithi016 conmon[32727]: 2022-03-21T17:46:07.895+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:07 smithi079 ceph-mon[23158]: pgmap v1019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:09.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:08 smithi016 conmon[32727]: 2022-03-21T17:46:08.879+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:09 smithi016 ceph-mon[22010]: pgmap v1020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:09 smithi016 conmon[32727]: 2022-03-21T17:46:09.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:10.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:09 smithi079 ceph-mon[23158]: pgmap v1020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:10.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:10 smithi016 conmon[32727]: 2022-03-21T17:46:10.899+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:11 smithi016 ceph-mon[22010]: pgmap v1021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:11 smithi016 conmon[32727]: 2022-03-21T17:46:11.855+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:11 smithi079 ceph-mon[23158]: pgmap v1021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:12 smithi016 conmon[32727]: 2022-03-21T17:46:12.881+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:13 smithi016 conmon[32727]: 2022-03-21T17:46:13.862+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: pgmap v1022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:46:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:46:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:46:14.390 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: pgmap v1022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:14.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:14.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:14.391 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:46:14.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:46:14.392 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:46:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:14 smithi016 conmon[32727]: 2022-03-21T17:46:14.837+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:15 smithi016 conmon[32727]: 2022-03-21T17:46:15.812+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:15 smithi016 ceph-mon[22010]: pgmap v1023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:15 smithi079 ceph-mon[23158]: pgmap v1023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:16.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:16 smithi016 conmon[32727]: 2022-03-21T17:46:16.789+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:17.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: 2022-03-21T17:46:17.037+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:17.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:46:17.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:17.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: yield 2022-03-21T17:46:17.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:17.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:17.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:17.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:46:17.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:17.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:46:17.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:17.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:46:17.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:17.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:46:17.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:17.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:46:17.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:17.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:17.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:17 smithi016 conmon[32727]: 2022-03-21T17:46:17.837+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:17 smithi016 ceph-mon[22010]: pgmap v1024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:18.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:17 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:18.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:46:18.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:18.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:46:18.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:18.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:18.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:18.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:46:18.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:18.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:46:18.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:18.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:46:18.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:18.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:18.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:18.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:46:18.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:18.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:18.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:18.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:17 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:46:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:17 smithi079 ceph-mon[23158]: pgmap v1024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:17 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:46:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:46:18.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:18.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:18.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:18.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:46:18.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:18.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:46:18.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:18.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:46:18.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:18.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:18.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:18.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:46:18.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:18.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:18.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:18.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:17 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:46:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:18 smithi016 conmon[32727]: 2022-03-21T17:46:18.842+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:19 smithi016 conmon[32727]: 2022-03-21T17:46:19.887+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:19 smithi016 ceph-mon[22010]: pgmap v1025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:20.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 2022-03-21T17:46:20.109+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:20.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:46:20.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:20.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: yield 2022-03-21T17:46:20.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:20.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:20.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:20.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:46:20.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:20.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:46:20.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:20.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:46:20.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:20.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:46:20.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:20.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:46:20.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:20.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:20.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:20.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 2022-03-21T17:46:20.110+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:46:20.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:46:20.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:20.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: yield 2022-03-21T17:46:20.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:20.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:20.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:20.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:46:20.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:20.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:46:20.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:20.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:46:20.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:20.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:46:20.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:20.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:46:20.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:20.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:20.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:20.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 2022-03-21T17:46:20.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:46:20.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: 2022-03-21T17:46:20.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:46:20.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:46:20.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:46:20.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:46:20.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:46:20.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:46:20.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:46:20.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:46:20.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:46:20.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:46:20.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:46:20.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:46:20.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:46:20.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:46:20.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:46:20.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:46:20.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:46:20.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:46:20.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:46:20.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:46:20.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:46:20.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:46:20.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise 2022-03-21T17:46:20.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:46:20.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:46:20.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:46:20.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:46:20.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:19 smithi079 ceph-mon[23158]: pgmap v1025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:20 smithi016 conmon[32727]: 2022-03-21T17:46:20.899+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:20 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:21.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:46:21.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:21.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:46:21.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:21.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:46:21.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:21.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:46:21.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:21.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:21.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:21.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:46:21.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:21.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:21.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:20 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:46:21.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:21.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:46:21.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:21.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:21.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:21.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:46:21.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:21.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:46:21.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:21.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:46:21.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:21.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:21.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:21.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:46:21.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:21.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:21.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:46:21.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:46:21.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:46:21.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:46:21.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:46:21.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:46:21.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:46:21.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:46:21.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:46:21.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:46:21.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:46:21.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:46:21.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:46:21.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:46:21.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:46:21.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:46:21.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:46:21.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:46:21.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:46:21.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:46:21.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:46:21.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:46:21.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:46:21.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:46:21.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:46:21.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:46:21.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:46:21.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:46:21.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:46:21.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:20 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:46:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:46:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:46:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:46:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:46:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:20 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:46:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:46:21.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:46:21.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:46:21.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:46:21.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:46:21.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:46:21.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:46:21.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:46:21.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:46:21.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:46:21.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:46:21.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:46:21.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:46:21.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:46:21.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:46:21.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:46:21.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:21.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:46:21.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:46:21.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:46:21.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:46:21.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:46:21.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:46:21.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:46:21.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:46:21.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:46:21.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:46:21.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:46:21.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:46:21.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:46:21.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:46:21.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:46:21.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:46:21.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:46:21.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:46:21.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:46:21.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:46:21.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:46:21.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:46:21.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:46:21.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:46:21.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:46:21.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:46:21.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:46:21.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:46:21.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:46:21.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:46:21.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:46:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:22 smithi016 ceph-mon[22010]: pgmap v1026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:22.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:21 smithi016 conmon[32727]: 2022-03-21T17:46:21.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:22 smithi079 ceph-mon[23158]: pgmap v1026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:22 smithi016 conmon[32727]: 2022-03-21T17:46:22.882+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:23 smithi016 conmon[32727]: 2022-03-21T17:46:23.909+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:24 smithi016 ceph-mon[22010]: pgmap v1027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1424 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:24 smithi079 ceph-mon[23158]: pgmap v1027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1424 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:24 smithi016 conmon[32727]: 2022-03-21T17:46:24.911+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:26 smithi016 ceph-mon[22010]: pgmap v1028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:25 smithi016 conmon[32727]: 2022-03-21T17:46:25.905+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:26 smithi079 ceph-mon[23158]: pgmap v1028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:27.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:26 smithi016 conmon[32727]: 2022-03-21T17:46:26.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:28 smithi016 ceph-mon[22010]: pgmap v1029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:27 smithi016 conmon[32727]: 2022-03-21T17:46:27.887+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:28 smithi079 ceph-mon[23158]: pgmap v1029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:28 smithi016 conmon[32727]: 2022-03-21T17:46:28.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:29 smithi016 conmon[32727]: 2022-03-21T17:46:29.874+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:30 smithi079 ceph-mon[23158]: pgmap v1030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:30 smithi016 ceph-mon[22010]: pgmap v1030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:30 smithi016 conmon[32727]: 2022-03-21T17:46:30.870+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:31 smithi016 conmon[32727]: 2022-03-21T17:46:31.824+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:32 smithi079 ceph-mon[23158]: pgmap v1031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:32 smithi016 ceph-mon[22010]: pgmap v1031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:32 smithi016 conmon[32727]: 2022-03-21T17:46:32.867+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:33 smithi016 conmon[32727]: 2022-03-21T17:46:33.856+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:34 smithi079 ceph-mon[23158]: pgmap v1032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:34 smithi016 ceph-mon[22010]: pgmap v1032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:34 smithi016 conmon[32727]: 2022-03-21T17:46:34.893+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:35 smithi016 conmon[32727]: 2022-03-21T17:46:35.852+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:36 smithi079 ceph-mon[23158]: pgmap v1033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:36 smithi016 ceph-mon[22010]: pgmap v1033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:36.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:37.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:36 smithi016 conmon[32727]: 2022-03-21T17:46:36.879+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:37 smithi016 conmon[32727]: 2022-03-21T17:46:37.890+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:38 smithi079 ceph-mon[23158]: pgmap v1034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:38 smithi016 ceph-mon[22010]: pgmap v1034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:38 smithi016 conmon[32727]: 2022-03-21T17:46:38.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:39 smithi016 conmon[32727]: 2022-03-21T17:46:39.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:40 smithi079 ceph-mon[23158]: pgmap v1035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:40 smithi016 ceph-mon[22010]: pgmap v1035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:40 smithi016 conmon[32727]: 2022-03-21T17:46:40.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:41 smithi016 conmon[32727]: 2022-03-21T17:46:41.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:42 smithi079 ceph-mon[23158]: pgmap v1036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:42 smithi016 ceph-mon[22010]: pgmap v1036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:42 smithi016 conmon[32727]: 2022-03-21T17:46:42.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:43 smithi016 conmon[32727]: 2022-03-21T17:46:43.836+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:44 smithi079 ceph-mon[23158]: pgmap v1037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:44 smithi016 ceph-mon[22010]: pgmap v1037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:44 smithi016 conmon[32727]: 2022-03-21T17:46:44.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:45 smithi016 conmon[32727]: 2022-03-21T17:46:45.868+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:46 smithi016 ceph-mon[22010]: pgmap v1038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:46 smithi079 ceph-mon[23158]: pgmap v1038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:46 smithi016 conmon[32727]: 2022-03-21T17:46:46.893+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:48.236 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:47 smithi016 conmon[32727]: 2022-03-21T17:46:47.928+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:48 smithi016 ceph-mon[22010]: pgmap v1039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:48 smithi079 ceph-mon[23158]: pgmap v1039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:48 smithi016 conmon[32727]: 2022-03-21T17:46:48.895+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:50.243 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:49 smithi016 conmon[32727]: 2022-03-21T17:46:49.914+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:50 smithi016 ceph-mon[22010]: pgmap v1040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:50 smithi079 ceph-mon[23158]: pgmap v1040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:50 smithi016 conmon[32727]: 2022-03-21T17:46:50.883+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:51 smithi016 conmon[32727]: 2022-03-21T17:46:51.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:52 smithi016 ceph-mon[22010]: pgmap v1041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:52.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:52 smithi079 ceph-mon[23158]: pgmap v1041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:52.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:52 smithi016 conmon[32727]: 2022-03-21T17:46:52.851+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:53 smithi016 conmon[32727]: 2022-03-21T17:46:53.814+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:54 smithi016 ceph-mon[22010]: pgmap v1042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:54 smithi079 ceph-mon[23158]: pgmap v1042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:54 smithi016 conmon[32727]: 2022-03-21T17:46:54.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:55 smithi016 conmon[32727]: 2022-03-21T17:46:55.751+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:56 smithi016 ceph-mon[22010]: pgmap v1043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:56 smithi079 ceph-mon[23158]: pgmap v1043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:46:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:56 smithi016 conmon[32727]: 2022-03-21T17:46:56.742+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:57 smithi016 conmon[32727]: 2022-03-21T17:46:57.730+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: pgmap v1044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:46:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:46:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:46:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:46:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: pgmap v1044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:46:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:46:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:46:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:46:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:46:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:46:59.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:58 smithi016 conmon[32727]: 2022-03-21T17:46:58.773+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:46:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:46:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:46:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:46:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:46:59 smithi016 conmon[32727]: 2022-03-21T17:46:59.797+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:00 smithi016 ceph-mon[22010]: pgmap v1045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:00 smithi079 ceph-mon[23158]: pgmap v1045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:00 smithi016 conmon[32727]: 2022-03-21T17:47:00.819+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:01 smithi016 conmon[32727]: 2022-03-21T17:47:01.784+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:02 smithi016 ceph-mon[22010]: pgmap v1046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:02 smithi079 ceph-mon[23158]: pgmap v1046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:02 smithi016 conmon[32727]: 2022-03-21T17:47:02.766+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:03 smithi016 conmon[32727]: 2022-03-21T17:47:03.808+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:04 smithi016 ceph-mon[22010]: pgmap v1047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:04 smithi079 ceph-mon[23158]: pgmap v1047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:04 smithi016 conmon[32727]: 2022-03-21T17:47:04.823+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:05 smithi016 conmon[32727]: 2022-03-21T17:47:05.799+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:06 smithi016 ceph-mon[22010]: pgmap v1048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:06 smithi079 ceph-mon[23158]: pgmap v1048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:06.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:06 smithi016 conmon[32727]: 2022-03-21T17:47:06.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:07 smithi016 conmon[32727]: 2022-03-21T17:47:07.848+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:08 smithi016 ceph-mon[22010]: pgmap v1049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1469 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:08 smithi079 ceph-mon[23158]: pgmap v1049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1469 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:08 smithi016 conmon[32727]: 2022-03-21T17:47:08.824+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:09 smithi016 conmon[32727]: 2022-03-21T17:47:09.874+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:10 smithi016 ceph-mon[22010]: pgmap v1050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:10 smithi079 ceph-mon[23158]: pgmap v1050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:10 smithi016 conmon[32727]: 2022-03-21T17:47:10.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:11 smithi016 conmon[32727]: 2022-03-21T17:47:11.880+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:12 smithi016 ceph-mon[22010]: pgmap v1051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:12 smithi079 ceph-mon[23158]: pgmap v1051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:12 smithi016 conmon[32727]: 2022-03-21T17:47:12.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:13 smithi016 conmon[32727]: 2022-03-21T17:47:13.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:14 smithi016 ceph-mon[22010]: pgmap v1052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:14 smithi079 ceph-mon[23158]: pgmap v1052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:14 smithi016 conmon[32727]: 2022-03-21T17:47:14.889+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:15 smithi016 conmon[32727]: 2022-03-21T17:47:15.904+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:16 smithi016 ceph-mon[22010]: pgmap v1053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:16 smithi079 ceph-mon[23158]: pgmap v1053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:17.331 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:16 smithi016 conmon[32727]: 2022-03-21T17:47:16.917+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:17 smithi016 ceph-mon[22010]: pgmap v1054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:17.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:17 smithi079 ceph-mon[23158]: pgmap v1054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:17.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:17 smithi016 conmon[32727]: 2022-03-21T17:47:17.870+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:18.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:18.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:19.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:18 smithi016 conmon[32727]: 2022-03-21T17:47:18.890+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:19 smithi016 ceph-mon[22010]: pgmap v1055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:19.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:19 smithi079 ceph-mon[23158]: pgmap v1055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:19 smithi016 conmon[32727]: 2022-03-21T17:47:19.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:20.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:20.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:47:20.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:47:20.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:47:20.571 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:20.572 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:47:20.572 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:47:20.573 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:47:20.909 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: 2022-03-21T17:47:20.589+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:20.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:47:20.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:20.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: yield 2022-03-21T17:47:20.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:20.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:20.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:20.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:47:20.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:20.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:47:20.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:20.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:47:20.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:20.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:47:20.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:47:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:20 smithi016 conmon[32727]: 2022-03-21T17:47:20.907+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:21 smithi016 ceph-mon[22010]: pgmap v1056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:21.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:47:21.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:21.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:47:21.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:21.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:21.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:21.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:47:21.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:21.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:47:21.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:21.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:47:21.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:21.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:21.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:21.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:47:21.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:21.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:21.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:21.676 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:47:21.676 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:21 smithi079 ceph-mon[23158]: pgmap v1056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:47:21.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:21.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:47:21.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:21.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:21.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:21.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:47:21.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:21.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:47:21.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:21.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:47:21.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:21.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:21.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:21.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:47:21.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:21.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:21.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:21.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:47:21.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:22.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:21 smithi016 conmon[32727]: 2022-03-21T17:47:21.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:22 smithi016 conmon[32727]: 2022-03-21T17:47:22.904+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:23.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:23 smithi016 ceph-mon[22010]: pgmap v1057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:23 smithi079 ceph-mon[23158]: pgmap v1057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:23.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[32727]: 2022-03-21T17:47:23.887+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:23.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 2022-03-21T17:47:23.661+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:23.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:47:23.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:23.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: yield 2022-03-21T17:47:23.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:23.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:23.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:23.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:47:23.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:23.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:47:23.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:23.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:47:23.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:23.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:47:23.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:23.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:47:23.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:23.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:23.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:23.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 2022-03-21T17:47:23.662+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:47:23.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:47:23.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:23.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: yield 2022-03-21T17:47:23.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:23.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:23.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:23.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:47:23.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:23.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:47:23.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:23.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:47:23.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:23.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:47:23.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:23.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:47:23.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:23.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:23.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:23.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 2022-03-21T17:47:23.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:47:23.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: 2022-03-21T17:47:23.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:47:23.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:47:23.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:47:23.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:47:23.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:47:23.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:47:23.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:47:23.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:47:23.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:47:23.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:47:23.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:47:23.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:47:23.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:47:23.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:47:23.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:47:23.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:47:23.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:47:23.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:47:23.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:47:23.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:47:23.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:47:23.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:47:23.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise 2022-03-21T17:47:23.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:47:23.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:47:23.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:47:23.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:47:23.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:47:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:47:24.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:24.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:24.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:24.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:47:24.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:24.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:47:24.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:24.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:47:24.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:24.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:24.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:24.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:47:24.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:24.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:24.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:47:24.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:24.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:47:24.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:24.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:24.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:24.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:47:24.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:24.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:47:24.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:24.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:47:24.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:24.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:24.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:24.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:47:24.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:24.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:24.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:47:24.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:47:24.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:47:24.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:47:24.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:47:24.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:47:24.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:47:24.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:47:24.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:47:24.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:47:24.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:47:24.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:47:24.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:47:24.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:47:24.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:47:24.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:47:24.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:47:24.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:47:24.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:47:24.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:47:24.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:47:24.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:47:24.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:47:24.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:47:24.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:47:24.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:47:24.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:47:24.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:47:24.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:47:24.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:47:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:24.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:24.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:24.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:47:24.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:24.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:47:24.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:24.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:47:24.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:24.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:24.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:24.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:47:24.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:24.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:24.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:47:24.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:47:24.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:47:24.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:47:24.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:47:24.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:47:24.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:47:24.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:47:24.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:47:24.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:47:24.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:47:24.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:47:24.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:47:24.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:47:24.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:47:24.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:47:24.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:47:24.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:47:24.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:47:24.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:47:24.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:47:24.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:47:24.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:47:24.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:47:24.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:47:24.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:47:24.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:47:24.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:47:24.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:47:24.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:47:24.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:47:24.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:47:24.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:47:24.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:47:24.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:47:24.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:47:24.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:47:24.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:47:24.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:47:24.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:47:24.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:47:24.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:47:24.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:47:24.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:47:24.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:47:24.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:47:24.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:47:24.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:47:24.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:24 smithi016 conmon[32727]: 2022-03-21T17:47:24.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:25 smithi016 ceph-mon[22010]: pgmap v1058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:25 smithi079 ceph-mon[23158]: pgmap v1058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:25 smithi016 conmon[32727]: 2022-03-21T17:47:25.870+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:26 smithi016 conmon[32727]: 2022-03-21T17:47:26.868+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:27 smithi016 ceph-mon[22010]: pgmap v1059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:27 smithi079 ceph-mon[23158]: pgmap v1059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:27 smithi016 conmon[32727]: 2022-03-21T17:47:27.885+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:29.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:28 smithi016 conmon[32727]: 2022-03-21T17:47:28.870+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:29 smithi016 ceph-mon[22010]: pgmap v1060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:29 smithi079 ceph-mon[23158]: pgmap v1060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:29 smithi016 conmon[32727]: 2022-03-21T17:47:29.911+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:30 smithi016 conmon[32727]: 2022-03-21T17:47:30.865+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:31 smithi016 ceph-mon[22010]: pgmap v1061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:31 smithi079 ceph-mon[23158]: pgmap v1061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:31 smithi016 conmon[32727]: 2022-03-21T17:47:31.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:32 smithi016 conmon[32727]: 2022-03-21T17:47:32.878+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:33 smithi016 ceph-mon[22010]: pgmap v1062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:33 smithi079 ceph-mon[23158]: pgmap v1062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:34.383 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:33 smithi016 conmon[32727]: 2022-03-21T17:47:33.920+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:35.388 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:34 smithi016 conmon[32727]: 2022-03-21T17:47:34.922+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:35 smithi016 ceph-mon[22010]: pgmap v1063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:35 smithi079 ceph-mon[23158]: pgmap v1063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:35 smithi016 conmon[32727]: 2022-03-21T17:47:35.874+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:36 smithi016 conmon[32727]: 2022-03-21T17:47:36.884+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:37 smithi016 ceph-mon[22010]: pgmap v1064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:37 smithi079 ceph-mon[23158]: pgmap v1064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:37 smithi016 conmon[32727]: 2022-03-21T17:47:37.852+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:38 smithi016 conmon[32727]: 2022-03-21T17:47:38.881+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:39 smithi016 ceph-mon[22010]: pgmap v1065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:39.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:39 smithi079 ceph-mon[23158]: pgmap v1065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:39 smithi016 conmon[32727]: 2022-03-21T17:47:39.846+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:40 smithi016 conmon[32727]: 2022-03-21T17:47:40.835+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:41 smithi016 ceph-mon[22010]: pgmap v1066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:41 smithi079 ceph-mon[23158]: pgmap v1066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:41 smithi016 conmon[32727]: 2022-03-21T17:47:41.882+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:42 smithi016 conmon[32727]: 2022-03-21T17:47:42.874+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:43 smithi016 ceph-mon[22010]: pgmap v1067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:43 smithi079 ceph-mon[23158]: pgmap v1067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:43 smithi016 conmon[32727]: 2022-03-21T17:47:43.835+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:44.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:44 smithi016 conmon[32727]: 2022-03-21T17:47:44.868+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:45 smithi079 ceph-mon[23158]: pgmap v1068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:45.875 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:45 smithi016 ceph-mon[22010]: pgmap v1068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:45.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:45 smithi016 conmon[32727]: 2022-03-21T17:47:45.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:46.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:46 smithi016 conmon[32727]: 2022-03-21T17:47:46.863+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:47 smithi079 ceph-mon[23158]: pgmap v1069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:47.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:47 smithi016 ceph-mon[22010]: pgmap v1069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:47.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:48.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:47 smithi016 conmon[32727]: 2022-03-21T17:47:47.868+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:48.845 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:48.845 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:48 smithi016 conmon[32727]: 2022-03-21T17:47:48.842+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:49 smithi079 ceph-mon[23158]: pgmap v1070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:49.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:49 smithi016 ceph-mon[22010]: pgmap v1070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:49.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:49 smithi016 conmon[32727]: 2022-03-21T17:47:49.814+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:50.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:50 smithi016 conmon[32727]: 2022-03-21T17:47:50.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:51 smithi079 ceph-mon[23158]: pgmap v1071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:51.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:51 smithi016 ceph-mon[22010]: pgmap v1071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:51.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:51 smithi016 conmon[32727]: 2022-03-21T17:47:51.849+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:52.853 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:52 smithi016 conmon[32727]: 2022-03-21T17:47:52.851+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:53 smithi079 ceph-mon[23158]: pgmap v1072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:53.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:53 smithi016 ceph-mon[22010]: pgmap v1072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:53.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:53.857 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:53 smithi016 conmon[32727]: 2022-03-21T17:47:53.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:54.893 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:54 smithi016 conmon[32727]: 2022-03-21T17:47:54.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:55 smithi079 ceph-mon[23158]: pgmap v1073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:55 smithi016 ceph-mon[22010]: pgmap v1073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:47:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:55 smithi016 conmon[32727]: 2022-03-21T17:47:55.918+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:56.906 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:56 smithi016 conmon[32727]: 2022-03-21T17:47:56.904+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:57 smithi079 ceph-mon[23158]: pgmap v1074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:57 smithi016 ceph-mon[22010]: pgmap v1074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:57 smithi016 conmon[32727]: 2022-03-21T17:47:57.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:47:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:47:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:47:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:47:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:47:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:47:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:47:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:47:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:47:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:58 smithi016 conmon[32727]: 2022-03-21T17:47:58.958+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:47:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:59 smithi079 ceph-mon[23158]: pgmap v1075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:47:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:47:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:59 smithi016 ceph-mon[22010]: pgmap v1075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:47:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:47:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:47:59 smithi016 conmon[32727]: 2022-03-21T17:47:59.955+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:00 smithi016 conmon[32727]: 2022-03-21T17:48:00.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:01 smithi016 ceph-mon[22010]: pgmap v1076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:01 smithi079 ceph-mon[23158]: pgmap v1076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:02 smithi016 conmon[32727]: 2022-03-21T17:48:02.033+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:03 smithi016 conmon[32727]: 2022-03-21T17:48:03.047+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:03 smithi016 ceph-mon[22010]: pgmap v1077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:03 smithi079 ceph-mon[23158]: pgmap v1077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:04 smithi016 conmon[32727]: 2022-03-21T17:48:04.088+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:05 smithi016 conmon[32727]: 2022-03-21T17:48:05.068+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:05 smithi016 ceph-mon[22010]: pgmap v1078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:05 smithi079 ceph-mon[23158]: pgmap v1078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:06 smithi016 conmon[32727]: 2022-03-21T17:48:06.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:07 smithi016 conmon[32727]: 2022-03-21T17:48:07.083+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:07 smithi016 ceph-mon[22010]: pgmap v1079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:07 smithi079 ceph-mon[23158]: pgmap v1079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:08 smithi016 conmon[32727]: 2022-03-21T17:48:08.046+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:09 smithi016 conmon[32727]: 2022-03-21T17:48:09.004+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:09 smithi016 ceph-mon[22010]: pgmap v1080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:09.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:09 smithi079 ceph-mon[23158]: pgmap v1080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:10 smithi016 conmon[32727]: 2022-03-21T17:48:10.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:11 smithi016 conmon[32727]: 2022-03-21T17:48:11.069+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:11 smithi016 ceph-mon[22010]: pgmap v1081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:11 smithi079 ceph-mon[23158]: pgmap v1081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:12 smithi016 conmon[32727]: 2022-03-21T17:48:12.079+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:13 smithi016 conmon[32727]: 2022-03-21T17:48:13.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:13 smithi016 ceph-mon[22010]: pgmap v1082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:13 smithi079 ceph-mon[23158]: pgmap v1082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:14 smithi016 conmon[32727]: 2022-03-21T17:48:14.024+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:14 smithi016 conmon[32727]: 2022-03-21T17:48:14.989+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:15 smithi016 ceph-mon[22010]: pgmap v1083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:15 smithi079 ceph-mon[23158]: pgmap v1083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:15 smithi016 conmon[32727]: 2022-03-21T17:48:15.974+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:16 smithi016 conmon[32727]: 2022-03-21T17:48:16.985+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:17 smithi016 ceph-mon[22010]: pgmap v1084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:17 smithi079 ceph-mon[23158]: pgmap v1084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:17 smithi016 conmon[32727]: 2022-03-21T17:48:17.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:18 smithi016 conmon[32727]: 2022-03-21T17:48:18.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:19 smithi016 ceph-mon[22010]: pgmap v1085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:19 smithi079 ceph-mon[23158]: pgmap v1085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:19 smithi016 conmon[32727]: 2022-03-21T17:48:19.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:20 smithi016 conmon[32727]: 2022-03-21T17:48:20.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:21 smithi016 ceph-mon[22010]: pgmap v1086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:21 smithi079 ceph-mon[23158]: pgmap v1086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:21 smithi016 conmon[32727]: 2022-03-21T17:48:21.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:23 smithi016 conmon[32727]: 2022-03-21T17:48:23.017+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:23 smithi016 ceph-mon[22010]: pgmap v1087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:23 smithi079 ceph-mon[23158]: pgmap v1087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:24 smithi016 conmon[32727]: 2022-03-21T17:48:24.031+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:48:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:48:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:48:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:24.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:48:24.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:24 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:48:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:48:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:48:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:48:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:24.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:48:24.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:24 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:48:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:25 smithi016 conmon[32727]: 2022-03-21T17:48:25.075+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:26 smithi016 ceph-mon[22010]: pgmap v1088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[32727]: 2022-03-21T17:48:26.072+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:26 smithi079 ceph-mon[23158]: pgmap v1088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:27.092 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: 2022-03-21T17:48:26.765+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:27.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:48:27.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:27.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: yield 2022-03-21T17:48:27.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:27.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:27.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:27.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:48:27.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:27.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:48:27.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:27.097 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:48:27.097 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:27.097 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:48:27.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:27.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:48:27.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:27.099 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:27.099 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:27 smithi016 conmon[32727]: 2022-03-21T17:48:27.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:28 smithi016 conmon[32727]: 2022-03-21T17:48:28.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:28 smithi016 ceph-mon[22010]: pgmap v1089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:48:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:48:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:28.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:48:28.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:28.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:48:28.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:28.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:48:28.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:28.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:28.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:28.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:48:28.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:28.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:28.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:28.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:28 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:48:28.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:28 smithi079 ceph-mon[23158]: pgmap v1089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:48:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:48:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:48:28.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:28.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:48:28.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:28.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:48:28.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:28.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:28.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:28.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:48:28.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:28.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:28.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:28.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:28 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:48:28.485 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[32727]: 2022-03-21T17:48:29.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:30 smithi016 conmon[32727]: 2022-03-21T17:48:30.070+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:30.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 2022-03-21T17:48:29.836+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:30.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:48:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: yield 2022-03-21T17:48:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:48:30.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:30.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:48:30.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:30.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:48:30.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:30.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:48:30.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:30.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:48:30.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:30.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:30.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:30.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 2022-03-21T17:48:29.837+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:48:30.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:48:30.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:30.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: yield 2022-03-21T17:48:30.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:30.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:30.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:30.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:48:30.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:30.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:48:30.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:30.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:48:30.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:30.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:48:30.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:30.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:48:30.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:30.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:30.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:30.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 2022-03-21T17:48:30.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:48:30.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: 2022-03-21T17:48:30.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:48:30.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:48:30.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:48:30.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:48:30.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:48:30.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:48:30.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:48:30.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:48:30.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:48:30.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:48:30.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:48:30.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:48:30.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:48:30.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:48:30.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:48:30.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:48:30.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:48:30.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:48:30.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:48:30.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:48:30.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:48:30.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:48:30.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise 2022-03-21T17:48:30.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:48:30.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:48:30.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:48:30.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:48:30.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:48:29 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:30 smithi079 ceph-mon[23158]: pgmap v1090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:30 smithi016 ceph-mon[22010]: pgmap v1090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:31 smithi016 conmon[32727]: 2022-03-21T17:48:31.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:48:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:48:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:48:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:48:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:48:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:48:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:48:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:48:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:48:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:48:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:48:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:48:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:48:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:48:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:48:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:48:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:48:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:48:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:48:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:48:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:48:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:48:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:48:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:48:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:48:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:48:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:48:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:48:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:48:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:48:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:48:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:48:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:48:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:48:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:48:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:48:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:48:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:48:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:48:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:48:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:48:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:48:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:48:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:48:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:48:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:48:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:48:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:48:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:48:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:48:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:48:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:48:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:48:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:48:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:48:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:48:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:48:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:48:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:48:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:48:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:48:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:48:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:48:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:48:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:48:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:48:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:48:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:48:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:48:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:48:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:48:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:48:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:48:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:48:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:48:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:48:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:48:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:48:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:48:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:48:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:48:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:48:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:48:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:48:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:48:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:48:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:48:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:48:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:48:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:48:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:48:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:48:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:48:31.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:32 smithi016 ceph-mon[22010]: pgmap v1091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:32 smithi016 conmon[32727]: 2022-03-21T17:48:32.145+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:32 smithi079 ceph-mon[23158]: pgmap v1091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:33 smithi016 conmon[32727]: 2022-03-21T17:48:33.123+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:33.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:34 smithi016 ceph-mon[22010]: pgmap v1092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:34 smithi016 conmon[32727]: 2022-03-21T17:48:34.083+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:34 smithi079 ceph-mon[23158]: pgmap v1092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:35.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:35 smithi016 conmon[32727]: 2022-03-21T17:48:35.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:36 smithi016 ceph-mon[22010]: pgmap v1093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:36 smithi016 conmon[32727]: 2022-03-21T17:48:36.112+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:36 smithi079 ceph-mon[23158]: pgmap v1093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:37 smithi016 conmon[32727]: 2022-03-21T17:48:37.099+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:38 smithi016 ceph-mon[22010]: pgmap v1094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:38 smithi016 conmon[32727]: 2022-03-21T17:48:38.063+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:38 smithi079 ceph-mon[23158]: pgmap v1094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:39 smithi016 conmon[32727]: 2022-03-21T17:48:39.049+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:40 smithi016 conmon[32727]: 2022-03-21T17:48:40.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:40 smithi016 ceph-mon[22010]: pgmap v1095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:40 smithi079 ceph-mon[23158]: pgmap v1095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:41 smithi016 conmon[32727]: 2022-03-21T17:48:41.070+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:42 smithi016 conmon[32727]: 2022-03-21T17:48:42.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:42 smithi016 ceph-mon[22010]: pgmap v1096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:42 smithi079 ceph-mon[23158]: pgmap v1096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:43 smithi016 conmon[32727]: 2022-03-21T17:48:43.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:44 smithi016 ceph-mon[22010]: pgmap v1097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:44 smithi016 conmon[32727]: 2022-03-21T17:48:44.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:44 smithi079 ceph-mon[23158]: pgmap v1097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:45 smithi016 conmon[32727]: 2022-03-21T17:48:45.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:46 smithi016 ceph-mon[22010]: pgmap v1098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:46 smithi016 conmon[32727]: 2022-03-21T17:48:46.058+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:46 smithi079 ceph-mon[23158]: pgmap v1098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:47 smithi016 conmon[32727]: 2022-03-21T17:48:47.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:48 smithi016 ceph-mon[22010]: pgmap v1099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:48 smithi016 conmon[32727]: 2022-03-21T17:48:48.062+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:48 smithi079 ceph-mon[23158]: pgmap v1099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:49 smithi016 conmon[32727]: 2022-03-21T17:48:49.084+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:50 smithi016 ceph-mon[22010]: pgmap v1100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:50 smithi016 conmon[32727]: 2022-03-21T17:48:50.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:50 smithi079 ceph-mon[23158]: pgmap v1100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:51.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:51 smithi016 conmon[32727]: 2022-03-21T17:48:51.173+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:52 smithi016 conmon[32727]: 2022-03-21T17:48:52.131+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:52 smithi016 ceph-mon[22010]: pgmap v1101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:52 smithi079 ceph-mon[23158]: pgmap v1101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:53 smithi016 conmon[32727]: 2022-03-21T17:48:53.088+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:53.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:53.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:54 smithi016 ceph-mon[22010]: pgmap v1102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:54.420 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:54 smithi016 conmon[32727]: 2022-03-21T17:48:54.052+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:54 smithi079 ceph-mon[23158]: pgmap v1102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:55 smithi016 conmon[32727]: 2022-03-21T17:48:55.083+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:56 smithi016 ceph-mon[22010]: pgmap v1103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:56 smithi016 conmon[32727]: 2022-03-21T17:48:56.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:56 smithi079 ceph-mon[23158]: pgmap v1103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:48:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:57 smithi016 conmon[32727]: 2022-03-21T17:48:57.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:58 smithi016 conmon[32727]: 2022-03-21T17:48:58.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: pgmap v1104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:48:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:48:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:48:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:48:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: pgmap v1104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:48:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:48:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:48:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:48:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:48:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:48:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:48:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:48:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:48:59 smithi016 conmon[32727]: 2022-03-21T17:48:59.147+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:48:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:48:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:00 smithi016 ceph-mon[22010]: pgmap v1105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:00 smithi016 conmon[32727]: 2022-03-21T17:49:00.130+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:00 smithi079 ceph-mon[23158]: pgmap v1105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:01 smithi016 conmon[32727]: 2022-03-21T17:49:01.171+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:02 smithi016 ceph-mon[22010]: pgmap v1106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:02 smithi016 conmon[32727]: 2022-03-21T17:49:02.212+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:02 smithi079 ceph-mon[23158]: pgmap v1106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:03.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:03 smithi016 conmon[32727]: 2022-03-21T17:49:03.217+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:04 smithi016 ceph-mon[22010]: pgmap v1107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:04 smithi016 conmon[32727]: 2022-03-21T17:49:04.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:04 smithi079 ceph-mon[23158]: pgmap v1107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:05 smithi016 conmon[32727]: 2022-03-21T17:49:05.166+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:06 smithi016 ceph-mon[22010]: pgmap v1108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:06 smithi016 conmon[32727]: 2022-03-21T17:49:06.140+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:06 smithi079 ceph-mon[23158]: pgmap v1108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:07 smithi016 conmon[32727]: 2022-03-21T17:49:07.123+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:08 smithi016 conmon[32727]: 2022-03-21T17:49:08.111+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:08 smithi016 ceph-mon[22010]: pgmap v1109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:08 smithi079 ceph-mon[23158]: pgmap v1109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:09 smithi016 conmon[32727]: 2022-03-21T17:49:09.159+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:10 smithi016 ceph-mon[22010]: pgmap v1110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:10 smithi016 conmon[32727]: 2022-03-21T17:49:10.131+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:10 smithi079 ceph-mon[23158]: pgmap v1110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:11 smithi016 conmon[32727]: 2022-03-21T17:49:11.109+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:12 smithi016 ceph-mon[22010]: pgmap v1111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:12 smithi016 conmon[32727]: 2022-03-21T17:49:12.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:12 smithi079 ceph-mon[23158]: pgmap v1111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:13 smithi016 conmon[32727]: 2022-03-21T17:49:13.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:13 smithi016 ceph-mon[22010]: pgmap v1112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:13 smithi079 ceph-mon[23158]: pgmap v1112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:14 smithi016 conmon[32727]: 2022-03-21T17:49:14.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:15 smithi016 ceph-mon[22010]: pgmap v1113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:15 smithi016 conmon[32727]: 2022-03-21T17:49:15.137+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:15 smithi079 ceph-mon[23158]: pgmap v1113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:16 smithi016 conmon[32727]: 2022-03-21T17:49:16.163+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:17 smithi016 ceph-mon[22010]: pgmap v1114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:17 smithi016 conmon[32727]: 2022-03-21T17:49:17.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:17 smithi079 ceph-mon[23158]: pgmap v1114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:18 smithi016 conmon[32727]: 2022-03-21T17:49:18.143+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:19 smithi016 ceph-mon[22010]: pgmap v1115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:19 smithi016 conmon[32727]: 2022-03-21T17:49:19.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:19 smithi079 ceph-mon[23158]: pgmap v1115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:20.370 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:20 smithi016 conmon[32727]: 2022-03-21T17:49:20.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:21.375 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:21 smithi016 conmon[32727]: 2022-03-21T17:49:21.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:21 smithi016 ceph-mon[22010]: pgmap v1116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:21 smithi079 ceph-mon[23158]: pgmap v1116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:22.376 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:22 smithi016 conmon[32727]: 2022-03-21T17:49:22.027+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:23.381 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:23 smithi016 conmon[32727]: 2022-03-21T17:49:23.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:23 smithi016 ceph-mon[22010]: pgmap v1117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:23 smithi079 ceph-mon[23158]: pgmap v1117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:24.382 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:24 smithi016 conmon[32727]: 2022-03-21T17:49:24.042+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:25.387 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:24 smithi016 conmon[32727]: 2022-03-21T17:49:24.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:25 smithi016 ceph-mon[22010]: pgmap v1118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:25 smithi079 ceph-mon[23158]: pgmap v1118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:26.389 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:25 smithi016 conmon[32727]: 2022-03-21T17:49:25.948+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:27.393 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:26 smithi016 conmon[32727]: 2022-03-21T17:49:26.955+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:27 smithi016 ceph-mon[22010]: pgmap v1119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:27 smithi079 ceph-mon[23158]: pgmap v1119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:28.395 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:27 smithi016 conmon[32727]: 2022-03-21T17:49:27.989+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:29.400 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:28 smithi016 conmon[32727]: 2022-03-21T17:49:28.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:29 smithi016 ceph-mon[22010]: pgmap v1120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:29 smithi079 ceph-mon[23158]: pgmap v1120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:30 smithi016 conmon[32727]: 2022-03-21T17:49:30.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:49:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:49:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:49:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:49:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:49:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:49:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:31.406 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:30 smithi016 conmon[32727]: 2022-03-21T17:49:30.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:31 smithi016 ceph-mon[22010]: pgmap v1121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:31 smithi079 ceph-mon[23158]: pgmap v1121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:32.408 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:31 smithi016 conmon[32727]: 2022-03-21T17:49:31.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:33.412 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[32727]: 2022-03-21T17:49:32.925+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:33.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: 2022-03-21T17:49:32.940+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:49:33.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:33.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: yield 2022-03-21T17:49:33.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:33.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:33.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:33.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:49:33.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:33.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:49:33.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:49:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:49:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:49:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:33 smithi016 ceph-mon[22010]: pgmap v1122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:33 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:49:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:49:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:33.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:33.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:33.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:49:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:49:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:33.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:49:33.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:49:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:33 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:49:33.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:33 smithi079 ceph-mon[23158]: pgmap v1122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:33 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:49:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:49:33.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:33.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:33.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:33.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:49:33.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:33.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:49:33.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:33.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:49:33.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:33.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:33.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:33.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:49:33.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:33.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:33.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:33.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:33 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:49:33.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:34.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:33 smithi016 conmon[32727]: 2022-03-21T17:49:33.961+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:34 smithi016 conmon[32727]: 2022-03-21T17:49:34.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:35 smithi079 ceph-mon[23158]: pgmap v1123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:35 smithi016 ceph-mon[22010]: pgmap v1123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[32727]: 2022-03-21T17:49:35.998+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:36.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 2022-03-21T17:49:36.012+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:49:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: yield 2022-03-21T17:49:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:49:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:49:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:49:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:49:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 2022-03-21T17:49:36.013+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:49:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:49:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: yield 2022-03-21T17:49:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:49:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:49:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:49:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:49:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 2022-03-21T17:49:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:49:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: 2022-03-21T17:49:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:49:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:49:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:49:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:49:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:49:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:49:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:49:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:49:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:49:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:49:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:49:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:49:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:49:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:49:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:49:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:49:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:49:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:49:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:49:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:49:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:49:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:49:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise 2022-03-21T17:49:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:49:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:49:36.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:49:36.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:49:36.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:49:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:49:36.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:49:36.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:49:36.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:49:36.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:36 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:49:36.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:49:36.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:49:36.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:49:36.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:49:36.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:49:36.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:49:36.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:49:36.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:49:36.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:49:36.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:49:36.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:49:36.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:49:36.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:49:36.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:49:36.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:49:36.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:49:36.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:49:36.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:49:36.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:49:36.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:49:36.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:49:36.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:49:36.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:49:36.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:49:36.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:49:36.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:49:36.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:49:36.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:49:36.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:49:36.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:49:36.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:49:36.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:49:36.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:49:36.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:49:36.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:49:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:49:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:49:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:49:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:49:36.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:36 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:49:36.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:49:36.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:49:36.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:49:36.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:49:36.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:49:36.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:49:36.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:49:36.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:49:36.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:49:36.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:49:36.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:49:36.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:49:36.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:49:36.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:49:36.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:49:36.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:49:36.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:36.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:49:36.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:49:36.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:49:36.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:49:36.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:49:36.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:49:36.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:49:36.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:49:36.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:49:36.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:49:36.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:49:36.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:49:36.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:49:36.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:49:36.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:49:36.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:49:36.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:49:36.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:49:36.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:49:36.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:49:36.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:49:36.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:49:36.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:49:36.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:49:36.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:49:36.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:49:36.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:49:36.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:49:36.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:49:36.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:49:36.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:49:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:37 smithi016 conmon[32727]: 2022-03-21T17:49:37.019+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:37 smithi079 ceph-mon[23158]: pgmap v1124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:37 smithi016 ceph-mon[22010]: pgmap v1124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:38 smithi016 conmon[32727]: 2022-03-21T17:49:38.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:39 smithi016 conmon[32727]: 2022-03-21T17:49:39.017+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:39 smithi079 ceph-mon[23158]: pgmap v1125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:39 smithi016 ceph-mon[22010]: pgmap v1125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:39 smithi016 conmon[32727]: 2022-03-21T17:49:39.976+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:41 smithi016 conmon[32727]: 2022-03-21T17:49:41.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:41 smithi079 ceph-mon[23158]: pgmap v1126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:41 smithi016 ceph-mon[22010]: pgmap v1126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:42 smithi016 conmon[32727]: 2022-03-21T17:49:42.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:43 smithi016 conmon[32727]: 2022-03-21T17:49:43.012+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:43 smithi079 ceph-mon[23158]: pgmap v1127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:43 smithi016 ceph-mon[22010]: pgmap v1127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T17:49:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:43.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:43 smithi016 conmon[32727]: 2022-03-21T17:49:43.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:44 smithi016 conmon[32727]: 2022-03-21T17:49:44.944+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:45 smithi079 ceph-mon[23158]: pgmap v1128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:45 smithi016 ceph-mon[22010]: pgmap v1128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:45 smithi016 conmon[32727]: 2022-03-21T17:49:45.944+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:46.905 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:46 smithi016 conmon[32727]: 2022-03-21T17:49:46.902+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:47 smithi079 ceph-mon[23158]: pgmap v1129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:47.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:47 smithi016 ceph-mon[22010]: pgmap v1129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:47.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:47 smithi016 conmon[32727]: 2022-03-21T17:49:47.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:48.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:48.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:48 smithi016 conmon[32727]: 2022-03-21T17:49:48.895+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:49 smithi079 ceph-mon[23158]: pgmap v1130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:49.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:49 smithi016 ceph-mon[22010]: pgmap v1130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:49.896 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:49 smithi016 conmon[32727]: 2022-03-21T17:49:49.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:50.857 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:50 smithi016 conmon[32727]: 2022-03-21T17:49:50.855+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:51 smithi079 ceph-mon[23158]: pgmap v1131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:51.879 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:51 smithi016 ceph-mon[22010]: pgmap v1131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:51.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:51 smithi016 conmon[32727]: 2022-03-21T17:49:51.877+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:52.894 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:52 smithi016 conmon[32727]: 2022-03-21T17:49:52.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:53 smithi016 ceph-mon[22010]: pgmap v1132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:53 smithi079 ceph-mon[23158]: pgmap v1132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:53 smithi016 conmon[32727]: 2022-03-21T17:49:53.916+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:54 smithi016 conmon[32727]: 2022-03-21T17:49:54.931+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:55 smithi016 ceph-mon[22010]: pgmap v1133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:55 smithi079 ceph-mon[23158]: pgmap v1133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:49:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:55 smithi016 conmon[32727]: 2022-03-21T17:49:55.943+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:56 smithi016 conmon[32727]: 2022-03-21T17:49:56.966+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:57 smithi016 ceph-mon[22010]: pgmap v1134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:57 smithi079 ceph-mon[23158]: pgmap v1134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:57 smithi016 conmon[32727]: 2022-03-21T17:49:57.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:49:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:49:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:49:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:49:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:49:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:49:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:49:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:49:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:49:59.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:49:59 smithi016 conmon[32727]: 2022-03-21T17:49:59.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:49:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:59 smithi016 ceph-mon[22010]: pgmap v1135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:49:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:49:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:59 smithi079 ceph-mon[23158]: pgmap v1135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:49:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:49:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:00 smithi016 conmon[32727]: 2022-03-21T17:50:00.046+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops 2022-03-21T17:50:01.502 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:50:01.503 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:50:01.503 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:50:01.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:50:01.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:50:01.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops 2022-03-21T17:50:01.505 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops 2022-03-21T17:50:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T17:50:01.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T17:50:01.507 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T17:50:01.507 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:50:01.507 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T17:50:01.508 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1640 sec, osd.4 has slow ops 2022-03-21T17:50:01.508 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:01.509 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:01 smithi016 conmon[32727]: 2022-03-21T17:50:01.047+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:01 smithi016 ceph-mon[22010]: pgmap v1136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:01 smithi079 ceph-mon[23158]: pgmap v1136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:02.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:02 smithi016 conmon[32727]: 2022-03-21T17:50:02.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:03 smithi016 conmon[32727]: 2022-03-21T17:50:03.042+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:03 smithi016 ceph-mon[22010]: pgmap v1137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:03 smithi079 ceph-mon[23158]: pgmap v1137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:04 smithi016 conmon[32727]: 2022-03-21T17:50:04.016+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:05 smithi016 conmon[32727]: 2022-03-21T17:50:05.029+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:05 smithi016 ceph-mon[22010]: pgmap v1138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:05 smithi079 ceph-mon[23158]: pgmap v1138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:06 smithi016 conmon[32727]: 2022-03-21T17:50:06.058+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:07 smithi016 conmon[32727]: 2022-03-21T17:50:07.019+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:07 smithi016 ceph-mon[22010]: pgmap v1139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:07 smithi079 ceph-mon[23158]: pgmap v1139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:08 smithi016 conmon[32727]: 2022-03-21T17:50:08.023+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:09 smithi016 conmon[32727]: 2022-03-21T17:50:09.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:09 smithi016 ceph-mon[22010]: pgmap v1140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:09 smithi079 ceph-mon[23158]: pgmap v1140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:10 smithi016 conmon[32727]: 2022-03-21T17:50:10.064+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:11 smithi016 conmon[32727]: 2022-03-21T17:50:11.076+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:11 smithi016 ceph-mon[22010]: pgmap v1141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:11 smithi079 ceph-mon[23158]: pgmap v1141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:12 smithi016 conmon[32727]: 2022-03-21T17:50:12.061+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:13 smithi016 conmon[32727]: 2022-03-21T17:50:13.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:13.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:13.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:13 smithi016 ceph-mon[22010]: pgmap v1142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:13.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:13 smithi079 ceph-mon[23158]: pgmap v1142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:14 smithi016 conmon[32727]: 2022-03-21T17:50:14.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:15 smithi016 conmon[32727]: 2022-03-21T17:50:15.121+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:15 smithi016 ceph-mon[22010]: pgmap v1143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:15 smithi079 ceph-mon[23158]: pgmap v1143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:16 smithi016 conmon[32727]: 2022-03-21T17:50:16.118+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:17 smithi016 conmon[32727]: 2022-03-21T17:50:17.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:17 smithi016 ceph-mon[22010]: pgmap v1144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:17 smithi079 ceph-mon[23158]: pgmap v1144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:18 smithi016 conmon[32727]: 2022-03-21T17:50:18.156+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:19 smithi016 conmon[32727]: 2022-03-21T17:50:19.153+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:19 smithi016 ceph-mon[22010]: pgmap v1145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:19 smithi079 ceph-mon[23158]: pgmap v1145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:20 smithi016 conmon[32727]: 2022-03-21T17:50:20.114+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:21 smithi016 conmon[32727]: 2022-03-21T17:50:21.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:21 smithi016 ceph-mon[22010]: pgmap v1146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:21 smithi079 ceph-mon[23158]: pgmap v1146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:22 smithi016 conmon[32727]: 2022-03-21T17:50:22.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:23 smithi016 conmon[32727]: 2022-03-21T17:50:23.125+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:23 smithi016 ceph-mon[22010]: pgmap v1147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:23 smithi079 ceph-mon[23158]: pgmap v1147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:24 smithi016 conmon[32727]: 2022-03-21T17:50:24.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:25 smithi016 conmon[32727]: 2022-03-21T17:50:25.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:25 smithi016 ceph-mon[22010]: pgmap v1148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:25 smithi079 ceph-mon[23158]: pgmap v1148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:26 smithi016 conmon[32727]: 2022-03-21T17:50:26.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:27 smithi016 conmon[32727]: 2022-03-21T17:50:27.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:27 smithi016 ceph-mon[22010]: pgmap v1149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:27 smithi079 ceph-mon[23158]: pgmap v1149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:28 smithi016 conmon[32727]: 2022-03-21T17:50:28.123+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:29.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:29 smithi016 conmon[32727]: 2022-03-21T17:50:29.164+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:29 smithi016 ceph-mon[22010]: pgmap v1150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:29 smithi079 ceph-mon[23158]: pgmap v1150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:30.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:30 smithi016 conmon[32727]: 2022-03-21T17:50:30.186+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:31.630 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:31 smithi016 conmon[32727]: 2022-03-21T17:50:31.182+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:31 smithi016 ceph-mon[22010]: pgmap v1151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:31 smithi079 ceph-mon[23158]: pgmap v1151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:32.635 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:32 smithi016 conmon[32727]: 2022-03-21T17:50:32.183+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:33.636 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:33 smithi016 conmon[32727]: 2022-03-21T17:50:33.230+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:33 smithi016 ceph-mon[22010]: pgmap v1152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:33 smithi079 ceph-mon[23158]: pgmap v1152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:34.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:34 smithi016 conmon[32727]: 2022-03-21T17:50:34.190+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:35.642 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:35 smithi016 conmon[32727]: 2022-03-21T17:50:35.182+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:35 smithi016 ceph-mon[22010]: pgmap v1153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:35 smithi079 ceph-mon[23158]: pgmap v1153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[32727]: 2022-03-21T17:50:36.185+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:50:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:50:36.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:50:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:50:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:50:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:50:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: 2022-03-21T17:50:36.620+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:50:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: yield 2022-03-21T17:50:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:50:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:50:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:50:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:50:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:50:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:37.648 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:37 smithi016 conmon[32727]: 2022-03-21T17:50:37.209+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:37 smithi016 ceph-mon[22010]: pgmap v1154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:50:37.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:37.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:50:37.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:37.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:37.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:37.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:50:37.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:37.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:50:37.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:37.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:50:37.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:37.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:37.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:37.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:50:37.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:37.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:37.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:37.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:37 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:50:37.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:37.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:37 smithi079 ceph-mon[23158]: pgmap v1154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:37.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:37.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:50:37.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:37.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:50:37.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:37.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:37.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:37.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:50:37.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:37.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:50:37.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:37.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:50:37.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:37.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:37.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:37.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:50:37.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:37.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:37.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:37.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:37 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:50:38.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:38 smithi016 conmon[32727]: 2022-03-21T17:50:38.208+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:39.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[32727]: 2022-03-21T17:50:39.246+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:39 smithi016 ceph-mon[22010]: pgmap v1155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:39.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 2022-03-21T17:50:39.692+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:39.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:50:39.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:39.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: yield 2022-03-21T17:50:39.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:39.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:39.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:39.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:50:39.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:39.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:50:39.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:39.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:50:39.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:39.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:50:39.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:39.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:50:39.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:39.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:39.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:39.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 2022-03-21T17:50:39.693+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:50:39.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:50:39.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:39.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: yield 2022-03-21T17:50:39.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:39.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:39.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:39.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:50:39.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:39.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:50:39.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:39.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:50:39.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:39.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:50:39.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:39.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:50:39.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:39.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:39.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:39.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 2022-03-21T17:50:39.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:50:39.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: 2022-03-21T17:50:39.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:50:39.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:50:39.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:50:39.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:50:39.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:50:39.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:50:39.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:50:39.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:50:39.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:50:39.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:50:39.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:50:39.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:50:39.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:50:39.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:50:39.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:50:39.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:50:39.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:50:39.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:50:39.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:50:39.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:50:39.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:50:39.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:50:39.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise 2022-03-21T17:50:39.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:50:39.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:50:39.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:50:39.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:50:39.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:50:39 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:39 smithi079 ceph-mon[23158]: pgmap v1155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:40.656 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:40 smithi016 conmon[32727]: 2022-03-21T17:50:40.264+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:40 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:50:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:50:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:40.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:50:40.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:40.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:50:40.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:40.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:40.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:40.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:50:40.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:40.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:40.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:40 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:50:40.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:40.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:50:40.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:40.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:40.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:40.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:50:40.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:40.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:50:40.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:40.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:50:40.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:40.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:40.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:40.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:50:40.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:40.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:40.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:50:40.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:50:40.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:50:40.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:50:40.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:50:40.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:50:40.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:50:40.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:50:40.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:50:40.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:50:40.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:50:40.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:50:40.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:50:40.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:50:40.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:50:40.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:50:40.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:50:40.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:50:40.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:50:40.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:50:40.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:50:40.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:50:40.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:50:40.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:50:40.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:50:40.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:50:40.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:50:40.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:50:40.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:50:40.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:40 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:50:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:40.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:40.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:50:40.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:40.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:50:40.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:40.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:50:40.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:40.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:40.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:40.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:50:40.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:40.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:40.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:40 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:50:40.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:50:40.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:50:40.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:50:40.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:50:40.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:50:40.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:50:40.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:50:40.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:50:40.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:50:40.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:50:40.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:50:40.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:50:40.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:50:40.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:50:40.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:50:40.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:50:40.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:40.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:50:40.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:50:40.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:50:40.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:50:40.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:50:40.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:50:40.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:50:40.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:50:40.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:50:40.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:50:40.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:50:40.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:50:40.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:50:40.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:50:40.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:50:40.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:50:40.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:50:40.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:50:40.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:50:41.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:50:41.008 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:50:41.008 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:50:41.009 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:50:41.009 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:50:41.010 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:50:41.010 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:50:41.011 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:50:41.011 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:50:41.011 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:50:41.012 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:50:41.012 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:50:41.658 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:41 smithi016 conmon[32727]: 2022-03-21T17:50:41.225+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:41 smithi016 ceph-mon[22010]: pgmap v1156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:41 smithi079 ceph-mon[23158]: pgmap v1156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:42.663 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:42 smithi016 conmon[32727]: 2022-03-21T17:50:42.208+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:43 smithi016 conmon[32727]: 2022-03-21T17:50:43.238+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:43 smithi016 ceph-mon[22010]: pgmap v1157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:43.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:43 smithi079 ceph-mon[23158]: pgmap v1157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:43.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:44 smithi016 conmon[32727]: 2022-03-21T17:50:44.270+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:45 smithi016 conmon[32727]: 2022-03-21T17:50:45.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:45 smithi079 ceph-mon[23158]: pgmap v1158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:45 smithi016 ceph-mon[22010]: pgmap v1158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:46 smithi016 conmon[32727]: 2022-03-21T17:50:46.323+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:47 smithi016 conmon[32727]: 2022-03-21T17:50:47.290+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:47 smithi079 ceph-mon[23158]: pgmap v1159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:47 smithi016 ceph-mon[22010]: pgmap v1159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:48 smithi016 conmon[32727]: 2022-03-21T17:50:48.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:49 smithi016 conmon[32727]: 2022-03-21T17:50:49.268+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:49 smithi079 ceph-mon[23158]: pgmap v1160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:49 smithi016 ceph-mon[22010]: pgmap v1160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:50 smithi016 conmon[32727]: 2022-03-21T17:50:50.237+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:51.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:51 smithi016 conmon[32727]: 2022-03-21T17:50:51.236+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:51 smithi079 ceph-mon[23158]: pgmap v1161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:51 smithi016 ceph-mon[22010]: pgmap v1161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:52 smithi016 conmon[32727]: 2022-03-21T17:50:52.200+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:53 smithi016 conmon[32727]: 2022-03-21T17:50:53.173+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:53 smithi079 ceph-mon[23158]: pgmap v1162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:53 smithi016 ceph-mon[22010]: pgmap v1162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:54 smithi016 conmon[32727]: 2022-03-21T17:50:54.168+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:55.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:55 smithi016 conmon[32727]: 2022-03-21T17:50:55.149+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:55 smithi079 ceph-mon[23158]: pgmap v1163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:55 smithi016 ceph-mon[22010]: pgmap v1163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:50:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:56 smithi016 conmon[32727]: 2022-03-21T17:50:56.164+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:57.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:57 smithi016 conmon[32727]: 2022-03-21T17:50:57.142+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:57 smithi079 ceph-mon[23158]: pgmap v1164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:57 smithi016 ceph-mon[22010]: pgmap v1164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:50:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:58 smithi016 conmon[32727]: 2022-03-21T17:50:58.167+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:50:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:50:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:50:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:50:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:50:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:50:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:50:59.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:50:59.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:50:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:50:59 smithi016 conmon[32727]: 2022-03-21T17:50:59.157+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:50:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:50:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:50:59 smithi079 ceph-mon[23158]: pgmap v1165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:50:59 smithi016 ceph-mon[22010]: pgmap v1165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:00 smithi016 conmon[32727]: 2022-03-21T17:51:00.198+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:01 smithi016 conmon[32727]: 2022-03-21T17:51:01.176+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:01 smithi079 ceph-mon[23158]: pgmap v1166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:01 smithi016 ceph-mon[22010]: pgmap v1166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:02 smithi016 conmon[32727]: 2022-03-21T17:51:02.182+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:03 smithi016 conmon[32727]: 2022-03-21T17:51:03.189+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:03 smithi079 ceph-mon[23158]: pgmap v1167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:03 smithi016 ceph-mon[22010]: pgmap v1167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:04 smithi016 conmon[32727]: 2022-03-21T17:51:04.169+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:05 smithi016 conmon[32727]: 2022-03-21T17:51:05.211+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:05 smithi016 ceph-mon[22010]: pgmap v1168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:05 smithi079 ceph-mon[23158]: pgmap v1168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:06 smithi016 conmon[32727]: 2022-03-21T17:51:06.224+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:07.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:07 smithi016 conmon[32727]: 2022-03-21T17:51:07.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:07 smithi016 ceph-mon[22010]: pgmap v1169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:07 smithi079 ceph-mon[23158]: pgmap v1169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:08 smithi016 conmon[32727]: 2022-03-21T17:51:08.210+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:09 smithi016 conmon[32727]: 2022-03-21T17:51:09.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:09 smithi016 ceph-mon[22010]: pgmap v1170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:09 smithi079 ceph-mon[23158]: pgmap v1170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:10 smithi016 conmon[32727]: 2022-03-21T17:51:10.246+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:11 smithi016 conmon[32727]: 2022-03-21T17:51:11.211+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:11 smithi016 ceph-mon[22010]: pgmap v1171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:11 smithi079 ceph-mon[23158]: pgmap v1171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:12 smithi016 conmon[32727]: 2022-03-21T17:51:12.189+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:13 smithi016 conmon[32727]: 2022-03-21T17:51:13.216+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:13 smithi016 ceph-mon[22010]: pgmap v1172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:13 smithi079 ceph-mon[23158]: pgmap v1172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:14 smithi016 conmon[32727]: 2022-03-21T17:51:14.243+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:15 smithi016 conmon[32727]: 2022-03-21T17:51:15.269+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:15 smithi016 ceph-mon[22010]: pgmap v1173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:15 smithi079 ceph-mon[23158]: pgmap v1173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:16 smithi016 conmon[32727]: 2022-03-21T17:51:16.234+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:17 smithi016 conmon[32727]: 2022-03-21T17:51:17.198+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:18.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:18.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:17 smithi016 ceph-mon[22010]: pgmap v1174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:17 smithi079 ceph-mon[23158]: pgmap v1174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:18 smithi016 conmon[32727]: 2022-03-21T17:51:18.160+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:19.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:19.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:19 smithi016 conmon[32727]: 2022-03-21T17:51:19.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:20.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:20.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:19 smithi016 ceph-mon[22010]: pgmap v1175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:19 smithi079 ceph-mon[23158]: pgmap v1175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:20 smithi016 conmon[32727]: 2022-03-21T17:51:20.153+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:21.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:21 smithi016 conmon[32727]: 2022-03-21T17:51:21.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:22.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:22.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:21 smithi016 ceph-mon[22010]: pgmap v1176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:21 smithi079 ceph-mon[23158]: pgmap v1176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:22 smithi016 conmon[32727]: 2022-03-21T17:51:22.069+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:23.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:23.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:23 smithi016 conmon[32727]: 2022-03-21T17:51:23.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:24.092 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:23 smithi016 ceph-mon[22010]: pgmap v1177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:24.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:24.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:23 smithi079 ceph-mon[23158]: pgmap v1177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:24.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:24 smithi016 conmon[32727]: 2022-03-21T17:51:24.090+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:25.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:25 smithi016 conmon[32727]: 2022-03-21T17:51:25.078+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:26.067 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:26.067 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:25 smithi016 ceph-mon[22010]: pgmap v1178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:25 smithi079 ceph-mon[23158]: pgmap v1178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:26 smithi016 conmon[32727]: 2022-03-21T17:51:26.065+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:27.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:27.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:27 smithi016 conmon[32727]: 2022-03-21T17:51:27.049+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:27 smithi016 ceph-mon[22010]: pgmap v1179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:28 smithi016 conmon[32727]: 2022-03-21T17:51:28.000+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:27 smithi079 ceph-mon[23158]: pgmap v1179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:29 smithi016 conmon[32727]: 2022-03-21T17:51:29.016+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:30 smithi016 conmon[32727]: 2022-03-21T17:51:30.023+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:29 smithi016 ceph-mon[22010]: pgmap v1180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:29 smithi079 ceph-mon[23158]: pgmap v1180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:30 smithi016 conmon[32727]: 2022-03-21T17:51:30.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:32 smithi016 conmon[32727]: 2022-03-21T17:51:32.007+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:31 smithi016 ceph-mon[22010]: pgmap v1181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:31 smithi079 ceph-mon[23158]: pgmap v1181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:33 smithi016 conmon[32727]: 2022-03-21T17:51:33.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:33 smithi016 conmon[32727]: 2022-03-21T17:51:33.973+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:33 smithi016 ceph-mon[22010]: pgmap v1182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:34.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:33 smithi079 ceph-mon[23158]: pgmap v1182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:35 smithi016 conmon[32727]: 2022-03-21T17:51:35.006+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:35 smithi016 ceph-mon[22010]: pgmap v1183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:36.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:35 smithi016 conmon[32727]: 2022-03-21T17:51:35.976+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:35 smithi079 ceph-mon[23158]: pgmap v1183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:36 smithi016 conmon[32727]: 2022-03-21T17:51:36.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:38 smithi016 conmon[32727]: 2022-03-21T17:51:38.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:37 smithi016 ceph-mon[22010]: pgmap v1184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:37 smithi079 ceph-mon[23158]: pgmap v1184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:39 smithi016 conmon[32727]: 2022-03-21T17:51:39.016+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:40.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:39 smithi016 ceph-mon[22010]: pgmap v1185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:40.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:40.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:51:40.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:51:40.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:51:40.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:39 smithi079 ceph-mon[23158]: pgmap v1185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:40.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:40.149 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:51:40.149 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:51:40.150 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:51:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:40 smithi016 conmon[32727]: 2022-03-21T17:51:40.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:41 smithi016 conmon[32727]: 2022-03-21T17:51:41.059+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[32727]: 2022-03-21T17:51:42.014+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:41 smithi016 ceph-mon[22010]: pgmap v1186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:41 smithi079 ceph-mon[23158]: pgmap v1186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:43 smithi016 conmon[32727]: 2022-03-21T17:51:43.035+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: 2022-03-21T17:51:42.796+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:51:43.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:43.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: yield 2022-03-21T17:51:43.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:43.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:43.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:43.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:51:43.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:43.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:51:43.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:43.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:51:43.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:43.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:51:43.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:43.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:51:43.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:43.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:43.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:44 smithi016 conmon[32727]: 2022-03-21T17:51:44.060+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:43 smithi016 ceph-mon[22010]: pgmap v1187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:51:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:51:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:51:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:51:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:51:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:51:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:43 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:51:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:43 smithi079 ceph-mon[23158]: pgmap v1187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:51:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:51:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:51:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:51:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:51:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:51:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:43 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:51:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[32727]: 2022-03-21T17:51:45.075+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:45 smithi016 ceph-mon[22010]: pgmap v1188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:46 smithi016 conmon[32727]: 2022-03-21T17:51:46.047+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 2022-03-21T17:51:45.868+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:51:46.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: yield 2022-03-21T17:51:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:46.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:51:46.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:46.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:51:46.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:46.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:51:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:51:46.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:51:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:46.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 2022-03-21T17:51:45.869+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:51:46.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:51:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: yield 2022-03-21T17:51:46.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:46.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:46.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:46.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:51:46.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:46.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:51:46.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:51:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:46.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:51:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:51:46.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:46.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:46.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:46.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 2022-03-21T17:51:46.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:51:46.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: 2022-03-21T17:51:46.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:51:46.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:51:46.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:51:46.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:51:46.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:51:46.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:51:46.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:51:46.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:51:46.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:51:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:51:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:51:46.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:51:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:51:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:51:46.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:51:46.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:51:46.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:51:46.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:51:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:51:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:51:46.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:51:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:51:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise 2022-03-21T17:51:46.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:51:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:51:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:51:46.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:51:46.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:51:45 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:45 smithi079 ceph-mon[23158]: pgmap v1188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:47 smithi016 conmon[32727]: 2022-03-21T17:51:47.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:51:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:51:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:51:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:51:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:51:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:47.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:46 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:51:47.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:47.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:51:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:51:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:51:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:51:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:47.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:47.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:47.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:51:47.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:47.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:47.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:51:47.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:51:47.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:51:47.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:51:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:51:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:51:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:51:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:51:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:51:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:51:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:51:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:51:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:51:47.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:51:47.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:51:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:51:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:51:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:51:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:51:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:51:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:51:47.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:51:47.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:51:47.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:51:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:51:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:51:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:51:47.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:51:47.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:51:47.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:51:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:51:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:51:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:51:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:51:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:46 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:51:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:51:47.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:51:47.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:51:47.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:51:47.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:51:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:51:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:51:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:51:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:51:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:51:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:51:47.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:51:47.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:51:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:51:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:51:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:51:47.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:51:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:51:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:51:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:51:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:51:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:51:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:51:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:51:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:51:47.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:51:47.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:51:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:51:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:51:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:51:47.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:51:47.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:51:47.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:51:47.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:51:47.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:51:47.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:51:47.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:51:47.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:51:47.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:51:47.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:51:47.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:51:47.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:51:47.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:51:47.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:51:47.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:51:47.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:51:47.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:51:47.254 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:47 smithi016 ceph-mon[22010]: pgmap v1189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:48 smithi016 conmon[32727]: 2022-03-21T17:51:48.083+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:47 smithi079 ceph-mon[23158]: pgmap v1189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:49 smithi016 conmon[32727]: 2022-03-21T17:51:49.042+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:49 smithi016 ceph-mon[22010]: pgmap v1190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:50 smithi016 conmon[32727]: 2022-03-21T17:51:50.022+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:49 smithi079 ceph-mon[23158]: pgmap v1190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:51 smithi016 conmon[32727]: 2022-03-21T17:51:51.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:51 smithi016 ceph-mon[22010]: pgmap v1191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:52.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:52 smithi016 conmon[32727]: 2022-03-21T17:51:52.079+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:51 smithi079 ceph-mon[23158]: pgmap v1191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:53 smithi016 conmon[32727]: 2022-03-21T17:51:53.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:53 smithi016 ceph-mon[22010]: pgmap v1192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:54 smithi016 conmon[32727]: 2022-03-21T17:51:54.105+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:53 smithi079 ceph-mon[23158]: pgmap v1192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:55 smithi016 conmon[32727]: 2022-03-21T17:51:55.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:56.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:56.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:55 smithi016 ceph-mon[22010]: pgmap v1193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:55 smithi079 ceph-mon[23158]: pgmap v1193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:51:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:56 smithi016 conmon[32727]: 2022-03-21T17:51:56.123+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:57 smithi016 conmon[32727]: 2022-03-21T17:51:57.122+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:58.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:58.151 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:57 smithi016 ceph-mon[22010]: pgmap v1194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:57 smithi079 ceph-mon[23158]: pgmap v1194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:51:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:58 smithi016 conmon[32727]: 2022-03-21T17:51:58.148+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:51:59 smithi016 conmon[32727]: 2022-03-21T17:51:59.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:51:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:51:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:51:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:51:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:51:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:51:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:51:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:51:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:00.151 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:00.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:51:59 smithi016 ceph-mon[22010]: pgmap v1195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:51:59 smithi079 ceph-mon[23158]: pgmap v1195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:00 smithi016 conmon[32727]: 2022-03-21T17:52:00.149+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:01 smithi016 conmon[32727]: 2022-03-21T17:52:01.195+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:01 smithi016 ceph-mon[22010]: pgmap v1196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:01 smithi079 ceph-mon[23158]: pgmap v1196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:02 smithi016 conmon[32727]: 2022-03-21T17:52:02.215+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:03 smithi016 conmon[32727]: 2022-03-21T17:52:03.260+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:03 smithi016 ceph-mon[22010]: pgmap v1197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:03 smithi079 ceph-mon[23158]: pgmap v1197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:04 smithi016 conmon[32727]: 2022-03-21T17:52:04.226+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:05 smithi016 conmon[32727]: 2022-03-21T17:52:05.207+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:05 smithi016 ceph-mon[22010]: pgmap v1198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:05 smithi079 ceph-mon[23158]: pgmap v1198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:06 smithi016 conmon[32727]: 2022-03-21T17:52:06.240+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:07 smithi016 conmon[32727]: 2022-03-21T17:52:07.192+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:08 smithi016 conmon[32727]: 2022-03-21T17:52:08.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:07 smithi016 ceph-mon[22010]: pgmap v1199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:07 smithi079 ceph-mon[23158]: pgmap v1199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:09.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:09.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:09 smithi016 conmon[32727]: 2022-03-21T17:52:09.157+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:09 smithi016 ceph-mon[22010]: pgmap v1200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:09 smithi079 ceph-mon[23158]: pgmap v1200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:10 smithi016 conmon[32727]: 2022-03-21T17:52:10.172+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:11.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:11 smithi016 conmon[32727]: 2022-03-21T17:52:11.202+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:11 smithi079 ceph-mon[23158]: pgmap v1201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:12.253 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:12.253 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:11 smithi016 ceph-mon[22010]: pgmap v1201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:12 smithi016 conmon[32727]: 2022-03-21T17:52:12.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:13.289 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:13 smithi016 conmon[32727]: 2022-03-21T17:52:13.287+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:13 smithi079 ceph-mon[23158]: pgmap v1202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:14.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:14.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:13 smithi016 ceph-mon[22010]: pgmap v1202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:14.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:14 smithi016 conmon[32727]: 2022-03-21T17:52:14.239+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:15.284 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:15 smithi016 conmon[32727]: 2022-03-21T17:52:15.283+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:15 smithi079 ceph-mon[23158]: pgmap v1203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:16.263 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:16.263 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:15 smithi016 ceph-mon[22010]: pgmap v1203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:16 smithi016 conmon[32727]: 2022-03-21T17:52:16.260+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:17.269 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:17 smithi016 conmon[32727]: 2022-03-21T17:52:17.267+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:18.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:18.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:17 smithi079 ceph-mon[23158]: pgmap v1204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:18.237 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:18.237 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:17 smithi016 ceph-mon[22010]: pgmap v1204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:18 smithi016 conmon[32727]: 2022-03-21T17:52:18.235+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:19.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:19 smithi016 conmon[32727]: 2022-03-21T17:52:19.186+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:19 smithi079 ceph-mon[23158]: pgmap v1205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:20.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:20.244 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:19 smithi016 ceph-mon[22010]: pgmap v1205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:20 smithi016 conmon[32727]: 2022-03-21T17:52:20.233+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:21.259 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:21 smithi016 conmon[32727]: 2022-03-21T17:52:21.257+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:21 smithi079 ceph-mon[23158]: pgmap v1206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:22.279 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:22.280 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:21 smithi016 ceph-mon[22010]: pgmap v1206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:22 smithi016 conmon[32727]: 2022-03-21T17:52:22.277+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:23.242 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:23 smithi016 conmon[32727]: 2022-03-21T17:52:23.240+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:24.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:24.218 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:23 smithi016 ceph-mon[22010]: pgmap v1207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:24.219 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:23 smithi079 ceph-mon[23158]: pgmap v1207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:24 smithi016 conmon[32727]: 2022-03-21T17:52:24.216+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:25 smithi016 conmon[32727]: 2022-03-21T17:52:25.199+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:25 smithi079 ceph-mon[23158]: pgmap v1208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:26 smithi016 conmon[32727]: 2022-03-21T17:52:26.159+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:25 smithi016 ceph-mon[22010]: pgmap v1208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:27 smithi016 conmon[32727]: 2022-03-21T17:52:27.146+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:27 smithi079 ceph-mon[23158]: pgmap v1209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:28 smithi016 conmon[32727]: 2022-03-21T17:52:28.158+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:27 smithi016 ceph-mon[22010]: pgmap v1209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:29 smithi016 conmon[32727]: 2022-03-21T17:52:29.115+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:29 smithi079 ceph-mon[23158]: pgmap v1210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:29 smithi016 ceph-mon[22010]: pgmap v1210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:30 smithi016 conmon[32727]: 2022-03-21T17:52:30.065+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:31 smithi016 conmon[32727]: 2022-03-21T17:52:31.044+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:31 smithi016 ceph-mon[22010]: pgmap v1211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:32 smithi016 conmon[32727]: 2022-03-21T17:52:32.075+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:31 smithi079 ceph-mon[23158]: pgmap v1211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:32.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:33.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:33.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:33 smithi016 conmon[32727]: 2022-03-21T17:52:33.118+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:33 smithi016 ceph-mon[22010]: pgmap v1212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:34 smithi016 conmon[32727]: 2022-03-21T17:52:34.076+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:33 smithi079 ceph-mon[23158]: pgmap v1212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:35 smithi016 conmon[32727]: 2022-03-21T17:52:35.083+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:35 smithi016 ceph-mon[22010]: pgmap v1213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:36 smithi016 conmon[32727]: 2022-03-21T17:52:36.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:36.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:35 smithi079 ceph-mon[23158]: pgmap v1213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:37 smithi016 conmon[32727]: 2022-03-21T17:52:37.001+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:37 smithi016 ceph-mon[22010]: pgmap v1214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:37 smithi016 conmon[32727]: 2022-03-21T17:52:37.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:37 smithi079 ceph-mon[23158]: pgmap v1214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:38 smithi016 conmon[32727]: 2022-03-21T17:52:38.994+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:39 smithi016 ceph-mon[22010]: pgmap v1215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:39 smithi016 conmon[32727]: 2022-03-21T17:52:39.988+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:39 smithi079 ceph-mon[23158]: pgmap v1215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:41 smithi016 conmon[32727]: 2022-03-21T17:52:41.038+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:41.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:42 smithi016 ceph-mon[22010]: pgmap v1216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:42.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:42 smithi016 conmon[32727]: 2022-03-21T17:52:41.999+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:42.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:42 smithi079 ceph-mon[23158]: pgmap v1216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:42.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:43.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:43 smithi016 conmon[32727]: 2022-03-21T17:52:43.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:44 smithi016 ceph-mon[22010]: pgmap v1217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:44.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:43 smithi016 conmon[32727]: 2022-03-21T17:52:43.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:44 smithi079 ceph-mon[23158]: pgmap v1217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:44 smithi016 conmon[32727]: 2022-03-21T17:52:44.943+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:46 smithi079 ceph-mon[23158]: pgmap v1218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:52:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:52:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:52:46.344 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:45 smithi016 conmon[32727]: 2022-03-21T17:52:45.958+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:46.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:46 smithi016 ceph-mon[22010]: pgmap v1218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:46.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:46.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:52:46.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:52:46.346 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:52:46.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: 2022-03-21T17:52:46.348+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:46.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:52:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: yield 2022-03-21T17:52:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:52:46.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:46.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:52:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:52:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:46.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:52:46.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:46.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:52:46.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:46.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:46.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:46 smithi016 conmon[32727]: 2022-03-21T17:52:46.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:47.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:47.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:47.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:52:47.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:47.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:52:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:47.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:52:47.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:47.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:52:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:52:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:52:47.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:47.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:47.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:47.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:47 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:52:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:52:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:52:47.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:47.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:52:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:47.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:52:47.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:52:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:52:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:47 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:52:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:47 smithi016 conmon[32727]: 2022-03-21T17:52:47.980+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:48 smithi016 ceph-mon[22010]: pgmap v1219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:48 smithi079 ceph-mon[23158]: pgmap v1219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:48 smithi016 conmon[32727]: 2022-03-21T17:52:48.932+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:49.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 2022-03-21T17:52:49.421+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:49.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:52:49.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:49.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: yield 2022-03-21T17:52:49.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:49.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:49.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:49.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:52:49.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:49.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:52:49.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:49.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:52:49.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:49.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:52:49.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:49.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:52:49.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:49.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:49.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:49.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 2022-03-21T17:52:49.422+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:52:49.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:52:49.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:49.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: yield 2022-03-21T17:52:49.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:49.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:49.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:49.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:52:49.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:49.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:52:49.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:49.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:52:49.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:49.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:52:49.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:49.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:52:49.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:49.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:49.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:49.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 2022-03-21T17:52:49.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:52:49.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: 2022-03-21T17:52:49.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:52:49.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:52:49.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:52:49.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:52:49.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:52:49.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:52:49.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:52:49.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:52:49.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:52:49.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:52:49.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:52:49.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:52:49.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:52:49.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:52:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:52:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:52:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:52:49.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:52:49.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:52:49.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:52:49.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:52:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:52:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise 2022-03-21T17:52:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:52:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:52:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:52:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:52:49.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:50 smithi016 ceph-mon[22010]: pgmap v1220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:49 smithi016 conmon[32727]: 2022-03-21T17:52:49.944+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:50 smithi079 ceph-mon[23158]: pgmap v1220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:50 smithi016 conmon[32727]: 2022-03-21T17:52:50.952+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:51.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:52:51.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:51.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:51.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:51.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:52:51.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:51.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:52:51.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:51.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:52:51.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:51.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:51.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:51.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:52:51.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:51.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:51.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:51 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:52:51.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:51.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:52:51.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:51.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:51.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:51.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:52:51.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:51.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:52:51.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:51.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:52:51.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:51.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:51.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:51.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:52:51.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:51.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:51.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:52:51.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:52:51.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:52:51.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:52:51.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:52:51.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:52:51.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:52:51.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:52:51.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:52:51.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:52:51.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:52:51.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:52:51.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:52:51.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:52:51.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:52:51.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:52:51.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:52:51.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:52:51.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:52:51.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:52:51.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:52:51.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:52:51.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:52:51.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:52:51.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:52:51.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:52:51.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:52:51.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:52:51.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:52:51.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:51.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:52:51.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:51.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:51.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:51.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:52:51.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:51.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:52:51.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:51.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:52:51.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:51.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:51.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:51.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:52:51.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:51.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:51.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:51 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:52:51.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:52:51.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:52:51.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:52:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:52:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:52:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:52:51.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:52:51.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:52:51.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:52:51.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:52:51.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:52:51.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:52:51.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:52:51.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:52:51.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:52:51.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:52:51.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:52:51.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:52:51.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:52:51.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:52:51.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:52:51.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:52:51.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:52:51.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:52:51.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:52:51.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:52:51.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:52:51.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:52:51.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:52:51.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:52:51.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:52:51.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:52:51.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:52:51.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:52:51.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:52:51.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:52:51.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:52:51.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:52:51.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:52:51.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:52:51.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:52:51.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:52:51.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:52:51.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:52:51.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:52:51.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:52:51.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:52:51.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:52 smithi016 ceph-mon[22010]: pgmap v1221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:51 smithi016 conmon[32727]: 2022-03-21T17:52:51.976+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:52 smithi079 ceph-mon[23158]: pgmap v1221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:52 smithi016 conmon[32727]: 2022-03-21T17:52:52.966+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:54 smithi016 ceph-mon[22010]: pgmap v1222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:54 smithi016 conmon[32727]: 2022-03-21T17:52:54.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:54 smithi079 ceph-mon[23158]: pgmap v1222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:54 smithi016 conmon[32727]: 2022-03-21T17:52:54.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:56 smithi016 ceph-mon[22010]: pgmap v1223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:56 smithi016 conmon[32727]: 2022-03-21T17:52:56.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:56 smithi079 ceph-mon[23158]: pgmap v1223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:52:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:56 smithi016 conmon[32727]: 2022-03-21T17:52:56.966+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:58 smithi016 conmon[32727]: 2022-03-21T17:52:58.013+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: pgmap v1224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:52:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:52:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:52:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:52:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: pgmap v1224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:52:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:52:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:52:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:52:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:52:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:52:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:52:59.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:59 smithi016 conmon[32727]: 2022-03-21T17:52:59.010+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:52:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:52:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:52:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:00 smithi016 ceph-mon[22010]: pgmap v1225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:00.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:52:59 smithi016 conmon[32727]: 2022-03-21T17:52:59.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:00 smithi079 ceph-mon[23158]: pgmap v1225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:01.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:01 smithi016 conmon[32727]: 2022-03-21T17:53:01.023+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:02 smithi016 ceph-mon[22010]: pgmap v1226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:02 smithi016 conmon[32727]: 2022-03-21T17:53:02.056+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:02 smithi079 ceph-mon[23158]: pgmap v1226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:03 smithi016 conmon[32727]: 2022-03-21T17:53:03.021+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:04 smithi016 ceph-mon[22010]: pgmap v1227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:04.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:04 smithi016 conmon[32727]: 2022-03-21T17:53:04.064+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:04 smithi079 ceph-mon[23158]: pgmap v1227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:05 smithi016 conmon[32727]: 2022-03-21T17:53:05.067+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:06 smithi016 ceph-mon[22010]: pgmap v1228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:06 smithi016 conmon[32727]: 2022-03-21T17:53:06.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:06 smithi079 ceph-mon[23158]: pgmap v1228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:07.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:07 smithi016 conmon[32727]: 2022-03-21T17:53:07.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:08 smithi016 ceph-mon[22010]: pgmap v1229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:08 smithi016 conmon[32727]: 2022-03-21T17:53:08.009+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:08 smithi079 ceph-mon[23158]: pgmap v1229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:09.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:08 smithi016 conmon[32727]: 2022-03-21T17:53:08.981+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:10 smithi016 ceph-mon[22010]: pgmap v1230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:09 smithi016 conmon[32727]: 2022-03-21T17:53:09.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:10 smithi079 ceph-mon[23158]: pgmap v1230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:10 smithi016 conmon[32727]: 2022-03-21T17:53:10.987+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:12 smithi016 ceph-mon[22010]: pgmap v1231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:12.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:11 smithi016 conmon[32727]: 2022-03-21T17:53:11.974+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:12 smithi079 ceph-mon[23158]: pgmap v1231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:13.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:12 smithi016 conmon[32727]: 2022-03-21T17:53:12.936+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:14 smithi016 ceph-mon[22010]: pgmap v1232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:14.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:13 smithi016 conmon[32727]: 2022-03-21T17:53:13.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:14 smithi079 ceph-mon[23158]: pgmap v1232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:15 smithi016 conmon[32727]: 2022-03-21T17:53:15.016+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:16 smithi016 ceph-mon[22010]: pgmap v1233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:16.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:15 smithi016 conmon[32727]: 2022-03-21T17:53:15.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:16 smithi079 ceph-mon[23158]: pgmap v1233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:17 smithi016 conmon[32727]: 2022-03-21T17:53:17.004+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:18 smithi016 ceph-mon[22010]: pgmap v1234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:18 smithi016 conmon[32727]: 2022-03-21T17:53:18.044+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:18 smithi079 ceph-mon[23158]: pgmap v1234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:19 smithi016 conmon[32727]: 2022-03-21T17:53:19.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:19.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:20 smithi016 ceph-mon[22010]: pgmap v1235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:20 smithi016 conmon[32727]: 2022-03-21T17:53:20.049+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:20 smithi079 ceph-mon[23158]: pgmap v1235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:21 smithi016 conmon[32727]: 2022-03-21T17:53:21.096+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:22 smithi016 ceph-mon[22010]: pgmap v1236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:22 smithi016 conmon[32727]: 2022-03-21T17:53:22.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:22 smithi079 ceph-mon[23158]: pgmap v1236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:23.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:23 smithi016 conmon[32727]: 2022-03-21T17:53:23.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:24 smithi016 conmon[32727]: 2022-03-21T17:53:24.066+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:24 smithi016 ceph-mon[22010]: pgmap v1237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:24 smithi079 ceph-mon[23158]: pgmap v1237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:25 smithi016 conmon[32727]: 2022-03-21T17:53:25.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:26 smithi016 ceph-mon[22010]: pgmap v1238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:26 smithi016 conmon[32727]: 2022-03-21T17:53:26.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:26 smithi079 ceph-mon[23158]: pgmap v1238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:27 smithi016 conmon[32727]: 2022-03-21T17:53:27.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:28 smithi016 conmon[32727]: 2022-03-21T17:53:28.092+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:28 smithi016 ceph-mon[22010]: pgmap v1239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:28 smithi079 ceph-mon[23158]: pgmap v1239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:29 smithi016 conmon[32727]: 2022-03-21T17:53:29.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:30.307 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:30 smithi016 conmon[32727]: 2022-03-21T17:53:30.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:30 smithi016 ceph-mon[22010]: pgmap v1240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:30 smithi079 ceph-mon[23158]: pgmap v1240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:31.312 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:31 smithi016 conmon[32727]: 2022-03-21T17:53:31.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:32.314 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:31 smithi016 conmon[32727]: 2022-03-21T17:53:31.998+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:32 smithi016 ceph-mon[22010]: pgmap v1241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:32 smithi079 ceph-mon[23158]: pgmap v1241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:33.319 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:33 smithi016 conmon[32727]: 2022-03-21T17:53:33.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:33.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:34.320 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:33 smithi016 conmon[32727]: 2022-03-21T17:53:33.985+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:34 smithi016 ceph-mon[22010]: pgmap v1242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:34 smithi079 ceph-mon[23158]: pgmap v1242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:35.326 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:35 smithi016 conmon[32727]: 2022-03-21T17:53:35.012+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:36.327 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:35 smithi016 conmon[32727]: 2022-03-21T17:53:35.989+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:36 smithi016 ceph-mon[22010]: pgmap v1243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:36.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:36 smithi079 ceph-mon[23158]: pgmap v1243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:37.332 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:36 smithi016 conmon[32727]: 2022-03-21T17:53:36.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:38.337 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:37 smithi016 conmon[32727]: 2022-03-21T17:53:37.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:38 smithi016 ceph-mon[22010]: pgmap v1244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:38.799 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:38.800 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:38.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:38 smithi079 ceph-mon[23158]: pgmap v1244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:38.801 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:38.802 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:39.342 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:38 smithi016 conmon[32727]: 2022-03-21T17:53:38.978+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:40.343 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:39 smithi016 conmon[32727]: 2022-03-21T17:53:39.963+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:40 smithi016 ceph-mon[22010]: pgmap v1245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:40 smithi079 ceph-mon[23158]: pgmap v1245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:41.348 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:40 smithi016 conmon[32727]: 2022-03-21T17:53:40.920+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:42.350 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:41 smithi016 conmon[32727]: 2022-03-21T17:53:41.936+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:42 smithi016 ceph-mon[22010]: pgmap v1246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:42 smithi079 ceph-mon[23158]: pgmap v1246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:43.354 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:42 smithi016 conmon[32727]: 2022-03-21T17:53:42.940+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:43 smithi016 conmon[32727]: 2022-03-21T17:53:43.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:44 smithi016 ceph-mon[22010]: pgmap v1247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:44.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:44.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:44 smithi079 ceph-mon[23158]: pgmap v1247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:44.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:45.360 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:44 smithi016 conmon[32727]: 2022-03-21T17:53:44.921+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:45 smithi016 conmon[32727]: 2022-03-21T17:53:45.898+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:46 smithi016 ceph-mon[22010]: pgmap v1248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:46 smithi079 ceph-mon[23158]: pgmap v1248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:47.367 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:46 smithi016 conmon[32727]: 2022-03-21T17:53:46.925+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:48.368 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:47 smithi016 conmon[32727]: 2022-03-21T17:53:47.934+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:48 smithi016 ceph-mon[22010]: pgmap v1249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:48 smithi079 ceph-mon[23158]: pgmap v1249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:49.373 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:48 smithi016 conmon[32727]: 2022-03-21T17:53:48.953+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:49 smithi016 conmon[32727]: 2022-03-21T17:53:49.953+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:50 smithi016 ceph-mon[22010]: pgmap v1250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:53:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:53:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:53:50.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:50 smithi079 ceph-mon[23158]: pgmap v1250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:53:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:53:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:53:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:51.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:50 smithi016 conmon[32727]: 2022-03-21T17:53:50.935+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:51 smithi016 conmon[32727]: 2022-03-21T17:53:51.902+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:52 smithi016 ceph-mon[22010]: pgmap v1251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:52.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: 2022-03-21T17:53:52.524+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:52.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:53:52.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:52.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: yield 2022-03-21T17:53:52.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:52.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:52.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:52.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:53:52.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:52.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:53:52.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:52.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:53:52.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:52.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:53:52.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:52.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:53:52.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:52.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:52.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:52 smithi079 ceph-mon[23158]: pgmap v1251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:52 smithi016 conmon[32727]: 2022-03-21T17:53:52.897+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:53 smithi016 conmon[32727]: 2022-03-21T17:53:53.859+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:53:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:53:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:53:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:53:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:53:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:53:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:53:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:54 smithi016 ceph-mon[22010]: pgmap v1252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:53:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:53:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:53:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:53:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:53:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:53:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:53:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:54 smithi079 ceph-mon[23158]: pgmap v1252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:54 smithi016 conmon[32727]: 2022-03-21T17:53:54.902+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:55 smithi016 ceph-mon[22010]: pgmap v1253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:55.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 2022-03-21T17:53:55.596+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:55.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:53:55.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:55.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: yield 2022-03-21T17:53:55.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:55.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:55.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:55.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:53:55.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:55.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:53:55.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:55.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:53:55.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:55.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:53:55.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:55.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:53:55.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:55.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:55.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:55.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 2022-03-21T17:53:55.597+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:53:55.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:53:55.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:55.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: yield 2022-03-21T17:53:55.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:55.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:55.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:55.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:53:55.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:55.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:53:55.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:55.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:53:55.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:55.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:53:55.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:55.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:53:55.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:55.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:55.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:55.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 2022-03-21T17:53:55.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:53:55.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: 2022-03-21T17:53:55.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:53:55.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:53:55.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:53:55.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:53:55.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:53:55.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:53:55.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:53:55.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:53:55.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:53:55.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:53:55.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:53:55.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:53:55.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:53:55.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:53:55.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:53:55.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:53:55.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:53:55.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:53:55.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:53:55.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:53:55.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:53:55.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:53:55.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise 2022-03-21T17:53:55.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:53:55.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:53:55.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:53:55.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:53:55.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:55 smithi079 ceph-mon[23158]: pgmap v1253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:53:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:56.395 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:55 smithi016 conmon[32727]: 2022-03-21T17:53:55.937+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:56.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:53:56.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:56.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:56.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:56.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:53:56.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:56.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:53:56.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:56.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:53:56.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:56.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:56.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:53:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:56 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:53:56.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:56.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:53:56.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:56.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:56.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:56.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:53:56.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:56.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:53:56.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:56.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:53:56.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:56.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:56.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:56.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:53:56.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:56.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:56.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:53:56.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:53:56.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:53:56.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:53:56.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:53:56.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:53:56.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:53:56.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:53:56.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:53:56.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:53:56.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:53:56.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:53:56.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:53:56.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:53:56.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:53:56.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:53:56.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:53:56.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:53:56.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:53:56.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:53:56.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:53:56.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:53:56.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:53:56.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:53:56.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:53:56.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:53:56.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:53:56.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:53:56.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:53:56.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.693 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:53:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:53:56.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:53:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:53:56.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:56.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:56.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:56.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:53:56.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:56.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:56.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:56 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:53:56.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:53:56.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:53:56.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:53:56.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:53:56.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:53:56.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:53:56.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:53:56.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:53:56.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:53:56.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:53:56.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:53:56.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:53:56.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:53:56.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:53:56.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:53:56.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:53:56.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:53:56.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:53:56.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:53:56.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:53:56.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:53:56.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:53:56.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:53:56.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:53:56.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:53:56.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:53:56.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:53:56.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:53:56.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:53:56.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:53:56.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:53:56.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:53:56.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:53:56.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:53:56.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:53:56.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:53:56.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:53:56.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:53:56.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:53:56.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:53:56.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:53:56.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:53:56.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:53:56.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:53:56.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:53:56.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:53:56.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:53:56.752 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:57.398 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:56 smithi016 conmon[32727]: 2022-03-21T17:53:56.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:57 smithi016 ceph-mon[22010]: pgmap v1254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:57 smithi079 ceph-mon[23158]: pgmap v1254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:58.400 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:57 smithi016 conmon[32727]: 2022-03-21T17:53:57.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:53:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:53:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:53:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:53:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:53:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:53:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:53:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:53:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:53:59.405 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:58 smithi016 conmon[32727]: 2022-03-21T17:53:58.959+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:53:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:59 smithi016 ceph-mon[22010]: pgmap v1255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:53:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:53:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:59 smithi079 ceph-mon[23158]: pgmap v1255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:53:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:53:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:00.406 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:53:59 smithi016 conmon[32727]: 2022-03-21T17:53:59.921+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:01.411 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:00 smithi016 conmon[32727]: 2022-03-21T17:54:00.935+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:01 smithi016 ceph-mon[22010]: pgmap v1256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:01 smithi079 ceph-mon[23158]: pgmap v1256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:02.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:01 smithi016 conmon[32727]: 2022-03-21T17:54:01.913+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:02 smithi016 conmon[32727]: 2022-03-21T17:54:02.909+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:03 smithi079 ceph-mon[23158]: pgmap v1257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:03 smithi016 ceph-mon[22010]: pgmap v1257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:03 smithi016 conmon[32727]: 2022-03-21T17:54:03.927+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:04 smithi016 conmon[32727]: 2022-03-21T17:54:04.958+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:05 smithi079 ceph-mon[23158]: pgmap v1258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:05 smithi016 ceph-mon[22010]: pgmap v1258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:06 smithi016 conmon[32727]: 2022-03-21T17:54:06.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:07 smithi016 conmon[32727]: 2022-03-21T17:54:07.024+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:07 smithi079 ceph-mon[23158]: pgmap v1259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:07 smithi016 ceph-mon[22010]: pgmap v1259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:08 smithi016 conmon[32727]: 2022-03-21T17:54:08.029+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:09 smithi016 conmon[32727]: 2022-03-21T17:54:09.033+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:09 smithi079 ceph-mon[23158]: pgmap v1260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:09 smithi016 ceph-mon[22010]: pgmap v1260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:10 smithi016 conmon[32727]: 2022-03-21T17:54:10.061+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:11 smithi016 conmon[32727]: 2022-03-21T17:54:11.093+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:11 smithi079 ceph-mon[23158]: pgmap v1261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:11 smithi016 ceph-mon[22010]: pgmap v1261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:12 smithi016 conmon[32727]: 2022-03-21T17:54:12.129+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:13.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:13 smithi016 conmon[32727]: 2022-03-21T17:54:13.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:13 smithi016 ceph-mon[22010]: pgmap v1262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:13 smithi079 ceph-mon[23158]: pgmap v1262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:13.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:14 smithi016 conmon[32727]: 2022-03-21T17:54:14.101+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:15.427 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:15 smithi016 conmon[32727]: 2022-03-21T17:54:15.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:15 smithi016 ceph-mon[22010]: pgmap v1263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:15 smithi079 ceph-mon[23158]: pgmap v1263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:16 smithi016 conmon[32727]: 2022-03-21T17:54:16.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:17 smithi016 conmon[32727]: 2022-03-21T17:54:17.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:17 smithi016 ceph-mon[22010]: pgmap v1264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:17.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:17 smithi079 ceph-mon[23158]: pgmap v1264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:17.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:18 smithi016 conmon[32727]: 2022-03-21T17:54:18.102+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:19 smithi016 conmon[32727]: 2022-03-21T17:54:19.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:19 smithi016 ceph-mon[22010]: pgmap v1265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:19 smithi079 ceph-mon[23158]: pgmap v1265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:20.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:20 smithi016 conmon[32727]: 2022-03-21T17:54:20.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:21 smithi016 conmon[32727]: 2022-03-21T17:54:21.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:21 smithi016 ceph-mon[22010]: pgmap v1266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:21 smithi079 ceph-mon[23158]: pgmap v1266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:22 smithi016 conmon[32727]: 2022-03-21T17:54:22.053+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:23 smithi016 conmon[32727]: 2022-03-21T17:54:23.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:23 smithi016 ceph-mon[22010]: pgmap v1267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:23 smithi079 ceph-mon[23158]: pgmap v1267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:24 smithi016 conmon[32727]: 2022-03-21T17:54:24.078+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:25 smithi016 conmon[32727]: 2022-03-21T17:54:25.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:25 smithi016 ceph-mon[22010]: pgmap v1268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:25 smithi079 ceph-mon[23158]: pgmap v1268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:26 smithi016 conmon[32727]: 2022-03-21T17:54:26.017+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:26.985 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:26 smithi016 conmon[32727]: 2022-03-21T17:54:26.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:27 smithi079 ceph-mon[23158]: pgmap v1269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:28.009 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:27 smithi016 ceph-mon[22010]: pgmap v1269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:28.010 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:28 smithi016 conmon[32727]: 2022-03-21T17:54:28.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:29.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:29.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:29 smithi016 conmon[32727]: 2022-03-21T17:54:29.055+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:29 smithi079 ceph-mon[23158]: pgmap v1270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:30.042 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:29 smithi016 ceph-mon[22010]: pgmap v1270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:30.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:30 smithi016 conmon[32727]: 2022-03-21T17:54:30.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:31.014 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:31 smithi016 conmon[32727]: 2022-03-21T17:54:31.012+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:31 smithi079 ceph-mon[23158]: pgmap v1271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:32.012 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:31 smithi016 ceph-mon[22010]: pgmap v1271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:32.013 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:32 smithi016 conmon[32727]: 2022-03-21T17:54:32.011+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:33.010 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:33 smithi016 conmon[32727]: 2022-03-21T17:54:33.008+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:33 smithi079 ceph-mon[23158]: pgmap v1272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:33.975 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:33 smithi016 ceph-mon[22010]: pgmap v1272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:33.976 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:33.976 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:33 smithi016 conmon[32727]: 2022-03-21T17:54:33.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:34.953 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:34 smithi016 conmon[32727]: 2022-03-21T17:54:34.951+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:35 smithi079 ceph-mon[23158]: pgmap v1273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:35.999 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:35 smithi016 ceph-mon[22010]: pgmap v1273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:36.000 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:35 smithi016 conmon[32727]: 2022-03-21T17:54:35.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:37.014 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:37 smithi016 conmon[32727]: 2022-03-21T17:54:37.012+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:37 smithi079 ceph-mon[23158]: pgmap v1274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:38.047 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:37 smithi016 ceph-mon[22010]: pgmap v1274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:38.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:38 smithi016 conmon[32727]: 2022-03-21T17:54:38.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:39.040 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:39.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:39 smithi016 conmon[32727]: 2022-03-21T17:54:39.038+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:39 smithi079 ceph-mon[23158]: pgmap v1275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:40.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:39 smithi016 ceph-mon[22010]: pgmap v1275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:40.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:40 smithi016 conmon[32727]: 2022-03-21T17:54:40.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:41.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:41 smithi016 conmon[32727]: 2022-03-21T17:54:41.121+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:41 smithi079 ceph-mon[23158]: pgmap v1276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:42.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:41 smithi016 ceph-mon[22010]: pgmap v1276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:42 smithi016 conmon[32727]: 2022-03-21T17:54:42.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:43.086 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:43.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:43 smithi016 conmon[32727]: 2022-03-21T17:54:43.084+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:43 smithi079 ceph-mon[23158]: pgmap v1277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:44.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:43 smithi016 ceph-mon[22010]: pgmap v1277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:44.089 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:44.089 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:44 smithi016 conmon[32727]: 2022-03-21T17:54:44.086+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:45.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:45 smithi016 conmon[32727]: 2022-03-21T17:54:45.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:45 smithi079 ceph-mon[23158]: pgmap v1278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:46.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:45 smithi016 ceph-mon[22010]: pgmap v1278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:46 smithi016 conmon[32727]: 2022-03-21T17:54:46.157+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:47.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:47 smithi016 conmon[32727]: 2022-03-21T17:54:47.136+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:48.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:48.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:47 smithi016 ceph-mon[22010]: pgmap v1279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:47 smithi079 ceph-mon[23158]: pgmap v1279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:48 smithi016 conmon[32727]: 2022-03-21T17:54:48.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:49.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:49.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:49 smithi016 conmon[32727]: 2022-03-21T17:54:49.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:50.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:50.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:49 smithi016 ceph-mon[22010]: pgmap v1280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:49 smithi079 ceph-mon[23158]: pgmap v1280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:50 smithi016 conmon[32727]: 2022-03-21T17:54:50.148+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:51.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:51 smithi016 conmon[32727]: 2022-03-21T17:54:51.149+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:51 smithi016 ceph-mon[22010]: pgmap v1281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:51 smithi079 ceph-mon[23158]: pgmap v1281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:52 smithi016 conmon[32727]: 2022-03-21T17:54:52.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:53 smithi016 conmon[32727]: 2022-03-21T17:54:53.223+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:53 smithi016 ceph-mon[22010]: pgmap v1282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:53 smithi079 ceph-mon[23158]: pgmap v1282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:54 smithi016 conmon[32727]: 2022-03-21T17:54:54.251+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:55.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:55 smithi016 conmon[32727]: 2022-03-21T17:54:55.254+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:55 smithi016 ceph-mon[22010]: pgmap v1283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:54:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:54:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:54:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:55 smithi079 ceph-mon[23158]: pgmap v1283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:54:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:54:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:54:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:54:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:56 smithi016 conmon[32727]: 2022-03-21T17:54:56.259+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:57 smithi016 conmon[32727]: 2022-03-21T17:54:57.216+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:57 smithi016 ceph-mon[22010]: pgmap v1284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:57 smithi079 ceph-mon[23158]: pgmap v1284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:54:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[32727]: 2022-03-21T17:54:58.239+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:54:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:54:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:54:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:54:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:54:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:59.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: 2022-03-21T17:54:58.700+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:54:59.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:54:59.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:54:59.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: yield 2022-03-21T17:54:59.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:54:59.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:54:59.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:54:59.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:54:59.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:54:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:54:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:54:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:54:59.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:54:59.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:54:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:54:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:54:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:54:59.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:54:59.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:54:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:54:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:54:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:54:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:54:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:54:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:54:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:54:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:54:59 smithi016 conmon[32727]: 2022-03-21T17:54:59.275+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:59 smithi016 ceph-mon[22010]: pgmap v1285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:59 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:55:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:55:00.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:00.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:00.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:00.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:55:00.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:55:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:00.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:55:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:00.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:55:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:00.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:00.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:00.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:54:59 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:55:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:59 smithi079 ceph-mon[23158]: pgmap v1285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:59 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:55:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:55:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:00.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:55:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:55:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:55:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:55:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:54:59 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:55:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:00 smithi016 conmon[32727]: 2022-03-21T17:55:00.259+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[32727]: 2022-03-21T17:55:01.280+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:01 smithi016 ceph-mon[22010]: pgmap v1286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:02.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 2022-03-21T17:55:01.772+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:02.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:55:02.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:02.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: yield 2022-03-21T17:55:02.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:02.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:02.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:55:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:02.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:55:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:55:02.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:02.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:55:02.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:02.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:55:02.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:02.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:02.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:02.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 2022-03-21T17:55:01.773+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:55:02.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:55:02.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:02.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: yield 2022-03-21T17:55:02.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:02.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:02.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:02.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:55:02.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:02.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:55:02.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:02.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:55:02.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:02.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:55:02.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:02.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:55:02.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:02.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:02.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:02.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 2022-03-21T17:55:02.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:55:02.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: 2022-03-21T17:55:02.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:55:02.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:55:02.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:55:02.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:55:02.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:55:02.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:55:02.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:55:02.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:55:02.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:55:02.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:55:02.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:55:02.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:55:02.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:55:02.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:55:02.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:55:02.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:55:02.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:55:02.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:55:02.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:55:02.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:55:02.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:55:02.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:55:02.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise 2022-03-21T17:55:02.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:55:02.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:55:02.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:55:02.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:55:02.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:55:01 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:01 smithi079 ceph-mon[23158]: pgmap v1286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:02 smithi016 conmon[32727]: 2022-03-21T17:55:02.296+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:55:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:55:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:55:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:55:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:55:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:02 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:55:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:03.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:55:03.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:55:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:55:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:03.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:55:03.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:03.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:03.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:03.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:55:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:55:03.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:55:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:55:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:55:03.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:55:03.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:55:03.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:55:03.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:55:03.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:55:03.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:55:03.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:55:03.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:55:03.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:55:03.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:55:03.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:55:03.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:55:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:55:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:55:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:55:03.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:55:03.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:55:03.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:55:03.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:55:03.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:55:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:55:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:55:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:55:03.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:55:03.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:55:03.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:55:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:55:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:55:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:55:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:55:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:02 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:55:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:55:03.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:55:03.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:55:03.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:55:03.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:55:03.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:55:03.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:55:03.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:55:03.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:55:03.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:55:03.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:55:03.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:55:03.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:55:03.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:55:03.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:55:03.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:55:03.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:55:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:55:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:55:03.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:55:03.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:55:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:55:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:55:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:55:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:55:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:55:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:55:03.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:55:03.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:55:03.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:55:03.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:55:03.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:55:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:55:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:55:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:55:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:55:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:55:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:55:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:55:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:55:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:55:03.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:55:03.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:55:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:55:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:55:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:55:03.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:55:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:03 smithi016 conmon[32727]: 2022-03-21T17:55:03.247+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:04.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:03 smithi016 ceph-mon[22010]: pgmap v1287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:04.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:03 smithi079 ceph-mon[23158]: pgmap v1287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:04 smithi016 conmon[32727]: 2022-03-21T17:55:04.253+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:05 smithi016 conmon[32727]: 2022-03-21T17:55:05.214+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:05 smithi016 ceph-mon[22010]: pgmap v1288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:05 smithi079 ceph-mon[23158]: pgmap v1288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:06 smithi016 conmon[32727]: 2022-03-21T17:55:06.200+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:07 smithi016 conmon[32727]: 2022-03-21T17:55:07.177+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:07 smithi016 ceph-mon[22010]: pgmap v1289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:07 smithi079 ceph-mon[23158]: pgmap v1289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:08 smithi016 conmon[32727]: 2022-03-21T17:55:08.184+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:09 smithi016 conmon[32727]: 2022-03-21T17:55:09.171+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:10.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:10.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:09 smithi016 ceph-mon[22010]: pgmap v1290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:09 smithi079 ceph-mon[23158]: pgmap v1290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:10 smithi016 conmon[32727]: 2022-03-21T17:55:10.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:11.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:11 smithi016 conmon[32727]: 2022-03-21T17:55:11.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:11 smithi016 ceph-mon[22010]: pgmap v1291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:11 smithi079 ceph-mon[23158]: pgmap v1291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:12 smithi016 conmon[32727]: 2022-03-21T17:55:12.166+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:13.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:13 smithi016 conmon[32727]: 2022-03-21T17:55:13.198+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:13 smithi016 ceph-mon[22010]: pgmap v1292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:13 smithi079 ceph-mon[23158]: pgmap v1292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:14 smithi016 conmon[32727]: 2022-03-21T17:55:14.196+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:15.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:15 smithi016 conmon[32727]: 2022-03-21T17:55:15.161+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:15 smithi016 ceph-mon[22010]: pgmap v1293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:15 smithi079 ceph-mon[23158]: pgmap v1293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:16.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:16 smithi016 conmon[32727]: 2022-03-21T17:55:16.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:17.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:17 smithi016 conmon[32727]: 2022-03-21T17:55:17.146+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:18.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:18.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:17 smithi016 ceph-mon[22010]: pgmap v1294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:17 smithi079 ceph-mon[23158]: pgmap v1294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:18 smithi016 conmon[32727]: 2022-03-21T17:55:18.146+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:19.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:19.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:19 smithi016 conmon[32727]: 2022-03-21T17:55:19.125+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:20.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:20.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:19 smithi016 ceph-mon[22010]: pgmap v1295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:19 smithi079 ceph-mon[23158]: pgmap v1295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:20 smithi016 conmon[32727]: 2022-03-21T17:55:20.129+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:21.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:21 smithi016 conmon[32727]: 2022-03-21T17:55:21.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:21 smithi016 ceph-mon[22010]: pgmap v1296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:22.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:22 smithi016 conmon[32727]: 2022-03-21T17:55:22.049+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:21 smithi079 ceph-mon[23158]: pgmap v1296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:23.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:23 smithi016 conmon[32727]: 2022-03-21T17:55:23.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:23 smithi016 ceph-mon[22010]: pgmap v1297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:24.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:24 smithi016 conmon[32727]: 2022-03-21T17:55:24.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:23 smithi079 ceph-mon[23158]: pgmap v1297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:24.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:25 smithi016 conmon[32727]: 2022-03-21T17:55:25.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:25 smithi016 ceph-mon[22010]: pgmap v1298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:26 smithi016 conmon[32727]: 2022-03-21T17:55:26.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:25 smithi079 ceph-mon[23158]: pgmap v1298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:27.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:26 smithi016 conmon[32727]: 2022-03-21T17:55:26.957+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:27 smithi016 ceph-mon[22010]: pgmap v1299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:28.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:27 smithi016 conmon[32727]: 2022-03-21T17:55:27.961+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:27 smithi079 ceph-mon[23158]: pgmap v1299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:28 smithi016 conmon[32727]: 2022-03-21T17:55:28.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:29 smithi016 ceph-mon[22010]: pgmap v1300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:29 smithi016 conmon[32727]: 2022-03-21T17:55:29.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:30.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:29 smithi079 ceph-mon[23158]: pgmap v1300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:30.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:31 smithi016 conmon[32727]: 2022-03-21T17:55:31.004+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:32.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:31 smithi016 ceph-mon[22010]: pgmap v1301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:32.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:32.174 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:31 smithi016 conmon[32727]: 2022-03-21T17:55:31.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:31 smithi079 ceph-mon[23158]: pgmap v1301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:32.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:32 smithi016 conmon[32727]: 2022-03-21T17:55:32.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:33 smithi016 ceph-mon[22010]: pgmap v1302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:34 smithi016 conmon[32727]: 2022-03-21T17:55:34.046+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:33 smithi079 ceph-mon[23158]: pgmap v1302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:35 smithi016 conmon[32727]: 2022-03-21T17:55:35.027+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:36 smithi016 conmon[32727]: 2022-03-21T17:55:36.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:35 smithi016 ceph-mon[22010]: pgmap v1303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:35 smithi079 ceph-mon[23158]: pgmap v1303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:37.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:36 smithi016 conmon[32727]: 2022-03-21T17:55:36.959+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:37 smithi016 ceph-mon[22010]: pgmap v1304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:37 smithi016 conmon[32727]: 2022-03-21T17:55:37.953+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:37 smithi079 ceph-mon[23158]: pgmap v1304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:38 smithi016 conmon[32727]: 2022-03-21T17:55:38.940+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:39 smithi016 ceph-mon[22010]: pgmap v1305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:39 smithi016 conmon[32727]: 2022-03-21T17:55:39.896+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:39 smithi079 ceph-mon[23158]: pgmap v1305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:41.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:40 smithi016 conmon[32727]: 2022-03-21T17:55:40.883+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:41 smithi016 ceph-mon[22010]: pgmap v1306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:41 smithi016 conmon[32727]: 2022-03-21T17:55:41.916+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:41 smithi079 ceph-mon[23158]: pgmap v1306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:42 smithi016 conmon[32727]: 2022-03-21T17:55:42.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:43 smithi016 conmon[32727]: 2022-03-21T17:55:43.853+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:43 smithi016 ceph-mon[22010]: pgmap v1307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:43 smithi079 ceph-mon[23158]: pgmap v1307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:44 smithi016 conmon[32727]: 2022-03-21T17:55:44.832+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:45 smithi016 ceph-mon[22010]: pgmap v1308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:45 smithi016 conmon[32727]: 2022-03-21T17:55:45.842+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:45 smithi079 ceph-mon[23158]: pgmap v1308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:46 smithi016 conmon[32727]: 2022-03-21T17:55:46.891+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:47 smithi016 ceph-mon[22010]: pgmap v1309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:47 smithi016 conmon[32727]: 2022-03-21T17:55:47.875+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:47 smithi079 ceph-mon[23158]: pgmap v1309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:48.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:48 smithi016 conmon[32727]: 2022-03-21T17:55:48.878+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:49 smithi016 ceph-mon[22010]: pgmap v1310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:50.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:49 smithi016 conmon[32727]: 2022-03-21T17:55:49.872+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:49 smithi079 ceph-mon[23158]: pgmap v1310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:50 smithi016 conmon[32727]: 2022-03-21T17:55:50.915+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:51 smithi016 ceph-mon[22010]: pgmap v1311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:51 smithi016 conmon[32727]: 2022-03-21T17:55:51.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:51 smithi079 ceph-mon[23158]: pgmap v1311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:52.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:53.643 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:52 smithi016 conmon[32727]: 2022-03-21T17:55:52.974+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:53.644 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:53 smithi016 ceph-mon[22010]: pgmap v1312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 1995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:54.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:54 smithi016 conmon[32727]: 2022-03-21T17:55:54.001+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:53 smithi079 ceph-mon[23158]: pgmap v1312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 1995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:54 smithi016 conmon[32727]: 2022-03-21T17:55:54.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:55 smithi079 ceph-mon[23158]: pgmap v1313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:55 smithi016 ceph-mon[22010]: pgmap v1313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:55:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:55 smithi016 conmon[32727]: 2022-03-21T17:55:55.953+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:56 smithi016 conmon[32727]: 2022-03-21T17:55:56.995+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:57 smithi079 ceph-mon[23158]: pgmap v1314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:57 smithi016 conmon[32727]: 2022-03-21T17:55:57.973+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:55:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:57 smithi016 ceph-mon[22010]: pgmap v1314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:55:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:55:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:55:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:55:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:55:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:55:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:55:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:55:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:55:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:55:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:55:59.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:58 smithi016 conmon[32727]: 2022-03-21T17:55:58.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:59 smithi079 ceph-mon[23158]: pgmap v1315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:55:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:59 smithi016 ceph-mon[22010]: pgmap v1315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:55:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:55:59 smithi016 conmon[32727]: 2022-03-21T17:55:59.965+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:01 smithi016 conmon[32727]: 2022-03-21T17:56:01.013+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[32727]: 2022-03-21T17:56:02.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:01 smithi016 ceph-mon[22010]: pgmap v1316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:56:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:56:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:56:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:01 smithi079 ceph-mon[23158]: pgmap v1316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:56:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:56:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:56:02.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: 2022-03-21T17:56:02.380+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:02.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:56:02.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:02.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: yield 2022-03-21T17:56:02.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:02.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:56:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:02.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:56:02.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:02.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:56:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:56:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:56:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:56:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:56:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:56:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:56:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:56:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:56:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:02 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:56:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:03 smithi016 conmon[32727]: 2022-03-21T17:56:03.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:56:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:56:03.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:03.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:56:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:56:03.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:03.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:56:03.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:03.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:03.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:03.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:56:03.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:03.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:03.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:03.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:02 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:56:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:03 smithi079 ceph-mon[23158]: pgmap v1317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:04 smithi016 conmon[32727]: 2022-03-21T17:56:04.066+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:03 smithi016 ceph-mon[22010]: pgmap v1317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[32727]: 2022-03-21T17:56:05.071+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:56:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:56:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:04 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:56:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:56:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:04 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T17:56:05.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 2022-03-21T17:56:05.454+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:05.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:56:05.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:05.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: yield 2022-03-21T17:56:05.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:05.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:56:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:05.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:56:05.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:56:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:56:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:56:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 2022-03-21T17:56:05.455+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:56:05.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:56:05.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: yield 2022-03-21T17:56:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:56:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:05.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:56:05.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:56:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:56:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:56:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 2022-03-21T17:56:05.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:56:05.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: 2022-03-21T17:56:05.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:56:05.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:56:05.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:56:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:56:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:56:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:56:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:56:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:56:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:56:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:56:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:56:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:56:05.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:56:05.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:56:05.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:56:05.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:56:05.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:56:05.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:56:05.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:56:05.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:56:05.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:56:05.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:56:05.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise 2022-03-21T17:56:05.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:56:05.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:56:05.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:56:05.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:56:05.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:56:05 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: pgmap v1318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:56:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:56:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:56:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:56:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:06 smithi016 conmon[32727]: 2022-03-21T17:56:06.092+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: pgmap v1318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T17:56:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T17:56:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T17:56:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T17:56:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:07.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:06 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:07.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:56:07.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:56:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:56:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:07.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:56:07.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:07.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:56:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:06 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:56:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:56:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:56:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:56:07.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:07.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:56:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:56:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:56:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:56:07.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:56:07.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:56:07.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:56:07.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:56:07.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:56:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:56:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:56:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:56:07.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:56:07.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:56:07.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:56:07.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:56:07.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:56:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:56:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:56:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:56:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:56:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:56:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:56:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:56:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:56:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:56:07.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:56:07.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:56:07.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:56:07.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:56:07.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:56:07.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.253 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:07 smithi016 conmon[32727]: 2022-03-21T17:56:07.135+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:06 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:07.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:56:07.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:07.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:07.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:56:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:56:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:56:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:56:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:06 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:56:07.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:56:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:56:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:56:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:56:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:56:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:56:07.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:56:07.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:56:07.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:56:07.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:56:07.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:56:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:56:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:56:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:56:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:56:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:56:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:56:07.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:56:07.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:56:07.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:56:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:56:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:56:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:56:07.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:56:07.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:56:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:56:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:56:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:56:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:56:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:56:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:56:07.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:56:07.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:56:07.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:56:07.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:56:07.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:56:07.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:56:07.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:56:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:56:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:56:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:56:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:56:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:56:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:56:07.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:56:07.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:56:07.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:56:07.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:07 smithi079 ceph-mon[23158]: pgmap v1319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:07 smithi016 ceph-mon[22010]: pgmap v1319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:08 smithi016 conmon[32727]: 2022-03-21T17:56:08.174+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:09.420 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:09 smithi016 conmon[32727]: 2022-03-21T17:56:09.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:10.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:09 smithi079 ceph-mon[23158]: pgmap v1320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:09 smithi016 ceph-mon[22010]: pgmap v1320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:10 smithi016 conmon[32727]: 2022-03-21T17:56:10.127+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:11 smithi016 conmon[32727]: 2022-03-21T17:56:11.151+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:11 smithi079 ceph-mon[23158]: pgmap v1321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:11 smithi016 ceph-mon[22010]: pgmap v1321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:12 smithi016 conmon[32727]: 2022-03-21T17:56:12.188+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:13.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:13 smithi016 conmon[32727]: 2022-03-21T17:56:13.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:13 smithi079 ceph-mon[23158]: pgmap v1322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:13 smithi016 ceph-mon[22010]: pgmap v1322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:14 smithi016 conmon[32727]: 2022-03-21T17:56:14.184+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:15 smithi016 conmon[32727]: 2022-03-21T17:56:15.184+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:16 smithi016 conmon[32727]: 2022-03-21T17:56:16.188+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:15 smithi016 ceph-mon[22010]: pgmap v1323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:15 smithi079 ceph-mon[23158]: pgmap v1323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:17.239 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:17 smithi016 conmon[32727]: 2022-03-21T17:56:17.237+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:17 smithi016 ceph-mon[22010]: pgmap v1324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:18.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:18 smithi016 conmon[32727]: 2022-03-21T17:56:18.226+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:17 smithi079 ceph-mon[23158]: pgmap v1324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:19.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:19 smithi016 conmon[32727]: 2022-03-21T17:56:19.226+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:19 smithi016 ceph-mon[22010]: pgmap v1325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:20 smithi016 conmon[32727]: 2022-03-21T17:56:20.185+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:19 smithi079 ceph-mon[23158]: pgmap v1325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:21 smithi016 conmon[32727]: 2022-03-21T17:56:21.192+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:21 smithi016 ceph-mon[22010]: pgmap v1326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:22 smithi016 conmon[32727]: 2022-03-21T17:56:22.213+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:21 smithi079 ceph-mon[23158]: pgmap v1326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:23 smithi016 conmon[32727]: 2022-03-21T17:56:23.199+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:24.967 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:23 smithi016 ceph-mon[22010]: pgmap v1327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:24.967 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:24.968 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:24 smithi016 conmon[32727]: 2022-03-21T17:56:24.231+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:24.969 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:24.970 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:23 smithi079 ceph-mon[23158]: pgmap v1327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:24.970 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:25 smithi016 conmon[32727]: 2022-03-21T17:56:25.202+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:26 smithi016 ceph-mon[22010]: pgmap v1328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:26 smithi016 conmon[32727]: 2022-03-21T17:56:26.228+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:26 smithi079 ceph-mon[23158]: pgmap v1328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:27 smithi016 conmon[32727]: 2022-03-21T17:56:27.189+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:28 smithi016 ceph-mon[22010]: pgmap v1329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:28 smithi016 conmon[32727]: 2022-03-21T17:56:28.240+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:28 smithi079 ceph-mon[23158]: pgmap v1329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:29 smithi016 conmon[32727]: 2022-03-21T17:56:29.259+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:30.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:30.272 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:30 smithi016 ceph-mon[22010]: pgmap v1330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:30 smithi079 ceph-mon[23158]: pgmap v1330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:30 smithi016 conmon[32727]: 2022-03-21T17:56:30.269+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:31.316 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:31 smithi016 conmon[32727]: 2022-03-21T17:56:31.314+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:32.326 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:32.327 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:32 smithi016 ceph-mon[22010]: pgmap v1331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:32 smithi079 ceph-mon[23158]: pgmap v1331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:32 smithi016 conmon[32727]: 2022-03-21T17:56:32.324+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:33.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:33 smithi016 conmon[32727]: 2022-03-21T17:56:33.306+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:34.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:34.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:34 smithi016 ceph-mon[22010]: pgmap v1332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:34.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:34 smithi079 ceph-mon[23158]: pgmap v1332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:34 smithi016 conmon[32727]: 2022-03-21T17:56:34.336+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:35.330 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:35 smithi016 conmon[32727]: 2022-03-21T17:56:35.328+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:36.341 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:36.342 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:36 smithi016 ceph-mon[22010]: pgmap v1333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:36 smithi079 ceph-mon[23158]: pgmap v1333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:36 smithi016 conmon[32727]: 2022-03-21T17:56:36.338+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:37.313 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:37 smithi016 conmon[32727]: 2022-03-21T17:56:37.312+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:38 smithi016 ceph-mon[22010]: pgmap v1334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:38 smithi016 conmon[32727]: 2022-03-21T17:56:38.272+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:38 smithi079 ceph-mon[23158]: pgmap v1334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:39 smithi016 conmon[32727]: 2022-03-21T17:56:39.232+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:40 smithi016 ceph-mon[22010]: pgmap v1335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:40 smithi016 conmon[32727]: 2022-03-21T17:56:40.242+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:40 smithi079 ceph-mon[23158]: pgmap v1335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:41.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:41 smithi016 conmon[32727]: 2022-03-21T17:56:41.262+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:42 smithi016 ceph-mon[22010]: pgmap v1336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:42.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:42 smithi016 conmon[32727]: 2022-03-21T17:56:42.288+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:42 smithi079 ceph-mon[23158]: pgmap v1336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:43 smithi016 conmon[32727]: 2022-03-21T17:56:43.259+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:44 smithi016 ceph-mon[22010]: pgmap v1337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:44 smithi016 conmon[32727]: 2022-03-21T17:56:44.278+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:44 smithi079 ceph-mon[23158]: pgmap v1337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:45.319 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:45 smithi016 conmon[32727]: 2022-03-21T17:56:45.316+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:46.367 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:46.368 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:46 smithi016 ceph-mon[22010]: pgmap v1338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:46 smithi079 ceph-mon[23158]: pgmap v1338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:46 smithi016 conmon[32727]: 2022-03-21T17:56:46.364+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:47.320 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:48.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:48.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:48 smithi016 ceph-mon[22010]: pgmap v1339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:48.348 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:47 smithi016 conmon[32727]: 2022-03-21T17:56:47.318+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:48 smithi079 ceph-mon[23158]: pgmap v1339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:48 smithi016 conmon[32727]: 2022-03-21T17:56:48.344+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:49 smithi016 conmon[32727]: 2022-03-21T17:56:49.309+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:50 smithi016 ceph-mon[22010]: pgmap v1340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:50.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:50 smithi016 conmon[32727]: 2022-03-21T17:56:50.280+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:50.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:50 smithi079 ceph-mon[23158]: pgmap v1340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:51 smithi016 conmon[32727]: 2022-03-21T17:56:51.290+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:52 smithi016 ceph-mon[22010]: pgmap v1341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:52 smithi016 conmon[32727]: 2022-03-21T17:56:52.306+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:52 smithi079 ceph-mon[23158]: pgmap v1341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:53.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:53 smithi016 conmon[32727]: 2022-03-21T17:56:53.345+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:54.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:54.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:54 smithi016 ceph-mon[22010]: pgmap v1342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:54.353 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:54 smithi079 ceph-mon[23158]: pgmap v1342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:54.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:54 smithi016 conmon[32727]: 2022-03-21T17:56:54.349+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:55.498 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:55 smithi016 conmon[32727]: 2022-03-21T17:56:55.304+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:55.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:56 smithi016 ceph-mon[22010]: pgmap v1343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:56 smithi016 conmon[32727]: 2022-03-21T17:56:56.302+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:56 smithi079 ceph-mon[23158]: pgmap v1343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:56:57.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:57 smithi016 conmon[32727]: 2022-03-21T17:56:57.342+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:58.387 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:58.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: pgmap v1344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:58.388 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:56:58.389 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:56:58.389 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:56:58.390 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:56:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: pgmap v1344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:56:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:56:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:56:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:56:58.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:56:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:58 smithi016 conmon[32727]: 2022-03-21T17:56:58.385+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:56:59.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:59.366 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:56:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:56:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:56:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:56:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:56:59 smithi016 conmon[32727]: 2022-03-21T17:56:59.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:00 smithi016 ceph-mon[22010]: pgmap v1345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:00 smithi016 conmon[32727]: 2022-03-21T17:57:00.337+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:00 smithi079 ceph-mon[23158]: pgmap v1345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:01.379 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:01 smithi016 conmon[32727]: 2022-03-21T17:57:01.376+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:02.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:02.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:02 smithi016 ceph-mon[22010]: pgmap v1346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:02 smithi079 ceph-mon[23158]: pgmap v1346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:02 smithi016 conmon[32727]: 2022-03-21T17:57:02.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:03 smithi016 conmon[32727]: 2022-03-21T17:57:03.326+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:04 smithi016 conmon[32727]: 2022-03-21T17:57:04.291+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:04 smithi016 ceph-mon[22010]: pgmap v1347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:04 smithi079 ceph-mon[23158]: pgmap v1347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:05 smithi016 conmon[32727]: 2022-03-21T17:57:05.333+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:06 smithi079 ceph-mon[23158]: pgmap v1348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:57:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:57:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:57:06.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:06.308 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:06 smithi016 ceph-mon[22010]: pgmap v1348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:06.309 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:57:06.309 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:57:06.309 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:57:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:06 smithi016 conmon[32727]: 2022-03-21T17:57:06.305+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:07.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:07 smithi016 conmon[32727]: 2022-03-21T17:57:07.270+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:08 smithi016 ceph-mon[22010]: pgmap v1349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[32727]: 2022-03-21T17:57:08.284+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:08 smithi079 ceph-mon[23158]: pgmap v1349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:08.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: 2022-03-21T17:57:08.557+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:08.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:57:08.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:08.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: yield 2022-03-21T17:57:08.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:08.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:08.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:08.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:57:08.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:08.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:57:08.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:08.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:57:08.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:08.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:57:08.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:08.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:57:08.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:08.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:08.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:08 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:09 smithi016 conmon[32727]: 2022-03-21T17:57:09.241+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:10 smithi016 conmon[32727]: 2022-03-21T17:57:10.265+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:10.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:57:10.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:10.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:57:10.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:10.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:10.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:10.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:57:10.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:10.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:57:10.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:10.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:57:10.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:10.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:10.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:10.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:57:10.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:10.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:10.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:10.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:10 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:57:10.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:10 smithi016 ceph-mon[22010]: pgmap v1350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:57:10.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:10.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:57:10.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:10.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:10.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:10.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:57:10.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:10.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:57:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:57:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:10.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:10.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:57:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:10.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:10.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:10 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:57:10.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:10 smithi079 ceph-mon[23158]: pgmap v1350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[32727]: 2022-03-21T17:57:11.243+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:11.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 2022-03-21T17:57:11.629+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:11.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:57:11.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:11.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: yield 2022-03-21T17:57:11.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:11.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:11.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:11.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:57:11.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:11.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:57:11.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:11.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:57:11.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:11.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:57:11.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:11.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:57:11.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:11.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:11.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:11.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 2022-03-21T17:57:11.630+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:57:11.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:57:11.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:11.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: yield 2022-03-21T17:57:11.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:11.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:11.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:11.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:57:11.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:11.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:57:11.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:11.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:57:11.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:11.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:57:11.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:11.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:57:11.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:11.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:11.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:11.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 2022-03-21T17:57:11.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:57:11.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: 2022-03-21T17:57:11.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:57:11.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:57:11.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:57:11.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:57:11.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:57:11.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:57:11.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:57:11.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:57:11.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:57:11.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:57:11.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:57:11.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:57:11.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:57:11.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:57:11.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:57:11.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:57:11.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:57:11.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:57:11.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:57:11.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:57:11.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:57:11.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:57:11.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise 2022-03-21T17:57:11.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:57:11.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:57:11.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:57:11.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:57:11.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:57:11 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:12 smithi016 ceph-mon[22010]: pgmap v1351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:12 smithi016 conmon[32727]: 2022-03-21T17:57:12.227+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:12 smithi079 ceph-mon[23158]: pgmap v1351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:13 smithi016 conmon[32727]: 2022-03-21T17:57:13.201+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:57:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:13.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:57:13.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:13.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:57:13.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:13.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:57:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:13.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:13.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:57:13.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:13.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:13.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:13 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:57:13.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:13.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:57:13.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:13.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:13.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:13.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:57:13.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:13.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:57:13.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:13.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:57:13.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:13.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:13.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:13.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:57:13.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:13.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:13.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:57:13.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:57:13.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:57:13.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:57:13.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:57:13.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:57:13.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:57:13.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:57:13.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:57:13.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:57:13.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:57:13.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:57:13.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:57:13.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:57:13.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:57:13.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:57:13.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:57:13.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:57:13.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:57:13.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:57:13.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:57:13.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:57:13.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:57:13.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:57:13.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:57:13.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:57:13.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:57:13.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:57:13.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:57:13.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:57:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:57:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:57:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:57:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:57:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:13 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:57:13.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:57:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:57:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:57:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:57:13.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:57:13.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:57:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:57:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:57:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:57:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:57:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:57:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:57:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:57:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:57:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:57:13.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:57:13.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:13.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:57:13.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:57:13.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:57:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:57:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:57:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:57:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:57:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:57:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:57:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:57:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:57:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:57:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:57:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:57:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:57:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:57:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:57:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:57:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:57:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:57:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:57:13.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:57:13.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:57:13.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:57:13.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:57:13.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:57:13.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:57:13.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:57:13.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:57:13.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:57:13.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:57:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:14 smithi016 ceph-mon[22010]: pgmap v1352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:14 smithi016 conmon[32727]: 2022-03-21T17:57:14.219+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:14 smithi079 ceph-mon[23158]: pgmap v1352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:15 smithi016 conmon[32727]: 2022-03-21T17:57:15.208+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:16 smithi016 conmon[32727]: 2022-03-21T17:57:16.200+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:16 smithi016 ceph-mon[22010]: pgmap v1353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:16.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:16 smithi079 ceph-mon[23158]: pgmap v1353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:16.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:17 smithi016 conmon[32727]: 2022-03-21T17:57:17.152+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:18 smithi016 ceph-mon[22010]: pgmap v1354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:18 smithi016 conmon[32727]: 2022-03-21T17:57:18.150+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:18 smithi079 ceph-mon[23158]: pgmap v1354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:19 smithi016 conmon[32727]: 2022-03-21T17:57:19.193+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:19.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:20 smithi016 ceph-mon[22010]: pgmap v1355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:20 smithi016 conmon[32727]: 2022-03-21T17:57:20.195+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:20 smithi079 ceph-mon[23158]: pgmap v1355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:21 smithi016 conmon[32727]: 2022-03-21T17:57:21.191+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:22 smithi016 ceph-mon[22010]: pgmap v1356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:22.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:22 smithi016 conmon[32727]: 2022-03-21T17:57:22.227+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:22 smithi079 ceph-mon[23158]: pgmap v1356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:23.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:23 smithi016 conmon[32727]: 2022-03-21T17:57:23.191+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:24 smithi016 ceph-mon[22010]: pgmap v1357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:24.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:24 smithi016 conmon[32727]: 2022-03-21T17:57:24.168+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:24 smithi079 ceph-mon[23158]: pgmap v1357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:25 smithi016 conmon[32727]: 2022-03-21T17:57:25.156+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:26 smithi016 conmon[32727]: 2022-03-21T17:57:26.169+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:26 smithi016 ceph-mon[22010]: pgmap v1358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:26 smithi079 ceph-mon[23158]: pgmap v1358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:27.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:27 smithi016 conmon[32727]: 2022-03-21T17:57:27.201+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:28 smithi016 ceph-mon[22010]: pgmap v1359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:28.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:28.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:28.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:28 smithi016 conmon[32727]: 2022-03-21T17:57:28.169+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:28 smithi079 ceph-mon[23158]: pgmap v1359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:29.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:29 smithi016 conmon[32727]: 2022-03-21T17:57:29.184+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:30 smithi016 ceph-mon[22010]: pgmap v1360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:30 smithi016 conmon[32727]: 2022-03-21T17:57:30.214+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:30 smithi079 ceph-mon[23158]: pgmap v1360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:31.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:31 smithi016 conmon[32727]: 2022-03-21T17:57:31.250+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:32 smithi016 ceph-mon[22010]: pgmap v1361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:32 smithi016 conmon[32727]: 2022-03-21T17:57:32.264+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:32 smithi079 ceph-mon[23158]: pgmap v1361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:33.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:33 smithi016 conmon[32727]: 2022-03-21T17:57:33.266+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:33.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:34 smithi016 ceph-mon[22010]: pgmap v1362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:34 smithi016 conmon[32727]: 2022-03-21T17:57:34.293+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:34 smithi079 ceph-mon[23158]: pgmap v1362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:35 smithi016 conmon[32727]: 2022-03-21T17:57:35.288+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:36 smithi016 ceph-mon[22010]: pgmap v1363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:36 smithi016 conmon[32727]: 2022-03-21T17:57:36.302+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:36 smithi079 ceph-mon[23158]: pgmap v1363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:37 smithi016 conmon[32727]: 2022-03-21T17:57:37.337+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:38 smithi016 ceph-mon[22010]: pgmap v1364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:38 smithi016 conmon[32727]: 2022-03-21T17:57:38.300+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:38 smithi079 ceph-mon[23158]: pgmap v1364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:39 smithi016 conmon[32727]: 2022-03-21T17:57:39.255+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:40 smithi016 ceph-mon[22010]: pgmap v1365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:40.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:40 smithi016 conmon[32727]: 2022-03-21T17:57:40.210+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:40 smithi079 ceph-mon[23158]: pgmap v1365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:41.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:41 smithi016 conmon[32727]: 2022-03-21T17:57:41.250+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:42 smithi016 ceph-mon[22010]: pgmap v1366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:42 smithi016 conmon[32727]: 2022-03-21T17:57:42.222+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:42 smithi079 ceph-mon[23158]: pgmap v1366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:43.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:43 smithi016 conmon[32727]: 2022-03-21T17:57:43.250+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:44 smithi016 ceph-mon[22010]: pgmap v1367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:44 smithi016 conmon[32727]: 2022-03-21T17:57:44.260+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:44 smithi079 ceph-mon[23158]: pgmap v1367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:44.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:45 smithi016 conmon[32727]: 2022-03-21T17:57:45.243+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:46 smithi016 ceph-mon[22010]: pgmap v1368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:46 smithi016 conmon[32727]: 2022-03-21T17:57:46.215+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:46 smithi079 ceph-mon[23158]: pgmap v1368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:47.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:47.672 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:47 smithi016 conmon[32727]: 2022-03-21T17:57:47.198+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:48 smithi016 conmon[32727]: 2022-03-21T17:57:48.157+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:48 smithi079 ceph-mon[23158]: pgmap v1369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:48 smithi016 ceph-mon[22010]: pgmap v1369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:49 smithi016 conmon[32727]: 2022-03-21T17:57:49.124+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:49 smithi079 ceph-mon[23158]: pgmap v1370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:49.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:49.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:49 smithi016 ceph-mon[22010]: pgmap v1370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:49.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:50 smithi016 conmon[32727]: 2022-03-21T17:57:50.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:51 smithi016 conmon[32727]: 2022-03-21T17:57:51.063+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:51 smithi079 ceph-mon[23158]: pgmap v1371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:51 smithi016 ceph-mon[22010]: pgmap v1371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:52 smithi016 conmon[32727]: 2022-03-21T17:57:52.023+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:53 smithi016 conmon[32727]: 2022-03-21T17:57:53.062+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:53 smithi079 ceph-mon[23158]: pgmap v1372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:53 smithi016 ceph-mon[22010]: pgmap v1372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:54 smithi016 conmon[32727]: 2022-03-21T17:57:54.086+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:55 smithi016 conmon[32727]: 2022-03-21T17:57:55.079+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:55 smithi079 ceph-mon[23158]: pgmap v1373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:55 smithi016 ceph-mon[22010]: pgmap v1373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:57:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:56 smithi016 conmon[32727]: 2022-03-21T17:57:56.100+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:57.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:57 smithi016 conmon[32727]: 2022-03-21T17:57:57.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:57 smithi079 ceph-mon[23158]: pgmap v1374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:57 smithi016 ceph-mon[22010]: pgmap v1374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:58 smithi016 conmon[32727]: 2022-03-21T17:57:58.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:57:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:57:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:57:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:57:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:57:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:57:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:57:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:57:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:57:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:57:59 smithi016 conmon[32727]: 2022-03-21T17:57:59.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:57:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:59 smithi079 ceph-mon[23158]: pgmap v1375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:57:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:57:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:59 smithi016 ceph-mon[22010]: pgmap v1375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:57:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:57:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:00 smithi016 conmon[32727]: 2022-03-21T17:58:00.165+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:01 smithi016 conmon[32727]: 2022-03-21T17:58:01.187+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:01 smithi079 ceph-mon[23158]: pgmap v1376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:01 smithi016 ceph-mon[22010]: pgmap v1376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:02 smithi016 conmon[32727]: 2022-03-21T17:58:02.200+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:03 smithi016 conmon[32727]: 2022-03-21T17:58:03.237+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:03 smithi079 ceph-mon[23158]: pgmap v1377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:03 smithi016 ceph-mon[22010]: pgmap v1377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:04 smithi016 conmon[32727]: 2022-03-21T17:58:04.258+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:05 smithi016 conmon[32727]: 2022-03-21T17:58:05.229+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:05 smithi079 ceph-mon[23158]: pgmap v1378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:05 smithi016 ceph-mon[22010]: pgmap v1378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:06 smithi016 conmon[32727]: 2022-03-21T17:58:06.241+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:07 smithi016 conmon[32727]: 2022-03-21T17:58:07.210+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:07 smithi079 ceph-mon[23158]: pgmap v1379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:07 smithi016 ceph-mon[22010]: pgmap v1379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:08 smithi016 conmon[32727]: 2022-03-21T17:58:08.168+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:09 smithi016 conmon[32727]: 2022-03-21T17:58:09.195+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:09 smithi016 ceph-mon[22010]: pgmap v1380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:09 smithi079 ceph-mon[23158]: pgmap v1380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:10.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:10 smithi016 conmon[32727]: 2022-03-21T17:58:10.213+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:11 smithi016 conmon[32727]: 2022-03-21T17:58:11.188+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:11 smithi079 ceph-mon[23158]: pgmap v1381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:58:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:58:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:58:12.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:12.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:11 smithi016 ceph-mon[22010]: pgmap v1381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:12.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:58:12.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:58:12.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:58:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[32727]: 2022-03-21T17:58:12.187+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:12.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: 2022-03-21T17:58:12.109+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:12.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:58:12.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:12.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: yield 2022-03-21T17:58:12.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:12.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:12.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:12.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:58:12.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:12.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:58:12.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:12.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:58:12.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:12.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:58:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:58:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:12.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:12.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:58:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:58:13.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:13.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:13.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:13.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:58:13.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:13.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:58:13.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:13.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:58:13.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:13.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:13.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:13.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:58:13.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:13.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:13.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:13.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:12 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:58:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:58:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:58:13.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:13.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:13.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:13.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:58:13.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:13.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:58:13.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:13.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:58:13.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:13.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:13.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:13.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:58:13.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:13.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:13.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:13.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:12 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:58:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:13 smithi016 conmon[32727]: 2022-03-21T17:58:13.190+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:13 smithi016 ceph-mon[22010]: pgmap v1382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:13 smithi079 ceph-mon[23158]: pgmap v1382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:14 smithi016 conmon[32727]: 2022-03-21T17:58:14.230+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[32727]: 2022-03-21T17:58:15.276+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:15.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 2022-03-21T17:58:15.181+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:15.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:58:15.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:15.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: yield 2022-03-21T17:58:15.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:58:15.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:15.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:58:15.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:15.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:58:15.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:15.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:58:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:58:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:15.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:15.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 2022-03-21T17:58:15.182+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:58:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:58:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:15.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: yield 2022-03-21T17:58:15.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:15.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:15.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:15.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:58:15.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:58:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:58:15.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:15.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:58:15.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:58:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 2022-03-21T17:58:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:58:15.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: 2022-03-21T17:58:15.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:58:15.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:58:15.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:58:15.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:58:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:58:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:58:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:58:15.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:58:15.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:58:15.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:58:15.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:58:15.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:58:15.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:58:15.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:58:15.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:58:15.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:58:15.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:58:15.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:58:15.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:58:15.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:58:15.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:58:15.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:58:15.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise 2022-03-21T17:58:15.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:58:15.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:58:15.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:58:15.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:58:15.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:58:15 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:15 smithi016 ceph-mon[22010]: pgmap v1383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:58:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:16.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:16.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:16.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:58:16.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:16.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:58:16.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:16.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:58:16.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:16.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:16.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:16.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:58:16.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:16.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:16.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:15 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:58:16.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:16.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:58:16.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:16.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:16.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:16.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:58:16.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:16.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:58:16.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:16.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:58:16.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:16.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:16.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:16.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:58:16.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:16.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:16.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:58:16.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:58:16.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:58:16.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:58:16.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:58:16.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:58:16.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:58:16.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:58:16.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:58:16.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:58:16.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:58:16.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:58:16.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:58:16.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:58:16.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:58:16.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:58:16.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:58:16.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:58:16.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:58:16.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:58:16.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:58:16.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:58:16.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:58:16.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:58:16.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:58:16.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:58:16.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:58:16.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:58:16.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:58:16.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:16.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:15 smithi079 ceph-mon[23158]: pgmap v1383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:16.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:16.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:58:16.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:16.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:16.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:16.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:58:16.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:16.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:58:16.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:16.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:58:16.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:16.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:16.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:16.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:58:16.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:16.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:16.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.237 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:15 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:58:16.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:58:16.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:58:16.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:58:16.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:58:16.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:58:16.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:58:16.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:58:16.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:58:16.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:58:16.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:58:16.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:58:16.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:58:16.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:58:16.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:58:16.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:58:16.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:58:16.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:58:16.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:58:16.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:58:16.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:58:16.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:58:16.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:58:16.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:58:16.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:58:16.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:58:16.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:58:16.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:58:16.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:58:16.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:58:16.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:58:16.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:58:16.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:58:16.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:58:16.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:58:16.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:58:16.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:58:16.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:58:16.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:58:16.265 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:58:16.266 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:58:16.266 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:58:16.267 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:58:16.267 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:58:16.268 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:58:16.269 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:58:16.270 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:58:16.271 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:58:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:16 smithi016 conmon[32727]: 2022-03-21T17:58:16.228+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:17 smithi016 conmon[32727]: 2022-03-21T17:58:17.205+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:17 smithi016 ceph-mon[22010]: pgmap v1384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:17 smithi079 ceph-mon[23158]: pgmap v1384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:18 smithi016 conmon[32727]: 2022-03-21T17:58:18.212+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:19 smithi016 conmon[32727]: 2022-03-21T17:58:19.166+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:20.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:20.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:19 smithi016 ceph-mon[22010]: pgmap v1385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:19 smithi079 ceph-mon[23158]: pgmap v1385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:20 smithi016 conmon[32727]: 2022-03-21T17:58:20.126+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:21.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:21.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:21 smithi016 conmon[32727]: 2022-03-21T17:58:21.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:22.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:21 smithi016 ceph-mon[22010]: pgmap v1386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:22.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:21 smithi079 ceph-mon[23158]: pgmap v1386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:22 smithi016 conmon[32727]: 2022-03-21T17:58:22.134+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:23.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:23 smithi016 conmon[32727]: 2022-03-21T17:58:23.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:24.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:23 smithi016 ceph-mon[22010]: pgmap v1387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:24.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:24.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:23 smithi079 ceph-mon[23158]: pgmap v1387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:24 smithi016 conmon[32727]: 2022-03-21T17:58:24.139+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:25.105 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:25 smithi016 conmon[32727]: 2022-03-21T17:58:25.104+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:25 smithi079 ceph-mon[23158]: pgmap v1388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:26.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:25 smithi016 ceph-mon[22010]: pgmap v1388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:26.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:26 smithi016 conmon[32727]: 2022-03-21T17:58:26.079+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:27.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:27 smithi016 conmon[32727]: 2022-03-21T17:58:27.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:28.088 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:27 smithi079 ceph-mon[23158]: pgmap v1389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:28.089 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:28.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:27 smithi016 ceph-mon[22010]: pgmap v1389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:28.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:28 smithi016 conmon[32727]: 2022-03-21T17:58:28.091+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:29.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:29.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:29.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:29.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:29 smithi016 conmon[32727]: 2022-03-21T17:58:29.106+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:30.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:29 smithi016 ceph-mon[22010]: pgmap v1390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:30.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:29 smithi079 ceph-mon[23158]: pgmap v1390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:30 smithi016 conmon[32727]: 2022-03-21T17:58:30.094+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:31.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:31 smithi016 conmon[32727]: 2022-03-21T17:58:31.130+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:32.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:31 smithi016 ceph-mon[22010]: pgmap v1391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:32.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:31 smithi079 ceph-mon[23158]: pgmap v1391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:32 smithi016 conmon[32727]: 2022-03-21T17:58:32.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:33.077 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:33 smithi016 conmon[32727]: 2022-03-21T17:58:33.075+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:34 smithi016 conmon[32727]: 2022-03-21T17:58:34.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:33 smithi016 ceph-mon[22010]: pgmap v1392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:33 smithi079 ceph-mon[23158]: pgmap v1392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:35 smithi016 conmon[32727]: 2022-03-21T17:58:35.011+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:35 smithi016 ceph-mon[22010]: pgmap v1393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:36 smithi016 conmon[32727]: 2022-03-21T17:58:36.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:35 smithi079 ceph-mon[23158]: pgmap v1393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:37 smithi016 conmon[32727]: 2022-03-21T17:58:37.022+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:37 smithi016 ceph-mon[22010]: pgmap v1394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:38 smithi016 conmon[32727]: 2022-03-21T17:58:38.032+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:37 smithi079 ceph-mon[23158]: pgmap v1394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:39.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:38 smithi016 conmon[32727]: 2022-03-21T17:58:38.995+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:39 smithi016 ceph-mon[22010]: pgmap v1395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:39 smithi016 conmon[32727]: 2022-03-21T17:58:39.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:39 smithi079 ceph-mon[23158]: pgmap v1395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:41 smithi016 conmon[32727]: 2022-03-21T17:58:41.000+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:41 smithi016 ceph-mon[22010]: pgmap v1396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:42 smithi016 conmon[32727]: 2022-03-21T17:58:42.000+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:41 smithi079 ceph-mon[23158]: pgmap v1396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:43 smithi016 conmon[32727]: 2022-03-21T17:58:43.026+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:44.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:44 smithi016 conmon[32727]: 2022-03-21T17:58:44.067+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:43 smithi016 ceph-mon[22010]: pgmap v1397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:43 smithi079 ceph-mon[23158]: pgmap v1397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:45 smithi016 conmon[32727]: 2022-03-21T17:58:45.042+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:45 smithi016 ceph-mon[22010]: pgmap v1398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:46 smithi016 conmon[32727]: 2022-03-21T17:58:46.000+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:45 smithi079 ceph-mon[23158]: pgmap v1398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:46 smithi016 conmon[32727]: 2022-03-21T17:58:46.972+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:47 smithi016 ceph-mon[22010]: pgmap v1399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:47 smithi016 conmon[32727]: 2022-03-21T17:58:47.971+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:47 smithi079 ceph-mon[23158]: pgmap v1399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:49.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:48 smithi016 conmon[32727]: 2022-03-21T17:58:48.950+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:49.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:49 smithi016 ceph-mon[22010]: pgmap v1400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:49 smithi016 conmon[32727]: 2022-03-21T17:58:49.981+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:49 smithi079 ceph-mon[23158]: pgmap v1400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:50 smithi016 conmon[32727]: 2022-03-21T17:58:50.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:51 smithi016 ceph-mon[22010]: pgmap v1401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:51 smithi016 conmon[32727]: 2022-03-21T17:58:51.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:51 smithi079 ceph-mon[23158]: pgmap v1401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:52 smithi016 conmon[32727]: 2022-03-21T17:58:52.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:53 smithi016 ceph-mon[22010]: pgmap v1402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:54.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:54.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:54 smithi016 conmon[32727]: 2022-03-21T17:58:54.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:53 smithi079 ceph-mon[23158]: pgmap v1402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:55 smithi016 conmon[32727]: 2022-03-21T17:58:55.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:55 smithi016 ceph-mon[22010]: pgmap v1403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:56 smithi016 conmon[32727]: 2022-03-21T17:58:56.009+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:55 smithi079 ceph-mon[23158]: pgmap v1403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:58:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:56 smithi016 conmon[32727]: 2022-03-21T17:58:56.969+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:57 smithi016 ceph-mon[22010]: pgmap v1404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:58.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:58 smithi016 conmon[32727]: 2022-03-21T17:58:58.014+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:57 smithi079 ceph-mon[23158]: pgmap v1404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:58:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:58 smithi016 conmon[32727]: 2022-03-21T17:58:58.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:58:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:58:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:58:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:58:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:58:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:59 smithi016 ceph-mon[22010]: pgmap v1405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:58:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:00.170 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:58:59 smithi016 conmon[32727]: 2022-03-21T17:58:59.941+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:59 smithi079 ceph-mon[23158]: pgmap v1405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:58:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:00 smithi016 conmon[32727]: 2022-03-21T17:59:00.931+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:01 smithi016 conmon[32727]: 2022-03-21T17:59:01.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:01 smithi016 ceph-mon[22010]: pgmap v1406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:01 smithi079 ceph-mon[23158]: pgmap v1406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:02 smithi016 conmon[32727]: 2022-03-21T17:59:02.924+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:03 smithi016 ceph-mon[22010]: pgmap v1407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:03 smithi016 conmon[32727]: 2022-03-21T17:59:03.907+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:03 smithi079 ceph-mon[23158]: pgmap v1407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:04 smithi016 conmon[32727]: 2022-03-21T17:59:04.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:05 smithi016 ceph-mon[22010]: pgmap v1408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:05 smithi016 conmon[32727]: 2022-03-21T17:59:05.856+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:05 smithi079 ceph-mon[23158]: pgmap v1408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:06 smithi016 conmon[32727]: 2022-03-21T17:59:06.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:07 smithi016 ceph-mon[22010]: pgmap v1409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:08.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:08.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:07 smithi016 conmon[32727]: 2022-03-21T17:59:07.874+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:07 smithi079 ceph-mon[23158]: pgmap v1409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:08.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:08 smithi016 conmon[32727]: 2022-03-21T17:59:08.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:09 smithi016 ceph-mon[22010]: pgmap v1410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:10.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:09 smithi016 conmon[32727]: 2022-03-21T17:59:09.891+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:10.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:09 smithi079 ceph-mon[23158]: pgmap v1410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:10.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:10 smithi016 conmon[32727]: 2022-03-21T17:59:10.878+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:11 smithi016 ceph-mon[22010]: pgmap v1411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:11 smithi016 conmon[32727]: 2022-03-21T17:59:11.915+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:11 smithi079 ceph-mon[23158]: pgmap v1411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:12 smithi016 conmon[32727]: 2022-03-21T17:59:12.935+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:13 smithi016 conmon[32727]: 2022-03-21T17:59:13.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:13 smithi016 ceph-mon[22010]: pgmap v1412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2196 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:13 smithi079 ceph-mon[23158]: pgmap v1412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2196 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:15.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:14 smithi016 conmon[32727]: 2022-03-21T17:59:14.936+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:15 smithi016 conmon[32727]: 2022-03-21T17:59:15.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: pgmap v1413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:59:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:59:16.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:59:16.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:59:16.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:15 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:59:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: pgmap v1413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T17:59:16.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T17:59:16.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T17:59:16.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:59:16.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:15 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T17:59:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:16 smithi016 conmon[32727]: 2022-03-21T17:59:16.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:17 smithi016 ceph-mon[22010]: pgmap v1414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:17 smithi016 conmon[32727]: 2022-03-21T17:59:17.858+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:17 smithi079 ceph-mon[23158]: pgmap v1414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:18.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: 2022-03-21T17:59:18.285+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:18.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:59:18.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:18.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: yield 2022-03-21T17:59:18.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:18.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:18.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:18.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:59:18.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:18.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:59:18.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:18.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:59:18.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:18.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:59:18.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:18.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:59:18.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:18.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:18.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:18 smithi016 conmon[32727]: 2022-03-21T17:59:18.847+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:59:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:19.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:59:19.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:19.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:59:19.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:59:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:19.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:59:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:19.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:19.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:59:19.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:19.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:59:19.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:59:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:59:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:59:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:59:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:59:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:59:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T17:59:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:19 smithi016 ceph-mon[22010]: pgmap v1415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:19 smithi016 conmon[32727]: 2022-03-21T17:59:19.808+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:19 smithi079 ceph-mon[23158]: pgmap v1415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:21.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:20 smithi016 conmon[32727]: 2022-03-21T17:59:20.825+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:21.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 2022-03-21T17:59:21.357+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:21.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:59:21.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: yield 2022-03-21T17:59:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:59:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:21.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:59:21.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:59:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:59:21.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:21.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:59:21.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:21.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:21.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:21.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 2022-03-21T17:59:21.358+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:59:21.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:59:21.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:21.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: yield 2022-03-21T17:59:21.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:21.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:21.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:21.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T17:59:21.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:21.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T17:59:21.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:21.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T17:59:21.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:21.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T17:59:21.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:21.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T17:59:21.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:21.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:21.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:21.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 2022-03-21T17:59:21.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T17:59:21.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: 2022-03-21T17:59:21.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T17:59:21.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:59:21.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return f(*arg) 2022-03-21T17:59:21.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:59:21.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T17:59:21.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:59:21.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:59:21.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:59:21.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:59:21.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:59:21.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T17:59:21.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:59:21.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:59:21.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:59:21.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T17:59:21.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:59:21.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise self._exception 2022-03-21T17:59:21.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:59:21.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:59:21.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:59:21.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T17:59:21.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:59:21.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise 2022-03-21T17:59:21.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:59:21.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T17:59:21.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:59:21.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T17:59:21.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:21 smithi016 conmon[32727]: 2022-03-21T17:59:21.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:21 smithi016 ceph-mon[22010]: pgmap v1416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:59:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:59:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:59:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:59:22.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:22.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:22.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:22.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:59:22.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:22.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:22.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.176 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:59:22.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:22.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T17:59:22.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:22.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:22.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:22.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T17:59:22.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:22.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T17:59:22.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:22.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T17:59:22.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:22.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:22.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:22.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T17:59:22.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:22.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:22.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:59:22.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:59:22.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T17:59:22.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:59:22.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T17:59:22.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:59:22.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T17:59:22.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:59:22.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:59:22.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:59:22.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:59:22.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:59:22.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:59:22.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:59:22.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:59:22.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:59:22.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T17:59:22.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:59:22.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T17:59:22.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:59:22.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:59:22.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:59:22.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:59:22.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:59:22.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T17:59:22.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:59:22.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:59:22.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:59:22.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T17:59:22.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:21 smithi079 ceph-mon[23158]: pgmap v1416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:59:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:59:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:59:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:59:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:59:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T17:59:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T17:59:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T17:59:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T17:59:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T17:59:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T17:59:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T17:59:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T17:59:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T17:59:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T17:59:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T17:59:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T17:59:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T17:59:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T17:59:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T17:59:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T17:59:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T17:59:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:59:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T17:59:22.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T17:59:22.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T17:59:22.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T17:59:22.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T17:59:22.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T17:59:22.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T17:59:22.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T17:59:22.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T17:59:22.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T17:59:22.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T17:59:22.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T17:59:22.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T17:59:22.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T17:59:22.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T17:59:22.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T17:59:22.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T17:59:22.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T17:59:22.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T17:59:22.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T17:59:22.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T17:59:22.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T17:59:22.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T17:59:22.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T17:59:22.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T17:59:22.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T17:59:22.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T17:59:22.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T17:59:22.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T17:59:22.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T17:59:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:22 smithi016 conmon[32727]: 2022-03-21T17:59:22.795+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:23 smithi016 conmon[32727]: 2022-03-21T17:59:23.778+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:23 smithi016 ceph-mon[22010]: pgmap v1417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2206 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:23 smithi079 ceph-mon[23158]: pgmap v1417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2206 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:24 smithi016 conmon[32727]: 2022-03-21T17:59:24.797+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:25 smithi016 conmon[32727]: 2022-03-21T17:59:25.784+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:25 smithi016 ceph-mon[22010]: pgmap v1418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:25 smithi079 ceph-mon[23158]: pgmap v1418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:26 smithi016 conmon[32727]: 2022-03-21T17:59:26.796+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:27 smithi016 ceph-mon[22010]: pgmap v1419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:27 smithi016 conmon[32727]: 2022-03-21T17:59:27.819+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:27 smithi079 ceph-mon[23158]: pgmap v1419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:29.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:28 smithi016 conmon[32727]: 2022-03-21T17:59:28.795+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:29 smithi016 ceph-mon[22010]: pgmap v1420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:29 smithi016 conmon[32727]: 2022-03-21T17:59:29.801+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:29 smithi079 ceph-mon[23158]: pgmap v1420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:30 smithi016 conmon[32727]: 2022-03-21T17:59:30.796+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:31 smithi016 ceph-mon[22010]: pgmap v1421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:31 smithi016 conmon[32727]: 2022-03-21T17:59:31.751+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:31 smithi079 ceph-mon[23158]: pgmap v1421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:32.967 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:32 smithi016 conmon[32727]: 2022-03-21T17:59:32.702+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:33 smithi016 ceph-mon[22010]: pgmap v1422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:33 smithi016 conmon[32727]: 2022-03-21T17:59:33.740+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:33 smithi079 ceph-mon[23158]: pgmap v1422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:34 smithi016 conmon[32727]: 2022-03-21T17:59:34.737+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:35 smithi016 ceph-mon[22010]: pgmap v1423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:35 smithi016 conmon[32727]: 2022-03-21T17:59:35.733+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:35 smithi079 ceph-mon[23158]: pgmap v1423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:36 smithi016 conmon[32727]: 2022-03-21T17:59:36.779+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:37 smithi016 ceph-mon[22010]: pgmap v1424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:37 smithi016 conmon[32727]: 2022-03-21T17:59:37.817+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:37 smithi079 ceph-mon[23158]: pgmap v1424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:38 smithi016 conmon[32727]: 2022-03-21T17:59:38.806+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:39 smithi016 ceph-mon[22010]: pgmap v1425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:39 smithi016 conmon[32727]: 2022-03-21T17:59:39.819+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:39 smithi079 ceph-mon[23158]: pgmap v1425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:40 smithi016 conmon[32727]: 2022-03-21T17:59:40.773+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:41 smithi016 ceph-mon[22010]: pgmap v1426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:41 smithi016 conmon[32727]: 2022-03-21T17:59:41.758+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:41 smithi079 ceph-mon[23158]: pgmap v1426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:42 smithi016 conmon[32727]: 2022-03-21T17:59:42.781+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:43 smithi016 conmon[32727]: 2022-03-21T17:59:43.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:43 smithi016 ceph-mon[22010]: pgmap v1427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:43 smithi079 ceph-mon[23158]: pgmap v1427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:44 smithi016 conmon[32727]: 2022-03-21T17:59:44.770+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:46.004 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:45 smithi016 conmon[32727]: 2022-03-21T17:59:45.750+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:46 smithi016 ceph-mon[22010]: pgmap v1428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:46 smithi079 ceph-mon[23158]: pgmap v1428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:46 smithi016 conmon[32727]: 2022-03-21T17:59:46.787+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:48.010 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:47 smithi016 conmon[32727]: 2022-03-21T17:59:47.751+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:48 smithi016 ceph-mon[22010]: pgmap v1429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:48 smithi079 ceph-mon[23158]: pgmap v1429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:48 smithi016 conmon[32727]: 2022-03-21T17:59:48.798+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:49 smithi016 conmon[32727]: 2022-03-21T17:59:49.801+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:50 smithi016 ceph-mon[22010]: pgmap v1430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:50.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:50 smithi079 ceph-mon[23158]: pgmap v1430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:50 smithi016 conmon[32727]: 2022-03-21T17:59:50.787+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:52 smithi016 ceph-mon[22010]: pgmap v1431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:51 smithi016 conmon[32727]: 2022-03-21T17:59:51.795+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:52 smithi079 ceph-mon[23158]: pgmap v1431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:52 smithi016 conmon[32727]: 2022-03-21T17:59:52.800+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:54 smithi016 ceph-mon[22010]: pgmap v1432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:54.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:53 smithi016 conmon[32727]: 2022-03-21T17:59:53.843+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:54 smithi079 ceph-mon[23158]: pgmap v1432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:54.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:54.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:54 smithi016 conmon[32727]: 2022-03-21T17:59:54.844+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:56 smithi016 ceph-mon[22010]: pgmap v1433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:55 smithi016 conmon[32727]: 2022-03-21T17:59:55.826+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:56 smithi079 ceph-mon[23158]: pgmap v1433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T17:59:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:56 smithi016 conmon[32727]: 2022-03-21T17:59:56.839+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:57.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:58 smithi016 ceph-mon[22010]: pgmap v1434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:57 smithi016 conmon[32727]: 2022-03-21T17:59:57.839+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:58 smithi079 ceph-mon[23158]: pgmap v1434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T17:59:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:58 smithi016 conmon[32727]: 2022-03-21T17:59:58.818+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T17:59:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:59:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:59:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:59:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:59:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 17:59:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T17:59:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T17:59:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:59:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:59:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T17:59:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T17:59:59.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 17:59:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 17:59:59 smithi016 conmon[32727]: 2022-03-21T17:59:59.859+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: pgmap v1435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops 2022-03-21T18:00:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:00:00.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:00:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:00:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T18:00:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T18:00:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops 2022-03-21T18:00:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: pgmap v1435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops 2022-03-21T18:00:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:00:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T18:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 3h, current state unknown, last acting [] 2022-03-21T18:00:00.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2240 sec, osd.4 has slow ops 2022-03-21T18:00:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:00 smithi016 conmon[32727]: 2022-03-21T18:00:00.896+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:02 smithi016 ceph-mon[22010]: pgmap v1436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:01 smithi016 conmon[32727]: 2022-03-21T18:00:01.886+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:02 smithi079 ceph-mon[23158]: pgmap v1436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:02 smithi016 conmon[32727]: 2022-03-21T18:00:02.896+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:04 smithi016 ceph-mon[22010]: pgmap v1437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:03 smithi016 conmon[32727]: 2022-03-21T18:00:03.879+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:04 smithi079 ceph-mon[23158]: pgmap v1437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:04 smithi016 conmon[32727]: 2022-03-21T18:00:04.843+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:06 smithi016 ceph-mon[22010]: pgmap v1438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:05 smithi016 conmon[32727]: 2022-03-21T18:00:05.877+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:06 smithi079 ceph-mon[23158]: pgmap v1438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:06 smithi016 conmon[32727]: 2022-03-21T18:00:06.831+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:08 smithi016 ceph-mon[22010]: pgmap v1439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:08.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:07 smithi016 conmon[32727]: 2022-03-21T18:00:07.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:08 smithi079 ceph-mon[23158]: pgmap v1439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2251 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:08 smithi016 conmon[32727]: 2022-03-21T18:00:08.866+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2251 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:10.071 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:09 smithi016 conmon[32727]: 2022-03-21T18:00:09.817+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:10 smithi016 ceph-mon[22010]: pgmap v1440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:10 smithi079 ceph-mon[23158]: pgmap v1440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:11.077 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:10 smithi016 conmon[32727]: 2022-03-21T18:00:10.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:12.078 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:11 smithi016 conmon[32727]: 2022-03-21T18:00:11.753+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:12 smithi016 ceph-mon[22010]: pgmap v1441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:12 smithi079 ceph-mon[23158]: pgmap v1441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:13.083 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:12 smithi016 conmon[32727]: 2022-03-21T18:00:12.760+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:14.084 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:13 smithi016 conmon[32727]: 2022-03-21T18:00:13.789+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:14 smithi016 ceph-mon[22010]: pgmap v1442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2256 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:14 smithi079 ceph-mon[23158]: pgmap v1442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2256 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:15.089 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:14 smithi016 conmon[32727]: 2022-03-21T18:00:14.830+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:16.091 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:15 smithi016 conmon[32727]: 2022-03-21T18:00:15.814+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:16 smithi016 ceph-mon[22010]: pgmap v1443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:16 smithi079 ceph-mon[23158]: pgmap v1443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:16 smithi016 conmon[32727]: 2022-03-21T18:00:16.846+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:18.098 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:17 smithi016 conmon[32727]: 2022-03-21T18:00:17.830+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:18 smithi016 ceph-mon[22010]: pgmap v1444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:18 smithi079 ceph-mon[23158]: pgmap v1444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:19.100 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:18 smithi016 conmon[32727]: 2022-03-21T18:00:18.784+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:20.101 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:19 smithi016 conmon[32727]: 2022-03-21T18:00:19.741+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:20 smithi016 ceph-mon[22010]: pgmap v1445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:20 smithi079 ceph-mon[23158]: pgmap v1445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:21.106 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:20 smithi016 conmon[32727]: 2022-03-21T18:00:20.772+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:21.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:21.404 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:22.107 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:21 smithi016 conmon[32727]: 2022-03-21T18:00:21.808+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:22 smithi016 ceph-mon[22010]: pgmap v1446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:00:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:00:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:00:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:22 smithi079 ceph-mon[23158]: pgmap v1446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:00:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:00:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:00:23.112 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:22 smithi016 conmon[32727]: 2022-03-21T18:00:22.772+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:24.117 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:23 smithi016 conmon[32727]: 2022-03-21T18:00:23.799+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:24 smithi016 ceph-mon[22010]: pgmap v1447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:24 smithi079 ceph-mon[23158]: pgmap v1447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:00:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:24.787 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: 2022-03-21T18:00:24.461+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:24.787 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:00:24.788 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:24.788 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: yield 2022-03-21T18:00:24.788 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:24.789 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:24.789 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:24.789 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:00:24.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:24.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:00:24.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:24.791 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:00:24.791 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:24.791 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:00:24.792 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:24.792 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:00:24.792 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:24.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:24.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:25.118 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:24 smithi016 conmon[32727]: 2022-03-21T18:00:24.784+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:26.120 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:25 smithi016 conmon[32727]: 2022-03-21T18:00:25.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:26 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:00:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:00:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:26.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:00:26.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:00:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:00:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:00:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:26.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:26.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:26 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:00:26.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:26 smithi016 ceph-mon[22010]: pgmap v1448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:26.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:26 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:00:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:00:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:00:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:00:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:00:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:00:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:26 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:00:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:26 smithi079 ceph-mon[23158]: pgmap v1448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:27.125 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:26 smithi016 conmon[32727]: 2022-03-21T18:00:26.777+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:27.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:27.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 2022-03-21T18:00:27.533+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:27.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:00:27.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:27.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: yield 2022-03-21T18:00:27.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:27.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:27.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:27.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:00:27.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:27.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:00:27.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:27.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:00:27.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:27.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:00:27.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:27.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:00:27.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:27.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:27.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:27.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 2022-03-21T18:00:27.534+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:00:27.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:00:27.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:27.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: yield 2022-03-21T18:00:27.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:27.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:27.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:27.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:00:27.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:27.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:00:27.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:27.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:00:27.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:27.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:00:27.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:27.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:00:27.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:27.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:27.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:27.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 2022-03-21T18:00:27.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:00:27.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: 2022-03-21T18:00:27.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:00:27.829 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:00:27.829 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:00:27.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:00:27.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:00:27.831 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:00:27.831 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:00:27.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:00:27.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:00:27.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:00:27.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:00:27.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:00:27.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:00:27.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:00:27.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:00:27.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:00:27.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:00:27.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:00:27.839 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:00:27.840 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:00:27.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:00:27.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:00:27.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise 2022-03-21T18:00:27.842 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:00:27.842 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:00:27.843 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:00:27.843 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:00:27.844 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:28.127 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:27 smithi016 conmon[32727]: 2022-03-21T18:00:27.805+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:28 smithi016 ceph-mon[22010]: pgmap v1449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:28 smithi079 ceph-mon[23158]: pgmap v1449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:29.128 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:28 smithi016 conmon[32727]: 2022-03-21T18:00:28.833+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:00:29.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:29.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:29.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:29.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:00:29.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:29.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:00:29.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:29.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:00:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:29.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:00:29.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:29.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:29 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:00:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:29.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:00:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:29.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:00:29.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:00:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:00:29.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:29.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:00:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:00:29.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:00:29.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:00:29.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:00:29.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:00:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:00:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:00:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:00:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:00:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:00:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:00:29.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:00:29.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:00:29.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:00:29.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:00:29.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:00:29.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:00:29.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:00:29.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:00:29.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:00:29.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:00:29.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:00:29.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:00:29.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:00:29.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:00:29.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:00:29.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:00:29.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:00:29.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:00:29.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:29.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:00:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:00:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:00:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:00:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:00:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:29.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:29.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:29 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:00:29.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:00:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:00:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:00:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:00:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:00:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:00:29.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:00:29.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:00:29.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:00:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:00:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:00:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:00:29.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:00:29.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:00:29.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:00:29.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:00:29.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:00:29.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:00:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:00:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:00:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:00:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:00:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:00:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:00:29.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:00:29.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:00:29.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:00:29.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:00:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:00:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:00:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:00:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:00:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:00:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:00:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:00:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:00:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:00:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:00:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:00:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:00:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:00:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:00:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:00:29.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:00:29.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:00:29.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:00:29.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:00:29.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:29.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:30.129 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:29 smithi016 conmon[32727]: 2022-03-21T18:00:29.861+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:30 smithi016 ceph-mon[22010]: pgmap v1450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:30 smithi079 ceph-mon[23158]: pgmap v1450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:31.135 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:30 smithi016 conmon[32727]: 2022-03-21T18:00:30.848+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:32.136 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:31 smithi016 conmon[32727]: 2022-03-21T18:00:31.845+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:32 smithi016 ceph-mon[22010]: pgmap v1451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:32 smithi079 ceph-mon[23158]: pgmap v1451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:33.141 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:32 smithi016 conmon[32727]: 2022-03-21T18:00:32.852+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:34.143 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:33 smithi016 conmon[32727]: 2022-03-21T18:00:33.812+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:34 smithi016 ceph-mon[22010]: pgmap v1452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:34 smithi079 ceph-mon[23158]: pgmap v1452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:34.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:34.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:35.148 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:34 smithi016 conmon[32727]: 2022-03-21T18:00:34.835+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:36.149 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:35 smithi016 conmon[32727]: 2022-03-21T18:00:35.852+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:36 smithi016 ceph-mon[22010]: pgmap v1453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:36 smithi079 ceph-mon[23158]: pgmap v1453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:37.155 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:36 smithi016 conmon[32727]: 2022-03-21T18:00:36.894+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:38.156 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:37 smithi016 conmon[32727]: 2022-03-21T18:00:37.845+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:38 smithi016 ceph-mon[22010]: pgmap v1454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:38 smithi079 ceph-mon[23158]: pgmap v1454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:39.157 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:38 smithi016 conmon[32727]: 2022-03-21T18:00:38.810+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:40.159 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:39 smithi016 conmon[32727]: 2022-03-21T18:00:39.761+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:40 smithi016 ceph-mon[22010]: pgmap v1455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:40 smithi079 ceph-mon[23158]: pgmap v1455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:40 smithi016 conmon[32727]: 2022-03-21T18:00:40.745+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:41 smithi016 conmon[32727]: 2022-03-21T18:00:41.734+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:42 smithi079 ceph-mon[23158]: pgmap v1456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:42 smithi016 ceph-mon[22010]: pgmap v1456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:42.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:42 smithi016 conmon[32727]: 2022-03-21T18:00:42.688+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:43 smithi016 conmon[32727]: 2022-03-21T18:00:43.692+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:44 smithi079 ceph-mon[23158]: pgmap v1457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:44 smithi016 ceph-mon[22010]: pgmap v1457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:44 smithi016 conmon[32727]: 2022-03-21T18:00:44.734+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:45 smithi016 conmon[32727]: 2022-03-21T18:00:45.766+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:46 smithi079 ceph-mon[23158]: pgmap v1458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:46 smithi016 ceph-mon[22010]: pgmap v1458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:46 smithi016 conmon[32727]: 2022-03-21T18:00:46.755+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:47 smithi016 conmon[32727]: 2022-03-21T18:00:47.723+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:48 smithi079 ceph-mon[23158]: pgmap v1459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:48 smithi016 ceph-mon[22010]: pgmap v1459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:48 smithi016 conmon[32727]: 2022-03-21T18:00:48.730+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:49 smithi016 conmon[32727]: 2022-03-21T18:00:49.721+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:50 smithi016 ceph-mon[22010]: pgmap v1460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:50 smithi079 ceph-mon[23158]: pgmap v1460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:50 smithi016 conmon[32727]: 2022-03-21T18:00:50.702+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:51 smithi016 conmon[32727]: 2022-03-21T18:00:51.679+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:52 smithi016 ceph-mon[22010]: pgmap v1461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:52 smithi079 ceph-mon[23158]: pgmap v1461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:52.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:52 smithi016 conmon[32727]: 2022-03-21T18:00:52.696+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:53 smithi016 conmon[32727]: 2022-03-21T18:00:53.722+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:54 smithi016 ceph-mon[22010]: pgmap v1462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:54 smithi079 ceph-mon[23158]: pgmap v1462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:54 smithi016 conmon[32727]: 2022-03-21T18:00:54.689+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:55 smithi016 conmon[32727]: 2022-03-21T18:00:55.704+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:56 smithi016 ceph-mon[22010]: pgmap v1463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:56 smithi079 ceph-mon[23158]: pgmap v1463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:00:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:56 smithi016 conmon[32727]: 2022-03-21T18:00:56.684+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:57 smithi016 conmon[32727]: 2022-03-21T18:00:57.727+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: pgmap v1464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:00:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:00:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:00:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:00:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: pgmap v1464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:00:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:00:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:00:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:00:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:00:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:58 smithi016 conmon[32727]: 2022-03-21T18:00:58.691+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:00:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:00:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:00:59 smithi016 conmon[32727]: 2022-03-21T18:00:59.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:00:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:00:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:00 smithi079 ceph-mon[23158]: pgmap v1465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:00 smithi016 ceph-mon[22010]: pgmap v1465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:00 smithi016 conmon[32727]: 2022-03-21T18:01:00.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:01 smithi016 conmon[32727]: 2022-03-21T18:01:01.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:02 smithi079 ceph-mon[23158]: pgmap v1466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:02 smithi016 ceph-mon[22010]: pgmap v1466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:02 smithi016 conmon[32727]: 2022-03-21T18:01:02.547+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:03 smithi016 conmon[32727]: 2022-03-21T18:01:03.549+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:04 smithi079 ceph-mon[23158]: pgmap v1467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:04 smithi016 ceph-mon[22010]: pgmap v1467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:04 smithi016 conmon[32727]: 2022-03-21T18:01:04.540+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:05 smithi016 conmon[32727]: 2022-03-21T18:01:05.537+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:06 smithi079 ceph-mon[23158]: pgmap v1468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:06 smithi016 conmon[32727]: 2022-03-21T18:01:06.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:06 smithi016 ceph-mon[22010]: pgmap v1468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:07 smithi016 conmon[32727]: 2022-03-21T18:01:07.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:08 smithi079 ceph-mon[23158]: pgmap v1469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:08 smithi016 ceph-mon[22010]: pgmap v1469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:08.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:08 smithi016 conmon[32727]: 2022-03-21T18:01:08.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:09 smithi016 conmon[32727]: 2022-03-21T18:01:09.546+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:10 smithi079 ceph-mon[23158]: pgmap v1470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:10 smithi016 ceph-mon[22010]: pgmap v1470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:10.921 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:10 smithi016 conmon[32727]: 2022-03-21T18:01:10.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:11 smithi016 conmon[32727]: 2022-03-21T18:01:11.590+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:12 smithi079 ceph-mon[23158]: pgmap v1471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:12 smithi016 ceph-mon[22010]: pgmap v1471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:12 smithi016 conmon[32727]: 2022-03-21T18:01:12.573+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:13 smithi079 ceph-mon[23158]: pgmap v1472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:13 smithi016 ceph-mon[22010]: pgmap v1472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:13.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:13.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:13 smithi016 conmon[32727]: 2022-03-21T18:01:13.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:14 smithi016 conmon[32727]: 2022-03-21T18:01:14.568+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:15 smithi079 ceph-mon[23158]: pgmap v1473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:15 smithi016 ceph-mon[22010]: pgmap v1473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:15.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:15 smithi016 conmon[32727]: 2022-03-21T18:01:15.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:16 smithi016 conmon[32727]: 2022-03-21T18:01:16.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:17 smithi016 conmon[32727]: 2022-03-21T18:01:17.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:17 smithi016 ceph-mon[22010]: pgmap v1474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:17 smithi079 ceph-mon[23158]: pgmap v1474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:18.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:18 smithi016 conmon[32727]: 2022-03-21T18:01:18.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:19 smithi016 conmon[32727]: 2022-03-21T18:01:19.700+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:19 smithi016 ceph-mon[22010]: pgmap v1475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:19 smithi079 ceph-mon[23158]: pgmap v1475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:20 smithi016 conmon[32727]: 2022-03-21T18:01:20.651+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:21 smithi016 conmon[32727]: 2022-03-21T18:01:21.610+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:21 smithi016 ceph-mon[22010]: pgmap v1476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:21 smithi079 ceph-mon[23158]: pgmap v1476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:22 smithi016 conmon[32727]: 2022-03-21T18:01:22.622+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:23.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:23 smithi016 conmon[32727]: 2022-03-21T18:01:23.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:23.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:23.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:23 smithi016 ceph-mon[22010]: pgmap v1477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:23.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:23 smithi079 ceph-mon[23158]: pgmap v1477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:24 smithi016 conmon[32727]: 2022-03-21T18:01:24.576+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:25 smithi016 conmon[32727]: 2022-03-21T18:01:25.604+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:25 smithi016 ceph-mon[22010]: pgmap v1478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:25 smithi079 ceph-mon[23158]: pgmap v1478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:26.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:26 smithi016 conmon[32727]: 2022-03-21T18:01:26.563+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:27 smithi016 ceph-mon[22010]: pgmap v1479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:27 smithi016 conmon[32727]: 2022-03-21T18:01:27.560+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:27 smithi079 ceph-mon[23158]: pgmap v1479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:28.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: 2022-03-21T18:01:28.141+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:01:28.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:28.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: yield 2022-03-21T18:01:28.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:28.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:28.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:28.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:01:28.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:28.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:01:28.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:28.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:01:28.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:28.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:01:28.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:28.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:01:28.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:28.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:28.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:28 smithi016 conmon[32727]: 2022-03-21T18:01:28.601+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:01:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:01:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:01:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:01:28.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:28.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:01:28.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:28.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:28.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:28.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:01:28.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:28.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:01:28.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:28.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:01:28.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:28.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:28.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:28.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:01:28.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:28.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:28.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:01:28.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:01:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:01:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:01:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:01:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:01:28.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:28.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:01:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:01:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:01:28.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:28.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:28.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:28.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:01:28.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:28.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:28.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:28.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:01:28.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:29 smithi016 conmon[32727]: 2022-03-21T18:01:29.613+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:29 smithi016 ceph-mon[22010]: pgmap v1480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:29 smithi079 ceph-mon[23158]: pgmap v1480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:30 smithi016 conmon[32727]: 2022-03-21T18:01:30.585+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:31.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 2022-03-21T18:01:31.213+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:01:31.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: yield 2022-03-21T18:01:31.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:01:31.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:01:31.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:01:31.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:01:31.523 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 2022-03-21T18:01:31.214+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:01:31.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:01:31.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: yield 2022-03-21T18:01:31.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.527 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.527 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:01:31.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:01:31.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:01:31.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:01:31.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 2022-03-21T18:01:31.533 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:01:31.533 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: 2022-03-21T18:01:31.533 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:01:31.534 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:01:31.534 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:01:31.535 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:01:31.535 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:01:31.536 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:01:31.536 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:01:31.536 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:01:31.537 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:01:31.537 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:01:31.537 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:01:31.538 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:01:31.538 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:01:31.539 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:01:31.539 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:01:31.539 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:01:31.540 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:01:31.540 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:01:31.540 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:01:31.541 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:01:31.541 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:01:31.542 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:01:31.542 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise 2022-03-21T18:01:31.542 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:01:31.543 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:01:31.543 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:01:31.543 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:01:31.544 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:31 smithi016 conmon[32727]: 2022-03-21T18:01:31.601+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:31 smithi016 ceph-mon[22010]: pgmap v1481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:01:31.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:01:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:01:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:01:31.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:01:31.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:01:31.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:01:31.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:01:31.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:01:31.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:01:31.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:01:31.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:01:31.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:01:31.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:01:31.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:01:31.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:01:31.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:01:31.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:01:31.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:01:31.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:01:31.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:01:31.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:01:31.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:01:31.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:01:31.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:01:31.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:01:31.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:01:31.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:01:31.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:01:31.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:01:31.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:01:31.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:01:31.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:01:31.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:01:31.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:01:31.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:01:31.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:01:31.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:01:31.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:01:31.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:01:31.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:31 smithi079 ceph-mon[23158]: pgmap v1481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:01:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:01:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:01:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:01:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:01:31.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:01:31.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:01:31.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:01:31.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:01:31.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:01:31.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:01:31.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:01:31.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:01:31.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:01:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:01:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:01:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:01:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:01:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:01:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:01:31.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:01:31.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:01:31.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:31.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:01:31.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:01:31.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:01:31.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:01:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:01:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:01:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:01:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:01:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:01:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:01:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:01:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:01:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:01:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:01:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:01:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:01:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:01:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:01:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:01:31.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:01:31.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:01:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:01:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:01:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:01:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:01:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:01:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:01:31.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:01:31.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:01:32.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:01:32.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:01:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:32 smithi016 conmon[32727]: 2022-03-21T18:01:32.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:33 smithi016 ceph-mon[22010]: pgmap v1482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:33.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:33.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:33 smithi016 conmon[32727]: 2022-03-21T18:01:33.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:33 smithi079 ceph-mon[23158]: pgmap v1482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:34.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:34 smithi016 conmon[32727]: 2022-03-21T18:01:34.593+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:34.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:35 smithi016 ceph-mon[22010]: pgmap v1483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:35 smithi016 conmon[32727]: 2022-03-21T18:01:35.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:35 smithi079 ceph-mon[23158]: pgmap v1483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:36 smithi016 conmon[32727]: 2022-03-21T18:01:36.505+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:37 smithi016 ceph-mon[22010]: pgmap v1484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:37 smithi016 conmon[32727]: 2022-03-21T18:01:37.503+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:37 smithi079 ceph-mon[23158]: pgmap v1484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:38 smithi016 conmon[32727]: 2022-03-21T18:01:38.544+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:39 smithi016 conmon[32727]: 2022-03-21T18:01:39.527+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:39 smithi016 ceph-mon[22010]: pgmap v1485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:39 smithi079 ceph-mon[23158]: pgmap v1485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:40 smithi016 conmon[32727]: 2022-03-21T18:01:40.490+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:41 smithi016 conmon[32727]: 2022-03-21T18:01:41.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:41 smithi016 ceph-mon[22010]: pgmap v1486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:41 smithi079 ceph-mon[23158]: pgmap v1486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:42 smithi016 conmon[32727]: 2022-03-21T18:01:42.546+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:43 smithi016 conmon[32727]: 2022-03-21T18:01:43.565+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:43 smithi016 ceph-mon[22010]: pgmap v1487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:43.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:43 smithi079 ceph-mon[23158]: pgmap v1487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:44 smithi016 conmon[32727]: 2022-03-21T18:01:44.571+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:45 smithi016 ceph-mon[22010]: pgmap v1488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:45 smithi016 conmon[32727]: 2022-03-21T18:01:45.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:45 smithi079 ceph-mon[23158]: pgmap v1488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:46 smithi016 conmon[32727]: 2022-03-21T18:01:46.561+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:47 smithi016 ceph-mon[22010]: pgmap v1489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:47 smithi016 conmon[32727]: 2022-03-21T18:01:47.573+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:47 smithi079 ceph-mon[23158]: pgmap v1489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:48 smithi016 conmon[32727]: 2022-03-21T18:01:48.604+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:49.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:49 smithi016 ceph-mon[22010]: pgmap v1490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:49 smithi016 conmon[32727]: 2022-03-21T18:01:49.582+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:49 smithi079 ceph-mon[23158]: pgmap v1490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:50 smithi016 conmon[32727]: 2022-03-21T18:01:50.605+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:51 smithi016 ceph-mon[22010]: pgmap v1491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:51 smithi016 conmon[32727]: 2022-03-21T18:01:51.628+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:51 smithi079 ceph-mon[23158]: pgmap v1491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:52.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:52 smithi016 conmon[32727]: 2022-03-21T18:01:52.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:53 smithi016 ceph-mon[22010]: pgmap v1492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:53.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:53 smithi016 conmon[32727]: 2022-03-21T18:01:53.636+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:53 smithi079 ceph-mon[23158]: pgmap v1492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:54 smithi016 conmon[32727]: 2022-03-21T18:01:54.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:55 smithi016 conmon[32727]: 2022-03-21T18:01:55.673+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:55 smithi016 ceph-mon[22010]: pgmap v1493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:55 smithi079 ceph-mon[23158]: pgmap v1493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:01:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:56 smithi016 conmon[32727]: 2022-03-21T18:01:56.677+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:57 smithi016 ceph-mon[22010]: pgmap v1494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:57.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:57 smithi016 conmon[32727]: 2022-03-21T18:01:57.668+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:57 smithi079 ceph-mon[23158]: pgmap v1494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:58 smithi016 conmon[32727]: 2022-03-21T18:01:58.671+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:01:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:01:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:01:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:01:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:01:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:01:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:01:59 smithi016 conmon[32727]: 2022-03-21T18:01:59.627+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:01:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:01:59 smithi016 ceph-mon[22010]: pgmap v1495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:01:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:01:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:01:59 smithi079 ceph-mon[23158]: pgmap v1495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:00 smithi016 conmon[32727]: 2022-03-21T18:02:00.646+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:01 smithi016 conmon[32727]: 2022-03-21T18:02:01.687+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:01 smithi016 ceph-mon[22010]: pgmap v1496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:01 smithi079 ceph-mon[23158]: pgmap v1496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:02 smithi016 conmon[32727]: 2022-03-21T18:02:02.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:03 smithi016 ceph-mon[22010]: pgmap v1497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:03 smithi016 conmon[32727]: 2022-03-21T18:02:03.624+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:03 smithi079 ceph-mon[23158]: pgmap v1497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:04 smithi016 conmon[32727]: 2022-03-21T18:02:04.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:05 smithi016 ceph-mon[22010]: pgmap v1498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:05.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:05 smithi016 conmon[32727]: 2022-03-21T18:02:05.632+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:05 smithi079 ceph-mon[23158]: pgmap v1498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:06 smithi016 conmon[32727]: 2022-03-21T18:02:06.593+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:07 smithi016 ceph-mon[22010]: pgmap v1499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:07.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:07 smithi016 conmon[32727]: 2022-03-21T18:02:07.609+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:07 smithi079 ceph-mon[23158]: pgmap v1499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:08 smithi016 conmon[32727]: 2022-03-21T18:02:08.573+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:09 smithi016 ceph-mon[22010]: pgmap v1500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:09.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:09 smithi016 conmon[32727]: 2022-03-21T18:02:09.612+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:09 smithi079 ceph-mon[23158]: pgmap v1500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:10 smithi016 conmon[32727]: 2022-03-21T18:02:10.611+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:11 smithi016 conmon[32727]: 2022-03-21T18:02:11.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:11 smithi016 ceph-mon[22010]: pgmap v1501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:11 smithi079 ceph-mon[23158]: pgmap v1501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:12 smithi016 conmon[32727]: 2022-03-21T18:02:12.664+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:13 smithi016 ceph-mon[22010]: pgmap v1502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:13 smithi016 conmon[32727]: 2022-03-21T18:02:13.682+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:13 smithi079 ceph-mon[23158]: pgmap v1502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:14 smithi016 conmon[32727]: 2022-03-21T18:02:14.710+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:15 smithi016 ceph-mon[22010]: pgmap v1503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:15.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:15 smithi016 conmon[32727]: 2022-03-21T18:02:15.717+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:15 smithi079 ceph-mon[23158]: pgmap v1503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:16 smithi016 conmon[32727]: 2022-03-21T18:02:16.760+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:17 smithi079 ceph-mon[23158]: pgmap v1504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:17 smithi016 ceph-mon[22010]: pgmap v1504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:17 smithi016 conmon[32727]: 2022-03-21T18:02:17.745+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:18 smithi016 conmon[32727]: 2022-03-21T18:02:18.716+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:19 smithi079 ceph-mon[23158]: pgmap v1505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:19 smithi016 ceph-mon[22010]: pgmap v1505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:19 smithi016 conmon[32727]: 2022-03-21T18:02:19.732+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:20 smithi016 conmon[32727]: 2022-03-21T18:02:20.720+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:21 smithi079 ceph-mon[23158]: pgmap v1506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:21 smithi016 ceph-mon[22010]: pgmap v1506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:21 smithi016 conmon[32727]: 2022-03-21T18:02:21.739+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:22.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:22 smithi016 conmon[32727]: 2022-03-21T18:02:22.776+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:23 smithi079 ceph-mon[23158]: pgmap v1507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:23 smithi016 ceph-mon[22010]: pgmap v1507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:24.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:23 smithi016 conmon[32727]: 2022-03-21T18:02:23.781+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:24 smithi016 conmon[32727]: 2022-03-21T18:02:24.763+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:25 smithi079 ceph-mon[23158]: pgmap v1508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:25 smithi016 ceph-mon[22010]: pgmap v1508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:25 smithi016 conmon[32727]: 2022-03-21T18:02:25.801+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:26 smithi016 conmon[32727]: 2022-03-21T18:02:26.779+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:27 smithi079 ceph-mon[23158]: pgmap v1509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:27 smithi016 ceph-mon[22010]: pgmap v1509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:28.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:28.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:27 smithi016 conmon[32727]: 2022-03-21T18:02:27.797+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:28 smithi016 conmon[32727]: 2022-03-21T18:02:28.750+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:29 smithi079 ceph-mon[23158]: pgmap v1510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:29 smithi016 conmon[32727]: 2022-03-21T18:02:29.736+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:29 smithi016 ceph-mon[22010]: pgmap v1510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:30 smithi016 conmon[32727]: 2022-03-21T18:02:30.752+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:31 smithi079 ceph-mon[23158]: pgmap v1511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:02:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:02:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:02:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:31 smithi016 conmon[32727]: 2022-03-21T18:02:31.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:31 smithi016 ceph-mon[22010]: pgmap v1511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:02:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:02:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:02:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:32 smithi016 conmon[32727]: 2022-03-21T18:02:32.797+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:33 smithi079 ceph-mon[23158]: pgmap v1512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:33 smithi016 ceph-mon[22010]: pgmap v1512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:33 smithi016 conmon[32727]: 2022-03-21T18:02:33.782+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:34.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: 2022-03-21T18:02:34.317+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:34.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:02:34.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:34.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: yield 2022-03-21T18:02:34.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:34.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:34.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:34.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:02:34.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:34.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:02:34.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:34.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:02:34.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:34.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:02:34.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:34.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:02:34.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:34.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:34.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:34 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:02:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:02:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:34.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:34.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:34.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:02:34.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:34.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:02:34.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:34.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:02:34.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:34.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:34.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:34.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:02:34.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:34.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:34.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:34.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:34 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:02:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:34 smithi016 conmon[32727]: 2022-03-21T18:02:34.827+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:34 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:02:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:02:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:02:35.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:35.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:02:35.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:35.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:02:35.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:35.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:35.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:35.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:02:35.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:35.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:35.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:35.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:34 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:02:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:35 smithi016 ceph-mon[22010]: pgmap v1513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:36.287 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:36.288 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:35 smithi016 conmon[32727]: 2022-03-21T18:02:35.781+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:36.289 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:35 smithi079 ceph-mon[23158]: pgmap v1513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:36.290 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:36 smithi016 conmon[32727]: 2022-03-21T18:02:36.830+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:37.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 2022-03-21T18:02:37.389+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:37.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:02:37.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:37.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: yield 2022-03-21T18:02:37.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:37.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:37.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:37.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:02:37.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:37.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:02:37.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:37.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:02:37.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:37.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:02:37.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:37.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:02:37.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:37.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:37.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:37.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 2022-03-21T18:02:37.390+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:02:37.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:02:37.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:37.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: yield 2022-03-21T18:02:37.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:37.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:37.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:37.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:02:37.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:37.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:02:37.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:37.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:02:37.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:37.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:02:37.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:37.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:02:37.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:37.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:37.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:37.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 2022-03-21T18:02:37.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:02:37.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: 2022-03-21T18:02:37.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:02:37.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:02:37.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:02:37.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:02:37.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:02:37.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:02:37.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:02:37.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:02:37.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:02:37.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:02:37.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:02:37.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:02:37.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:02:37.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:02:37.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:02:37.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:02:37.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:02:37.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:02:37.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:02:37.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:02:37.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:02:37.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:02:37.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise 2022-03-21T18:02:37.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:02:37.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:02:37.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:02:37.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:02:37.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:37 smithi016 conmon[32727]: 2022-03-21T18:02:37.854+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:37 smithi016 ceph-mon[22010]: pgmap v1514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:38.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:02:38.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:38.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:38.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:38.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:02:38.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:38.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:02:38.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:38.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:02:38.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:38.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:38.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:38.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:02:38.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:38.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:38.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:37 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:02:38.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:38.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:02:38.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:38.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:38.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:38.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:02:38.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:38.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:02:38.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:38.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:02:38.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:38.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:38.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:38.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:02:38.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:38.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:38.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:02:38.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:02:38.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:02:38.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:02:38.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:02:38.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:02:38.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:02:38.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:02:38.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:02:38.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:02:38.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:02:38.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:02:38.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:02:38.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:02:38.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:02:38.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:02:38.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:02:38.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:02:38.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:02:38.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:02:38.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:02:38.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:02:38.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:02:38.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:02:38.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:02:38.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:02:38.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:02:38.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:02:38.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:02:38.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:37 smithi079 ceph-mon[23158]: pgmap v1514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:38.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:02:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:02:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:02:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:02:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:02:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:37 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:02:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:02:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:02:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:02:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:02:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:02:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:02:38.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:02:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:02:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:02:38.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:02:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:02:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:02:38.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:02:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:02:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:02:38.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:02:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:02:38.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:02:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:02:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:02:38.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:02:38.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:02:38.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:02:38.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:02:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:02:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:02:38.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:02:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:02:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:02:38.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:02:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:02:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:02:38.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:02:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:02:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:02:38.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:02:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:02:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:02:38.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:02:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:02:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:02:38.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:02:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:02:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:02:38.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:02:38.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:02:38.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:02:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:38 smithi016 conmon[32727]: 2022-03-21T18:02:38.813+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:39 smithi016 ceph-mon[22010]: pgmap v1515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:39 smithi016 conmon[32727]: 2022-03-21T18:02:39.818+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:39 smithi079 ceph-mon[23158]: pgmap v1515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:40 smithi016 conmon[32727]: 2022-03-21T18:02:40.863+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:41 smithi016 ceph-mon[22010]: pgmap v1516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:41 smithi016 conmon[32727]: 2022-03-21T18:02:41.844+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:41 smithi079 ceph-mon[23158]: pgmap v1516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:42 smithi016 conmon[32727]: 2022-03-21T18:02:42.804+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:43 smithi016 ceph-mon[22010]: pgmap v1517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:44.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:43 smithi016 conmon[32727]: 2022-03-21T18:02:43.850+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:43 smithi079 ceph-mon[23158]: pgmap v1517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:44 smithi016 conmon[32727]: 2022-03-21T18:02:44.865+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:45 smithi016 ceph-mon[22010]: pgmap v1518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:45 smithi016 conmon[32727]: 2022-03-21T18:02:45.859+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:45 smithi079 ceph-mon[23158]: pgmap v1518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:46 smithi016 conmon[32727]: 2022-03-21T18:02:46.893+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:47 smithi016 ceph-mon[22010]: pgmap v1519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:47 smithi016 conmon[32727]: 2022-03-21T18:02:47.847+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:47 smithi079 ceph-mon[23158]: pgmap v1519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:48 smithi016 conmon[32727]: 2022-03-21T18:02:48.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2411 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2411 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:49 smithi016 ceph-mon[22010]: pgmap v1520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:49 smithi016 conmon[32727]: 2022-03-21T18:02:49.905+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:49 smithi079 ceph-mon[23158]: pgmap v1520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:50 smithi016 conmon[32727]: 2022-03-21T18:02:50.947+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:51 smithi016 ceph-mon[22010]: pgmap v1521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:51 smithi016 conmon[32727]: 2022-03-21T18:02:51.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:51 smithi079 ceph-mon[23158]: pgmap v1521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:52 smithi016 conmon[32727]: 2022-03-21T18:02:52.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:53 smithi016 conmon[32727]: 2022-03-21T18:02:53.994+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:53 smithi016 ceph-mon[22010]: pgmap v1522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2416 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:53 smithi079 ceph-mon[23158]: pgmap v1522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2416 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:54 smithi016 conmon[32727]: 2022-03-21T18:02:54.963+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:55 smithi016 conmon[32727]: 2022-03-21T18:02:55.929+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:55 smithi016 ceph-mon[22010]: pgmap v1523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:55 smithi079 ceph-mon[23158]: pgmap v1523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:02:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:56 smithi016 conmon[32727]: 2022-03-21T18:02:56.881+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:57 smithi016 conmon[32727]: 2022-03-21T18:02:57.906+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:57 smithi016 ceph-mon[22010]: pgmap v1524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:57 smithi079 ceph-mon[23158]: pgmap v1524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:02:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:58 smithi016 conmon[32727]: 2022-03-21T18:02:58.862+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:02:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:02:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:02:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:02:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:02:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:02:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:02:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:02:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:59 smithi016 ceph-mon[22010]: pgmap v1525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:02:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:00.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:02:59 smithi016 conmon[32727]: 2022-03-21T18:02:59.905+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:59 smithi079 ceph-mon[23158]: pgmap v1525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:02:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:00 smithi016 conmon[32727]: 2022-03-21T18:03:00.892+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:01 smithi016 ceph-mon[22010]: pgmap v1526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:02.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:01 smithi016 conmon[32727]: 2022-03-21T18:03:01.853+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:01 smithi079 ceph-mon[23158]: pgmap v1526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:03.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:02 smithi016 conmon[32727]: 2022-03-21T18:03:02.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:03 smithi016 ceph-mon[22010]: pgmap v1527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:04.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2426 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:04.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:03 smithi016 conmon[32727]: 2022-03-21T18:03:03.832+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:03 smithi079 ceph-mon[23158]: pgmap v1527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2426 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:04 smithi016 conmon[32727]: 2022-03-21T18:03:04.865+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:05 smithi016 ceph-mon[22010]: pgmap v1528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:05 smithi016 conmon[32727]: 2022-03-21T18:03:05.828+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:05 smithi079 ceph-mon[23158]: pgmap v1528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:07.266 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:06 smithi016 conmon[32727]: 2022-03-21T18:03:06.782+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:07.267 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:07 smithi016 ceph-mon[22010]: pgmap v1529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:08.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:08.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:07 smithi016 conmon[32727]: 2022-03-21T18:03:07.740+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:07 smithi079 ceph-mon[23158]: pgmap v1529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:08 smithi016 conmon[32727]: 2022-03-21T18:03:08.719+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:09 smithi016 ceph-mon[22010]: pgmap v1530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:09 smithi016 conmon[32727]: 2022-03-21T18:03:09.682+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:09 smithi079 ceph-mon[23158]: pgmap v1530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:10.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:10 smithi016 conmon[32727]: 2022-03-21T18:03:10.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:11 smithi016 ceph-mon[22010]: pgmap v1531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:11 smithi016 conmon[32727]: 2022-03-21T18:03:11.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:11 smithi079 ceph-mon[23158]: pgmap v1531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:12 smithi016 conmon[32727]: 2022-03-21T18:03:12.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:13 smithi016 ceph-mon[22010]: pgmap v1532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:13 smithi016 conmon[32727]: 2022-03-21T18:03:13.628+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:13 smithi079 ceph-mon[23158]: pgmap v1532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:14 smithi016 conmon[32727]: 2022-03-21T18:03:14.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:15 smithi016 ceph-mon[22010]: pgmap v1533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:15 smithi016 conmon[32727]: 2022-03-21T18:03:15.669+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:15 smithi079 ceph-mon[23158]: pgmap v1533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:16 smithi016 conmon[32727]: 2022-03-21T18:03:16.670+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:17 smithi016 conmon[32727]: 2022-03-21T18:03:17.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:17 smithi016 ceph-mon[22010]: pgmap v1534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:17 smithi079 ceph-mon[23158]: pgmap v1534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:18 smithi016 conmon[32727]: 2022-03-21T18:03:18.611+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:19 smithi016 conmon[32727]: 2022-03-21T18:03:19.606+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:19 smithi016 ceph-mon[22010]: pgmap v1535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:19 smithi079 ceph-mon[23158]: pgmap v1535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:20 smithi016 conmon[32727]: 2022-03-21T18:03:20.646+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:21.861 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:21 smithi016 conmon[32727]: 2022-03-21T18:03:21.607+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:21 smithi016 ceph-mon[22010]: pgmap v1536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:21 smithi079 ceph-mon[23158]: pgmap v1536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:22.866 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:22 smithi016 conmon[32727]: 2022-03-21T18:03:22.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:23.867 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:23 smithi016 conmon[32727]: 2022-03-21T18:03:23.568+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:23 smithi016 ceph-mon[22010]: pgmap v1537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:23 smithi079 ceph-mon[23158]: pgmap v1537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:24.872 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:24 smithi016 conmon[32727]: 2022-03-21T18:03:24.534+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:25.874 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:25 smithi016 conmon[32727]: 2022-03-21T18:03:25.502+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:25 smithi016 ceph-mon[22010]: pgmap v1538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:26.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:26.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:25 smithi079 ceph-mon[23158]: pgmap v1538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:26.879 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:26 smithi016 conmon[32727]: 2022-03-21T18:03:26.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:27.881 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:27 smithi016 conmon[32727]: 2022-03-21T18:03:27.488+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:27 smithi016 ceph-mon[22010]: pgmap v1539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:27 smithi079 ceph-mon[23158]: pgmap v1539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:28.882 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:28 smithi016 conmon[32727]: 2022-03-21T18:03:28.506+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:29.883 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:29 smithi016 conmon[32727]: 2022-03-21T18:03:29.469+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:29 smithi016 ceph-mon[22010]: pgmap v1540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:29 smithi079 ceph-mon[23158]: pgmap v1540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:30.888 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:30 smithi016 conmon[32727]: 2022-03-21T18:03:30.482+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:31.889 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:31 smithi016 conmon[32727]: 2022-03-21T18:03:31.518+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:31 smithi016 ceph-mon[22010]: pgmap v1541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:31 smithi079 ceph-mon[23158]: pgmap v1541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:32.895 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:32 smithi016 conmon[32727]: 2022-03-21T18:03:32.491+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:33.896 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:33 smithi016 conmon[32727]: 2022-03-21T18:03:33.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:33 smithi016 ceph-mon[22010]: pgmap v1542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:33 smithi079 ceph-mon[23158]: pgmap v1542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:34.901 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:34 smithi016 conmon[32727]: 2022-03-21T18:03:34.466+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:35.903 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:35 smithi016 conmon[32727]: 2022-03-21T18:03:35.464+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:35 smithi016 ceph-mon[22010]: pgmap v1543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:35 smithi079 ceph-mon[23158]: pgmap v1543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:36.909 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:36 smithi016 conmon[32727]: 2022-03-21T18:03:36.430+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[32727]: 2022-03-21T18:03:37.396+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:37 smithi016 ceph-mon[22010]: pgmap v1544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:03:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:03:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:03:38.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: 2022-03-21T18:03:37.869+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:38.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:03:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: yield 2022-03-21T18:03:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:03:38.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:38.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:03:38.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:38.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:03:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:03:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:03:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:38.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:37 smithi079 ceph-mon[23158]: pgmap v1544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:03:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:03:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:03:38.912 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:38 smithi016 conmon[32727]: 2022-03-21T18:03:38.440+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:03:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:03:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:03:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:03:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:03:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:03:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:38 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:03:39.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:03:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:03:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:03:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:03:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:03:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:03:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:39.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:38 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:03:39.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:39.913 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:39 smithi016 conmon[32727]: 2022-03-21T18:03:39.414+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:39 smithi016 ceph-mon[22010]: pgmap v1545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:39 smithi079 ceph-mon[23158]: pgmap v1545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[32727]: 2022-03-21T18:03:40.438+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:41.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 2022-03-21T18:03:40.941+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:41.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:03:41.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:41.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: yield 2022-03-21T18:03:41.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:41.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:41.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:41.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:03:41.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:41.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:03:41.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:41.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:03:41.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:41.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:03:41.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:41.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:03:41.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:41.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:41.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:41.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 2022-03-21T18:03:40.942+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:03:41.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:03:41.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:41.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: yield 2022-03-21T18:03:41.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:41.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:41.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:41.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:03:41.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:41.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:03:41.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:41.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:03:41.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:41.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:03:41.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:41.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:03:41.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:41.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:41.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:41.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 2022-03-21T18:03:41.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:03:41.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: 2022-03-21T18:03:41.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:03:41.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:03:41.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:03:41.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:03:41.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:03:41.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:03:41.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:03:41.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:03:41.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:03:41.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:03:41.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:03:41.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:03:41.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:03:41.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:03:41.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:03:41.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:03:41.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:03:41.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:03:41.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:03:41.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:03:41.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:03:41.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:03:41.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise 2022-03-21T18:03:41.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:03:41.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:03:41.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:03:41.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:03:41.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:03:40 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:41 smithi016 conmon[32727]: 2022-03-21T18:03:41.485+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:41 smithi079 ceph-mon[23158]: pgmap v1546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:41 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:42.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:03:42.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:42.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:42.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:42.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:03:42.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:42.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:03:42.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:42.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:03:42.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:42.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:42.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:42.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:03:42.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:42.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:42.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.238 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:41 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:03:42.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:42.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:03:42.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:42.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:42.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:42.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:03:42.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:42.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:03:42.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:42.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:03:42.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:42.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:42.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:42.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:03:42.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:42.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:42.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:03:42.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:03:42.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:03:42.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:03:42.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:03:42.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:03:42.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:03:42.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:03:42.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:03:42.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:03:42.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:03:42.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:03:42.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:03:42.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:03:42.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:03:42.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:03:42.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:03:42.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:03:42.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:03:42.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:03:42.265 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:03:42.265 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:03:42.266 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:03:42.267 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:03:42.267 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:03:42.268 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:03:42.269 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:03:42.269 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:03:42.270 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:03:42.271 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:41 smithi016 ceph-mon[22010]: pgmap v1546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:41 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:03:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:42.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:03:42.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:42.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:03:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:03:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:42.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:42.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:42.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:03:42.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:42.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:41 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:03:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:03:42.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:03:42.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:03:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:03:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:03:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:03:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:03:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:03:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:03:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:03:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:03:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:03:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:03:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:03:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:03:42.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:03:42.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:03:42.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:03:42.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:03:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:03:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:03:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:03:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:03:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:03:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:03:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:03:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:03:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:03:42.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:03:42.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:03:42.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:03:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:03:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:03:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:03:42.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:03:42.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:03:42.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:03:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:03:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:03:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:03:42.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:03:42.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:03:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:03:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:03:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:03:42.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:03:42.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:03:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:42 smithi016 conmon[32727]: 2022-03-21T18:03:42.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:43 smithi016 conmon[32727]: 2022-03-21T18:03:43.438+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:43 smithi079 ceph-mon[23158]: pgmap v1547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:43 smithi016 ceph-mon[22010]: pgmap v1547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:44 smithi016 conmon[32727]: 2022-03-21T18:03:44.473+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:45 smithi016 conmon[32727]: 2022-03-21T18:03:45.498+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:45 smithi079 ceph-mon[23158]: pgmap v1548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:45 smithi016 ceph-mon[22010]: pgmap v1548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:46 smithi016 conmon[32727]: 2022-03-21T18:03:46.478+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:47 smithi016 conmon[32727]: 2022-03-21T18:03:47.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:47 smithi079 ceph-mon[23158]: pgmap v1549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:47 smithi016 ceph-mon[22010]: pgmap v1549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:48 smithi016 conmon[32727]: 2022-03-21T18:03:48.496+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:49 smithi016 conmon[32727]: 2022-03-21T18:03:49.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:49 smithi079 ceph-mon[23158]: pgmap v1550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:49 smithi016 ceph-mon[22010]: pgmap v1550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:50 smithi016 conmon[32727]: 2022-03-21T18:03:50.498+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:51 smithi016 conmon[32727]: 2022-03-21T18:03:51.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:51 smithi079 ceph-mon[23158]: pgmap v1551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:51 smithi016 ceph-mon[22010]: pgmap v1551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:52 smithi016 conmon[32727]: 2022-03-21T18:03:52.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:53 smithi016 conmon[32727]: 2022-03-21T18:03:53.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:53 smithi079 ceph-mon[23158]: pgmap v1552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:53 smithi016 ceph-mon[22010]: pgmap v1552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:54 smithi016 conmon[32727]: 2022-03-21T18:03:54.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:55 smithi016 conmon[32727]: 2022-03-21T18:03:55.584+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:55 smithi079 ceph-mon[23158]: pgmap v1553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:55 smithi016 ceph-mon[22010]: pgmap v1553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:03:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:56 smithi016 conmon[32727]: 2022-03-21T18:03:56.592+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:57 smithi016 conmon[32727]: 2022-03-21T18:03:57.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:57 smithi079 ceph-mon[23158]: pgmap v1554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:57 smithi016 ceph-mon[22010]: pgmap v1554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:03:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:58 smithi016 conmon[32727]: 2022-03-21T18:03:58.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:03:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:03:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:03:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:03:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:03:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:03:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:03:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:03:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:03:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:03:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:03:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:03:59 smithi016 conmon[32727]: 2022-03-21T18:03:59.591+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:03:59 smithi079 ceph-mon[23158]: pgmap v1555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:03:59 smithi016 ceph-mon[22010]: pgmap v1555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:00 smithi016 conmon[32727]: 2022-03-21T18:04:00.592+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:01 smithi016 conmon[32727]: 2022-03-21T18:04:01.585+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:01 smithi016 ceph-mon[22010]: pgmap v1556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:02.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:02.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:01 smithi079 ceph-mon[23158]: pgmap v1556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:02.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:02 smithi016 conmon[32727]: 2022-03-21T18:04:02.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:03 smithi016 conmon[32727]: 2022-03-21T18:04:03.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:03 smithi016 ceph-mon[22010]: pgmap v1557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:03 smithi079 ceph-mon[23158]: pgmap v1557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:04 smithi016 conmon[32727]: 2022-03-21T18:04:04.587+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:05 smithi016 conmon[32727]: 2022-03-21T18:04:05.598+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:05 smithi016 ceph-mon[22010]: pgmap v1558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:05 smithi079 ceph-mon[23158]: pgmap v1558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:06 smithi016 conmon[32727]: 2022-03-21T18:04:06.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:07 smithi016 conmon[32727]: 2022-03-21T18:04:07.629+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:07 smithi016 ceph-mon[22010]: pgmap v1559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:07 smithi079 ceph-mon[23158]: pgmap v1559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:08 smithi016 conmon[32727]: 2022-03-21T18:04:08.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:09 smithi016 conmon[32727]: 2022-03-21T18:04:09.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:09 smithi016 ceph-mon[22010]: pgmap v1560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:09 smithi079 ceph-mon[23158]: pgmap v1560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:10 smithi016 conmon[32727]: 2022-03-21T18:04:10.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:11 smithi016 conmon[32727]: 2022-03-21T18:04:11.604+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:12 smithi016 ceph-mon[22010]: pgmap v1561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:12 smithi079 ceph-mon[23158]: pgmap v1561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:12 smithi016 conmon[32727]: 2022-03-21T18:04:12.609+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:13 smithi016 conmon[32727]: 2022-03-21T18:04:13.576+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:14 smithi016 ceph-mon[22010]: pgmap v1562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:14 smithi079 ceph-mon[23158]: pgmap v1562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:14.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:14 smithi016 conmon[32727]: 2022-03-21T18:04:14.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:15 smithi016 conmon[32727]: 2022-03-21T18:04:15.607+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:16 smithi016 ceph-mon[22010]: pgmap v1563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:16 smithi079 ceph-mon[23158]: pgmap v1563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:16 smithi016 conmon[32727]: 2022-03-21T18:04:16.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:17 smithi016 conmon[32727]: 2022-03-21T18:04:17.622+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:18 smithi016 ceph-mon[22010]: pgmap v1564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:18 smithi079 ceph-mon[23158]: pgmap v1564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:18 smithi016 conmon[32727]: 2022-03-21T18:04:18.636+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:19 smithi016 conmon[32727]: 2022-03-21T18:04:19.641+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:20 smithi016 ceph-mon[22010]: pgmap v1565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:20 smithi079 ceph-mon[23158]: pgmap v1565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:20 smithi016 conmon[32727]: 2022-03-21T18:04:20.630+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:21 smithi016 conmon[32727]: 2022-03-21T18:04:21.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:22 smithi016 ceph-mon[22010]: pgmap v1566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:22 smithi079 ceph-mon[23158]: pgmap v1566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:22 smithi016 conmon[32727]: 2022-03-21T18:04:22.569+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:23 smithi016 conmon[32727]: 2022-03-21T18:04:23.612+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:24 smithi016 ceph-mon[22010]: pgmap v1567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:24 smithi079 ceph-mon[23158]: pgmap v1567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:24 smithi016 conmon[32727]: 2022-03-21T18:04:24.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:25 smithi016 conmon[32727]: 2022-03-21T18:04:25.659+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:26 smithi016 ceph-mon[22010]: pgmap v1568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:26 smithi079 ceph-mon[23158]: pgmap v1568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:26 smithi016 conmon[32727]: 2022-03-21T18:04:26.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:28.054 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:27 smithi016 conmon[32727]: 2022-03-21T18:04:27.664+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:28 smithi016 ceph-mon[22010]: pgmap v1569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:28 smithi079 ceph-mon[23158]: pgmap v1569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:28 smithi016 conmon[32727]: 2022-03-21T18:04:28.642+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:29 smithi016 conmon[32727]: 2022-03-21T18:04:29.608+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:30 smithi016 ceph-mon[22010]: pgmap v1570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:30 smithi079 ceph-mon[23158]: pgmap v1570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:30 smithi016 conmon[32727]: 2022-03-21T18:04:30.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:31 smithi016 conmon[32727]: 2022-03-21T18:04:31.534+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:32 smithi016 ceph-mon[22010]: pgmap v1571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:32 smithi079 ceph-mon[23158]: pgmap v1571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:32 smithi016 conmon[32727]: 2022-03-21T18:04:32.517+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:33 smithi016 conmon[32727]: 2022-03-21T18:04:33.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:34 smithi016 ceph-mon[22010]: pgmap v1572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:34 smithi079 ceph-mon[23158]: pgmap v1572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:34 smithi016 conmon[32727]: 2022-03-21T18:04:34.617+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:35 smithi016 conmon[32727]: 2022-03-21T18:04:35.593+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:36 smithi016 ceph-mon[22010]: pgmap v1573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:36 smithi079 ceph-mon[23158]: pgmap v1573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:36 smithi016 conmon[32727]: 2022-03-21T18:04:36.578+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:37 smithi016 conmon[32727]: 2022-03-21T18:04:37.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:38 smithi016 ceph-mon[22010]: pgmap v1574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:38 smithi079 ceph-mon[23158]: pgmap v1574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:38 smithi016 conmon[32727]: 2022-03-21T18:04:38.542+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:39 smithi016 conmon[32727]: 2022-03-21T18:04:39.512+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:40 smithi016 ceph-mon[22010]: pgmap v1575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:40 smithi079 ceph-mon[23158]: pgmap v1575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:40 smithi016 conmon[32727]: 2022-03-21T18:04:40.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:41.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:41.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:04:41.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:04:41.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:04:41.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:41.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:04:41.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:04:41.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:04:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:41 smithi016 conmon[32727]: 2022-03-21T18:04:41.554+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:42 smithi016 ceph-mon[22010]: pgmap v1576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:42 smithi079 ceph-mon[23158]: pgmap v1576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:42 smithi016 conmon[32727]: 2022-03-21T18:04:42.559+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:43 smithi016 conmon[32727]: 2022-03-21T18:04:43.535+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:44 smithi016 ceph-mon[22010]: pgmap v1577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:44.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: 2022-03-21T18:04:44.045+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:44.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:04:44.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:44.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: yield 2022-03-21T18:04:44.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:44.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:04:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:04:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:04:44.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:44.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:04:44.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:44.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:04:44.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:44.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:44.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:44.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:44 smithi079 ceph-mon[23158]: pgmap v1577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:44.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:44 smithi016 conmon[32727]: 2022-03-21T18:04:44.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:45 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:04:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:04:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:04:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:04:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:45.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:04:45.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:45.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:45.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:04:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:45.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:45.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:45 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:04:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:45 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:04:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:04:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:45.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:04:45.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:45.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:04:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:04:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:04:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:45.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:45 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:04:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:45 smithi016 conmon[32727]: 2022-03-21T18:04:45.585+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:46 smithi016 ceph-mon[22010]: pgmap v1578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:46 smithi079 ceph-mon[23158]: pgmap v1578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:46 smithi016 conmon[32727]: 2022-03-21T18:04:46.557+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:47.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 2022-03-21T18:04:47.117+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:47.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:04:47.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:47.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: yield 2022-03-21T18:04:47.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:47.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:47.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:47.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:04:47.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:47.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:04:47.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:47.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:04:47.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:47.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:04:47.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:47.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:04:47.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:47.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:47.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:47.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 2022-03-21T18:04:47.118+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:04:47.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:04:47.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:47.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: yield 2022-03-21T18:04:47.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:47.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:47.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:47.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:04:47.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:47.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:04:47.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:47.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:04:47.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:47.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:04:47.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:04:47.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:47.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:47.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:47.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 2022-03-21T18:04:47.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:04:47.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: 2022-03-21T18:04:47.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:04:47.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:04:47.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:04:47.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:04:47.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:04:47.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:04:47.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:04:47.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:04:47.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:04:47.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:04:47.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:04:47.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:04:47.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:04:47.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:04:47.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:04:47.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:04:47.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:04:47.457 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:04:47.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:04:47.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:04:47.459 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:04:47.460 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:04:47.460 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise 2022-03-21T18:04:47.461 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:04:47.462 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:04:47.462 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:04:47.463 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:04:47.463 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:47 smithi016 conmon[32727]: 2022-03-21T18:04:47.603+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:48 smithi016 ceph-mon[22010]: pgmap v1579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:48.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:04:48.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:48.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:48.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:48.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:04:48.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:48.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:04:48.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:48.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:04:48.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:48.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:48.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:48.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:04:48.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:48.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:48.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:48 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:04:48.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:48.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:04:48.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:48.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:48.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:48.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:04:48.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:48.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:04:48.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:48.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:04:48.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:48.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:48.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:48.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:04:48.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:48.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:48.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:04:48.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:04:48.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:04:48.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:04:48.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:04:48.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:04:48.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:04:48.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:04:48.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:04:48.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:04:48.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:04:48.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:04:48.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:04:48.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:04:48.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:04:48.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:04:48.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:04:48.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:04:48.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:04:48.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:04:48.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:04:48.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:04:48.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:04:48.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:04:48.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:04:48.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:04:48.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:04:48.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:04:48.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:04:48.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:48 smithi079 ceph-mon[23158]: pgmap v1579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:04:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:48.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:48.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:04:48.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:48.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:04:48.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:48.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:04:48.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:48.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:48.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:48.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:04:48.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:48.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:48.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:48 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:04:48.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:04:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:04:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:04:48.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:04:48.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:04:48.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:04:48.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:04:48.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:04:48.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:04:48.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:04:48.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:04:48.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:04:48.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:04:48.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:04:48.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:04:48.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:04:48.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:04:48.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:04:48.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:04:48.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:04:48.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:04:48.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:04:48.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:04:48.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:04:48.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:04:48.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:04:48.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:04:48.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:04:48.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:04:48.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:04:48.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:04:48.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:04:48.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:04:48.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:04:48.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:04:48.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:04:48.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:04:48.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:04:48.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:04:48.508 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:04:48.509 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:04:48.509 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:04:48.510 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:04:48.511 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:04:48.512 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:04:48.512 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:04:48.512 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:04:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:48 smithi016 conmon[32727]: 2022-03-21T18:04:48.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:49 smithi016 conmon[32727]: 2022-03-21T18:04:49.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:50 smithi016 ceph-mon[22010]: pgmap v1580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:50 smithi079 ceph-mon[23158]: pgmap v1580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:50 smithi016 conmon[32727]: 2022-03-21T18:04:50.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:51 smithi016 conmon[32727]: 2022-03-21T18:04:51.634+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:52.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:52.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:52 smithi016 ceph-mon[22010]: pgmap v1581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:52.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:52.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:52 smithi079 ceph-mon[23158]: pgmap v1581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:52 smithi016 conmon[32727]: 2022-03-21T18:04:52.586+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:53 smithi016 conmon[32727]: 2022-03-21T18:04:53.537+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:54 smithi016 ceph-mon[22010]: pgmap v1582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:54 smithi079 ceph-mon[23158]: pgmap v1582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:54 smithi016 conmon[32727]: 2022-03-21T18:04:54.570+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:55 smithi016 conmon[32727]: 2022-03-21T18:04:55.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:56 smithi016 ceph-mon[22010]: pgmap v1583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:56 smithi079 ceph-mon[23158]: pgmap v1583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:04:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:56 smithi016 conmon[32727]: 2022-03-21T18:04:56.533+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:57 smithi016 conmon[32727]: 2022-03-21T18:04:57.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: pgmap v1584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:04:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:04:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:04:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:04:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: pgmap v1584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:04:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:04:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:04:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:04:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:04:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:58 smithi016 conmon[32727]: 2022-03-21T18:04:58.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:04:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:04:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:04:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:04:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:04:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:04:59 smithi016 conmon[32727]: 2022-03-21T18:04:59.588+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:00 smithi016 ceph-mon[22010]: pgmap v1585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:00 smithi079 ceph-mon[23158]: pgmap v1585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:00 smithi016 conmon[32727]: 2022-03-21T18:05:00.581+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:01 smithi016 conmon[32727]: 2022-03-21T18:05:01.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:02 smithi016 ceph-mon[22010]: pgmap v1586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:02 smithi079 ceph-mon[23158]: pgmap v1586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:02 smithi016 conmon[32727]: 2022-03-21T18:05:02.648+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:03 smithi016 conmon[32727]: 2022-03-21T18:05:03.654+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:04 smithi016 ceph-mon[22010]: pgmap v1587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:04 smithi079 ceph-mon[23158]: pgmap v1587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:04 smithi016 conmon[32727]: 2022-03-21T18:05:04.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:05 smithi016 conmon[32727]: 2022-03-21T18:05:05.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:06 smithi016 ceph-mon[22010]: pgmap v1588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:06 smithi079 ceph-mon[23158]: pgmap v1588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:06 smithi016 conmon[32727]: 2022-03-21T18:05:06.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:07.588 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:07 smithi016 conmon[32727]: 2022-03-21T18:05:07.586+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:08 smithi079 ceph-mon[23158]: pgmap v1589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:08.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:08.543 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:08 smithi016 ceph-mon[22010]: pgmap v1589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:08 smithi016 conmon[32727]: 2022-03-21T18:05:08.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:09.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:09.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:09 smithi016 conmon[32727]: 2022-03-21T18:05:09.492+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:10 smithi079 ceph-mon[23158]: pgmap v1590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:10.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:10.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:10 smithi016 ceph-mon[22010]: pgmap v1590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:10 smithi016 conmon[32727]: 2022-03-21T18:05:10.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:11.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:11 smithi016 conmon[32727]: 2022-03-21T18:05:11.532+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:12 smithi079 ceph-mon[23158]: pgmap v1591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:12.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:12.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:12 smithi016 ceph-mon[22010]: pgmap v1591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:12 smithi016 conmon[32727]: 2022-03-21T18:05:12.574+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:13.543 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:13 smithi016 conmon[32727]: 2022-03-21T18:05:13.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:14 smithi079 ceph-mon[23158]: pgmap v1592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:14.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:14.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:14 smithi016 ceph-mon[22010]: pgmap v1592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:14.527 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:14 smithi016 conmon[32727]: 2022-03-21T18:05:14.523+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:15.516 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:15 smithi016 conmon[32727]: 2022-03-21T18:05:15.512+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:16 smithi079 ceph-mon[23158]: pgmap v1593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:16.479 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:16.479 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:16 smithi016 ceph-mon[22010]: pgmap v1593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:16 smithi016 conmon[32727]: 2022-03-21T18:05:16.477+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:17 smithi016 conmon[32727]: 2022-03-21T18:05:17.436+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:18 smithi079 ceph-mon[23158]: pgmap v1594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:18 smithi016 ceph-mon[22010]: pgmap v1594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:18 smithi016 conmon[32727]: 2022-03-21T18:05:18.441+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:19 smithi016 conmon[32727]: 2022-03-21T18:05:19.424+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:20 smithi079 ceph-mon[23158]: pgmap v1595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:20.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:20.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:20 smithi016 ceph-mon[22010]: pgmap v1595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:20 smithi016 conmon[32727]: 2022-03-21T18:05:20.471+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:21.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:21 smithi016 conmon[32727]: 2022-03-21T18:05:21.505+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:22 smithi079 ceph-mon[23158]: pgmap v1596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:22.514 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:22.514 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:22 smithi016 ceph-mon[22010]: pgmap v1596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:22 smithi016 conmon[32727]: 2022-03-21T18:05:22.512+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:23.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:23 smithi016 conmon[32727]: 2022-03-21T18:05:23.494+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:24 smithi079 ceph-mon[23158]: pgmap v1597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:24.524 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:24.524 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:24 smithi016 ceph-mon[22010]: pgmap v1597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:24.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:24 smithi016 conmon[32727]: 2022-03-21T18:05:24.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:25.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:25 smithi016 conmon[32727]: 2022-03-21T18:05:25.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:26 smithi079 ceph-mon[23158]: pgmap v1598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:26.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:26.535 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:26 smithi016 ceph-mon[22010]: pgmap v1598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:26 smithi016 conmon[32727]: 2022-03-21T18:05:26.532+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:27.551 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:27 smithi016 conmon[32727]: 2022-03-21T18:05:27.549+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:28.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:28.505 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:28 smithi016 ceph-mon[22010]: pgmap v1599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:28 smithi079 ceph-mon[23158]: pgmap v1599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:28 smithi016 conmon[32727]: 2022-03-21T18:05:28.502+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:29.521 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:29.522 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:29 smithi016 conmon[32727]: 2022-03-21T18:05:29.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:30.493 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:30.494 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:30 smithi016 ceph-mon[22010]: pgmap v1600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:30 smithi079 ceph-mon[23158]: pgmap v1600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:30 smithi016 conmon[32727]: 2022-03-21T18:05:30.491+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:31.520 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:31 smithi016 conmon[32727]: 2022-03-21T18:05:31.518+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:32 smithi016 ceph-mon[22010]: pgmap v1601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:32 smithi016 conmon[32727]: 2022-03-21T18:05:32.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:32 smithi079 ceph-mon[23158]: pgmap v1601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:33.522 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:33 smithi016 conmon[32727]: 2022-03-21T18:05:33.520+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:34.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:34.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:34 smithi016 ceph-mon[22010]: pgmap v1602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:34.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:34.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:34 smithi079 ceph-mon[23158]: pgmap v1602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:34 smithi016 conmon[32727]: 2022-03-21T18:05:34.551+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:35.596 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:35 smithi016 conmon[32727]: 2022-03-21T18:05:35.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:36.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:36 smithi016 ceph-mon[22010]: pgmap v1603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:36.602 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:36 smithi079 ceph-mon[23158]: pgmap v1603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:36 smithi016 conmon[32727]: 2022-03-21T18:05:36.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:37.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:37 smithi016 conmon[32727]: 2022-03-21T18:05:37.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:38.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:38 smithi016 ceph-mon[22010]: pgmap v1604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:38.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:38 smithi079 ceph-mon[23158]: pgmap v1604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:38 smithi016 conmon[32727]: 2022-03-21T18:05:38.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:39.585 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:39.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:39 smithi016 conmon[32727]: 2022-03-21T18:05:39.583+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:40 smithi079 ceph-mon[23158]: pgmap v1605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:40 smithi016 ceph-mon[22010]: pgmap v1605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:40 smithi016 conmon[32727]: 2022-03-21T18:05:40.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:41 smithi016 conmon[32727]: 2022-03-21T18:05:41.548+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:42 smithi079 ceph-mon[23158]: pgmap v1606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:42.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:42 smithi016 conmon[32727]: 2022-03-21T18:05:42.596+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:42 smithi016 ceph-mon[22010]: pgmap v1606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:43 smithi016 conmon[32727]: 2022-03-21T18:05:43.598+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:44 smithi079 ceph-mon[23158]: pgmap v1607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:44 smithi016 conmon[32727]: 2022-03-21T18:05:44.617+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:44 smithi016 ceph-mon[22010]: pgmap v1607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:45 smithi016 conmon[32727]: 2022-03-21T18:05:45.591+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:46 smithi079 ceph-mon[23158]: pgmap v1608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:46 smithi016 ceph-mon[22010]: pgmap v1608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:46.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:46 smithi016 conmon[32727]: 2022-03-21T18:05:46.607+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:47 smithi016 conmon[32727]: 2022-03-21T18:05:47.583+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:05:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:05:47.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:05:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:05:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:05:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:05:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:48 smithi016 ceph-mon[22010]: pgmap v1609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:48.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:48 smithi016 conmon[32727]: 2022-03-21T18:05:48.576+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:48 smithi079 ceph-mon[23158]: pgmap v1609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:49 smithi016 conmon[32727]: 2022-03-21T18:05:49.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:50.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: 2022-03-21T18:05:50.221+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:50.483 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:05:50.483 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:50.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: yield 2022-03-21T18:05:50.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:50.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:50.485 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:50.485 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:05:50.486 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:50.486 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:05:50.486 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:50.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:05:50.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:50.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:05:50.488 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:50.488 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:05:50.488 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:50.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:50.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:50 smithi016 conmon[32727]: 2022-03-21T18:05:50.615+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:50 smithi016 ceph-mon[22010]: pgmap v1610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:50 smithi079 ceph-mon[23158]: pgmap v1610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:51 smithi016 conmon[32727]: 2022-03-21T18:05:51.645+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:05:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:51.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:05:51.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:05:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:05:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:05:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:05:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:51 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:05:51.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:05:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:51.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:05:51.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:51.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:51.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:51.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:05:51.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:51.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:05:51.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:51.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:05:51.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:51.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:51.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:51.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:05:51.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:51.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:51.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:51.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:51 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:05:51.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:52 smithi016 ceph-mon[22010]: pgmap v1611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:52.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:52 smithi016 conmon[32727]: 2022-03-21T18:05:52.597+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:52 smithi079 ceph-mon[23158]: pgmap v1611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:53.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:53.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:53.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 2022-03-21T18:05:53.293+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:53.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:05:53.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:53.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: yield 2022-03-21T18:05:53.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:53.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:53.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:53.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:05:53.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:53.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:05:53.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:53.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:05:53.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:53.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:05:53.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:53.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:05:53.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:53.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:53.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:53.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 2022-03-21T18:05:53.294+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:05:53.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:05:53.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:53.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: yield 2022-03-21T18:05:53.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:53.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:53.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:53.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:05:53.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:53.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:05:53.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:53.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:05:53.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:53.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:05:53.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:53.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:05:53.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:53.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:53.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:53.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 2022-03-21T18:05:53.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:05:53.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: 2022-03-21T18:05:53.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:05:53.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:05:53.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:05:53.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:05:53.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:05:53.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:05:53.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:05:53.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:05:53.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:05:53.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:05:53.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:05:53.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:05:53.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:05:53.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:05:53.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:05:53.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:05:53.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:05:53.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:05:53.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:05:53.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:05:53.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:05:53.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:05:53.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise 2022-03-21T18:05:53.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:05:53.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:05:53.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:05:53.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:05:53.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:53 smithi016 conmon[32727]: 2022-03-21T18:05:53.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:54 smithi016 conmon[32727]: 2022-03-21T18:05:54.681+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:54 smithi016 ceph-mon[22010]: pgmap v1612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:05:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:05:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:05:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:05:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:05:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:54 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:05:54.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:54.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:05:54.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:54.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:54.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:54.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:05:54.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:54.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:05:54.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:54.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:05:54.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:54.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:54.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:54.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:05:54.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:54.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:54.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:05:54.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:05:54.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:05:54.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:05:54.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:05:54.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:05:54.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:05:54.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:05:54.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:05:54.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:05:54.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:05:54.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:05:54.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:05:54.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:05:54.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:05:54.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:05:54.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:05:54.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:05:54.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:05:54.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:05:54.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:05:54.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:05:54.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:05:54.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:05:54.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:05:54.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:05:54.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:05:54.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:05:54.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:05:54.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:54 smithi079 ceph-mon[23158]: pgmap v1612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:05:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:05:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:05:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:05:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:05:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:54 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:05:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:05:54.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:05:54.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:05:54.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:05:54.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:05:54.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:05:54.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:05:54.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:05:54.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:05:54.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:05:54.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:05:54.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:05:54.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:05:54.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:05:54.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:05:54.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:05:54.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:05:54.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:05:54.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:05:54.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:05:54.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:05:54.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:05:54.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:05:54.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:05:54.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:05:54.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:05:54.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:05:54.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:05:54.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:05:54.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:05:54.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:05:54.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:05:54.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:05:54.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:05:54.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:05:54.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:05:54.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:05:54.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:05:54.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:05:54.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:05:54.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:05:54.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:05:54.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:05:54.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:05:54.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:05:54.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:05:54.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:05:54.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:55 smithi016 ceph-mon[22010]: pgmap v1613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:55.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:55 smithi016 conmon[32727]: 2022-03-21T18:05:55.665+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:55 smithi079 ceph-mon[23158]: pgmap v1613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:05:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:56 smithi016 conmon[32727]: 2022-03-21T18:05:56.689+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:57 smithi016 ceph-mon[22010]: pgmap v1614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:57.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:57 smithi016 conmon[32727]: 2022-03-21T18:05:57.701+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:57 smithi079 ceph-mon[23158]: pgmap v1614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:58 smithi016 conmon[32727]: 2022-03-21T18:05:58.684+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:05:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:05:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:05:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:05:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:05:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:05:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:05:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:05:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:05:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:05:59 smithi016 ceph-mon[22010]: pgmap v1615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:05:59.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:05:59 smithi016 conmon[32727]: 2022-03-21T18:05:59.678+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:05:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:05:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:05:59 smithi079 ceph-mon[23158]: pgmap v1615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:00 smithi016 conmon[32727]: 2022-03-21T18:06:00.674+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:01 smithi016 ceph-mon[22010]: pgmap v1616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:01 smithi016 conmon[32727]: 2022-03-21T18:06:01.700+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:01 smithi079 ceph-mon[23158]: pgmap v1616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:02 smithi016 conmon[32727]: 2022-03-21T18:06:02.667+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:03 smithi016 ceph-mon[22010]: pgmap v1617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:03 smithi016 conmon[32727]: 2022-03-21T18:06:03.708+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:03 smithi079 ceph-mon[23158]: pgmap v1617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:04 smithi016 conmon[32727]: 2022-03-21T18:06:04.697+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:05 smithi016 ceph-mon[22010]: pgmap v1618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:05 smithi016 conmon[32727]: 2022-03-21T18:06:05.699+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:05 smithi079 ceph-mon[23158]: pgmap v1618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:06 smithi016 conmon[32727]: 2022-03-21T18:06:06.694+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:06.955 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:07 smithi016 ceph-mon[22010]: pgmap v1619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:07 smithi016 conmon[32727]: 2022-03-21T18:06:07.678+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:07 smithi079 ceph-mon[23158]: pgmap v1619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:08 smithi016 conmon[32727]: 2022-03-21T18:06:08.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:09 smithi016 ceph-mon[22010]: pgmap v1620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:09 smithi016 conmon[32727]: 2022-03-21T18:06:09.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:09 smithi079 ceph-mon[23158]: pgmap v1620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:10 smithi016 conmon[32727]: 2022-03-21T18:06:10.677+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:11 smithi016 ceph-mon[22010]: pgmap v1621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:11 smithi016 conmon[32727]: 2022-03-21T18:06:11.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:11 smithi079 ceph-mon[23158]: pgmap v1621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:12 smithi016 conmon[32727]: 2022-03-21T18:06:12.666+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:13 smithi016 ceph-mon[22010]: pgmap v1622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:13.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:13.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:13 smithi016 conmon[32727]: 2022-03-21T18:06:13.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:13 smithi079 ceph-mon[23158]: pgmap v1622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:13.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:14 smithi016 conmon[32727]: 2022-03-21T18:06:14.599+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:15 smithi016 ceph-mon[22010]: pgmap v1623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:15 smithi016 conmon[32727]: 2022-03-21T18:06:15.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:15 smithi079 ceph-mon[23158]: pgmap v1623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:16 smithi016 conmon[32727]: 2022-03-21T18:06:16.506+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:17 smithi016 ceph-mon[22010]: pgmap v1624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:17 smithi016 conmon[32727]: 2022-03-21T18:06:17.471+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:17 smithi079 ceph-mon[23158]: pgmap v1624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:18 smithi016 conmon[32727]: 2022-03-21T18:06:18.423+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:19 smithi016 ceph-mon[22010]: pgmap v1625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:19 smithi016 conmon[32727]: 2022-03-21T18:06:19.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:19 smithi079 ceph-mon[23158]: pgmap v1625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:20 smithi016 conmon[32727]: 2022-03-21T18:06:20.440+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:21 smithi016 ceph-mon[22010]: pgmap v1626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:21 smithi016 conmon[32727]: 2022-03-21T18:06:21.391+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:21 smithi079 ceph-mon[23158]: pgmap v1626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:22 smithi016 conmon[32727]: 2022-03-21T18:06:22.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:23 smithi016 ceph-mon[22010]: pgmap v1627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:23.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:23 smithi016 conmon[32727]: 2022-03-21T18:06:23.373+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:23 smithi079 ceph-mon[23158]: pgmap v1627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:24 smithi016 conmon[32727]: 2022-03-21T18:06:24.394+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:25 smithi016 ceph-mon[22010]: pgmap v1628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:25.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:25 smithi016 conmon[32727]: 2022-03-21T18:06:25.358+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:25 smithi079 ceph-mon[23158]: pgmap v1628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:26.601 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:26 smithi016 conmon[32727]: 2022-03-21T18:06:26.311+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:27.606 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:27 smithi016 conmon[32727]: 2022-03-21T18:06:27.301+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:27 smithi016 ceph-mon[22010]: pgmap v1629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:27 smithi079 ceph-mon[23158]: pgmap v1629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:28.607 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:28 smithi016 conmon[32727]: 2022-03-21T18:06:28.284+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:29.612 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:29 smithi016 conmon[32727]: 2022-03-21T18:06:29.262+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:29 smithi016 ceph-mon[22010]: pgmap v1630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:29 smithi079 ceph-mon[23158]: pgmap v1630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:30.614 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:30 smithi016 conmon[32727]: 2022-03-21T18:06:30.300+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:31.619 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:31 smithi016 conmon[32727]: 2022-03-21T18:06:31.255+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:31 smithi016 ceph-mon[22010]: pgmap v1631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:31 smithi079 ceph-mon[23158]: pgmap v1631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:32.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:32 smithi016 conmon[32727]: 2022-03-21T18:06:32.293+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:33.625 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:33 smithi016 conmon[32727]: 2022-03-21T18:06:33.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:33 smithi016 ceph-mon[22010]: pgmap v1632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:33 smithi079 ceph-mon[23158]: pgmap v1632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:34.627 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:34 smithi016 conmon[32727]: 2022-03-21T18:06:34.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:35.631 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:35 smithi016 conmon[32727]: 2022-03-21T18:06:35.356+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:35 smithi016 ceph-mon[22010]: pgmap v1633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:35 smithi079 ceph-mon[23158]: pgmap v1633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:36.632 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:36 smithi016 conmon[32727]: 2022-03-21T18:06:36.348+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:37.638 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:37 smithi016 conmon[32727]: 2022-03-21T18:06:37.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:37 smithi016 ceph-mon[22010]: pgmap v1634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:37 smithi079 ceph-mon[23158]: pgmap v1634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:38.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:38 smithi016 conmon[32727]: 2022-03-21T18:06:38.331+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:39.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:39 smithi016 conmon[32727]: 2022-03-21T18:06:39.297+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:39 smithi016 ceph-mon[22010]: pgmap v1635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:39 smithi079 ceph-mon[23158]: pgmap v1635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:40.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:40 smithi016 conmon[32727]: 2022-03-21T18:06:40.278+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:41.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:41 smithi016 conmon[32727]: 2022-03-21T18:06:41.298+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:41 smithi016 ceph-mon[22010]: pgmap v1636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:41 smithi079 ceph-mon[23158]: pgmap v1636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:42.653 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:42 smithi016 conmon[32727]: 2022-03-21T18:06:42.326+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:42.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:43.660 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:43 smithi016 conmon[32727]: 2022-03-21T18:06:43.369+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:43 smithi016 ceph-mon[22010]: pgmap v1637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:43 smithi079 ceph-mon[23158]: pgmap v1637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:44.661 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:44 smithi016 conmon[32727]: 2022-03-21T18:06:44.407+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:45 smithi016 ceph-mon[22010]: pgmap v1638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:45 smithi016 conmon[32727]: 2022-03-21T18:06:45.438+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:45 smithi079 ceph-mon[23158]: pgmap v1638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:46 smithi016 conmon[32727]: 2022-03-21T18:06:46.437+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:47 smithi016 conmon[32727]: 2022-03-21T18:06:47.399+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:47 smithi079 ceph-mon[23158]: pgmap v1639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:47 smithi016 ceph-mon[22010]: pgmap v1639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:48 smithi016 conmon[32727]: 2022-03-21T18:06:48.445+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:49 smithi016 ceph-mon[22010]: pgmap v1640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:49 smithi016 conmon[32727]: 2022-03-21T18:06:49.459+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:49 smithi079 ceph-mon[23158]: pgmap v1640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:50 smithi016 conmon[32727]: 2022-03-21T18:06:50.472+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:51.685 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:51 smithi016 conmon[32727]: 2022-03-21T18:06:51.431+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:51 smithi079 ceph-mon[23158]: pgmap v1641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:51 smithi016 ceph-mon[22010]: pgmap v1641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:52.921 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:52 smithi016 conmon[32727]: 2022-03-21T18:06:52.435+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[32727]: 2022-03-21T18:06:53.442+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: pgmap v1642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:06:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:06:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:06:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: pgmap v1642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:06:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:06:54.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:06:54.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:54.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: 2022-03-21T18:06:53.901+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:06:54.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:54.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: yield 2022-03-21T18:06:54.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:54.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:54.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:54.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:06:54.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:54.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:06:54.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:54.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:06:54.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:54.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:06:54.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:54.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:06:54.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:54.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:54.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:54 smithi016 conmon[32727]: 2022-03-21T18:06:54.452+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:06:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:06:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:06:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:06:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:06:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:06:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:54.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:06:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:06:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:06:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:06:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:06:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:06:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:06:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:54.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:54.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:06:55.864 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:55 smithi016 conmon[32727]: 2022-03-21T18:06:55.481+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:55.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:55 smithi016 ceph-mon[22010]: pgmap v1643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:55.965 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:55.966 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:55 smithi079 ceph-mon[23158]: pgmap v1643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:06:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[32727]: 2022-03-21T18:06:56.433+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:06:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:06:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:06:57.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:06:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:06:57.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:56 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:06:57.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 2022-03-21T18:06:56.973+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:57.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:06:57.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:57.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: yield 2022-03-21T18:06:57.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:57.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:57.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:57.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:06:57.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:57.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:06:57.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:57.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:06:57.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:57.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:06:57.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:57.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:06:57.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:57.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:57.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:57.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 2022-03-21T18:06:56.974+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:06:57.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:06:57.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:57.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: yield 2022-03-21T18:06:57.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:57.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:57.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:57.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:06:57.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:57.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:06:57.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:57.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:06:57.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:57.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:06:57.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:57.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:06:57.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:57.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:57.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:57.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 2022-03-21T18:06:57.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:06:57.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: 2022-03-21T18:06:57.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:06:57.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:06:57.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:06:57.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:06:57.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:06:57.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:06:57.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:06:57.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:06:57.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:06:57.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:06:57.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:06:57.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:06:57.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:06:57.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:06:57.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:06:57.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:06:57.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:06:57.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:06:57.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:06:57.200 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:06:57.200 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:06:57.200 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:06:57.201 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise 2022-03-21T18:06:57.201 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:06:57.201 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:06:57.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:06:57.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:06:57.203 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:06:56 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:06:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:06:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:06:57.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:06:57.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:06:57.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:06:57.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:56 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:06:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:57 smithi016 conmon[32727]: 2022-03-21T18:06:57.450+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:57 smithi079 ceph-mon[23158]: pgmap v1644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:58.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:06:58.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:58.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:58.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:58.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:06:58.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:58.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:06:58.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:58.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:06:58.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:58.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:58.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:58.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:06:58.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:58.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:58.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:06:58.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:58.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:06:58.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:58.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:58.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:58.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:06:58.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:58.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:06:58.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:58.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:06:58.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:58.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:58.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:58.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:06:58.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:58.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:58.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:06:58.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:06:58.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:06:58.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:06:58.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:06:58.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:06:58.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:06:58.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:06:58.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:06:58.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:06:58.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:06:58.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:06:58.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:06:58.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:06:58.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:06:58.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:06:58.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:06:58.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:06:58.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:06:58.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:06:58.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:06:58.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:06:58.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:06:58.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:06:58.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:06:58.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:06:58.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:06:58.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:06:58.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:06:58.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:57 smithi016 ceph-mon[22010]: pgmap v1644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:06:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:06:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:06:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:58.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:06:58.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:58.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:06:58.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:58.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:58.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:58.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:06:58.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:58.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:58.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:06:58.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:06:58.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:06:58.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:06:58.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:06:58.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:06:58.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:06:58.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:06:58.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:06:58.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:06:58.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:06:58.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:06:58.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:06:58.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:06:58.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:06:58.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:06:58.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:06:58.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:06:58.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:06:58.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:06:58.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:06:58.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:06:58.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:06:58.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:06:58.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:06:58.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:06:58.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:06:58.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:06:58.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:06:58.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:06:58.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:06:58.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:06:58.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:06:58.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:06:58.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:06:58.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:06:58.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:06:58.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:06:58.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:06:58.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:06:58.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:06:58.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:06:58.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:06:58.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:06:58.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:06:58.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:06:58.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:06:58.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:06:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:58 smithi016 conmon[32727]: 2022-03-21T18:06:58.469+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:06:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:06:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:06:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:06:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:06:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:06:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:06:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:06:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:06:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:06:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:06:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:06:59 smithi016 conmon[32727]: 2022-03-21T18:06:59.493+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:06:59 smithi079 ceph-mon[23158]: pgmap v1645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:06:59 smithi016 ceph-mon[22010]: pgmap v1645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:00 smithi016 conmon[32727]: 2022-03-21T18:07:00.515+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:01 smithi016 conmon[32727]: 2022-03-21T18:07:01.538+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:01 smithi079 ceph-mon[23158]: pgmap v1646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:01 smithi016 ceph-mon[22010]: pgmap v1646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:02 smithi016 conmon[32727]: 2022-03-21T18:07:02.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:03 smithi016 conmon[32727]: 2022-03-21T18:07:03.536+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:03 smithi079 ceph-mon[23158]: pgmap v1647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:03 smithi016 ceph-mon[22010]: pgmap v1647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:04 smithi016 conmon[32727]: 2022-03-21T18:07:04.551+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:05 smithi016 conmon[32727]: 2022-03-21T18:07:05.557+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:05 smithi079 ceph-mon[23158]: pgmap v1648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:05 smithi016 ceph-mon[22010]: pgmap v1648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:06 smithi016 conmon[32727]: 2022-03-21T18:07:06.530+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:07 smithi016 conmon[32727]: 2022-03-21T18:07:07.552+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:07 smithi079 ceph-mon[23158]: pgmap v1649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:07 smithi016 ceph-mon[22010]: pgmap v1649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:08 smithi016 conmon[32727]: 2022-03-21T18:07:08.503+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:09 smithi016 conmon[32727]: 2022-03-21T18:07:09.477+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:10.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:09 smithi016 ceph-mon[22010]: pgmap v1650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:09 smithi079 ceph-mon[23158]: pgmap v1650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:10 smithi016 conmon[32727]: 2022-03-21T18:07:10.518+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:11 smithi016 conmon[32727]: 2022-03-21T18:07:11.544+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:12.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:11 smithi016 ceph-mon[22010]: pgmap v1651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:11 smithi079 ceph-mon[23158]: pgmap v1651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:12 smithi016 conmon[32727]: 2022-03-21T18:07:12.567+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:13 smithi016 conmon[32727]: 2022-03-21T18:07:13.590+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:13 smithi016 ceph-mon[22010]: pgmap v1652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:13 smithi079 ceph-mon[23158]: pgmap v1652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:14 smithi016 conmon[32727]: 2022-03-21T18:07:14.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:15.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:15 smithi016 conmon[32727]: 2022-03-21T18:07:15.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:15 smithi016 ceph-mon[22010]: pgmap v1653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:15 smithi079 ceph-mon[23158]: pgmap v1653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:16 smithi016 conmon[32727]: 2022-03-21T18:07:16.624+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:17 smithi016 conmon[32727]: 2022-03-21T18:07:17.661+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:17 smithi016 ceph-mon[22010]: pgmap v1654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:17 smithi079 ceph-mon[23158]: pgmap v1654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:18 smithi016 conmon[32727]: 2022-03-21T18:07:18.625+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:19 smithi016 conmon[32727]: 2022-03-21T18:07:19.588+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:20 smithi016 ceph-mon[22010]: pgmap v1655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:20 smithi079 ceph-mon[23158]: pgmap v1655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:20 smithi016 conmon[32727]: 2022-03-21T18:07:20.565+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:21 smithi016 conmon[32727]: 2022-03-21T18:07:21.610+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:22 smithi016 ceph-mon[22010]: pgmap v1656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:22 smithi079 ceph-mon[23158]: pgmap v1656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:22 smithi016 conmon[32727]: 2022-03-21T18:07:22.604+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:23 smithi016 conmon[32727]: 2022-03-21T18:07:23.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:24 smithi016 ceph-mon[22010]: pgmap v1657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:24 smithi079 ceph-mon[23158]: pgmap v1657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:24 smithi016 conmon[32727]: 2022-03-21T18:07:24.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:25 smithi016 conmon[32727]: 2022-03-21T18:07:25.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:26 smithi016 ceph-mon[22010]: pgmap v1658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:26 smithi079 ceph-mon[23158]: pgmap v1658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:27.026 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:26 smithi016 conmon[32727]: 2022-03-21T18:07:26.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:27 smithi016 conmon[32727]: 2022-03-21T18:07:27.654+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:28 smithi016 ceph-mon[22010]: pgmap v1659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:28 smithi079 ceph-mon[23158]: pgmap v1659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:28 smithi016 conmon[32727]: 2022-03-21T18:07:28.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:29 smithi016 conmon[32727]: 2022-03-21T18:07:29.655+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:30 smithi016 ceph-mon[22010]: pgmap v1660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:30 smithi079 ceph-mon[23158]: pgmap v1660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:30 smithi016 conmon[32727]: 2022-03-21T18:07:30.609+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:31 smithi016 conmon[32727]: 2022-03-21T18:07:31.621+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:32 smithi016 ceph-mon[22010]: pgmap v1661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:32 smithi079 ceph-mon[23158]: pgmap v1661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:32 smithi016 conmon[32727]: 2022-03-21T18:07:32.599+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:33 smithi016 conmon[32727]: 2022-03-21T18:07:33.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:34 smithi016 ceph-mon[22010]: pgmap v1662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:34 smithi079 ceph-mon[23158]: pgmap v1662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:34 smithi016 conmon[32727]: 2022-03-21T18:07:34.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:35 smithi016 conmon[32727]: 2022-03-21T18:07:35.513+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:36 smithi016 ceph-mon[22010]: pgmap v1663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:36 smithi079 ceph-mon[23158]: pgmap v1663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:36 smithi016 conmon[32727]: 2022-03-21T18:07:36.558+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:37 smithi016 conmon[32727]: 2022-03-21T18:07:37.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:38 smithi016 ceph-mon[22010]: pgmap v1664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:38 smithi079 ceph-mon[23158]: pgmap v1664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:38 smithi016 conmon[32727]: 2022-03-21T18:07:38.540+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:39 smithi016 conmon[32727]: 2022-03-21T18:07:39.556+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:40 smithi016 ceph-mon[22010]: pgmap v1665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:40 smithi079 ceph-mon[23158]: pgmap v1665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:40 smithi016 conmon[32727]: 2022-03-21T18:07:40.575+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:41 smithi016 conmon[32727]: 2022-03-21T18:07:41.622+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:42 smithi016 ceph-mon[22010]: pgmap v1666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:42 smithi079 ceph-mon[23158]: pgmap v1666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:42 smithi016 conmon[32727]: 2022-03-21T18:07:42.650+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:43 smithi016 conmon[32727]: 2022-03-21T18:07:43.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:44 smithi016 ceph-mon[22010]: pgmap v1667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:44 smithi079 ceph-mon[23158]: pgmap v1667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:45.078 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:44 smithi016 conmon[32727]: 2022-03-21T18:07:44.690+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:46.080 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:45 smithi016 conmon[32727]: 2022-03-21T18:07:45.724+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:46 smithi016 ceph-mon[22010]: pgmap v1668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:46 smithi079 ceph-mon[23158]: pgmap v1668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:47.085 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:46 smithi016 conmon[32727]: 2022-03-21T18:07:46.707+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:48.086 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:47 smithi016 conmon[32727]: 2022-03-21T18:07:47.745+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:48 smithi016 ceph-mon[22010]: pgmap v1669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:48 smithi079 ceph-mon[23158]: pgmap v1669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:49.088 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:48 smithi016 conmon[32727]: 2022-03-21T18:07:48.712+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:50.090 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:49 smithi016 conmon[32727]: 2022-03-21T18:07:49.706+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:50 smithi016 ceph-mon[22010]: pgmap v1670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:50 smithi079 ceph-mon[23158]: pgmap v1670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:51.094 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:50 smithi016 conmon[32727]: 2022-03-21T18:07:50.716+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:52.095 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:51 smithi016 conmon[32727]: 2022-03-21T18:07:51.746+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:52.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:52.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:52 smithi016 ceph-mon[22010]: pgmap v1671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:52 smithi079 ceph-mon[23158]: pgmap v1671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:53.100 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:52 smithi016 conmon[32727]: 2022-03-21T18:07:52.738+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:54.102 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:53 smithi016 conmon[32727]: 2022-03-21T18:07:53.696+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:54 smithi016 ceph-mon[22010]: pgmap v1672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:54.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:54.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:54 smithi079 ceph-mon[23158]: pgmap v1672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:54.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:55.107 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:54 smithi016 conmon[32727]: 2022-03-21T18:07:54.729+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:56.109 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:55 smithi016 conmon[32727]: 2022-03-21T18:07:55.694+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:56 smithi016 ceph-mon[22010]: pgmap v1673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:56 smithi079 ceph-mon[23158]: pgmap v1673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:07:57.022 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:56 smithi016 conmon[32727]: 2022-03-21T18:07:56.705+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:07:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:07:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:07:57.432 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:57.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:07:57.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:07:57.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:07:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:57 smithi016 conmon[32727]: 2022-03-21T18:07:57.656+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: pgmap v1674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:07:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:07:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:07:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:07:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: pgmap v1674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:07:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:07:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:07:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:07:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:07:59.116 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:58 smithi016 conmon[32727]: 2022-03-21T18:07:58.704+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:07:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:07:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:07:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:07:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:07:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:00.079 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:07:59 smithi016 conmon[32727]: 2022-03-21T18:07:59.665+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:00 smithi016 ceph-mon[22010]: pgmap v1675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:00.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: 2022-03-21T18:08:00.077+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:00.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:08:00.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:00.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: yield 2022-03-21T18:08:00.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:00.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:00.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:00.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:08:00.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:00.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:08:00.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:00.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:08:00.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:00.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:08:00.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:00.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:08:00.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:00.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:00.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:00 smithi079 ceph-mon[23158]: pgmap v1675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:00 smithi016 conmon[32727]: 2022-03-21T18:08:00.638+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:08:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:08:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:08:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:08:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:08:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:08:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:01.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:01.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:08:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:08:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:08:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:08:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:08:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:08:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:08:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:08:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:01 smithi016 conmon[32727]: 2022-03-21T18:08:01.649+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:02.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:02.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:02 smithi016 ceph-mon[22010]: pgmap v1676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:02.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:02 smithi079 ceph-mon[23158]: pgmap v1676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:02 smithi016 conmon[32727]: 2022-03-21T18:08:02.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 2022-03-21T18:08:03.148+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:08:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:03.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: yield 2022-03-21T18:08:03.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:08:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:08:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:08:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:08:03.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:03.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:08:03.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:03.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:03.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:03.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 2022-03-21T18:08:03.149+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:08:03.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:08:03.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:03.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: yield 2022-03-21T18:08:03.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:03.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:03.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:03.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:08:03.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:03.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:08:03.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:03.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:08:03.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:03.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:08:03.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:03.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:08:03.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:03.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:03.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:03.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 2022-03-21T18:08:03.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:08:03.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: 2022-03-21T18:08:03.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:08:03.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:08:03.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:08:03.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:08:03.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:08:03.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:08:03.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:08:03.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:08:03.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:08:03.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:08:03.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:08:03.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:08:03.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:08:03.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:08:03.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:08:03.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:08:03.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:08:03.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:08:03.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:08:03.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:08:03.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:08:03.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:08:03.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise 2022-03-21T18:08:03.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:08:03.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:08:03.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:08:03.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:08:03.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:03.923 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:03 smithi016 conmon[32727]: 2022-03-21T18:08:03.614+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:04 smithi016 ceph-mon[22010]: pgmap v1677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:08:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:08:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:08:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:08:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:08:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:04 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:08:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:08:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:08:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:04.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:08:04.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:04.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:08:04.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:04.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:08:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:08:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:08:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:08:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:08:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:08:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:08:04.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:08:04.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:08:04.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:08:04.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:08:04.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:08:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:08:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:08:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:08:04.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:08:04.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:08:04.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:08:04.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:08:04.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:08:04.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:08:04.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:08:04.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:08:04.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:08:04.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:08:04.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:08:04.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:08:04.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:08:04.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:08:04.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:08:04.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:04 smithi079 ceph-mon[23158]: pgmap v1677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:04.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:08:04.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:04.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:04.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:04.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:08:04.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:04.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:08:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:08:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:04.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:04.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:04.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:08:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.485 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:04 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:08:04.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:08:04.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:08:04.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:08:04.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:08:04.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:08:04.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:08:04.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:08:04.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:08:04.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:08:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:08:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:08:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:08:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:08:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:08:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:08:04.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:08:04.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:08:04.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:08:04.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:08:04.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:08:04.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:08:04.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:08:04.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:08:04.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:08:04.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:08:04.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:08:04.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:08:04.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:08:04.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:08:04.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:08:04.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:08:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:08:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:08:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:08:04.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:08:04.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:08:04.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:08:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:08:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:08:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:08:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:08:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:08:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:08:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:08:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:08:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:08:04.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:08:04.502 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:04 smithi016 conmon[32727]: 2022-03-21T18:08:04.640+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:05 smithi016 conmon[32727]: 2022-03-21T18:08:05.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:06 smithi016 ceph-mon[22010]: pgmap v1678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:06 smithi079 ceph-mon[23158]: pgmap v1678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:06 smithi016 conmon[32727]: 2022-03-21T18:08:06.654+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:07 smithi016 conmon[32727]: 2022-03-21T18:08:07.618+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:08 smithi016 ceph-mon[22010]: pgmap v1679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:08 smithi079 ceph-mon[23158]: pgmap v1679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:08 smithi016 conmon[32727]: 2022-03-21T18:08:08.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:09 smithi016 conmon[32727]: 2022-03-21T18:08:09.642+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:10 smithi016 ceph-mon[22010]: pgmap v1680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:10.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:10 smithi079 ceph-mon[23158]: pgmap v1680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:10 smithi016 conmon[32727]: 2022-03-21T18:08:10.660+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:12.152 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:11 smithi016 conmon[32727]: 2022-03-21T18:08:11.697+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:12 smithi016 ceph-mon[22010]: pgmap v1681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:12 smithi079 ceph-mon[23158]: pgmap v1681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:12 smithi016 conmon[32727]: 2022-03-21T18:08:12.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:13.927 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:13 smithi016 conmon[32727]: 2022-03-21T18:08:13.613+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:14 smithi016 ceph-mon[22010]: pgmap v1682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:14 smithi079 ceph-mon[23158]: pgmap v1682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:14.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:14 smithi016 conmon[32727]: 2022-03-21T18:08:14.590+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:15 smithi016 conmon[32727]: 2022-03-21T18:08:15.620+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:16 smithi079 ceph-mon[23158]: pgmap v1683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:16.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:16.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:16 smithi016 ceph-mon[22010]: pgmap v1683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:16 smithi016 conmon[32727]: 2022-03-21T18:08:16.574+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:17.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:17 smithi016 conmon[32727]: 2022-03-21T18:08:17.577+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:18 smithi079 ceph-mon[23158]: pgmap v1684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:18.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:18.622 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:18 smithi016 ceph-mon[22010]: pgmap v1684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:18 smithi016 conmon[32727]: 2022-03-21T18:08:18.619+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:19.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:19.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:19 smithi016 conmon[32727]: 2022-03-21T18:08:19.663+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:20 smithi079 ceph-mon[23158]: pgmap v1685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:20.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:20.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:20 smithi016 ceph-mon[22010]: pgmap v1685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:20 smithi016 conmon[32727]: 2022-03-21T18:08:20.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:21 smithi016 conmon[32727]: 2022-03-21T18:08:21.681+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:22 smithi079 ceph-mon[23158]: pgmap v1686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:22 smithi016 ceph-mon[22010]: pgmap v1686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:22 smithi016 conmon[32727]: 2022-03-21T18:08:22.682+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:23.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:23 smithi016 conmon[32727]: 2022-03-21T18:08:23.658+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:24 smithi079 ceph-mon[23158]: pgmap v1687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:24.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:24.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:24 smithi016 ceph-mon[22010]: pgmap v1687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:24.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:24 smithi016 conmon[32727]: 2022-03-21T18:08:24.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:25 smithi016 conmon[32727]: 2022-03-21T18:08:25.686+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:26 smithi079 ceph-mon[23158]: pgmap v1688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:26 smithi016 ceph-mon[22010]: pgmap v1688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:26 smithi016 conmon[32727]: 2022-03-21T18:08:26.720+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:27 smithi016 conmon[32727]: 2022-03-21T18:08:27.675+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:28 smithi079 ceph-mon[23158]: pgmap v1689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:28 smithi016 ceph-mon[22010]: pgmap v1689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:28 smithi016 conmon[32727]: 2022-03-21T18:08:28.673+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:29 smithi016 conmon[32727]: 2022-03-21T18:08:29.719+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:30 smithi079 ceph-mon[23158]: pgmap v1690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:30 smithi016 ceph-mon[22010]: pgmap v1690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:30 smithi016 conmon[32727]: 2022-03-21T18:08:30.698+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:31 smithi016 conmon[32727]: 2022-03-21T18:08:31.727+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:32 smithi079 ceph-mon[23158]: pgmap v1691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:32 smithi016 ceph-mon[22010]: pgmap v1691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:32 smithi016 conmon[32727]: 2022-03-21T18:08:32.690+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:33 smithi016 conmon[32727]: 2022-03-21T18:08:33.685+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:34 smithi079 ceph-mon[23158]: pgmap v1692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:34 smithi016 ceph-mon[22010]: pgmap v1692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:34 smithi016 conmon[32727]: 2022-03-21T18:08:34.699+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:35 smithi016 conmon[32727]: 2022-03-21T18:08:35.673+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:36 smithi016 ceph-mon[22010]: pgmap v1693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:36 smithi079 ceph-mon[23158]: pgmap v1693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:36 smithi016 conmon[32727]: 2022-03-21T18:08:36.689+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:37 smithi016 conmon[32727]: 2022-03-21T18:08:37.702+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:38 smithi016 ceph-mon[22010]: pgmap v1694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:38 smithi079 ceph-mon[23158]: pgmap v1694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:38 smithi016 conmon[32727]: 2022-03-21T18:08:38.677+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:39.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:39 smithi016 conmon[32727]: 2022-03-21T18:08:39.680+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:40 smithi016 ceph-mon[22010]: pgmap v1695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:40 smithi079 ceph-mon[23158]: pgmap v1695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:40 smithi016 conmon[32727]: 2022-03-21T18:08:40.678+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:41.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:41 smithi016 conmon[32727]: 2022-03-21T18:08:41.635+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:42.649 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:42.649 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:42 smithi016 ceph-mon[22010]: pgmap v1696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:42 smithi079 ceph-mon[23158]: pgmap v1696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:42 smithi016 conmon[32727]: 2022-03-21T18:08:42.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:43 smithi016 conmon[32727]: 2022-03-21T18:08:43.686+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:44.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:44.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:44 smithi016 ceph-mon[22010]: pgmap v1697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:44.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:44 smithi079 ceph-mon[23158]: pgmap v1697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:44 smithi016 conmon[32727]: 2022-03-21T18:08:44.637+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:45.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:45 smithi016 conmon[32727]: 2022-03-21T18:08:45.653+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:46 smithi016 ceph-mon[22010]: pgmap v1698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:46 smithi079 ceph-mon[23158]: pgmap v1698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:46 smithi016 conmon[32727]: 2022-03-21T18:08:46.683+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:47 smithi016 conmon[32727]: 2022-03-21T18:08:47.669+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:48.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:48.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:48 smithi016 ceph-mon[22010]: pgmap v1699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:48 smithi079 ceph-mon[23158]: pgmap v1699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:48 smithi016 conmon[32727]: 2022-03-21T18:08:48.644+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:49 smithi016 conmon[32727]: 2022-03-21T18:08:49.679+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:50 smithi016 ceph-mon[22010]: pgmap v1700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:50 smithi079 ceph-mon[23158]: pgmap v1700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:50 smithi016 conmon[32727]: 2022-03-21T18:08:50.682+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:51.634 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:51 smithi016 conmon[32727]: 2022-03-21T18:08:51.633+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:52.618 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:52.618 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:52 smithi016 ceph-mon[22010]: pgmap v1701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:52 smithi079 ceph-mon[23158]: pgmap v1701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:52 smithi016 conmon[32727]: 2022-03-21T18:08:52.616+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:53.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:53 smithi016 conmon[32727]: 2022-03-21T18:08:53.591+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:54.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:54.564 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:54 smithi016 ceph-mon[22010]: pgmap v1702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:54.564 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:54 smithi079 ceph-mon[23158]: pgmap v1702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:54 smithi016 conmon[32727]: 2022-03-21T18:08:54.562+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:55.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:55 smithi016 conmon[32727]: 2022-03-21T18:08:55.595+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:56.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:56.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:56 smithi016 ceph-mon[22010]: pgmap v1703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:56.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:56 smithi079 ceph-mon[23158]: pgmap v1703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:08:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:56 smithi016 conmon[32727]: 2022-03-21T18:08:56.580+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:57.544 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:57 smithi016 conmon[32727]: 2022-03-21T18:08:57.543+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:58 smithi016 conmon[32727]: 2022-03-21T18:08:58.512+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:08:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: pgmap v1704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:08:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:08:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:08:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:08:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: pgmap v1704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:08:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:08:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:08:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:08:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:08:59.541 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:59.541 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:08:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:08:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:08:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:08:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:08:59 smithi016 conmon[32727]: 2022-03-21T18:08:59.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:00 smithi016 ceph-mon[22010]: pgmap v1705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:00 smithi016 conmon[32727]: 2022-03-21T18:09:00.498+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:00 smithi079 ceph-mon[23158]: pgmap v1705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:01.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:01.923 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:01 smithi016 conmon[32727]: 2022-03-21T18:09:01.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:02 smithi016 ceph-mon[22010]: pgmap v1706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:02 smithi016 conmon[32727]: 2022-03-21T18:09:02.489+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:02 smithi079 ceph-mon[23158]: pgmap v1706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:02.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[32727]: 2022-03-21T18:09:03.467+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:09:03.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:09:03.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:09:03.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:03.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: 2022-03-21T18:09:03.628+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:03.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:09:03.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:03.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: yield 2022-03-21T18:09:03.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:03.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:09:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:09:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:09:03.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:03.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:09:03.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:03.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:09:03.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:03.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:03.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:09:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:09:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:09:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:04 smithi016 ceph-mon[22010]: pgmap v1707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:04.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:04 smithi016 conmon[32727]: 2022-03-21T18:09:04.475+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:04 smithi079 ceph-mon[23158]: pgmap v1707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:05 smithi016 conmon[32727]: 2022-03-21T18:09:05.517+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:09:05.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:05.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:09:05.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:09:05.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:05.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:09:05.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:09:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:09:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:05.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:05.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:05.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:05 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:09:05.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:09:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:09:05.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:05.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:05.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:05.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:09:05.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:05.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:09:05.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:05.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:09:05.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:05.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:05.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:05.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:05.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:05 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:09:05.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:06 smithi016 ceph-mon[22010]: pgmap v1708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[32727]: 2022-03-21T18:09:06.527+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:06.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 2022-03-21T18:09:06.701+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:06.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:09:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: yield 2022-03-21T18:09:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:09:06.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:06.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:09:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:09:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:06.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:09:06.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:06.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:09:06.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:06.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:06.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:06.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 2022-03-21T18:09:06.701+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:09:06.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:09:06.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:06.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: yield 2022-03-21T18:09:06.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:06.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:06.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:06.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:09:06.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:06.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:09:06.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:06.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:09:06.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:06.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:09:06.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:06.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:09:06.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:06.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:06.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:06.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 2022-03-21T18:09:06.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:09:06.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: 2022-03-21T18:09:06.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:09:06.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:09:06.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:09:06.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:09:06.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:09:06.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:09:06.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:09:06.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:09:06.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:09:06.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:09:06.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:09:06.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:09:06.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:09:06.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:09:06.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:09:06.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:09:06.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:09:06.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:09:06.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:09:06.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:09:06.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:09:06.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:09:06.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise 2022-03-21T18:09:06.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:09:06.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:09:06.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:09:06.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:09:06.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:09:06 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:06 smithi079 ceph-mon[23158]: pgmap v1708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:07 smithi016 conmon[32727]: 2022-03-21T18:09:07.573+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:08 smithi016 conmon[32727]: 2022-03-21T18:09:08.612+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:08.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:09:08.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:08.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:08.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:08.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:09:08.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:08.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:09:08.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:08.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:09:08.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:08.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:08.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:08.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:09:08.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:08.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:08.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:08 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:09:08.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:08.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:09:08.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:08.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:08.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:08.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:09:08.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:08.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:09:08.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:08.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:09:08.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:08.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:08.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:08.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:09:08.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:08.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:08.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:09:08.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:09:08.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:09:08.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:09:08.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:09:08.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:09:08.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:09:08.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:09:08.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:09:08.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:09:08.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:09:08.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:09:08.950 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:09:08.950 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:09:08.951 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:09:08.951 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:09:08.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:09:08.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:09:08.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:09:08.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:09:08.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:09:08.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:09:08.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:09:08.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:09:08.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:09:08.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:09:08.956 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:09:08.956 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:09:08.956 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:09:08.957 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.957 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:08 smithi016 ceph-mon[22010]: pgmap v1709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:08.957 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:08.958 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:09:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:08.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:09:08.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:08.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:09:08.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:08.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:09:08.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:08.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:08.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:08.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:09:08.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:08.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:08.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:08 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:09:08.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:09:08.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:09:08.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:09:08.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:09:08.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:09:08.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:09:08.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:09:08.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:09:08.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:09:08.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:09:08.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:09:08.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:09:08.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:09:08.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:09:08.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:09:08.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:09:08.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:08.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:09:08.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:09:08.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:09:08.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:09:08.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:09:08.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:09:08.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:09:08.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:09:08.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:09:08.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:09:08.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:09:08.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:09:08.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:09:08.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:09:08.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:09:08.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:09:08.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:09:08.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:09:08.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:09:08.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:09:08.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:09:08.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:09:08.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:09:08.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:09:08.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:09:08.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:09:08.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:09:08.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:09:08.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:09:08.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:09:09.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:09:09.000 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:08 smithi079 ceph-mon[23158]: pgmap v1709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:09.001 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:09.001 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:09 smithi016 ceph-mon[22010]: pgmap v1710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:09 smithi016 conmon[32727]: 2022-03-21T18:09:09.608+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:09 smithi079 ceph-mon[23158]: pgmap v1710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:10 smithi016 conmon[32727]: 2022-03-21T18:09:10.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:11 smithi016 ceph-mon[22010]: pgmap v1711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:11 smithi016 conmon[32727]: 2022-03-21T18:09:11.674+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:11 smithi079 ceph-mon[23158]: pgmap v1711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:12 smithi016 conmon[32727]: 2022-03-21T18:09:12.659+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:12.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:13 smithi016 ceph-mon[22010]: pgmap v1712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:13.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:13 smithi016 conmon[32727]: 2022-03-21T18:09:13.670+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:13 smithi079 ceph-mon[23158]: pgmap v1712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:14.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:14 smithi016 conmon[32727]: 2022-03-21T18:09:14.713+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:15 smithi016 ceph-mon[22010]: pgmap v1713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:15.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:15 smithi016 conmon[32727]: 2022-03-21T18:09:15.732+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:15 smithi079 ceph-mon[23158]: pgmap v1713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:16 smithi016 conmon[32727]: 2022-03-21T18:09:16.748+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:17 smithi016 ceph-mon[22010]: pgmap v1714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:17 smithi016 conmon[32727]: 2022-03-21T18:09:17.730+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:17.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:17.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:17 smithi079 ceph-mon[23158]: pgmap v1714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:18 smithi016 conmon[32727]: 2022-03-21T18:09:18.757+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:19 smithi016 conmon[32727]: 2022-03-21T18:09:19.803+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:19 smithi016 ceph-mon[22010]: pgmap v1715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:19 smithi079 ceph-mon[23158]: pgmap v1715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:20 smithi016 conmon[32727]: 2022-03-21T18:09:20.780+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:21.818 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:21.819 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:21 smithi016 ceph-mon[22010]: pgmap v1716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:21 smithi079 ceph-mon[23158]: pgmap v1716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:21 smithi016 conmon[32727]: 2022-03-21T18:09:21.816+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:22.845 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:22 smithi016 conmon[32727]: 2022-03-21T18:09:22.842+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:23 smithi016 conmon[32727]: 2022-03-21T18:09:23.815+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:23 smithi016 ceph-mon[22010]: pgmap v1717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:23.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:23 smithi079 ceph-mon[23158]: pgmap v1717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:24 smithi016 conmon[32727]: 2022-03-21T18:09:24.804+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:25.852 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:25.853 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:25 smithi016 ceph-mon[22010]: pgmap v1718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:25 smithi079 ceph-mon[23158]: pgmap v1718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:25 smithi016 conmon[32727]: 2022-03-21T18:09:25.850+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:26 smithi016 conmon[32727]: 2022-03-21T18:09:26.825+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:27 smithi016 ceph-mon[22010]: pgmap v1719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:27 smithi016 conmon[32727]: 2022-03-21T18:09:27.801+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:27 smithi079 ceph-mon[23158]: pgmap v1719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:28.852 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:28.853 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:28 smithi016 conmon[32727]: 2022-03-21T18:09:28.850+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:29.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:29.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:29 smithi016 ceph-mon[22010]: pgmap v1720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:29 smithi079 ceph-mon[23158]: pgmap v1720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:29 smithi016 conmon[32727]: 2022-03-21T18:09:29.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:30 smithi016 conmon[32727]: 2022-03-21T18:09:30.821+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:31 smithi016 ceph-mon[22010]: pgmap v1721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:31 smithi016 conmon[32727]: 2022-03-21T18:09:31.799+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:31 smithi079 ceph-mon[23158]: pgmap v1721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:32 smithi016 conmon[32727]: 2022-03-21T18:09:32.809+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:33 smithi016 conmon[32727]: 2022-03-21T18:09:33.766+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:33 smithi016 ceph-mon[22010]: pgmap v1722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2816 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:33 smithi079 ceph-mon[23158]: pgmap v1722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2816 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:34 smithi016 conmon[32727]: 2022-03-21T18:09:34.810+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:35 smithi016 conmon[32727]: 2022-03-21T18:09:35.780+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:35 smithi016 ceph-mon[22010]: pgmap v1723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:35 smithi079 ceph-mon[23158]: pgmap v1723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:36 smithi016 conmon[32727]: 2022-03-21T18:09:36.774+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:37 smithi016 conmon[32727]: 2022-03-21T18:09:37.734+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:37 smithi016 ceph-mon[22010]: pgmap v1724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:37 smithi079 ceph-mon[23158]: pgmap v1724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:38.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:38 smithi016 conmon[32727]: 2022-03-21T18:09:38.764+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:39 smithi016 ceph-mon[22010]: pgmap v1725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:39 smithi016 conmon[32727]: 2022-03-21T18:09:39.804+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:39 smithi079 ceph-mon[23158]: pgmap v1725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:40 smithi016 conmon[32727]: 2022-03-21T18:09:40.847+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:41 smithi016 ceph-mon[22010]: pgmap v1726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:41.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:41 smithi016 conmon[32727]: 2022-03-21T18:09:41.818+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:41 smithi079 ceph-mon[23158]: pgmap v1726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:42 smithi016 conmon[32727]: 2022-03-21T18:09:42.787+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:43 smithi016 conmon[32727]: 2022-03-21T18:09:43.777+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:43.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:43.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:43 smithi016 ceph-mon[22010]: pgmap v1727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:43.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:43 smithi079 ceph-mon[23158]: pgmap v1727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:43.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:44 smithi016 conmon[32727]: 2022-03-21T18:09:44.743+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:45 smithi016 conmon[32727]: 2022-03-21T18:09:45.714+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:45 smithi016 ceph-mon[22010]: pgmap v1728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:45 smithi079 ceph-mon[23158]: pgmap v1728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:46 smithi016 conmon[32727]: 2022-03-21T18:09:46.679+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:47 smithi016 conmon[32727]: 2022-03-21T18:09:47.722+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:47 smithi016 ceph-mon[22010]: pgmap v1729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:47 smithi079 ceph-mon[23158]: pgmap v1729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:48 smithi016 conmon[32727]: 2022-03-21T18:09:48.735+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:49 smithi016 ceph-mon[22010]: pgmap v1730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:49 smithi016 conmon[32727]: 2022-03-21T18:09:49.782+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:49 smithi079 ceph-mon[23158]: pgmap v1730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:50 smithi016 conmon[32727]: 2022-03-21T18:09:50.793+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:51 smithi016 ceph-mon[22010]: pgmap v1731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:51 smithi016 conmon[32727]: 2022-03-21T18:09:51.807+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:51 smithi079 ceph-mon[23158]: pgmap v1731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:52 smithi016 conmon[32727]: 2022-03-21T18:09:52.814+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:53 smithi079 ceph-mon[23158]: pgmap v1732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:53 smithi016 conmon[32727]: 2022-03-21T18:09:53.771+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:54.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:53 smithi016 ceph-mon[22010]: pgmap v1732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:54.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:54 smithi016 conmon[32727]: 2022-03-21T18:09:54.806+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:55 smithi079 ceph-mon[23158]: pgmap v1733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:55 smithi016 ceph-mon[22010]: pgmap v1733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:09:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:55 smithi016 conmon[32727]: 2022-03-21T18:09:55.789+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:56 smithi016 conmon[32727]: 2022-03-21T18:09:56.831+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:57 smithi079 ceph-mon[23158]: pgmap v1734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:57 smithi016 ceph-mon[22010]: pgmap v1734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:09:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:57 smithi016 conmon[32727]: 2022-03-21T18:09:57.864+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:09:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:09:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:09:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:09:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:58 smithi016 conmon[32727]: 2022-03-21T18:09:58.868+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:09:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:09:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:09:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:09:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:09:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:09:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:09:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:09:59 smithi079 ceph-mon[23158]: pgmap v1735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:09:59 smithi016 ceph-mon[22010]: pgmap v1735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:09:59 smithi016 conmon[32727]: 2022-03-21T18:09:59.838+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops 2022-03-21T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:10:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:10:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:10:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:10:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops 2022-03-21T18:10:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:00 smithi016 conmon[32727]: 2022-03-21T18:10:00.836+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops 2022-03-21T18:10:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:10:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:10:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:10:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:10:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:10:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2840 sec, osd.4 has slow ops 2022-03-21T18:10:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:01 smithi079 ceph-mon[23158]: pgmap v1736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:01 smithi016 ceph-mon[22010]: pgmap v1736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:01 smithi016 conmon[32727]: 2022-03-21T18:10:01.872+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:02.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:02 smithi016 conmon[32727]: 2022-03-21T18:10:02.869+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:03 smithi079 ceph-mon[23158]: pgmap v1737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2846 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:03 smithi016 ceph-mon[22010]: pgmap v1737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2846 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:03 smithi016 conmon[32727]: 2022-03-21T18:10:03.822+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:04 smithi016 conmon[32727]: 2022-03-21T18:10:04.838+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:05.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:05.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:05 smithi079 ceph-mon[23158]: pgmap v1738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:05 smithi016 ceph-mon[22010]: pgmap v1738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:06.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:05 smithi016 conmon[32727]: 2022-03-21T18:10:05.859+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:06.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:07.125 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:06 smithi016 conmon[32727]: 2022-03-21T18:10:06.848+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:07.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:07.879 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:10:07.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:10:07.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:10:07.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:07.881 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: pgmap v1739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:07.881 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:07.881 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:07 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:10:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:10:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:10:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: pgmap v1739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:07 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:08.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:07 smithi016 conmon[32727]: 2022-03-21T18:10:07.876+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:09.174 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:08 smithi016 conmon[32727]: 2022-03-21T18:10:08.915+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:09 smithi079 ceph-mon[23158]: pgmap v1740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:09.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:10.002 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:09 smithi016 ceph-mon[22010]: pgmap v1740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:10.003 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:10.003 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: 2022-03-21T18:10:09.804+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:10.004 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:10:10.004 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:10.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: yield 2022-03-21T18:10:10.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:10.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:10.006 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:10.006 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:10:10.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:10.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:10:10.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:10.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:10:10.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:10.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:10:10.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:10.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:10:10.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:10.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:10.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:10.011 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:09 smithi016 conmon[32727]: 2022-03-21T18:10:09.877+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:10:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:10:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:10.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:10.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:10.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:10:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:10:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:10:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:10:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:10.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:10.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:10.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:10 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:10:10.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:10.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:10 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T18:10:10.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:10 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:10 smithi016 conmon[32727]: 2022-03-21T18:10:10.865+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:10:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:10:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:10:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:10:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:10:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:10:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:10 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:10:11.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:11.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:10 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T18:10:11.175 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:10 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:10:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:11 smithi079 ceph-mon[23158]: pgmap v1741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:11 smithi016 conmon[32727]: 2022-03-21T18:10:11.873+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:11 smithi016 ceph-mon[22010]: pgmap v1741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[32727]: 2022-03-21T18:10:12.918+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:13.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 2022-03-21T18:10:12.876+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:13.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:10:13.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:13.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: yield 2022-03-21T18:10:13.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:13.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:13.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:13.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:10:13.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:13.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:10:13.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:13.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:10:13.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:13.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:10:13.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:13.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:10:13.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:13.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:13.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:13.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 2022-03-21T18:10:12.877+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:10:13.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:10:13.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:13.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: yield 2022-03-21T18:10:13.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:13.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:13.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:13.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:10:13.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:13.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:10:13.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:13.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:10:13.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:13.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:10:13.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:13.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:10:13.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:13.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:13.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:13.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 2022-03-21T18:10:13.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:10:13.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: 2022-03-21T18:10:13.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:10:13.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:10:13.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:10:13.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:10:13.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:10:13.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:10:13.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:10:13.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:10:13.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:10:13.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:10:13.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:10:13.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:10:13.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:10:13.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:10:13.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:10:13.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:10:13.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:10:13.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:10:13.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:10:13.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:10:13.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:10:13.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:10:13.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise 2022-03-21T18:10:13.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:10:13.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:10:13.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:10:13.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:10:13.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:10:12 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:13 smithi016 conmon[32727]: 2022-03-21T18:10:13.937+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:13 smithi016 ceph-mon[22010]: pgmap v1742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:10:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:10:14.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:10:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:10:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:14.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:14.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:10:14.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:14.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:14.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:13 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:10:14.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:14.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:10:14.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:14.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:14.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:14.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:10:14.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:14.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:10:14.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:14.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:10:14.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:14.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:14.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:14.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:10:14.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:14.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:14.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:10:14.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:10:14.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:10:14.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:10:14.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:10:14.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:10:14.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:10:14.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:10:14.196 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:10:14.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:10:14.197 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:10:14.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:10:14.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:10:14.198 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:10:14.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:10:14.199 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:10:14.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:10:14.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:10:14.200 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:10:14.201 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:10:14.201 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:10:14.201 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:10:14.202 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:10:14.202 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:10:14.202 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:10:14.203 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:10:14.203 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:10:14.203 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:10:14.204 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:10:14.204 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:14.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2856 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:13 smithi079 ceph-mon[23158]: pgmap v1742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:10:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:10:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:10:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:10:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:10:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:13 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:10:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:10:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:10:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:10:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:10:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:10:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:10:14.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:10:14.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:10:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:10:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:10:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:10:14.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:10:14.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:10:14.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:10:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:10:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:10:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:10:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:10:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:10:14.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:10:14.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:10:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:10:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:10:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:10:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:10:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:10:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:10:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:10:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:10:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:10:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:10:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:10:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:10:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:10:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:10:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:10:14.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:10:14.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:10:14.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:10:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:10:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:10:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:10:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:10:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:10:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:10:14.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:10:14.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:10:14.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:14.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2856 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:14 smithi016 conmon[32727]: 2022-03-21T18:10:14.901+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:15 smithi016 ceph-mon[22010]: pgmap v1743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:15 smithi016 conmon[32727]: 2022-03-21T18:10:15.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:15 smithi079 ceph-mon[23158]: pgmap v1743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:16.989 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:16 smithi016 conmon[32727]: 2022-03-21T18:10:16.987+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:17.993 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:17 smithi016 ceph-mon[22010]: pgmap v1744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:17.994 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:17 smithi079 ceph-mon[23158]: pgmap v1744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:17 smithi016 conmon[32727]: 2022-03-21T18:10:17.990+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2861 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:18 smithi016 conmon[32727]: 2022-03-21T18:10:18.974+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2861 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:19 smithi016 ceph-mon[22010]: pgmap v1745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:19 smithi016 conmon[32727]: 2022-03-21T18:10:19.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:19 smithi079 ceph-mon[23158]: pgmap v1745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:20 smithi016 conmon[32727]: 2022-03-21T18:10:20.937+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:21 smithi016 ceph-mon[22010]: pgmap v1746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:21 smithi016 conmon[32727]: 2022-03-21T18:10:21.927+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:21 smithi079 ceph-mon[23158]: pgmap v1746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:22 smithi016 conmon[32727]: 2022-03-21T18:10:22.885+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:23 smithi016 ceph-mon[22010]: pgmap v1747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:23 smithi016 conmon[32727]: 2022-03-21T18:10:23.923+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:23 smithi079 ceph-mon[23158]: pgmap v1747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:24 smithi016 conmon[32727]: 2022-03-21T18:10:24.927+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:25 smithi016 ceph-mon[22010]: pgmap v1748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:25 smithi016 conmon[32727]: 2022-03-21T18:10:25.898+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:25 smithi079 ceph-mon[23158]: pgmap v1748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:26 smithi016 conmon[32727]: 2022-03-21T18:10:26.857+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:27 smithi016 ceph-mon[22010]: pgmap v1749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:27 smithi016 conmon[32727]: 2022-03-21T18:10:27.905+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:27 smithi079 ceph-mon[23158]: pgmap v1749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2871 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:28 smithi016 conmon[32727]: 2022-03-21T18:10:28.942+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2871 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:29 smithi016 ceph-mon[22010]: pgmap v1750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:29 smithi016 conmon[32727]: 2022-03-21T18:10:29.960+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:29 smithi079 ceph-mon[23158]: pgmap v1750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:30 smithi016 conmon[32727]: 2022-03-21T18:10:30.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:31 smithi016 ceph-mon[22010]: pgmap v1751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:31 smithi016 conmon[32727]: 2022-03-21T18:10:31.979+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:31 smithi079 ceph-mon[23158]: pgmap v1751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:33 smithi016 conmon[32727]: 2022-03-21T18:10:33.020+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:34.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:33 smithi016 ceph-mon[22010]: pgmap v1752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:34.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:34.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2876 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:33 smithi079 ceph-mon[23158]: pgmap v1752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2876 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:34 smithi016 conmon[32727]: 2022-03-21T18:10:34.053+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:35.067 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:35 smithi016 conmon[32727]: 2022-03-21T18:10:35.064+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:35 smithi016 ceph-mon[22010]: pgmap v1753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:36 smithi016 conmon[32727]: 2022-03-21T18:10:36.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:35 smithi079 ceph-mon[23158]: pgmap v1753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:37 smithi016 conmon[32727]: 2022-03-21T18:10:37.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:38.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:37 smithi016 ceph-mon[22010]: pgmap v1754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:38.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:37 smithi079 ceph-mon[23158]: pgmap v1754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:38 smithi016 conmon[32727]: 2022-03-21T18:10:38.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:39.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:39.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:39 smithi016 conmon[32727]: 2022-03-21T18:10:39.080+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:40 smithi016 conmon[32727]: 2022-03-21T18:10:40.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:39 smithi016 ceph-mon[22010]: pgmap v1755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:39 smithi079 ceph-mon[23158]: pgmap v1755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:41 smithi016 conmon[32727]: 2022-03-21T18:10:41.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:42.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:41 smithi016 ceph-mon[22010]: pgmap v1756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:42.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:41 smithi079 ceph-mon[23158]: pgmap v1756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:42 smithi016 conmon[32727]: 2022-03-21T18:10:42.092+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:43.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:43 smithi016 conmon[32727]: 2022-03-21T18:10:43.074+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:44 smithi016 conmon[32727]: 2022-03-21T18:10:44.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:43 smithi016 ceph-mon[22010]: pgmap v1757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:43 smithi079 ceph-mon[23158]: pgmap v1757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:45 smithi016 conmon[32727]: 2022-03-21T18:10:45.007+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:45 smithi016 ceph-mon[22010]: pgmap v1758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:46 smithi016 conmon[32727]: 2022-03-21T18:10:46.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:45 smithi079 ceph-mon[23158]: pgmap v1758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:47 smithi016 conmon[32727]: 2022-03-21T18:10:47.039+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:47 smithi016 ceph-mon[22010]: pgmap v1759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:48 smithi016 conmon[32727]: 2022-03-21T18:10:48.005+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:47 smithi079 ceph-mon[23158]: pgmap v1759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:49.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:49 smithi016 conmon[32727]: 2022-03-21T18:10:49.022+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:49 smithi016 ceph-mon[22010]: pgmap v1760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:50 smithi016 conmon[32727]: 2022-03-21T18:10:50.030+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:49 smithi079 ceph-mon[23158]: pgmap v1760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:51 smithi016 conmon[32727]: 2022-03-21T18:10:50.999+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:51 smithi016 ceph-mon[22010]: pgmap v1761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:52 smithi016 conmon[32727]: 2022-03-21T18:10:52.045+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:51 smithi079 ceph-mon[23158]: pgmap v1761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:53 smithi016 conmon[32727]: 2022-03-21T18:10:53.074+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:53.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:54.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:53 smithi016 ceph-mon[22010]: pgmap v1762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:54.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:54.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:53 smithi079 ceph-mon[23158]: pgmap v1762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:10:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:54 smithi016 conmon[32727]: 2022-03-21T18:10:54.124+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:55.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:55 smithi016 conmon[32727]: 2022-03-21T18:10:55.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:55 smithi016 ceph-mon[22010]: pgmap v1763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:55 smithi079 ceph-mon[23158]: pgmap v1763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:10:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:56 smithi016 conmon[32727]: 2022-03-21T18:10:56.164+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:57.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:57 smithi016 conmon[32727]: 2022-03-21T18:10:57.161+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:57 smithi016 ceph-mon[22010]: pgmap v1764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:57 smithi079 ceph-mon[23158]: pgmap v1764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:10:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:58 smithi016 conmon[32727]: 2022-03-21T18:10:58.194+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:10:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:10:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:10:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:10:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:10:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:10:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:10:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:10:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:10:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:10:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:10:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:10:59 smithi016 conmon[32727]: 2022-03-21T18:10:59.243+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:59 smithi016 ceph-mon[22010]: pgmap v1765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:10:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:59 smithi079 ceph-mon[23158]: pgmap v1765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:10:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:00 smithi016 conmon[32727]: 2022-03-21T18:11:00.248+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:01 smithi016 conmon[32727]: 2022-03-21T18:11:01.265+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:01 smithi016 ceph-mon[22010]: pgmap v1766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:01 smithi079 ceph-mon[23158]: pgmap v1766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:02 smithi016 conmon[32727]: 2022-03-21T18:11:02.312+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:03 smithi016 conmon[32727]: 2022-03-21T18:11:03.291+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:03 smithi079 ceph-mon[23158]: pgmap v1767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:04.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:04.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:03 smithi016 ceph-mon[22010]: pgmap v1767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:04.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:04.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:04 smithi016 conmon[32727]: 2022-03-21T18:11:04.342+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:05 smithi016 conmon[32727]: 2022-03-21T18:11:05.330+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:05 smithi079 ceph-mon[23158]: pgmap v1768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:06.350 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:06.350 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:05 smithi016 ceph-mon[22010]: pgmap v1768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:06 smithi016 conmon[32727]: 2022-03-21T18:11:06.348+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:07.375 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:07 smithi016 conmon[32727]: 2022-03-21T18:11:07.372+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:07 smithi079 ceph-mon[23158]: pgmap v1769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:08.343 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:08.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:07 smithi016 ceph-mon[22010]: pgmap v1769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:08 smithi016 conmon[32727]: 2022-03-21T18:11:08.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:09.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:09.370 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:09 smithi016 conmon[32727]: 2022-03-21T18:11:09.367+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:09 smithi079 ceph-mon[23158]: pgmap v1770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:10.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:10.349 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:09 smithi016 ceph-mon[22010]: pgmap v1770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:10 smithi016 conmon[32727]: 2022-03-21T18:11:10.345+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:11.331 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:11 smithi016 conmon[32727]: 2022-03-21T18:11:11.326+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:11 smithi079 ceph-mon[23158]: pgmap v1771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:12.355 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:12.356 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:11 smithi016 ceph-mon[22010]: pgmap v1771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:12 smithi016 conmon[32727]: 2022-03-21T18:11:12.353+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:11:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:11:13.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:11:13.354 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:13.354 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:11:13.355 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:11:13.355 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:11:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:13 smithi016 conmon[32727]: 2022-03-21T18:11:13.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:13 smithi079 ceph-mon[23158]: pgmap v1772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:13 smithi016 ceph-mon[22010]: pgmap v1772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:14 smithi016 conmon[32727]: 2022-03-21T18:11:14.417+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:15.383 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:15 smithi079 ceph-mon[23158]: pgmap v1773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[32727]: 2022-03-21T18:11:15.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:16.435 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:16.436 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:15 smithi016 ceph-mon[22010]: pgmap v1773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:16.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: 2022-03-21T18:11:15.980+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:16.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:11:16.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:16.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: yield 2022-03-21T18:11:16.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:16.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:16.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:16.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:11:16.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:16.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:11:16.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:16.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:11:16.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:16.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:11:16.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:16.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:11:16.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:16.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:16.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:16 smithi016 conmon[32727]: 2022-03-21T18:11:16.423+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:17.397 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:17.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:17 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:17.398 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:11:17.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:17.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:11:17.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:17.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:17.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:17.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:11:17.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:17.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:11:17.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:17.403 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:11:17.403 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:17.404 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:17.404 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:17.404 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:11:17.405 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:17.405 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:17.405 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:17.406 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:17 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:11:17.406 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:17 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T18:11:17.406 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:17 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:11:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:17 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:11:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:11:17.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:17.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:17.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:17.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:11:17.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:17.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:11:17.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:17.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:11:17.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:17.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:17.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:17.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:11:17.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:17.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:17.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:17.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:17 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:11:17.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:17 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T18:11:17.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:17 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:11:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:17 smithi016 conmon[32727]: 2022-03-21T18:11:17.395+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:18.363 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:18.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:18 smithi016 ceph-mon[22010]: pgmap v1774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:18 smithi079 ceph-mon[23158]: pgmap v1774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:18 smithi016 conmon[32727]: 2022-03-21T18:11:18.356+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:19.316 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:19.317 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:19.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 2022-03-21T18:11:19.052+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:19.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:11:19.318 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:19.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: yield 2022-03-21T18:11:19.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:19.319 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:19.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:19.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:11:19.320 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:19.321 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:11:19.321 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:19.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:11:19.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:19.322 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:11:19.323 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:19.323 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:11:19.323 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:19.324 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:19.324 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:19.324 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 2022-03-21T18:11:19.053+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:11:19.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:11:19.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:19.325 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: yield 2022-03-21T18:11:19.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:19.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:19.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:19.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:11:19.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:19.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:11:19.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:19.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:11:19.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:19.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:11:19.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:19.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:11:19.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:19.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:19.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:19.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 2022-03-21T18:11:19.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:11:19.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: 2022-03-21T18:11:19.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:11:19.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:11:19.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:11:19.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:11:19.333 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:11:19.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:11:19.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:11:19.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:11:19.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:11:19.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:11:19.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:11:19.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:11:19.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:11:19.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:11:19.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:11:19.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:11:19.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:11:19.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:11:19.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:11:19.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:11:19.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:11:19.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:11:19.341 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise 2022-03-21T18:11:19.341 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:11:19.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:11:19.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:11:19.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:11:19.343 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:19 smithi016 conmon[32727]: 2022-03-21T18:11:19.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:20.315 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:20.315 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:20 smithi016 ceph-mon[22010]: pgmap v1775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:20.316 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:20 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.316 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.317 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:20.317 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:11:20.317 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:20.318 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:20.318 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:20.319 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:11:20.319 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:20.320 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:11:20.320 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:20.320 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:11:20.321 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:20.321 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:20.321 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:20.322 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:11:20.322 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:20.323 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:20.323 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.324 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:20 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:11:20.324 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.324 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:20.325 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:11:20.325 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:20.325 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:20.326 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:20.326 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:11:20.326 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:20.327 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:11:20.327 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:20.327 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:11:20.328 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:20.328 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:20.328 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:20.329 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:11:20.329 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:20.329 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:20.330 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.330 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:11:20.331 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:11:20.331 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:11:20.331 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.332 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:11:20.332 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:11:20.333 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:11:20.333 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:11:20.334 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:11:20.334 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:11:20.335 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:11:20.336 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:11:20.336 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:11:20.336 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:11:20.337 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:11:20.338 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:11:20.338 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:11:20.339 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:11:20.339 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:11:20.339 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:11:20.340 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:11:20.340 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:11:20.341 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:11:20.341 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:11:20.342 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:11:20.342 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:11:20.343 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:11:20.343 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:11:20.344 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:11:20.344 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:11:20.345 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:20 smithi079 ceph-mon[23158]: pgmap v1775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:20 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:20.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:11:20.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:20.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:20.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:20.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:11:20.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:20.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:11:20.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:20.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:11:20.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:20.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:20.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:20.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:11:20.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:20.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:20.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.487 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:20 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:11:20.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:11:20.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:11:20.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:11:20.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:11:20.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:11:20.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:11:20.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:11:20.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:11:20.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:11:20.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:11:20.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:11:20.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:11:20.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:11:20.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:11:20.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:11:20.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:11:20.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:11:20.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:11:20.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:11:20.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:11:20.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:11:20.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:11:20.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:11:20.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:11:20.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:11:20.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:11:20.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:11:20.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:11:20.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:11:20.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:11:20.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:11:20.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:11:20.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:11:20.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:11:20.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:11:20.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:11:20.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:11:20.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:11:20.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:11:20.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:11:20.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:11:20.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:11:20.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:11:20.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:11:20.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:11:20.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:11:20.508 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:11:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:20 smithi016 conmon[32727]: 2022-03-21T18:11:20.312+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:21.284 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:21 smithi016 conmon[32727]: 2022-03-21T18:11:21.282+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:22 smithi016 conmon[32727]: 2022-03-21T18:11:22.268+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:22 smithi016 ceph-mon[22010]: pgmap v1776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:22 smithi079 ceph-mon[23158]: pgmap v1776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:23.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:23 smithi016 conmon[32727]: 2022-03-21T18:11:23.301+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:24.313 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:24 smithi016 ceph-mon[22010]: pgmap v1777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:24.314 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:24.314 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:24 smithi079 ceph-mon[23158]: pgmap v1777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:24 smithi016 conmon[32727]: 2022-03-21T18:11:24.311+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:25.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:25 smithi016 conmon[32727]: 2022-03-21T18:11:25.313+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:26.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:26.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:26 smithi016 ceph-mon[22010]: pgmap v1778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:26 smithi079 ceph-mon[23158]: pgmap v1778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:26.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:26 smithi016 conmon[32727]: 2022-03-21T18:11:26.341+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:27 smithi016 conmon[32727]: 2022-03-21T18:11:27.390+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:28.383 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:28.384 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:28 smithi016 ceph-mon[22010]: pgmap v1779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:28 smithi079 ceph-mon[23158]: pgmap v1779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:28 smithi016 conmon[32727]: 2022-03-21T18:11:28.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:29 smithi016 conmon[32727]: 2022-03-21T18:11:29.417+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:30.379 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:30.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:30 smithi016 ceph-mon[22010]: pgmap v1780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:30 smithi079 ceph-mon[23158]: pgmap v1780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:30 smithi016 conmon[32727]: 2022-03-21T18:11:30.377+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:31.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:31 smithi016 conmon[32727]: 2022-03-21T18:11:31.402+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:32.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:32.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:32 smithi016 ceph-mon[22010]: pgmap v1781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:32 smithi079 ceph-mon[23158]: pgmap v1781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:32 smithi016 conmon[32727]: 2022-03-21T18:11:32.430+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:33 smithi016 conmon[32727]: 2022-03-21T18:11:33.470+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:34 smithi016 ceph-mon[22010]: pgmap v1782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:34 smithi079 ceph-mon[23158]: pgmap v1782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:34 smithi016 conmon[32727]: 2022-03-21T18:11:34.434+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:35.408 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:35 smithi016 conmon[32727]: 2022-03-21T18:11:35.406+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:36 smithi016 ceph-mon[22010]: pgmap v1783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:36 smithi079 ceph-mon[23158]: pgmap v1783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:36 smithi016 conmon[32727]: 2022-03-21T18:11:36.441+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:37 smithi016 conmon[32727]: 2022-03-21T18:11:37.478+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:38 smithi016 ceph-mon[22010]: pgmap v1784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:38.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:38.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:38 smithi079 ceph-mon[23158]: pgmap v1784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:38 smithi016 conmon[32727]: 2022-03-21T18:11:38.518+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:39 smithi016 conmon[32727]: 2022-03-21T18:11:39.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:40 smithi016 ceph-mon[22010]: pgmap v1785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:40 smithi079 ceph-mon[23158]: pgmap v1785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:40 smithi016 conmon[32727]: 2022-03-21T18:11:40.463+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:41 smithi016 conmon[32727]: 2022-03-21T18:11:41.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:42 smithi016 ceph-mon[22010]: pgmap v1786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:42 smithi079 ceph-mon[23158]: pgmap v1786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:42 smithi016 conmon[32727]: 2022-03-21T18:11:42.528+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:43 smithi016 conmon[32727]: 2022-03-21T18:11:43.504+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:44 smithi016 ceph-mon[22010]: pgmap v1787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:44.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:44 smithi079 ceph-mon[23158]: pgmap v1787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:44 smithi016 conmon[32727]: 2022-03-21T18:11:44.551+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:45 smithi016 conmon[32727]: 2022-03-21T18:11:45.510+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:46 smithi016 ceph-mon[22010]: pgmap v1788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:46 smithi079 ceph-mon[23158]: pgmap v1788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:46 smithi016 conmon[32727]: 2022-03-21T18:11:46.539+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:47 smithi016 conmon[32727]: 2022-03-21T18:11:47.560+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:48 smithi016 ceph-mon[22010]: pgmap v1789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:48 smithi079 ceph-mon[23158]: pgmap v1789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:48.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:48 smithi016 conmon[32727]: 2022-03-21T18:11:48.523+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:49 smithi016 conmon[32727]: 2022-03-21T18:11:49.520+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:50.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:50.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:50 smithi016 ceph-mon[22010]: pgmap v1790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:50 smithi079 ceph-mon[23158]: pgmap v1790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:50 smithi016 conmon[32727]: 2022-03-21T18:11:50.546+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:51 smithi016 conmon[32727]: 2022-03-21T18:11:51.545+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:52.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:52.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:52 smithi016 ceph-mon[22010]: pgmap v1791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:52.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:52.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:52 smithi079 ceph-mon[23158]: pgmap v1791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:52 smithi016 conmon[32727]: 2022-03-21T18:11:52.527+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:53 smithi016 conmon[32727]: 2022-03-21T18:11:53.508+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:54 smithi016 ceph-mon[22010]: pgmap v1792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:54 smithi079 ceph-mon[23158]: pgmap v1792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:54 smithi016 conmon[32727]: 2022-03-21T18:11:54.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:55.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:55 smithi016 conmon[32727]: 2022-03-21T18:11:55.484+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:56 smithi016 ceph-mon[22010]: pgmap v1793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:56 smithi079 ceph-mon[23158]: pgmap v1793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:11:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:56 smithi016 conmon[32727]: 2022-03-21T18:11:56.487+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:57 smithi016 conmon[32727]: 2022-03-21T18:11:57.494+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:58.431 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:58.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: pgmap v1794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:58.433 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:11:58.433 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:11:58.434 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:11:58.434 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:11:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: pgmap v1794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:11:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:11:58.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:11:58.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:11:58.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:11:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:58 smithi016 conmon[32727]: 2022-03-21T18:11:58.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:11:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:11:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:11:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:11:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:11:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:11:59 smithi016 conmon[32727]: 2022-03-21T18:11:59.482+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:00 smithi016 ceph-mon[22010]: pgmap v1795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:00 smithi079 ceph-mon[23158]: pgmap v1795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:00 smithi016 conmon[32727]: 2022-03-21T18:12:00.438+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:01 smithi016 conmon[32727]: 2022-03-21T18:12:01.439+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:02 smithi016 ceph-mon[22010]: pgmap v1796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:02 smithi079 ceph-mon[23158]: pgmap v1796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:02 smithi016 conmon[32727]: 2022-03-21T18:12:02.453+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:03 smithi016 conmon[32727]: 2022-03-21T18:12:03.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:04 smithi016 ceph-mon[22010]: pgmap v1797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:04 smithi079 ceph-mon[23158]: pgmap v1797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:04 smithi016 conmon[32727]: 2022-03-21T18:12:04.483+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:05 smithi016 conmon[32727]: 2022-03-21T18:12:05.453+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:06 smithi016 ceph-mon[22010]: pgmap v1798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:06 smithi079 ceph-mon[23158]: pgmap v1798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:06 smithi016 conmon[32727]: 2022-03-21T18:12:06.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:07 smithi016 conmon[32727]: 2022-03-21T18:12:07.456+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:08 smithi016 ceph-mon[22010]: pgmap v1799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:08 smithi079 ceph-mon[23158]: pgmap v1799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:08 smithi016 conmon[32727]: 2022-03-21T18:12:08.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:09 smithi016 conmon[32727]: 2022-03-21T18:12:09.448+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:10 smithi016 ceph-mon[22010]: pgmap v1800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:10 smithi079 ceph-mon[23158]: pgmap v1800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:10 smithi016 conmon[32727]: 2022-03-21T18:12:10.495+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:11.463 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:11 smithi016 conmon[32727]: 2022-03-21T18:12:11.460+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:12.456 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:12.457 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:12 smithi016 ceph-mon[22010]: pgmap v1801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:12 smithi079 ceph-mon[23158]: pgmap v1801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:12 smithi016 conmon[32727]: 2022-03-21T18:12:12.454+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:13.470 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:13 smithi016 conmon[32727]: 2022-03-21T18:12:13.468+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:14 smithi079 ceph-mon[23158]: pgmap v1802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:14.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:14.519 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:14 smithi016 ceph-mon[22010]: pgmap v1802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:14.520 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:14 smithi016 conmon[32727]: 2022-03-21T18:12:14.515+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:15.480 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:15 smithi016 conmon[32727]: 2022-03-21T18:12:15.478+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:16 smithi079 ceph-mon[23158]: pgmap v1803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:16.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:16.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:16 smithi016 ceph-mon[22010]: pgmap v1803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:16 smithi016 conmon[32727]: 2022-03-21T18:12:16.506+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:17.479 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:17 smithi016 conmon[32727]: 2022-03-21T18:12:17.477+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:18.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:18 smithi079 ceph-mon[23158]: pgmap v1804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:18 smithi016 ceph-mon[22010]: pgmap v1804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:18 smithi016 conmon[32727]: 2022-03-21T18:12:18.433+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:19.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:12:19.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:12:19.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:12:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:12:19.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:12:19.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:12:19.923 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[32727]: 2022-03-21T18:12:19.450+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:19.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: 2022-03-21T18:12:19.660+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:19.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:12:19.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:19.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: yield 2022-03-21T18:12:19.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:19.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:19.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:19.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:12:19.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:19.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:12:19.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:19.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:12:19.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:19.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:12:19.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:19.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:12:19.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:19.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:19.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:20 smithi079 ceph-mon[23158]: pgmap v1805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:20.488 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:20.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:20 smithi016 ceph-mon[22010]: pgmap v1805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:20 smithi016 conmon[32727]: 2022-03-21T18:12:20.486+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:12:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:12:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:12:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:12:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:12:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:12:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:21.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:12:21.517 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:21.517 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:21.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:12:21.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:21.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:12:21.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:21.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:21.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:21.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:12:21.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:21.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:12:21.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:21.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:12:21.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:21.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:21.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:21.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:12:21.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:21.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:21.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:21.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:12:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:21 smithi016 conmon[32727]: 2022-03-21T18:12:21.514+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:22.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:22 smithi079 ceph-mon[23158]: pgmap v1806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:22.540 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:22.540 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:22 smithi016 ceph-mon[22010]: pgmap v1806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[32727]: 2022-03-21T18:12:22.537+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:22.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 2022-03-21T18:12:22.732+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:22.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:12:22.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:22.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: yield 2022-03-21T18:12:22.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:22.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:22.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:22.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:12:22.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:22.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:12:22.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:22.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:12:22.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:22.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:12:22.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:22.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:12:22.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:22.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:22.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:22.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 2022-03-21T18:12:22.733+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:12:22.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:12:22.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:22.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: yield 2022-03-21T18:12:22.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:22.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:22.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:22.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:12:22.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:22.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:12:22.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:22.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:12:22.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:22.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:12:22.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:22.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:12:22.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:22.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:22.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:22.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 2022-03-21T18:12:22.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:12:22.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: 2022-03-21T18:12:22.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:12:22.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:12:22.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:12:22.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:12:22.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:12:22.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:12:22.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:12:22.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:12:22.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:12:22.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:12:22.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:12:22.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:12:22.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:12:22.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:12:22.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:12:22.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:12:22.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:12:22.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:12:22.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:12:22.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:12:22.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:12:22.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:12:22.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise 2022-03-21T18:12:22.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:12:22.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:12:22.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:12:22.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:12:22.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:12:22 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:23.547 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:23 smithi016 conmon[32727]: 2022-03-21T18:12:23.544+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:12:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:12:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:12:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:12:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:12:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:12:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:12:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:12:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:12:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:12:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:12:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:12:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:12:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:12:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:12:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:12:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:12:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:12:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:12:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:12:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:12:24.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:12:24.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:12:24.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:12:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:12:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:12:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:12:24.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:12:24.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:12:24.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:12:24.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:12:24.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:12:24.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:12:24.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:12:24.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:12:24.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:12:24.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:12:24.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:12:24.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:12:24.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:12:24.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.507 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:24 smithi079 ceph-mon[23158]: pgmap v1807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:24.508 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:24.509 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:24.509 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.509 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.510 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:24.510 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:12:24.510 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:24.511 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:24.511 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:24.512 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:12:24.512 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:24.512 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:12:24.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:24.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:12:24.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:24.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:24.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:24.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:12:24.515 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:24.515 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:24.515 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.516 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:12:24.516 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.516 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:12:24.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:12:24.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:12:24.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:12:24.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:12:24.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:12:24.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:12:24.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:12:24.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:12:24.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:12:24.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:12:24.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:12:24.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:12:24.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:12:24.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:12:24.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:12:24.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:12:24.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:12:24.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:12:24.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:12:24.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:12:24.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:12:24.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:12:24.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:12:24.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:12:24.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:12:24.526 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:12:24.526 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:12:24.526 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:12:24.527 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:12:24.527 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:12:24.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:12:24.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:12:24.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:12:24.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:12:24.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:12:24.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:12:24.530 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:12:24.530 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:12:24.530 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:12:24.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:12:24.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:12:24.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:12:24.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:12:24.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:12:24.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:12:24.533 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:12:24.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:24 smithi016 ceph-mon[22010]: pgmap v1807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:24.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:24 smithi016 conmon[32727]: 2022-03-21T18:12:24.497+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:25.488 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:25 smithi016 conmon[32727]: 2022-03-21T18:12:25.485+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:26 smithi079 ceph-mon[23158]: pgmap v1808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:26.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:26.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:26 smithi016 ceph-mon[22010]: pgmap v1808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:26 smithi016 conmon[32727]: 2022-03-21T18:12:26.487+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:27.499 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:27 smithi016 conmon[32727]: 2022-03-21T18:12:27.497+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:28 smithi079 ceph-mon[23158]: pgmap v1809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:28.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:28.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:28 smithi016 ceph-mon[22010]: pgmap v1809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:28 smithi016 conmon[32727]: 2022-03-21T18:12:28.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:29.565 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:29.566 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:29 smithi016 conmon[32727]: 2022-03-21T18:12:29.563+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:30 smithi079 ceph-mon[23158]: pgmap v1810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:30.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:30.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:30 smithi016 ceph-mon[22010]: pgmap v1810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:30 smithi016 conmon[32727]: 2022-03-21T18:12:30.577+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:31.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:31 smithi016 conmon[32727]: 2022-03-21T18:12:31.579+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:32.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:32.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:32 smithi016 ceph-mon[22010]: pgmap v1811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:32 smithi079 ceph-mon[23158]: pgmap v1811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:32 smithi016 conmon[32727]: 2022-03-21T18:12:32.623+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:33.619 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:33 smithi016 conmon[32727]: 2022-03-21T18:12:33.617+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:34.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:34.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:34 smithi016 ceph-mon[22010]: pgmap v1812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:34.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 2995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:34 smithi079 ceph-mon[23158]: pgmap v1812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 2995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:34 smithi016 conmon[32727]: 2022-03-21T18:12:34.577+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:35.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:35 smithi016 conmon[32727]: 2022-03-21T18:12:35.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:36.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:36.564 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:36 smithi016 ceph-mon[22010]: pgmap v1813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:36 smithi079 ceph-mon[23158]: pgmap v1813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:36 smithi016 conmon[32727]: 2022-03-21T18:12:36.561+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:37.551 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:37 smithi016 conmon[32727]: 2022-03-21T18:12:37.549+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:38.522 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:38.522 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:38 smithi016 ceph-mon[22010]: pgmap v1814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:38 smithi079 ceph-mon[23158]: pgmap v1814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:38.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:38 smithi016 conmon[32727]: 2022-03-21T18:12:38.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:39.539 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:39.539 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:39 smithi016 conmon[32727]: 2022-03-21T18:12:39.537+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:40.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:40.583 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:40 smithi016 ceph-mon[22010]: pgmap v1815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:40 smithi079 ceph-mon[23158]: pgmap v1815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:40 smithi016 conmon[32727]: 2022-03-21T18:12:40.581+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:41.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:41 smithi016 conmon[32727]: 2022-03-21T18:12:41.584+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:42.552 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:42.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:42 smithi016 ceph-mon[22010]: pgmap v1816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:42 smithi079 ceph-mon[23158]: pgmap v1816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:42 smithi016 conmon[32727]: 2022-03-21T18:12:42.550+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:43.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:43 smithi016 conmon[32727]: 2022-03-21T18:12:43.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:44.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:44.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:44 smithi016 ceph-mon[22010]: pgmap v1817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:44.527 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:44 smithi079 ceph-mon[23158]: pgmap v1817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:44.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:44.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:44 smithi016 conmon[32727]: 2022-03-21T18:12:44.524+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:45.517 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:45 smithi016 conmon[32727]: 2022-03-21T18:12:45.516+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:46 smithi016 ceph-mon[22010]: pgmap v1818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:46 smithi016 conmon[32727]: 2022-03-21T18:12:46.512+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:46 smithi079 ceph-mon[23158]: pgmap v1818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:47 smithi016 conmon[32727]: 2022-03-21T18:12:47.470+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:48 smithi016 ceph-mon[22010]: pgmap v1819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:48.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:48 smithi016 conmon[32727]: 2022-03-21T18:12:48.505+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:48 smithi079 ceph-mon[23158]: pgmap v1819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:49.528 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:49.528 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:49 smithi016 conmon[32727]: 2022-03-21T18:12:49.526+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:50.541 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:50.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:50 smithi016 ceph-mon[22010]: pgmap v1820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:50 smithi079 ceph-mon[23158]: pgmap v1820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:50 smithi016 conmon[32727]: 2022-03-21T18:12:50.540+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:51.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:51 smithi016 conmon[32727]: 2022-03-21T18:12:51.531+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:52 smithi016 ceph-mon[22010]: pgmap v1821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:52.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:52 smithi016 conmon[32727]: 2022-03-21T18:12:52.529+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:52 smithi079 ceph-mon[23158]: pgmap v1821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:53 smithi016 conmon[32727]: 2022-03-21T18:12:53.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:54 smithi016 ceph-mon[22010]: pgmap v1822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:54.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:54 smithi016 conmon[32727]: 2022-03-21T18:12:54.532+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:54 smithi079 ceph-mon[23158]: pgmap v1822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:55 smithi016 conmon[32727]: 2022-03-21T18:12:55.483+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:56.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:56.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:56 smithi016 ceph-mon[22010]: pgmap v1823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:56.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:56 smithi016 conmon[32727]: 2022-03-21T18:12:56.509+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:56 smithi079 ceph-mon[23158]: pgmap v1823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:12:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:57 smithi016 conmon[32727]: 2022-03-21T18:12:57.461+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:58 smithi016 conmon[32727]: 2022-03-21T18:12:58.444+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: pgmap v1824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:12:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:12:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:12:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:12:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: pgmap v1824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:12:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:12:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:12:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:12:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:12:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:12:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:12:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:12:59 smithi016 conmon[32727]: 2022-03-21T18:12:59.420+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:12:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:12:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:12:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:00 smithi016 conmon[32727]: 2022-03-21T18:13:00.453+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:00 smithi016 ceph-mon[22010]: pgmap v1825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:00 smithi079 ceph-mon[23158]: pgmap v1825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:01 smithi016 conmon[32727]: 2022-03-21T18:13:01.484+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:02 smithi016 ceph-mon[22010]: pgmap v1826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:02 smithi016 conmon[32727]: 2022-03-21T18:13:02.507+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:02 smithi079 ceph-mon[23158]: pgmap v1826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:03 smithi016 conmon[32727]: 2022-03-21T18:13:03.492+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:04 smithi016 conmon[32727]: 2022-03-21T18:13:04.497+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:04 smithi016 ceph-mon[22010]: pgmap v1827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:04 smithi079 ceph-mon[23158]: pgmap v1827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:04.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:05 smithi016 conmon[32727]: 2022-03-21T18:13:05.455+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:06 smithi016 conmon[32727]: 2022-03-21T18:13:06.444+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:06 smithi016 ceph-mon[22010]: pgmap v1828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:06.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:06 smithi079 ceph-mon[23158]: pgmap v1828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:07.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:07 smithi016 conmon[32727]: 2022-03-21T18:13:07.406+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:08 smithi016 conmon[32727]: 2022-03-21T18:13:08.416+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:08 smithi016 ceph-mon[22010]: pgmap v1829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:08 smithi079 ceph-mon[23158]: pgmap v1829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:09 smithi016 conmon[32727]: 2022-03-21T18:13:09.451+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:10 smithi016 ceph-mon[22010]: pgmap v1830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:10.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:10 smithi016 conmon[32727]: 2022-03-21T18:13:10.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:10 smithi079 ceph-mon[23158]: pgmap v1830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:11.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:11 smithi016 conmon[32727]: 2022-03-21T18:13:11.383+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:12 smithi016 ceph-mon[22010]: pgmap v1831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:12 smithi016 conmon[32727]: 2022-03-21T18:13:12.339+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:12 smithi079 ceph-mon[23158]: pgmap v1831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:13 smithi016 ceph-mon[22010]: pgmap v1832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:13 smithi016 conmon[32727]: 2022-03-21T18:13:13.388+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:13 smithi079 ceph-mon[23158]: pgmap v1832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:14 smithi016 conmon[32727]: 2022-03-21T18:13:14.346+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:15 smithi016 ceph-mon[22010]: pgmap v1833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:15 smithi016 conmon[32727]: 2022-03-21T18:13:15.358+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:15 smithi079 ceph-mon[23158]: pgmap v1833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:16 smithi016 conmon[32727]: 2022-03-21T18:13:16.405+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:17 smithi016 ceph-mon[22010]: pgmap v1834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:17 smithi016 conmon[32727]: 2022-03-21T18:13:17.386+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:17 smithi079 ceph-mon[23158]: pgmap v1834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:18 smithi016 conmon[32727]: 2022-03-21T18:13:18.391+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:19 smithi016 ceph-mon[22010]: pgmap v1835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:19 smithi016 conmon[32727]: 2022-03-21T18:13:19.378+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:19 smithi079 ceph-mon[23158]: pgmap v1835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:20 smithi016 conmon[32727]: 2022-03-21T18:13:20.396+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:21 smithi016 ceph-mon[22010]: pgmap v1836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:21.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:21 smithi016 conmon[32727]: 2022-03-21T18:13:21.422+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:21 smithi079 ceph-mon[23158]: pgmap v1836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:22 smithi016 conmon[32727]: 2022-03-21T18:13:22.390+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:23.607 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:23 smithi016 conmon[32727]: 2022-03-21T18:13:23.429+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:13:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:13:23.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:13:23.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: pgmap v1837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:23.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:13:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:13:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:13:23.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: pgmap v1837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:23.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:24 smithi016 conmon[32727]: 2022-03-21T18:13:24.409+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:25 smithi016 ceph-mon[22010]: pgmap v1838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:25.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[32727]: 2022-03-21T18:13:25.383+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:25 smithi079 ceph-mon[23158]: pgmap v1838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:26.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: 2022-03-21T18:13:25.836+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:13:26.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:26.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: yield 2022-03-21T18:13:26.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:26.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:26.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:26.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:13:26.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:26.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:13:26.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:26.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:13:26.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:26.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:13:26.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:26.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:13:26.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:26.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:26.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:25 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.616 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:26 smithi016 conmon[32727]: 2022-03-21T18:13:26.342+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:26 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:13:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:13:26.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:26.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:26.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:26.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:13:26.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:26.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:13:26.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:26.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:13:26.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:26.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:26.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:26.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:13:26.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:26.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:26.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:26 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:13:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:26 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:13:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:13:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:26.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:26.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:26.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:13:26.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:26.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:13:26.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:26.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:13:26.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:26.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:26.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:26.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:13:26.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:26.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:26.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:26.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:26 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:13:27.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:27 smithi016 conmon[32727]: 2022-03-21T18:13:27.354+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:27 smithi016 ceph-mon[22010]: pgmap v1839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:27 smithi079 ceph-mon[23158]: pgmap v1839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[32727]: 2022-03-21T18:13:28.395+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:29.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 2022-03-21T18:13:28.908+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:13:29.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: yield 2022-03-21T18:13:29.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:13:29.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:13:29.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:13:29.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:13:29.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 2022-03-21T18:13:28.909+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:13:29.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:13:29.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: yield 2022-03-21T18:13:29.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:13:29.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:13:29.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:13:29.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:13:29.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 2022-03-21T18:13:29.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:13:29.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: 2022-03-21T18:13:29.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:13:29.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:13:29.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:13:29.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:13:29.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:13:29.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:13:29.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:13:29.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:13:29.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:13:29.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:13:29.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:13:29.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:13:29.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:13:29.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:13:29.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:13:29.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:13:29.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:13:29.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:13:29.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:13:29.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:13:29.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:13:29.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:13:29.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise 2022-03-21T18:13:29.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:13:29.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:13:29.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:13:29.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:13:29.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:13:28 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:29 smithi016 conmon[32727]: 2022-03-21T18:13:29.363+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:13:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:13:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:13:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:13:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:13:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:29 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:13:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:13:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:13:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:13:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:13:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:13:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:13:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:13:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:13:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:13:29.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:13:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:13:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:13:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:13:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:13:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:13:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:13:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:13:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:13:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:13:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:13:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:13:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:13:29.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:13:29.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:13:29.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:13:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:13:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:13:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:13:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:13:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:13:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:13:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:13:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:13:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:13:29.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:29 smithi016 ceph-mon[22010]: pgmap v1840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:13:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:13:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:13:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:13:29.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:13:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:29 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:13:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:13:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:13:29.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:13:29.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:13:29.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:13:29.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:13:29.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:13:29.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:13:29.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:13:29.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:13:29.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:13:29.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:13:29.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:13:29.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:13:29.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:13:29.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:13:29.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:13:29.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:13:29.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:13:29.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:13:29.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:13:29.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:13:29.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:13:29.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:13:29.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:13:29.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:13:29.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:13:29.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:13:29.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:13:29.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:13:29.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:13:29.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:13:29.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:13:29.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:13:29.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:13:29.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:13:29.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:13:29.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:13:29.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:13:29.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:13:29.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:13:29.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:13:29.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:13:29.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:13:29.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:13:29.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:13:29.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:13:29.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:29 smithi079 ceph-mon[23158]: pgmap v1840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:30.629 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:30 smithi016 conmon[32727]: 2022-03-21T18:13:30.322+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:31.634 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:31 smithi016 conmon[32727]: 2022-03-21T18:13:31.370+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:31 smithi016 ceph-mon[22010]: pgmap v1841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:31 smithi079 ceph-mon[23158]: pgmap v1841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:32 smithi016 conmon[32727]: 2022-03-21T18:13:32.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:33 smithi016 ceph-mon[22010]: pgmap v1842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:33 smithi016 conmon[32727]: 2022-03-21T18:13:33.449+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:33 smithi079 ceph-mon[23158]: pgmap v1842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:34 smithi016 conmon[32727]: 2022-03-21T18:13:34.420+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:35 smithi016 ceph-mon[22010]: pgmap v1843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:35 smithi016 conmon[32727]: 2022-03-21T18:13:35.416+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:35 smithi079 ceph-mon[23158]: pgmap v1843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:36 smithi016 conmon[32727]: 2022-03-21T18:13:36.447+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:37 smithi016 ceph-mon[22010]: pgmap v1844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:37 smithi016 conmon[32727]: 2022-03-21T18:13:37.483+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:37 smithi079 ceph-mon[23158]: pgmap v1844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:38.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:38 smithi016 conmon[32727]: 2022-03-21T18:13:38.444+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:39 smithi016 ceph-mon[22010]: pgmap v1845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:39.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:39 smithi016 conmon[32727]: 2022-03-21T18:13:39.461+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:39 smithi079 ceph-mon[23158]: pgmap v1845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:40.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:40 smithi016 conmon[32727]: 2022-03-21T18:13:40.449+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:41 smithi016 conmon[32727]: 2022-03-21T18:13:41.401+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:41 smithi079 ceph-mon[23158]: pgmap v1846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:41 smithi016 ceph-mon[22010]: pgmap v1846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:42 smithi016 conmon[32727]: 2022-03-21T18:13:42.388+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:43 smithi016 conmon[32727]: 2022-03-21T18:13:43.374+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:43 smithi079 ceph-mon[23158]: pgmap v1847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:43 smithi016 ceph-mon[22010]: pgmap v1847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:44 smithi016 conmon[32727]: 2022-03-21T18:13:44.381+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:45 smithi016 conmon[32727]: 2022-03-21T18:13:45.361+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:45 smithi079 ceph-mon[23158]: pgmap v1848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:45 smithi016 ceph-mon[22010]: pgmap v1848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:46 smithi016 conmon[32727]: 2022-03-21T18:13:46.362+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:47 smithi016 conmon[32727]: 2022-03-21T18:13:47.374+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:47 smithi079 ceph-mon[23158]: pgmap v1849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:47 smithi016 ceph-mon[22010]: pgmap v1849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:48.684 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:48 smithi016 conmon[32727]: 2022-03-21T18:13:48.424+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:49.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:49.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:49 smithi016 conmon[32727]: 2022-03-21T18:13:49.456+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:49.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:49.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:49 smithi016 ceph-mon[22010]: pgmap v1850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:49 smithi079 ceph-mon[23158]: pgmap v1850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:50.690 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:50 smithi016 conmon[32727]: 2022-03-21T18:13:50.417+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:50.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:51.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:51 smithi016 conmon[32727]: 2022-03-21T18:13:51.371+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:51 smithi079 ceph-mon[23158]: pgmap v1851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:51 smithi016 ceph-mon[22010]: pgmap v1851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:52 smithi016 conmon[32727]: 2022-03-21T18:13:52.411+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:53.701 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:53 smithi016 conmon[32727]: 2022-03-21T18:13:53.423+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:53 smithi079 ceph-mon[23158]: pgmap v1852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:53 smithi016 ceph-mon[22010]: pgmap v1852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:54 smithi016 conmon[32727]: 2022-03-21T18:13:54.409+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:55.709 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:55 smithi016 conmon[32727]: 2022-03-21T18:13:55.420+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:55 smithi079 ceph-mon[23158]: pgmap v1853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:55 smithi016 ceph-mon[22010]: pgmap v1853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:13:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:56 smithi016 conmon[32727]: 2022-03-21T18:13:56.467+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:57.714 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:57 smithi016 conmon[32727]: 2022-03-21T18:13:57.457+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:57 smithi079 ceph-mon[23158]: pgmap v1854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:58.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:58.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:57 smithi016 ceph-mon[22010]: pgmap v1854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:13:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:58 smithi016 conmon[32727]: 2022-03-21T18:13:58.464+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:13:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:13:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:13:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:13:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:13:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:13:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:13:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:13:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:13:59.721 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:13:59 smithi016 conmon[32727]: 2022-03-21T18:13:59.444+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:13:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:13:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:13:59 smithi079 ceph-mon[23158]: pgmap v1855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:13:59 smithi016 ceph-mon[22010]: pgmap v1855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:00 smithi016 conmon[32727]: 2022-03-21T18:14:00.476+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:01 smithi016 ceph-mon[22010]: pgmap v1856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:01 smithi016 conmon[32727]: 2022-03-21T18:14:01.509+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:01 smithi079 ceph-mon[23158]: pgmap v1856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:02.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:02 smithi016 conmon[32727]: 2022-03-21T18:14:02.513+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:03 smithi016 ceph-mon[22010]: pgmap v1857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:03 smithi016 conmon[32727]: 2022-03-21T18:14:03.556+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:03 smithi079 ceph-mon[23158]: pgmap v1857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:04 smithi016 conmon[32727]: 2022-03-21T18:14:04.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:05 smithi016 ceph-mon[22010]: pgmap v1858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:05 smithi016 conmon[32727]: 2022-03-21T18:14:05.533+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:05 smithi079 ceph-mon[23158]: pgmap v1858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:06 smithi016 conmon[32727]: 2022-03-21T18:14:06.561+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:07 smithi016 ceph-mon[22010]: pgmap v1859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:07 smithi016 conmon[32727]: 2022-03-21T18:14:07.535+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:07 smithi079 ceph-mon[23158]: pgmap v1859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:08 smithi016 conmon[32727]: 2022-03-21T18:14:08.499+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:09 smithi016 ceph-mon[22010]: pgmap v1860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:09 smithi016 conmon[32727]: 2022-03-21T18:14:09.530+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:09 smithi079 ceph-mon[23158]: pgmap v1860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:10 smithi016 conmon[32727]: 2022-03-21T18:14:10.565+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:11 smithi016 ceph-mon[22010]: pgmap v1861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:11 smithi016 conmon[32727]: 2022-03-21T18:14:11.574+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:11 smithi079 ceph-mon[23158]: pgmap v1861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:12 smithi016 conmon[32727]: 2022-03-21T18:14:12.532+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:13.766 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:13 smithi016 conmon[32727]: 2022-03-21T18:14:13.496+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:13 smithi016 ceph-mon[22010]: pgmap v1862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:13 smithi079 ceph-mon[23158]: pgmap v1862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:14 smithi016 conmon[32727]: 2022-03-21T18:14:14.522+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:15 smithi016 ceph-mon[22010]: pgmap v1863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:15 smithi016 conmon[32727]: 2022-03-21T18:14:15.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:15 smithi079 ceph-mon[23158]: pgmap v1863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:16.774 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:16 smithi016 conmon[32727]: 2022-03-21T18:14:16.518+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:17.778 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:17 smithi016 conmon[32727]: 2022-03-21T18:14:17.497+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:17 smithi016 ceph-mon[22010]: pgmap v1864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:17 smithi079 ceph-mon[23158]: pgmap v1864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:18 smithi016 conmon[32727]: 2022-03-21T18:14:18.543+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:19.785 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:19 smithi016 conmon[32727]: 2022-03-21T18:14:19.504+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:19 smithi016 ceph-mon[22010]: pgmap v1865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:19 smithi079 ceph-mon[23158]: pgmap v1865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:20 smithi016 conmon[32727]: 2022-03-21T18:14:20.552+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:21 smithi016 conmon[32727]: 2022-03-21T18:14:21.568+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:21 smithi016 ceph-mon[22010]: pgmap v1866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:22.326 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:21 smithi079 ceph-mon[23158]: pgmap v1866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:22 smithi016 conmon[32727]: 2022-03-21T18:14:22.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:23 smithi016 ceph-mon[22010]: pgmap v1867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:23 smithi016 conmon[32727]: 2022-03-21T18:14:23.612+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:24.235 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:24.236 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:23 smithi079 ceph-mon[23158]: pgmap v1867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:24.236 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:24 smithi016 conmon[32727]: 2022-03-21T18:14:24.594+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:25 smithi016 ceph-mon[22010]: pgmap v1868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:25 smithi016 conmon[32727]: 2022-03-21T18:14:25.640+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:25 smithi079 ceph-mon[23158]: pgmap v1868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:26 smithi016 conmon[32727]: 2022-03-21T18:14:26.595+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:27.811 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:27 smithi016 conmon[32727]: 2022-03-21T18:14:27.552+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:27 smithi016 ceph-mon[22010]: pgmap v1869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:27 smithi079 ceph-mon[23158]: pgmap v1869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:28 smithi016 conmon[32727]: 2022-03-21T18:14:28.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:29.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: 2022-03-21T18:14:29.388+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:29.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:14:29.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:29.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: yield 2022-03-21T18:14:29.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:29.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:29.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:29.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:14:29.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:29.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:14:29.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:29.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:14:29.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:29.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:14:29.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:29.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:14:29.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:29.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:29.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:29 smithi016 conmon[32727]: 2022-03-21T18:14:29.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:14:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:14:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:14:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: pgmap v1870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:14:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:14:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:14:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:14:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:14:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:14:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:14:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:14:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:29 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:14:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:14:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:14:30.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:14:30.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: pgmap v1870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:30.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:30.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:14:30.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:30.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:14:30.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:30.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:30.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:30.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:14:30.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:30.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:14:30.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:30.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:14:30.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:30.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:30.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:30.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:14:30.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:30.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:30.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:30.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:14:30.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:14:30.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:29 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:14:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:30 smithi016 conmon[32727]: 2022-03-21T18:14:30.657+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:31 smithi016 ceph-mon[22010]: pgmap v1871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:31 smithi016 conmon[32727]: 2022-03-21T18:14:31.646+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:31 smithi079 ceph-mon[23158]: pgmap v1871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:32.824 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[32727]: 2022-03-21T18:14:32.640+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:32.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 2022-03-21T18:14:32.460+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:32.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:14:32.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:32.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: yield 2022-03-21T18:14:32.829 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:32.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:32.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:32.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:14:32.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:32.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:14:32.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:32.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:14:32.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:32.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:14:32.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:32.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:14:32.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:32.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:32.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:32.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 2022-03-21T18:14:32.461+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:14:32.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:14:32.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:32.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: yield 2022-03-21T18:14:32.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:32.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:32.839 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:32.839 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:14:32.839 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:32.840 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:14:32.840 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:32.840 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:14:32.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:32.842 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:14:32.843 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:32.843 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:14:32.844 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:32.844 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:32.845 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:32.846 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 2022-03-21T18:14:32.847 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:14:32.847 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: 2022-03-21T18:14:32.848 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:14:32.849 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:14:32.849 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:14:32.850 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:14:32.850 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:14:32.851 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:14:32.852 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:14:32.852 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:14:32.853 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:14:32.853 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:14:32.854 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:14:32.855 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:14:32.856 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:14:32.856 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:14:32.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:14:32.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:14:32.858 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:14:32.859 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:14:32.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:14:32.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:14:32.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:14:32.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:14:32.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise 2022-03-21T18:14:32.863 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:14:32.864 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:14:32.864 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:14:32.865 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:14:32.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:14:32 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:32 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:33.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:14:33.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:33.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:33.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:14:33.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:33.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:14:33.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:33.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:14:33.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:33.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:33.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:33.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:14:33.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:33.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:33.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:32 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:14:33.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:33.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:14:33.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:33.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:33.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:33.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:14:33.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:33.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:14:33.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:33.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:14:33.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:33.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:33.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:33.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:14:33.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:33.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:33.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:14:33.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:14:33.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:14:33.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:14:33.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:14:33.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:14:33.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:14:33.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:14:33.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:14:33.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:14:33.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:14:33.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:14:33.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:14:33.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:14:33.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:14:33.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:14:33.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:14:33.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:14:33.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:14:33.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:14:33.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:14:33.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:14:33.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:14:33.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:14:33.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:14:33.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:14:33.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:14:33.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:14:33.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:14:33.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:32 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:33.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:14:33.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:33.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:33.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:33.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:14:33.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:33.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:14:33.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:33.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:14:33.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:33.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:33.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:33.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:14:33.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:33.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:33.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:32 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:14:33.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:14:33.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:14:33.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:14:33.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:14:33.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:14:33.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:14:33.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:14:33.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:14:33.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:14:33.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:14:33.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:14:33.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:14:33.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:14:33.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:14:33.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:14:33.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:14:33.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:33.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:14:33.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:14:33.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:14:33.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:14:33.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:14:33.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:14:33.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:14:33.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:14:33.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:14:33.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:14:33.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:14:33.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:14:33.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:14:33.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:14:33.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:14:33.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:14:33.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:14:33.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:14:33.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:14:33.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:14:33.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:14:33.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:14:33.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:14:33.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:14:33.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:14:33.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:14:33.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:14:33.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:14:33.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:14:33.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:14:33.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:14:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:33 smithi016 ceph-mon[22010]: pgmap v1872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:33 smithi016 conmon[32727]: 2022-03-21T18:14:33.681+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:33 smithi079 ceph-mon[23158]: pgmap v1872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:34 smithi016 conmon[32727]: 2022-03-21T18:14:34.667+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:35 smithi016 ceph-mon[22010]: pgmap v1873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:35 smithi016 conmon[32727]: 2022-03-21T18:14:35.685+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:35 smithi079 ceph-mon[23158]: pgmap v1873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:36 smithi016 conmon[32727]: 2022-03-21T18:14:36.651+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:37 smithi016 ceph-mon[22010]: pgmap v1874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:37 smithi016 conmon[32727]: 2022-03-21T18:14:37.631+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:37 smithi079 ceph-mon[23158]: pgmap v1874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:38 smithi016 conmon[32727]: 2022-03-21T18:14:38.665+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:39 smithi016 conmon[32727]: 2022-03-21T18:14:39.686+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:39 smithi016 ceph-mon[22010]: pgmap v1875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:39 smithi079 ceph-mon[23158]: pgmap v1875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:40 smithi016 conmon[32727]: 2022-03-21T18:14:40.647+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:41 smithi016 ceph-mon[22010]: pgmap v1876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:41 smithi016 conmon[32727]: 2022-03-21T18:14:41.655+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:41 smithi079 ceph-mon[23158]: pgmap v1876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:42 smithi016 conmon[32727]: 2022-03-21T18:14:42.610+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:43.861 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:43 smithi016 conmon[32727]: 2022-03-21T18:14:43.563+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:43 smithi016 ceph-mon[22010]: pgmap v1877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:43 smithi079 ceph-mon[23158]: pgmap v1877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:44.862 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:44 smithi016 conmon[32727]: 2022-03-21T18:14:44.525+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:45.867 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:45 smithi016 conmon[32727]: 2022-03-21T18:14:45.555+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:45 smithi016 ceph-mon[22010]: pgmap v1878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:45 smithi079 ceph-mon[23158]: pgmap v1878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:46.868 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:46 smithi016 conmon[32727]: 2022-03-21T18:14:46.572+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:47.875 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:47 smithi016 conmon[32727]: 2022-03-21T18:14:47.530+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:47 smithi016 ceph-mon[22010]: pgmap v1879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:47 smithi079 ceph-mon[23158]: pgmap v1879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:48.875 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:48 smithi016 conmon[32727]: 2022-03-21T18:14:48.502+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:49.880 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:49 smithi016 conmon[32727]: 2022-03-21T18:14:49.479+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:49 smithi016 ceph-mon[22010]: pgmap v1880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:49 smithi079 ceph-mon[23158]: pgmap v1880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:50.882 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:50 smithi016 conmon[32727]: 2022-03-21T18:14:50.485+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:51.887 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:51 smithi016 conmon[32727]: 2022-03-21T18:14:51.534+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:51 smithi016 ceph-mon[22010]: pgmap v1881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:51 smithi079 ceph-mon[23158]: pgmap v1881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:52.888 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:52 smithi016 conmon[32727]: 2022-03-21T18:14:52.542+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:53.283 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:53.893 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:53 smithi016 conmon[32727]: 2022-03-21T18:14:53.519+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:53 smithi016 ceph-mon[22010]: pgmap v1882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:53 smithi079 ceph-mon[23158]: pgmap v1882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:54.895 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:54 smithi016 conmon[32727]: 2022-03-21T18:14:54.491+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:55.900 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:55 smithi016 conmon[32727]: 2022-03-21T18:14:55.469+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:55 smithi016 ceph-mon[22010]: pgmap v1883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:55 smithi079 ceph-mon[23158]: pgmap v1883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:14:56.901 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:56 smithi016 conmon[32727]: 2022-03-21T18:14:56.514+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:57.907 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:57 smithi016 conmon[32727]: 2022-03-21T18:14:57.556+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:57 smithi016 ceph-mon[22010]: pgmap v1884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:57 smithi079 ceph-mon[23158]: pgmap v1884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:14:58.908 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:58 smithi016 conmon[32727]: 2022-03-21T18:14:58.509+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:14:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:14:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:14:59.913 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:14:59 smithi016 conmon[32727]: 2022-03-21T18:14:59.515+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:14:59 smithi016 ceph-mon[22010]: pgmap v1885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:14:59 smithi079 ceph-mon[23158]: pgmap v1885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:00 smithi016 conmon[32727]: 2022-03-21T18:15:00.533+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:01 smithi016 conmon[32727]: 2022-03-21T18:15:01.535+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:01 smithi079 ceph-mon[23158]: pgmap v1886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:01 smithi016 ceph-mon[22010]: pgmap v1886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:02 smithi016 conmon[32727]: 2022-03-21T18:15:02.557+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:03 smithi016 conmon[32727]: 2022-03-21T18:15:03.600+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:03 smithi079 ceph-mon[23158]: pgmap v1887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:03 smithi016 ceph-mon[22010]: pgmap v1887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:04 smithi016 conmon[32727]: 2022-03-21T18:15:04.584+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:05 smithi016 conmon[32727]: 2022-03-21T18:15:05.629+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:05 smithi079 ceph-mon[23158]: pgmap v1888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:05 smithi016 ceph-mon[22010]: pgmap v1888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:06 smithi016 conmon[32727]: 2022-03-21T18:15:06.602+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:07 smithi016 conmon[32727]: 2022-03-21T18:15:07.643+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:07 smithi079 ceph-mon[23158]: pgmap v1889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:07 smithi016 ceph-mon[22010]: pgmap v1889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:08.940 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:08 smithi016 conmon[32727]: 2022-03-21T18:15:08.679+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:09 smithi016 conmon[32727]: 2022-03-21T18:15:09.706+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:10.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:09 smithi016 ceph-mon[22010]: pgmap v1890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:09 smithi079 ceph-mon[23158]: pgmap v1890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:10.946 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:10 smithi016 conmon[32727]: 2022-03-21T18:15:10.692+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:12.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:11 smithi016 ceph-mon[22010]: pgmap v1891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:11 smithi016 conmon[32727]: 2022-03-21T18:15:11.708+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:11 smithi079 ceph-mon[23158]: pgmap v1891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:12 smithi016 conmon[32727]: 2022-03-21T18:15:12.748+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:13 smithi016 ceph-mon[22010]: pgmap v1892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:14.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:13 smithi016 conmon[32727]: 2022-03-21T18:15:13.788+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:13 smithi079 ceph-mon[23158]: pgmap v1892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:14 smithi016 conmon[32727]: 2022-03-21T18:15:14.827+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:15 smithi016 ceph-mon[22010]: pgmap v1893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:16.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:15 smithi016 conmon[32727]: 2022-03-21T18:15:15.827+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:15 smithi079 ceph-mon[23158]: pgmap v1893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:16 smithi016 conmon[32727]: 2022-03-21T18:15:16.871+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:17 smithi016 ceph-mon[22010]: pgmap v1894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:17 smithi016 conmon[32727]: 2022-03-21T18:15:17.904+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:18.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:17 smithi079 ceph-mon[23158]: pgmap v1894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:18 smithi016 conmon[32727]: 2022-03-21T18:15:18.916+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:19 smithi016 ceph-mon[22010]: pgmap v1895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:19 smithi016 conmon[32727]: 2022-03-21T18:15:19.867+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:19 smithi079 ceph-mon[23158]: pgmap v1895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:21.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:20 smithi016 conmon[32727]: 2022-03-21T18:15:20.849+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:21 smithi016 ceph-mon[22010]: pgmap v1896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:21 smithi016 conmon[32727]: 2022-03-21T18:15:21.859+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:21 smithi079 ceph-mon[23158]: pgmap v1896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:22 smithi016 conmon[32727]: 2022-03-21T18:15:22.887+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:23 smithi016 conmon[32727]: 2022-03-21T18:15:23.912+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:24.519 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:24.520 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:23 smithi016 ceph-mon[22010]: pgmap v1897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:24.520 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:24.521 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:24.521 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:23 smithi079 ceph-mon[23158]: pgmap v1897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:24.522 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:24 smithi016 conmon[32727]: 2022-03-21T18:15:24.942+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:25 smithi016 ceph-mon[22010]: pgmap v1898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:26.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:25 smithi016 conmon[32727]: 2022-03-21T18:15:25.980+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:25 smithi079 ceph-mon[23158]: pgmap v1898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:27 smithi016 conmon[32727]: 2022-03-21T18:15:27.024+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:27 smithi016 ceph-mon[22010]: pgmap v1899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:28 smithi016 conmon[32727]: 2022-03-21T18:15:28.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:27 smithi079 ceph-mon[23158]: pgmap v1899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:29 smithi016 conmon[32727]: 2022-03-21T18:15:29.040+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:30 smithi016 ceph-mon[22010]: pgmap v1900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:30.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:30 smithi016 conmon[32727]: 2022-03-21T18:15:30.055+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:30 smithi079 ceph-mon[23158]: pgmap v1900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:31.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:31 smithi016 conmon[32727]: 2022-03-21T18:15:31.086+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:32 smithi016 ceph-mon[22010]: pgmap v1901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:32 smithi016 conmon[32727]: 2022-03-21T18:15:32.068+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:32 smithi079 ceph-mon[23158]: pgmap v1901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:15:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:15:33.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:15:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:33 smithi016 conmon[32727]: 2022-03-21T18:15:33.039+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:15:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:15:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:15:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:34 smithi016 ceph-mon[22010]: pgmap v1902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:34 smithi016 conmon[32727]: 2022-03-21T18:15:34.089+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:34 smithi079 ceph-mon[23158]: pgmap v1902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[32727]: 2022-03-21T18:15:35.125+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:35.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: 2022-03-21T18:15:35.565+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:35.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:15:35.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:35.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: yield 2022-03-21T18:15:35.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:35.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:35.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:35.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:15:35.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:35.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:15:35.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:35.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:15:35.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:35.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:15:35.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:35.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:15:35.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:35.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:35.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:35 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:36 smithi016 conmon[32727]: 2022-03-21T18:15:36.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:36 smithi016 ceph-mon[22010]: pgmap v1903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:36.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:15:36.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:36.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:15:36.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:36.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:36.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:36.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:15:36.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:36.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:15:36.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:36.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:15:36.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:36.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:36.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:36.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:15:36.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:36.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:36.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:36.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:15:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:36 smithi079 ceph-mon[23158]: pgmap v1903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:15:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:36.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:15:36.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:36.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:36.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:36.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:15:36.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:36.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:15:36.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:36.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:15:36.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:36.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:36.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:36.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:15:36.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:36.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:36.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:36.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:15:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:37 smithi016 conmon[32727]: 2022-03-21T18:15:37.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:38 smithi016 ceph-mon[22010]: pgmap v1904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[32727]: 2022-03-21T18:15:38.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:38 smithi079 ceph-mon[23158]: pgmap v1904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:38.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 2022-03-21T18:15:38.637+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:38.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:15:38.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:38.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: yield 2022-03-21T18:15:38.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:38.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:38.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:38.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:15:38.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:38.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:15:38.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:38.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:15:38.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:38.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:15:38.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:38.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:15:38.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:38.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:38.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:38.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 2022-03-21T18:15:38.638+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:15:38.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:15:38.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:38.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: yield 2022-03-21T18:15:38.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:38.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:38.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:38.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:15:38.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:38.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:15:38.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:38.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:15:38.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:38.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:15:38.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:38.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:15:38.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:38.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:38.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:38.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 2022-03-21T18:15:38.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:15:38.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: 2022-03-21T18:15:38.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:15:38.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:15:38.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:15:38.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:15:38.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:15:38.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:15:38.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:15:38.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:15:38.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:15:38.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:15:38.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:15:38.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:15:38.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:15:38.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:15:38.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:15:38.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:15:38.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:15:38.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:15:38.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:15:38.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:15:38.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:15:38.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:15:38.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise 2022-03-21T18:15:38.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:15:38.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:15:38.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:15:38.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:15:38.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:15:38 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:39 smithi016 conmon[32727]: 2022-03-21T18:15:39.113+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:40 smithi016 conmon[32727]: 2022-03-21T18:15:40.117+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:40 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:15:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:40.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:15:40.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:40.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:15:40.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:40.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:15:40.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:40.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:15:40.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:40.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:40.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:40 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:15:40.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:40.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:15:40.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:40.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:40.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:40.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:15:40.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:40.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:15:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:15:40.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:40.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:15:40.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:40.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:40.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:15:40.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:15:40.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:15:40.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:15:40.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:15:40.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:15:40.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:15:40.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:15:40.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:15:40.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:15:40.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:15:40.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:15:40.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:15:40.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:15:40.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:15:40.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:15:40.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:15:40.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:15:40.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:15:40.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:15:40.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:15:40.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:15:40.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:15:40.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:15:40.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:15:40.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:15:40.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:15:40.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:15:40.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:15:40.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.452 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:40 smithi016 ceph-mon[22010]: pgmap v1905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:40.452 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:40 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:15:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:15:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:40.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:15:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:15:40.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:40.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:15:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:40.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:40 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:15:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:15:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:15:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:15:40.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:15:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:15:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:15:40.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:15:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:15:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:15:40.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:15:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:15:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:15:40.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:15:40.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:15:40.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:15:40.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:15:40.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:15:40.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:15:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:15:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:15:40.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:15:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:15:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:15:40.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:15:40.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:15:40.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:15:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:15:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:15:40.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:15:40.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:15:40.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:15:40.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:15:40.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:15:40.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:15:40.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:15:40.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:15:40.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:15:40.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:15:40.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:15:40.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:15:40.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:15:40.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:15:40.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:15:40.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:15:40.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:15:40.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:15:40.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:15:40.506 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:40 smithi079 ceph-mon[23158]: pgmap v1905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:40.507 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:41 smithi016 conmon[32727]: 2022-03-21T18:15:41.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:42 smithi016 ceph-mon[22010]: pgmap v1906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:42 smithi016 conmon[32727]: 2022-03-21T18:15:42.156+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:42 smithi079 ceph-mon[23158]: pgmap v1906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:43 smithi016 conmon[32727]: 2022-03-21T18:15:43.112+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:44 smithi016 conmon[32727]: 2022-03-21T18:15:44.073+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:44 smithi016 ceph-mon[22010]: pgmap v1907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:44 smithi079 ceph-mon[23158]: pgmap v1907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:44.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:45 smithi016 conmon[32727]: 2022-03-21T18:15:45.077+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:46 smithi016 ceph-mon[22010]: pgmap v1908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:46 smithi016 conmon[32727]: 2022-03-21T18:15:46.031+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:46 smithi079 ceph-mon[23158]: pgmap v1908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:47 smithi016 conmon[32727]: 2022-03-21T18:15:47.014+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:48 smithi016 ceph-mon[22010]: pgmap v1909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:47 smithi016 conmon[32727]: 2022-03-21T18:15:47.964+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:48 smithi079 ceph-mon[23158]: pgmap v1909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:49.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:48 smithi016 conmon[32727]: 2022-03-21T18:15:48.995+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:50 smithi016 ceph-mon[22010]: pgmap v1910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:49 smithi016 conmon[32727]: 2022-03-21T18:15:49.950+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:50 smithi079 ceph-mon[23158]: pgmap v1910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:50 smithi016 conmon[32727]: 2022-03-21T18:15:50.931+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:52 smithi016 ceph-mon[22010]: pgmap v1911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:51 smithi016 conmon[32727]: 2022-03-21T18:15:51.977+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:52 smithi079 ceph-mon[23158]: pgmap v1911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:52 smithi016 conmon[32727]: 2022-03-21T18:15:52.975+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:54 smithi016 ceph-mon[22010]: pgmap v1912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:53 smithi016 conmon[32727]: 2022-03-21T18:15:53.993+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:54 smithi079 ceph-mon[23158]: pgmap v1912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:55 smithi016 conmon[32727]: 2022-03-21T18:15:55.039+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:56 smithi016 ceph-mon[22010]: pgmap v1913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:56 smithi016 conmon[32727]: 2022-03-21T18:15:56.028+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:56 smithi079 ceph-mon[23158]: pgmap v1913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:15:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:57.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:56 smithi016 conmon[32727]: 2022-03-21T18:15:56.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:58 smithi016 ceph-mon[22010]: pgmap v1914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:57 smithi016 conmon[32727]: 2022-03-21T18:15:57.989+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:58 smithi079 ceph-mon[23158]: pgmap v1914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:15:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:58 smithi016 conmon[32727]: 2022-03-21T18:15:58.947+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:15:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:15:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:15:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:15:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:15:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:15:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:15:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:15:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:15:59.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:15:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:00 smithi016 ceph-mon[22010]: pgmap v1915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:15:59 smithi016 conmon[32727]: 2022-03-21T18:15:59.900+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:00 smithi079 ceph-mon[23158]: pgmap v1915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:00 smithi016 conmon[32727]: 2022-03-21T18:16:00.945+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:02 smithi016 ceph-mon[22010]: pgmap v1916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:01 smithi016 conmon[32727]: 2022-03-21T18:16:01.932+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:02 smithi079 ceph-mon[23158]: pgmap v1916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:02 smithi016 conmon[32727]: 2022-03-21T18:16:02.921+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:04 smithi016 ceph-mon[22010]: pgmap v1917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:03 smithi016 conmon[32727]: 2022-03-21T18:16:03.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:04 smithi079 ceph-mon[23158]: pgmap v1917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:04 smithi016 conmon[32727]: 2022-03-21T18:16:04.953+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:06 smithi016 ceph-mon[22010]: pgmap v1918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:05 smithi016 conmon[32727]: 2022-03-21T18:16:05.958+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:06 smithi079 ceph-mon[23158]: pgmap v1918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:06 smithi016 conmon[32727]: 2022-03-21T18:16:06.944+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:08 smithi016 ceph-mon[22010]: pgmap v1919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:07 smithi016 conmon[32727]: 2022-03-21T18:16:07.940+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:08 smithi079 ceph-mon[23158]: pgmap v1919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:08 smithi016 conmon[32727]: 2022-03-21T18:16:08.899+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:10 smithi016 ceph-mon[22010]: pgmap v1920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:09 smithi016 conmon[32727]: 2022-03-21T18:16:09.923+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:10 smithi079 ceph-mon[23158]: pgmap v1920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:10 smithi016 conmon[32727]: 2022-03-21T18:16:10.919+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:11.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:12 smithi016 ceph-mon[22010]: pgmap v1921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:11 smithi016 conmon[32727]: 2022-03-21T18:16:11.930+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:12 smithi079 ceph-mon[23158]: pgmap v1921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:13.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:12 smithi016 conmon[32727]: 2022-03-21T18:16:12.956+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:14 smithi016 ceph-mon[22010]: pgmap v1922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:13 smithi016 conmon[32727]: 2022-03-21T18:16:13.929+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:14 smithi079 ceph-mon[23158]: pgmap v1922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:14 smithi016 conmon[32727]: 2022-03-21T18:16:14.884+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:16 smithi016 ceph-mon[22010]: pgmap v1923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:15 smithi016 conmon[32727]: 2022-03-21T18:16:15.933+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:16 smithi079 ceph-mon[23158]: pgmap v1923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:16 smithi016 conmon[32727]: 2022-03-21T18:16:16.930+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:18 smithi016 ceph-mon[22010]: pgmap v1924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:17 smithi016 conmon[32727]: 2022-03-21T18:16:17.909+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:18 smithi079 ceph-mon[23158]: pgmap v1924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:18 smithi016 conmon[32727]: 2022-03-21T18:16:18.955+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:20 smithi016 ceph-mon[22010]: pgmap v1925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:20.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:19 smithi016 conmon[32727]: 2022-03-21T18:16:19.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:20 smithi079 ceph-mon[23158]: pgmap v1925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:20 smithi016 conmon[32727]: 2022-03-21T18:16:20.969+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:22 smithi016 ceph-mon[22010]: pgmap v1926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:21 smithi016 conmon[32727]: 2022-03-21T18:16:21.969+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:22 smithi079 ceph-mon[23158]: pgmap v1926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:23 smithi016 conmon[32727]: 2022-03-21T18:16:23.006+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:24 smithi016 ceph-mon[22010]: pgmap v1927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:24.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:24 smithi016 conmon[32727]: 2022-03-21T18:16:24.014+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:24 smithi079 ceph-mon[23158]: pgmap v1927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:25 smithi016 conmon[32727]: 2022-03-21T18:16:25.012+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:26 smithi016 ceph-mon[22010]: pgmap v1928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:26.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:26.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:26 smithi016 conmon[32727]: 2022-03-21T18:16:26.006+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:26 smithi079 ceph-mon[23158]: pgmap v1928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:27 smithi016 conmon[32727]: 2022-03-21T18:16:27.000+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:28 smithi016 ceph-mon[22010]: pgmap v1929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:28.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:28 smithi016 conmon[32727]: 2022-03-21T18:16:28.021+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:28 smithi079 ceph-mon[23158]: pgmap v1929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:29.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:29 smithi016 conmon[32727]: 2022-03-21T18:16:29.043+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:30 smithi016 ceph-mon[22010]: pgmap v1930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:30 smithi016 conmon[32727]: 2022-03-21T18:16:30.041+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:30 smithi079 ceph-mon[23158]: pgmap v1930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:31 smithi016 conmon[32727]: 2022-03-21T18:16:31.087+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:32 smithi016 ceph-mon[22010]: pgmap v1931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:32 smithi016 conmon[32727]: 2022-03-21T18:16:32.133+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:32 smithi079 ceph-mon[23158]: pgmap v1931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:32.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:33.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:33.422 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:33 smithi016 conmon[32727]: 2022-03-21T18:16:33.097+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:34 smithi016 ceph-mon[22010]: pgmap v1932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:34 smithi016 conmon[32727]: 2022-03-21T18:16:34.138+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:34 smithi079 ceph-mon[23158]: pgmap v1932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:35 smithi016 conmon[32727]: 2022-03-21T18:16:35.099+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:36 smithi016 ceph-mon[22010]: pgmap v1933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:36 smithi016 conmon[32727]: 2022-03-21T18:16:36.129+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:36 smithi079 ceph-mon[23158]: pgmap v1933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:37 smithi016 conmon[32727]: 2022-03-21T18:16:37.103+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:38 smithi016 ceph-mon[22010]: pgmap v1934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:38 smithi016 conmon[32727]: 2022-03-21T18:16:38.127+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:38 smithi079 ceph-mon[23158]: pgmap v1934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:39 smithi016 conmon[32727]: 2022-03-21T18:16:39.135+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:16:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:16:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:16:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:16:39.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:16:39.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:16:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:40 smithi016 conmon[32727]: 2022-03-21T18:16:40.092+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:40 smithi016 ceph-mon[22010]: pgmap v1935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:40 smithi079 ceph-mon[23158]: pgmap v1935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[32727]: 2022-03-21T18:16:41.052+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:42.067 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: 2022-03-21T18:16:41.741+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:42.068 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:16:42.068 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:42.069 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: yield 2022-03-21T18:16:42.069 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:42.069 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:42.070 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:42.070 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:16:42.071 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:42.071 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:16:42.072 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:42.072 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:16:42.073 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:42.073 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:16:42.073 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:42.074 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:16:42.074 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:42.075 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:42.076 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:42 smithi016 ceph-mon[22010]: pgmap v1936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:42 smithi016 conmon[32727]: 2022-03-21T18:16:42.065+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:42 smithi079 ceph-mon[23158]: pgmap v1936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:43 smithi016 conmon[32727]: 2022-03-21T18:16:43.048+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:16:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:16:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:16:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:16:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:16:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:16:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:43 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:16:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:16:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:16:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:16:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:43.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:16:43.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:43.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:16:43.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:43.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:43.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:16:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:43 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:16:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:44 smithi016 ceph-mon[22010]: pgmap v1937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[32727]: 2022-03-21T18:16:44.015+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:44 smithi079 ceph-mon[23158]: pgmap v1937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:45 smithi016 conmon[32727]: 2022-03-21T18:16:45.016+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:45.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 2022-03-21T18:16:44.813+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:45.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:16:45.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:45.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: yield 2022-03-21T18:16:45.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:45.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:45.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:45.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:16:45.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:45.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:16:45.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:45.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:16:45.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:45.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:16:45.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:45.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:16:45.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:45.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:45.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:45.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 2022-03-21T18:16:44.814+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:16:45.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:16:45.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:45.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: yield 2022-03-21T18:16:45.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:45.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:45.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:45.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:16:45.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:45.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:16:45.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:45.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:16:45.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:45.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:16:45.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:45.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:16:45.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:45.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:45.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:45.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 2022-03-21T18:16:45.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:16:45.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: 2022-03-21T18:16:45.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:16:45.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:16:45.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:16:45.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:16:45.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:16:45.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:16:45.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:16:45.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:16:45.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:16:45.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:16:45.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:16:45.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:16:45.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:16:45.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:16:45.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:16:45.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:16:45.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:16:45.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:16:45.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:16:45.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:16:45.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:16:45.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:16:45.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise 2022-03-21T18:16:45.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:16:45.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:16:45.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:16:45.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:16:45.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:16:44 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:46 smithi016 conmon[32727]: 2022-03-21T18:16:46.004+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:16:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:16:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:16:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:16:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:16:46.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:46.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:46 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:16:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:16:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:16:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:16:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:16:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:16:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:16:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:16:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:16:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:16:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:16:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:16:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:16:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:16:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:16:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:16:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:16:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:16:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:16:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:16:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:16:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:16:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:16:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:16:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:16:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:16:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:16:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:16:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:16:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:16:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:16:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:16:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:16:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:16:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:16:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:46 smithi016 ceph-mon[22010]: pgmap v1938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:16:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:46.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:16:46.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:16:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:16:46.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:46.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:46.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:46.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:16:46.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:46 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:16:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:16:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:16:46.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:16:46.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:16:46.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:16:46.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:16:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:16:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:16:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:16:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:16:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:16:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:16:46.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:16:46.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:16:46.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:16:46.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:16:46.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:16:46.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:16:46.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:16:46.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:16:46.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:16:46.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:16:46.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:16:46.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:16:46.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:16:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:16:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:16:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:16:46.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:16:46.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:16:46.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:16:46.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:16:46.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:16:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:16:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:16:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:16:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:16:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:16:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:16:46.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:16:46.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:16:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:16:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:16:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:16:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:46 smithi079 ceph-mon[23158]: pgmap v1938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:47 smithi016 conmon[32727]: 2022-03-21T18:16:47.001+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:47.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:48.226 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:47 smithi016 conmon[32727]: 2022-03-21T18:16:47.951+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:48 smithi016 ceph-mon[22010]: pgmap v1939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:48 smithi079 ceph-mon[23158]: pgmap v1939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:48 smithi016 conmon[32727]: 2022-03-21T18:16:48.910+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:50.229 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:49 smithi016 conmon[32727]: 2022-03-21T18:16:49.954+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:50 smithi016 ceph-mon[22010]: pgmap v1940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:50 smithi079 ceph-mon[23158]: pgmap v1940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:50 smithi016 conmon[32727]: 2022-03-21T18:16:50.984+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:52.235 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:51 smithi016 conmon[32727]: 2022-03-21T18:16:51.936+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:52 smithi016 ceph-mon[22010]: pgmap v1941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:52 smithi079 ceph-mon[23158]: pgmap v1941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:53.240 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:52 smithi016 conmon[32727]: 2022-03-21T18:16:52.949+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:54 smithi016 ceph-mon[22010]: pgmap v1942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:53 smithi016 conmon[32727]: 2022-03-21T18:16:53.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:54 smithi079 ceph-mon[23158]: pgmap v1942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:55 smithi016 conmon[32727]: 2022-03-21T18:16:55.021+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:56.250 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:55 smithi016 conmon[32727]: 2022-03-21T18:16:55.993+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:56 smithi016 ceph-mon[22010]: pgmap v1943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:56 smithi079 ceph-mon[23158]: pgmap v1943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:16:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:57.253 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:56 smithi016 conmon[32727]: 2022-03-21T18:16:56.983+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:58.254 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:58 smithi016 conmon[32727]: 2022-03-21T18:16:58.002+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: pgmap v1944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:16:58.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:16:58.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:16:58.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:16:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: pgmap v1944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:16:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:16:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:16:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:16:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:16:59.256 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:58 smithi016 conmon[32727]: 2022-03-21T18:16:58.997+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:16:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:16:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:16:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:16:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:16:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:00.257 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:16:59 smithi016 conmon[32727]: 2022-03-21T18:16:59.967+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:00 smithi016 ceph-mon[22010]: pgmap v1945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:00 smithi079 ceph-mon[23158]: pgmap v1945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:01.262 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:00 smithi016 conmon[32727]: 2022-03-21T18:17:00.982+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:02 smithi016 conmon[32727]: 2022-03-21T18:17:02.018+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:02.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:02 smithi016 ceph-mon[22010]: pgmap v1946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:02.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:02 smithi079 ceph-mon[23158]: pgmap v1946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:03.269 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:03 smithi016 conmon[32727]: 2022-03-21T18:17:03.017+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:04 smithi016 conmon[32727]: 2022-03-21T18:17:04.050+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:04 smithi016 ceph-mon[22010]: pgmap v1947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:04 smithi079 ceph-mon[23158]: pgmap v1947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:05 smithi016 conmon[32727]: 2022-03-21T18:17:05.070+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:06 smithi016 ceph-mon[22010]: pgmap v1948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:06 smithi016 conmon[32727]: 2022-03-21T18:17:06.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:06.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:06 smithi079 ceph-mon[23158]: pgmap v1948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:06.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:07 smithi016 conmon[32727]: 2022-03-21T18:17:07.070+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:08 smithi016 ceph-mon[22010]: pgmap v1949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:08 smithi016 conmon[32727]: 2022-03-21T18:17:08.085+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:08 smithi079 ceph-mon[23158]: pgmap v1949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:09.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:09 smithi016 conmon[32727]: 2022-03-21T18:17:09.120+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:10 smithi016 ceph-mon[22010]: pgmap v1950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:10 smithi016 conmon[32727]: 2022-03-21T18:17:10.128+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:10 smithi079 ceph-mon[23158]: pgmap v1950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:11 smithi016 conmon[32727]: 2022-03-21T18:17:11.119+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:12 smithi016 ceph-mon[22010]: pgmap v1951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:12.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:12 smithi016 conmon[32727]: 2022-03-21T18:17:12.098+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:12 smithi079 ceph-mon[23158]: pgmap v1951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:13.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:13 smithi016 conmon[32727]: 2022-03-21T18:17:13.061+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:14.300 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:14 smithi016 conmon[32727]: 2022-03-21T18:17:14.035+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:14 smithi016 ceph-mon[22010]: pgmap v1952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:14 smithi079 ceph-mon[23158]: pgmap v1952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:15.305 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:14 smithi016 conmon[32727]: 2022-03-21T18:17:14.993+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:16.306 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:15 smithi016 conmon[32727]: 2022-03-21T18:17:15.996+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:16 smithi016 ceph-mon[22010]: pgmap v1953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:16 smithi079 ceph-mon[23158]: pgmap v1953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:17.311 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:17 smithi016 conmon[32727]: 2022-03-21T18:17:17.036+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:17.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:18.313 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:18 smithi016 conmon[32727]: 2022-03-21T18:17:18.003+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:18 smithi016 ceph-mon[22010]: pgmap v1954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:18 smithi079 ceph-mon[23158]: pgmap v1954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:19.315 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:19 smithi016 conmon[32727]: 2022-03-21T18:17:19.051+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:20 smithi016 ceph-mon[22010]: pgmap v1955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:20 smithi016 conmon[32727]: 2022-03-21T18:17:20.081+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:20 smithi079 ceph-mon[23158]: pgmap v1955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:21 smithi016 conmon[32727]: 2022-03-21T18:17:21.095+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:22 smithi016 conmon[32727]: 2022-03-21T18:17:22.116+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:22 smithi016 ceph-mon[22010]: pgmap v1956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:22 smithi079 ceph-mon[23158]: pgmap v1956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:23 smithi016 conmon[32727]: 2022-03-21T18:17:23.084+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:24 smithi016 conmon[32727]: 2022-03-21T18:17:24.089+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:24 smithi016 ceph-mon[22010]: pgmap v1957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:24 smithi079 ceph-mon[23158]: pgmap v1957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:25 smithi016 conmon[32727]: 2022-03-21T18:17:25.084+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:26.335 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:26 smithi016 conmon[32727]: 2022-03-21T18:17:26.061+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:26 smithi016 ceph-mon[22010]: pgmap v1958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:26 smithi079 ceph-mon[23158]: pgmap v1958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:27.340 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:27 smithi016 conmon[32727]: 2022-03-21T18:17:27.057+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:28.342 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:28 smithi016 conmon[32727]: 2022-03-21T18:17:28.013+0000 7ff70c0ba700 -1 osd.4 112 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:28 smithi016 ceph-mon[22010]: pgmap v1959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:28 smithi079 ceph-mon[23158]: pgmap v1959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:29.343 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:29 smithi016 conmon[32727]: 2022-03-21T18:17:29.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:29 smithi016 ceph-mon[22010]: osdmap e113: 12 total, 8 up, 8 in 2022-03-21T18:17:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:29 smithi079 ceph-mon[23158]: osdmap e113: 12 total, 8 up, 8 in 2022-03-21T18:17:30.345 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:29 smithi016 conmon[32727]: 2022-03-21T18:17:29.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:30 smithi016 ceph-mon[22010]: pgmap v1961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:30 smithi079 ceph-mon[23158]: pgmap v1961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:31.350 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:30 smithi016 conmon[32727]: 2022-03-21T18:17:30.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:32.351 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:32 smithi016 conmon[32727]: 2022-03-21T18:17:32.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:32 smithi016 ceph-mon[22010]: pgmap v1962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:32 smithi079 ceph-mon[23158]: pgmap v1962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:33.356 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:33 smithi016 conmon[32727]: 2022-03-21T18:17:33.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:34.358 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:34 smithi016 conmon[32727]: 2022-03-21T18:17:34.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:34 smithi016 ceph-mon[22010]: pgmap v1963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:34.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:34 smithi079 ceph-mon[23158]: pgmap v1963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:35.364 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:35 smithi016 conmon[32727]: 2022-03-21T18:17:35.068+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:36.365 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:36 smithi016 conmon[32727]: 2022-03-21T18:17:36.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:36 smithi016 ceph-mon[22010]: pgmap v1964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:36 smithi079 ceph-mon[23158]: pgmap v1964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:37.370 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:37 smithi016 conmon[32727]: 2022-03-21T18:17:37.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:38.371 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:38 smithi016 conmon[32727]: 2022-03-21T18:17:38.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:38 smithi016 ceph-mon[22010]: pgmap v1965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:38 smithi079 ceph-mon[23158]: pgmap v1965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-03-21T18:17:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:39.373 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:39 smithi016 conmon[32727]: 2022-03-21T18:17:39.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:40.374 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:40 smithi016 conmon[32727]: 2022-03-21T18:17:40.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:40 smithi016 ceph-mon[22010]: pgmap v1966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2 op/s 2022-03-21T18:17:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:40 smithi079 ceph-mon[23158]: pgmap v1966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2 op/s 2022-03-21T18:17:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:41.379 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:41 smithi016 conmon[32727]: 2022-03-21T18:17:41.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:42.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:42 smithi016 conmon[32727]: 2022-03-21T18:17:42.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:42 smithi016 ceph-mon[22010]: pgmap v1967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:42 smithi079 ceph-mon[23158]: pgmap v1967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:43.385 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:43 smithi016 conmon[32727]: 2022-03-21T18:17:43.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:44.387 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:44 smithi016 conmon[32727]: 2022-03-21T18:17:44.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:44 smithi016 ceph-mon[22010]: pgmap v1968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:44 smithi079 ceph-mon[23158]: pgmap v1968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:44.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[32727]: 2022-03-21T18:17:45.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:17:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:17:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:17:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:17:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:17:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:17:45.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: 2022-03-21T18:17:45.421+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:17:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: yield 2022-03-21T18:17:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:17:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:17:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:17:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:17:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:17:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:45.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:45.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:46 smithi016 conmon[32727]: 2022-03-21T18:17:46.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:46 smithi016 ceph-mon[22010]: pgmap v1969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:17:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:17:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:17:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:17:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:17:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:17:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:46.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:17:46.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:46 smithi079 ceph-mon[23158]: pgmap v1969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:17:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:17:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:17:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:17:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:17:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:17:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:17:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:47 smithi016 conmon[32727]: 2022-03-21T18:17:47.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:48.495 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[32727]: 2022-03-21T18:17:48.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:48.496 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: pgmap v1970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:48.497 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:48.498 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:17:48.503 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:17:48.503 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.505 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.505 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.506 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.506 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.509 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.509 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.509 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.510 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.510 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.511 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.512 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.513 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 2022-03-21T18:17:48.493+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:48.514 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:17:48.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:48.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: yield 2022-03-21T18:17:48.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:48.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:48.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:48.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:17:48.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:48.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:17:48.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:48.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:17:48.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:48.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:17:48.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:48.523 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:17:48.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:48.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:48.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:48.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 2022-03-21T18:17:48.494+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:17:48.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:17:48.527 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:48.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: yield 2022-03-21T18:17:48.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:48.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:48.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:48.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:17:48.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:48.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:17:48.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:48.533 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:17:48.533 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:48.534 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:17:48.535 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:48.536 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:17:48.536 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:48.537 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:48.537 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:48.538 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 2022-03-21T18:17:48.539 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:17:48.540 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: 2022-03-21T18:17:48.540 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:17:48.541 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:17:48.541 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:17:48.542 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:17:48.543 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:17:48.543 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:17:48.544 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:17:48.545 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:17:48.546 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:17:48.546 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:17:48.547 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:17:48.547 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:17:48.548 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:17:48.549 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:17:48.550 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:17:48.550 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:17:48.551 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:17:48.551 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:17:48.552 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:17:48.552 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:17:48.553 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:17:48.554 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:17:48.555 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise 2022-03-21T18:17:48.555 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:17:48.556 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:17:48.556 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:17:48.557 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:17:48.558 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:17:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: pgmap v1970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:17:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:17:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.735 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:48.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:17:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:49 smithi016 conmon[32727]: 2022-03-21T18:17:49.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:17:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:17:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:17:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:17:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:49.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:17:49.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:49.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:17:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:17:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:17:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:17:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:49.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:17:49.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:17:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:49.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:49.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:17:49.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:49.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:17:49.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:49.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:17:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:17:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:17:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:17:49.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:17:49.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:17:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:17:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:17:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:17:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:17:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:17:49.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:17:49.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:17:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:17:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:17:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:17:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:17:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:17:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:17:49.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:17:49.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:17:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:17:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:17:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:17:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:17:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:17:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:17:49.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:17:49.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:17:49.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:17:49.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:17:49.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.693 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:17:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:17:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:17:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:17:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:49.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:17:49.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:17:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:17:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:17:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:17:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:17:49.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:17:49.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:17:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:17:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:17:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:17:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:17:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:17:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:17:49.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:17:49.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:17:49.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:17:49.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:17:49.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:17:49.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:17:49.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:17:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:17:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:17:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:17:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:17:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:17:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:17:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:17:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:17:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:17:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:17:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:17:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:17:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:17:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:17:49.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:17:49.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:17:49.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:17:49.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:17:49.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:17:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:17:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:17:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:17:49.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:17:49.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:17:49.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:17:49.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:17:49.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:17:49.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:17:49.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:17:49.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:17:49.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:17:49.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:17:49.753 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:50 smithi016 conmon[32727]: 2022-03-21T18:17:50.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:50 smithi016 ceph-mon[22010]: pgmap v1971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:50 smithi079 ceph-mon[23158]: pgmap v1971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:51 smithi016 conmon[32727]: 2022-03-21T18:17:51.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:52 smithi016 conmon[32727]: 2022-03-21T18:17:52.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:52 smithi016 ceph-mon[22010]: pgmap v1972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:52.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:52 smithi079 ceph-mon[23158]: pgmap v1972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:53 smithi016 conmon[32727]: 2022-03-21T18:17:53.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:54 smithi016 ceph-mon[22010]: pgmap v1973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:54 smithi016 conmon[32727]: 2022-03-21T18:17:54.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:54 smithi079 ceph-mon[23158]: pgmap v1973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:55 smithi016 conmon[32727]: 2022-03-21T18:17:55.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:56 smithi016 ceph-mon[22010]: pgmap v1974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:56 smithi016 conmon[32727]: 2022-03-21T18:17:56.230+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:56 smithi079 ceph-mon[23158]: pgmap v1974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:17:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:57 smithi016 conmon[32727]: 2022-03-21T18:17:57.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:58 smithi016 conmon[32727]: 2022-03-21T18:17:58.250+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: pgmap v1975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:17:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:17:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:17:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:17:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: pgmap v1975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:17:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:17:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:17:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:17:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:17:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:17:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:17:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:17:59 smithi016 conmon[32727]: 2022-03-21T18:17:59.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:17:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:17:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:17:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:00 smithi016 ceph-mon[22010]: pgmap v1976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:00 smithi016 conmon[32727]: 2022-03-21T18:18:00.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:00 smithi079 ceph-mon[23158]: pgmap v1976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:01 smithi016 conmon[32727]: 2022-03-21T18:18:01.243+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:02 smithi016 ceph-mon[22010]: pgmap v1977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:02 smithi016 conmon[32727]: 2022-03-21T18:18:02.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:02 smithi079 ceph-mon[23158]: pgmap v1977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:03 smithi016 conmon[32727]: 2022-03-21T18:18:03.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:04.409 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:04 smithi016 conmon[32727]: 2022-03-21T18:18:04.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:04 smithi016 ceph-mon[22010]: pgmap v1978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:04 smithi079 ceph-mon[23158]: pgmap v1978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:04.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:05 smithi016 conmon[32727]: 2022-03-21T18:18:05.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:06 smithi016 conmon[32727]: 2022-03-21T18:18:06.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:06 smithi079 ceph-mon[23158]: pgmap v1979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:06 smithi016 ceph-mon[22010]: pgmap v1979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:07 smithi016 conmon[32727]: 2022-03-21T18:18:07.104+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:08 smithi016 conmon[32727]: 2022-03-21T18:18:08.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:08 smithi079 ceph-mon[23158]: pgmap v1980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:08 smithi016 ceph-mon[22010]: pgmap v1980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:09 smithi016 conmon[32727]: 2022-03-21T18:18:09.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:10 smithi016 conmon[32727]: 2022-03-21T18:18:10.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:10 smithi079 ceph-mon[23158]: pgmap v1981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:10 smithi016 ceph-mon[22010]: pgmap v1981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:11 smithi016 conmon[32727]: 2022-03-21T18:18:11.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:12 smithi016 conmon[32727]: 2022-03-21T18:18:12.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:12 smithi079 ceph-mon[23158]: pgmap v1982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:12 smithi016 ceph-mon[22010]: pgmap v1982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:13 smithi016 conmon[32727]: 2022-03-21T18:18:13.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:14 smithi016 conmon[32727]: 2022-03-21T18:18:14.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:14 smithi079 ceph-mon[23158]: pgmap v1983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:14 smithi016 ceph-mon[22010]: pgmap v1983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:15 smithi016 conmon[32727]: 2022-03-21T18:18:15.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:16 smithi016 conmon[32727]: 2022-03-21T18:18:16.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:16 smithi079 ceph-mon[23158]: pgmap v1984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:16 smithi016 ceph-mon[22010]: pgmap v1984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:17 smithi016 conmon[32727]: 2022-03-21T18:18:17.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:18.451 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:18 smithi016 conmon[32727]: 2022-03-21T18:18:18.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:18 smithi079 ceph-mon[23158]: pgmap v1985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:18 smithi016 ceph-mon[22010]: pgmap v1985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:19.453 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:19 smithi016 conmon[32727]: 2022-03-21T18:18:19.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:20 smithi016 conmon[32727]: 2022-03-21T18:18:20.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:20 smithi079 ceph-mon[23158]: pgmap v1986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:20 smithi016 ceph-mon[22010]: pgmap v1986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:21 smithi016 conmon[32727]: 2022-03-21T18:18:21.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:22 smithi016 conmon[32727]: 2022-03-21T18:18:22.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:22 smithi079 ceph-mon[23158]: pgmap v1987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:22 smithi016 ceph-mon[22010]: pgmap v1987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:23 smithi016 conmon[32727]: 2022-03-21T18:18:23.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:24 smithi016 conmon[32727]: 2022-03-21T18:18:24.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:24 smithi079 ceph-mon[23158]: pgmap v1988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:24 smithi016 ceph-mon[22010]: pgmap v1988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:25 smithi016 conmon[32727]: 2022-03-21T18:18:25.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:26 smithi016 conmon[32727]: 2022-03-21T18:18:26.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:26 smithi016 ceph-mon[22010]: pgmap v1989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:26 smithi079 ceph-mon[23158]: pgmap v1989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:27 smithi016 conmon[32727]: 2022-03-21T18:18:27.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:28 smithi016 conmon[32727]: 2022-03-21T18:18:28.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:28 smithi016 ceph-mon[22010]: pgmap v1990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:28 smithi079 ceph-mon[23158]: pgmap v1990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:29 smithi016 conmon[32727]: 2022-03-21T18:18:29.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:30 smithi016 conmon[32727]: 2022-03-21T18:18:30.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:30 smithi016 ceph-mon[22010]: pgmap v1991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:30 smithi079 ceph-mon[23158]: pgmap v1991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:31 smithi016 conmon[32727]: 2022-03-21T18:18:31.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:32 smithi016 conmon[32727]: 2022-03-21T18:18:32.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:32 smithi016 ceph-mon[22010]: pgmap v1992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:32 smithi079 ceph-mon[23158]: pgmap v1992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:33 smithi016 conmon[32727]: 2022-03-21T18:18:33.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:34 smithi016 conmon[32727]: 2022-03-21T18:18:34.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:34 smithi016 ceph-mon[22010]: pgmap v1993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:34 smithi079 ceph-mon[23158]: pgmap v1993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:35 smithi016 conmon[32727]: 2022-03-21T18:18:35.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:36 smithi016 conmon[32727]: 2022-03-21T18:18:36.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:36 smithi016 ceph-mon[22010]: pgmap v1994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:36 smithi079 ceph-mon[23158]: pgmap v1994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:37.508 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:37 smithi016 conmon[32727]: 2022-03-21T18:18:37.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:38 smithi016 conmon[32727]: 2022-03-21T18:18:38.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:38 smithi016 ceph-mon[22010]: pgmap v1995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:38 smithi079 ceph-mon[23158]: pgmap v1995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:39 smithi016 conmon[32727]: 2022-03-21T18:18:39.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:40 smithi016 conmon[32727]: 2022-03-21T18:18:40.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:40 smithi016 ceph-mon[22010]: pgmap v1996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:40 smithi079 ceph-mon[23158]: pgmap v1996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:41 smithi016 conmon[32727]: 2022-03-21T18:18:41.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:42 smithi016 conmon[32727]: 2022-03-21T18:18:42.048+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:42 smithi016 ceph-mon[22010]: pgmap v1997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:42 smithi079 ceph-mon[23158]: pgmap v1997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:43 smithi016 conmon[32727]: 2022-03-21T18:18:43.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:44 smithi016 conmon[32727]: 2022-03-21T18:18:44.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:44 smithi016 ceph-mon[22010]: pgmap v1998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:44 smithi079 ceph-mon[23158]: pgmap v1998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:45 smithi016 conmon[32727]: 2022-03-21T18:18:45.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:46 smithi016 conmon[32727]: 2022-03-21T18:18:46.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:46 smithi016 ceph-mon[22010]: pgmap v1999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:46 smithi079 ceph-mon[23158]: pgmap v1999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:47 smithi016 conmon[32727]: 2022-03-21T18:18:47.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:48 smithi016 conmon[32727]: 2022-03-21T18:18:48.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:48 smithi016 ceph-mon[22010]: pgmap v2000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:18:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:18:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:18:48.951 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:48 smithi079 ceph-mon[23158]: pgmap v2000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:48.951 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:48.952 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:18:48.952 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:18:48.952 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:18:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:49 smithi016 conmon[32727]: 2022-03-21T18:18:49.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:50 smithi016 conmon[32727]: 2022-03-21T18:18:50.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:50 smithi016 ceph-mon[22010]: pgmap v2001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:50 smithi079 ceph-mon[23158]: pgmap v2001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[32727]: 2022-03-21T18:18:51.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:51.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: 2022-03-21T18:18:51.597+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:51.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:18:51.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: yield 2022-03-21T18:18:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:51.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:18:51.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:18:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:51.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:18:51.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:51.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:18:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:18:51.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:51.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:51.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:52 smithi016 conmon[32727]: 2022-03-21T18:18:52.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:52 smithi016 ceph-mon[22010]: pgmap v2002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:52 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:18:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:18:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:52.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:18:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:18:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:18:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:18:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:52 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:18:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:52 smithi079 ceph-mon[23158]: pgmap v2002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:52 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:18:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:18:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:18:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:18:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:18:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:18:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:52 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:18:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:53 smithi016 conmon[32727]: 2022-03-21T18:18:53.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[32727]: 2022-03-21T18:18:54.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:54 smithi016 ceph-mon[22010]: pgmap v2003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 2022-03-21T18:18:54.669+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:18:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: yield 2022-03-21T18:18:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:18:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:18:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:18:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:18:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:18:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 2022-03-21T18:18:54.670+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:18:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:18:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: yield 2022-03-21T18:18:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:18:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:18:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:18:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:18:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:18:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 2022-03-21T18:18:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:18:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: 2022-03-21T18:18:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:18:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:18:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:18:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:18:54.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:18:54.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:18:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:18:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:18:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:18:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:18:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:18:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:18:54.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:18:54.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:18:54.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:18:54.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:18:54.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:18:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:18:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:18:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:18:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:18:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:18:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise 2022-03-21T18:18:54.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:18:54.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:18:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:18:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:18:54.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:18:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:54 smithi079 ceph-mon[23158]: pgmap v2003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:55 smithi016 conmon[32727]: 2022-03-21T18:18:55.053+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:56 smithi016 conmon[32727]: 2022-03-21T18:18:56.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:18:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:18:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:18:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:18:56.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:56.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:56.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:56.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:18:56.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:56.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:56.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:56 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:18:56.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:56.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:18:56.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:56.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:56.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:56.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:18:56.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:56.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:18:56.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:56.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:18:56.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:56.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:56.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:56.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:18:56.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:56.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:56.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:18:56.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:18:56.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:18:56.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:18:56.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:18:56.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:18:56.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:18:56.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:18:56.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:18:56.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:18:56.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:18:56.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:18:56.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:18:56.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:18:56.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:18:56.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:18:56.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:18:56.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:18:56.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:18:56.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:18:56.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:18:56.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:18:56.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:18:56.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:18:56.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:18:56.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:18:56.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:18:56.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:18:56.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:18:56.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:56 smithi016 ceph-mon[22010]: pgmap v2004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:56.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:18:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:18:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:18:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:18:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:18:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:56 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:18:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:18:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:18:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:18:56.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:18:56.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:18:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:18:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:18:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:18:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:18:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:18:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:18:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:18:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:18:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:18:56.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:18:56.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:18:56.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:18:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:18:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:18:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:18:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:18:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:18:56.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:18:56.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:18:56.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:18:56.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:18:56.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:18:56.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:18:56.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:18:56.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:18:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:18:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:18:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:18:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:18:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:18:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:18:56.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:18:56.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:18:56.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:18:56.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:18:56.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:18:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:18:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:18:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:18:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:18:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:18:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:18:56.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:56 smithi079 ceph-mon[23158]: pgmap v2004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:18:56.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:57 smithi016 conmon[32727]: 2022-03-21T18:18:57.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:58 smithi016 conmon[32727]: 2022-03-21T18:18:58.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: pgmap v2005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:18:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:18:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:18:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:18:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: pgmap v2005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:18:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:18:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:18:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:18:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:18:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:18:59 smithi016 conmon[32727]: 2022-03-21T18:18:59.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:18:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:18:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:18:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:18:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:18:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:00 smithi016 conmon[32727]: 2022-03-21T18:19:00.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:00 smithi016 ceph-mon[22010]: pgmap v2006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:00 smithi079 ceph-mon[23158]: pgmap v2006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:01 smithi016 conmon[32727]: 2022-03-21T18:19:01.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:02 smithi016 conmon[32727]: 2022-03-21T18:19:02.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:02 smithi016 ceph-mon[22010]: pgmap v2007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:02 smithi079 ceph-mon[23158]: pgmap v2007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:03 smithi016 conmon[32727]: 2022-03-21T18:19:03.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:04.582 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:04 smithi016 conmon[32727]: 2022-03-21T18:19:04.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:04 smithi016 ceph-mon[22010]: pgmap v2008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:04 smithi079 ceph-mon[23158]: pgmap v2008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:05 smithi016 conmon[32727]: 2022-03-21T18:19:05.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:06.590 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:06 smithi016 conmon[32727]: 2022-03-21T18:19:06.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:06 smithi016 ceph-mon[22010]: pgmap v2009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:06 smithi079 ceph-mon[23158]: pgmap v2009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:07.594 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:07 smithi016 conmon[32727]: 2022-03-21T18:19:07.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:08.596 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:08 smithi016 conmon[32727]: 2022-03-21T18:19:08.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:08 smithi016 ceph-mon[22010]: pgmap v2010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:08 smithi079 ceph-mon[23158]: pgmap v2010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:09.597 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:09 smithi016 conmon[32727]: 2022-03-21T18:19:09.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:10 smithi016 conmon[32727]: 2022-03-21T18:19:10.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:10 smithi016 ceph-mon[22010]: pgmap v2011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:10 smithi079 ceph-mon[23158]: pgmap v2011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:11 smithi016 conmon[32727]: 2022-03-21T18:19:11.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:12 smithi016 conmon[32727]: 2022-03-21T18:19:12.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:12 smithi016 ceph-mon[22010]: pgmap v2012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:12 smithi079 ceph-mon[23158]: pgmap v2012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:13 smithi016 conmon[32727]: 2022-03-21T18:19:13.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:14 smithi016 conmon[32727]: 2022-03-21T18:19:14.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:14 smithi016 ceph-mon[22010]: pgmap v2013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:14 smithi079 ceph-mon[23158]: pgmap v2013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:15 smithi016 conmon[32727]: 2022-03-21T18:19:15.138+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:16.618 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:16 smithi016 conmon[32727]: 2022-03-21T18:19:16.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:16 smithi016 ceph-mon[22010]: pgmap v2014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:16 smithi079 ceph-mon[23158]: pgmap v2014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:17.623 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:17 smithi016 conmon[32727]: 2022-03-21T18:19:17.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:18 smithi016 conmon[32727]: 2022-03-21T18:19:18.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:18 smithi016 ceph-mon[22010]: pgmap v2015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:18 smithi079 ceph-mon[23158]: pgmap v2015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:19 smithi016 conmon[32727]: 2022-03-21T18:19:19.124+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:19 smithi016 ceph-mon[22010]: pgmap v2016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:19 smithi079 ceph-mon[23158]: pgmap v2016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:20 smithi016 conmon[32727]: 2022-03-21T18:19:20.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:21 smithi016 conmon[32727]: 2022-03-21T18:19:21.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:21 smithi016 ceph-mon[22010]: pgmap v2017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:21 smithi079 ceph-mon[23158]: pgmap v2017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:22 smithi016 conmon[32727]: 2022-03-21T18:19:22.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:23 smithi016 conmon[32727]: 2022-03-21T18:19:23.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:23 smithi016 ceph-mon[22010]: pgmap v2018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:23.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:23 smithi079 ceph-mon[23158]: pgmap v2018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:24 smithi016 conmon[32727]: 2022-03-21T18:19:24.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:25 smithi016 conmon[32727]: 2022-03-21T18:19:25.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:25 smithi016 ceph-mon[22010]: pgmap v2019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:25 smithi079 ceph-mon[23158]: pgmap v2019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:26 smithi016 conmon[32727]: 2022-03-21T18:19:26.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:27.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:27 smithi016 conmon[32727]: 2022-03-21T18:19:27.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:27 smithi079 ceph-mon[23158]: pgmap v2020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:28.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:27 smithi016 ceph-mon[22010]: pgmap v2020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:28.113 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:28 smithi016 conmon[32727]: 2022-03-21T18:19:28.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:29.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:29.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:29 smithi016 conmon[32727]: 2022-03-21T18:19:29.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:29 smithi079 ceph-mon[23158]: pgmap v2021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:30.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:29 smithi016 ceph-mon[22010]: pgmap v2021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:30.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:30 smithi016 conmon[32727]: 2022-03-21T18:19:30.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:31.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:31 smithi016 conmon[32727]: 2022-03-21T18:19:31.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:31 smithi079 ceph-mon[23158]: pgmap v2022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:32.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:31 smithi016 ceph-mon[22010]: pgmap v2022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:32.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:32 smithi016 conmon[32727]: 2022-03-21T18:19:32.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:32.999 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:32 smithi016 conmon[32727]: 2022-03-21T18:19:32.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:33 smithi079 ceph-mon[23158]: pgmap v2023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:33.978 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:33.979 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:33 smithi016 ceph-mon[22010]: pgmap v2023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:19:33.979 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:33 smithi016 conmon[32727]: 2022-03-21T18:19:33.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:34.971 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:34 smithi016 conmon[32727]: 2022-03-21T18:19:34.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:35 smithi079 ceph-mon[23158]: pgmap v2024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:36.004 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:36.005 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:35 smithi016 ceph-mon[22010]: pgmap v2024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:36 smithi016 conmon[32727]: 2022-03-21T18:19:36.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:37.053 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:37 smithi016 conmon[32727]: 2022-03-21T18:19:37.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:37 smithi079 ceph-mon[23158]: pgmap v2025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:38.073 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:37 smithi016 ceph-mon[22010]: pgmap v2025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:38.073 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:38 smithi016 conmon[32727]: 2022-03-21T18:19:38.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:39.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:39.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:39 smithi016 conmon[32727]: 2022-03-21T18:19:39.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:39 smithi079 ceph-mon[23158]: pgmap v2026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:40.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:40.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:39 smithi016 ceph-mon[22010]: pgmap v2026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:40 smithi016 conmon[32727]: 2022-03-21T18:19:40.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:41.067 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:41 smithi016 conmon[32727]: 2022-03-21T18:19:41.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:41 smithi079 ceph-mon[23158]: pgmap v2027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:42.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:41 smithi016 ceph-mon[22010]: pgmap v2027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:42.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:42 smithi016 conmon[32727]: 2022-03-21T18:19:42.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:43.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:43 smithi016 conmon[32727]: 2022-03-21T18:19:43.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:43 smithi079 ceph-mon[23158]: pgmap v2028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:44.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:43 smithi016 ceph-mon[22010]: pgmap v2028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:44.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:44.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:44 smithi016 conmon[32727]: 2022-03-21T18:19:44.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:45.086 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:45 smithi016 conmon[32727]: 2022-03-21T18:19:45.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:46.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:45 smithi016 ceph-mon[22010]: pgmap v2029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:46.052 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:45 smithi079 ceph-mon[23158]: pgmap v2029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:46 smithi016 conmon[32727]: 2022-03-21T18:19:46.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:47.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:47 smithi016 conmon[32727]: 2022-03-21T18:19:47.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:48.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:47 smithi016 ceph-mon[22010]: pgmap v2030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:48.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:47 smithi079 ceph-mon[23158]: pgmap v2030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:48 smithi016 conmon[32727]: 2022-03-21T18:19:48.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:49.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:49.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:49 smithi016 conmon[32727]: 2022-03-21T18:19:49.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:50.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:49 smithi016 ceph-mon[22010]: pgmap v2031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:50.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:49 smithi079 ceph-mon[23158]: pgmap v2031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:50 smithi016 conmon[32727]: 2022-03-21T18:19:50.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:51.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:51 smithi016 conmon[32727]: 2022-03-21T18:19:51.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:52.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:51 smithi016 ceph-mon[22010]: pgmap v2032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:52.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:51 smithi079 ceph-mon[23158]: pgmap v2032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:52 smithi016 conmon[32727]: 2022-03-21T18:19:52.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:53.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:53 smithi016 conmon[32727]: 2022-03-21T18:19:53.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:54.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:53 smithi016 ceph-mon[22010]: pgmap v2033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:54.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:54.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:53 smithi079 ceph-mon[23158]: pgmap v2033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:54.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:54 smithi016 conmon[32727]: 2022-03-21T18:19:54.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:55.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:55.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:19:55.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:19:55.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:19:55.129 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:55.129 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:19:55.130 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:19:55.130 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:19:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[32727]: 2022-03-21T18:19:55.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:55.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: 2022-03-21T18:19:55.150+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:19:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: yield 2022-03-21T18:19:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:19:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:19:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:19:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:19:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:19:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:56.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:55 smithi016 ceph-mon[22010]: pgmap v2034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:56.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:56.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:56.088 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:19:56.089 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:56.089 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:19:56.089 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:56.090 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:56.090 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:56.091 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:19:56.091 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:56.091 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:19:56.092 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:56.092 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:19:56.092 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:56.093 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:56.093 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:56.093 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:19:56.094 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:56.094 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:56.094 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:56.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:55 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:19:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:55 smithi079 ceph-mon[23158]: pgmap v2034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:19:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:19:56.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:19:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:56.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:19:56.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:19:56.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:19:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:56.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:19:56.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:56.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:55 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:19:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:56 smithi016 conmon[32727]: 2022-03-21T18:19:56.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:57.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:57 smithi016 conmon[32727]: 2022-03-21T18:19:57.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:58.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:57 smithi016 ceph-mon[22010]: pgmap v2035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:58.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:57 smithi079 ceph-mon[23158]: pgmap v2035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:19:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[32727]: 2022-03-21T18:19:58.132+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:19:58.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 2022-03-21T18:19:58.221+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:58.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:19:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: yield 2022-03-21T18:19:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:58.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:58.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:58.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:19:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:19:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:19:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:19:58.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:58.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:19:58.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:58.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:58.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 2022-03-21T18:19:58.222+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:19:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:19:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: yield 2022-03-21T18:19:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:19:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:19:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:19:58.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:58.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:19:58.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:58.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:19:58.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 2022-03-21T18:19:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:19:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: 2022-03-21T18:19:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:19:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:19:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:19:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:19:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:19:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:19:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:19:58.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:19:58.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:19:58.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:19:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:19:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:19:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:19:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:19:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:19:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:19:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:19:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:19:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:19:58.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:19:58.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:19:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:19:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise 2022-03-21T18:19:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:19:58.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:19:58.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:19:58.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:19:58.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:19:58 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:59.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:19:59.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:19:59.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:19:59.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:19:59.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.146 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.146 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:59.147 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:19:59.147 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:59.147 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:59.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:59.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:19:59.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:59.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:19:59.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:59.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:19:59.150 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:59.150 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:59.150 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:59.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:19:59.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:59.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:59.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:19:59.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:59.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:19:59.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:59.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:59.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:59.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:19:59.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:59.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:19:59.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:59.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:19:59.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:59.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:59.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:59.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:19:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:59.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:19:59.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:19:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:19:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:19:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:19:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:19:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:19:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:19:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:19:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:19:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:19:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:19:59.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:19:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:19:59.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:19:59.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:19:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:19:59.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:19:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:19:59.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:19:59.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:19:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:19:59.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:19:59.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:19:59.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:19:59.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:19:59.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:19:59.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:19:59.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:19:59.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.182 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:19:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:19:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:19:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:59.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:19:59.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:59.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:59.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:59.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:19:59.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:59.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:19:59.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:59.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:19:59.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:59.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:59.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:59.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:19:59.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:59.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:59.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.239 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:19:59.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:19:59.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:19:59.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:19:59.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:19:59.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:19:59.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:19:59.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:19:59.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:19:59.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:19:59.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:19:59.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:19:59.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:19:59.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:19:59.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:19:59.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:19:59.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:19:59.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:19:59.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:19:59.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:19:59.253 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:19:59.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:19:59.254 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:19:59.255 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:19:59.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:19:59.256 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:19:59.257 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:19:59.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:19:59.258 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:19:59.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:19:59.259 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:19:59.260 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:19:59.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:19:59.261 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:19:59.262 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:19:59.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:19:59.263 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:19:59.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:19:59.264 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:19:59.265 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:19:59.266 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:19:59.266 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:19:59.267 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:19:59.268 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:19:59.268 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:19:59.269 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:19:59.269 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:19:59.270 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:19:59.271 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:19:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:19:59 smithi016 conmon[32727]: 2022-03-21T18:19:59.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:00.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:59 smithi016 ceph-mon[22010]: pgmap v2036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:00.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:19:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:59 smithi079 ceph-mon[23158]: pgmap v2036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:19:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:00 smithi016 conmon[32727]: 2022-03-21T18:20:00.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops 2022-03-21T18:20:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:20:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:20:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:20:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:20:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:20:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops 2022-03-21T18:20:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops 2022-03-21T18:20:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:20:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:20:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:20:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:20:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:20:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops 2022-03-21T18:20:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:01 smithi016 conmon[32727]: 2022-03-21T18:20:01.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:02.151 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:01 smithi016 ceph-mon[22010]: pgmap v2037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:02.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:01 smithi079 ceph-mon[23158]: pgmap v2037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:02 smithi016 conmon[32727]: 2022-03-21T18:20:02.149+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:03.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:03 smithi016 conmon[32727]: 2022-03-21T18:20:03.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:04.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:03 smithi016 ceph-mon[22010]: pgmap v2038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:04.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:04.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:03 smithi079 ceph-mon[23158]: pgmap v2038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:04 smithi016 conmon[32727]: 2022-03-21T18:20:04.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:05.140 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:05 smithi016 conmon[32727]: 2022-03-21T18:20:05.138+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:06.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:05 smithi016 ceph-mon[22010]: pgmap v2039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:06.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:05 smithi079 ceph-mon[23158]: pgmap v2039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:06 smithi016 conmon[32727]: 2022-03-21T18:20:06.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:07 smithi016 conmon[32727]: 2022-03-21T18:20:07.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:08.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:07 smithi016 ceph-mon[22010]: pgmap v2040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:08.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:07 smithi079 ceph-mon[23158]: pgmap v2040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:08 smithi016 conmon[32727]: 2022-03-21T18:20:08.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:09.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:09.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:09 smithi016 conmon[32727]: 2022-03-21T18:20:09.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:10.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:09 smithi016 ceph-mon[22010]: pgmap v2041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:10.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:09 smithi079 ceph-mon[23158]: pgmap v2041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:10 smithi016 conmon[32727]: 2022-03-21T18:20:10.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:11.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:11 smithi016 conmon[32727]: 2022-03-21T18:20:11.081+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:12.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:11 smithi016 ceph-mon[22010]: pgmap v2042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:12.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:11 smithi079 ceph-mon[23158]: pgmap v2042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:12 smithi016 conmon[32727]: 2022-03-21T18:20:12.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:13.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:13 smithi016 conmon[32727]: 2022-03-21T18:20:13.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:14.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:13 smithi016 ceph-mon[22010]: pgmap v2043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:14.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:14.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:13 smithi079 ceph-mon[23158]: pgmap v2043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:14 smithi016 conmon[32727]: 2022-03-21T18:20:14.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:15.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:15 smithi016 conmon[32727]: 2022-03-21T18:20:15.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:16.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:15 smithi016 ceph-mon[22010]: pgmap v2044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:16.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:15 smithi079 ceph-mon[23158]: pgmap v2044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:16 smithi016 conmon[32727]: 2022-03-21T18:20:16.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:17.125 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:17 smithi016 conmon[32727]: 2022-03-21T18:20:17.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:18.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:17 smithi016 ceph-mon[22010]: pgmap v2045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:18.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:17 smithi079 ceph-mon[23158]: pgmap v2045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:18 smithi016 conmon[32727]: 2022-03-21T18:20:18.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:19.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:19.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:19 smithi016 conmon[32727]: 2022-03-21T18:20:19.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:20.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:19 smithi016 ceph-mon[22010]: pgmap v2046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:20.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:19 smithi079 ceph-mon[23158]: pgmap v2046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:20 smithi016 conmon[32727]: 2022-03-21T18:20:20.081+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:21.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:21 smithi016 conmon[32727]: 2022-03-21T18:20:21.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:22 smithi016 conmon[32727]: 2022-03-21T18:20:22.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:21 smithi016 ceph-mon[22010]: pgmap v2047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:21 smithi079 ceph-mon[23158]: pgmap v2047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:23 smithi016 conmon[32727]: 2022-03-21T18:20:23.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:24 smithi016 conmon[32727]: 2022-03-21T18:20:24.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:23 smithi016 ceph-mon[22010]: pgmap v2048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:23 smithi079 ceph-mon[23158]: pgmap v2048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:25 smithi016 conmon[32727]: 2022-03-21T18:20:25.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:25 smithi016 ceph-mon[22010]: pgmap v2049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:26 smithi016 conmon[32727]: 2022-03-21T18:20:26.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:25 smithi079 ceph-mon[23158]: pgmap v2049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:27 smithi016 conmon[32727]: 2022-03-21T18:20:27.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:28 smithi016 conmon[32727]: 2022-03-21T18:20:28.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:27 smithi016 ceph-mon[22010]: pgmap v2050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:27 smithi079 ceph-mon[23158]: pgmap v2050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:29 smithi016 conmon[32727]: 2022-03-21T18:20:29.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:29 smithi016 ceph-mon[22010]: pgmap v2051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:30 smithi016 conmon[32727]: 2022-03-21T18:20:30.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:29 smithi079 ceph-mon[23158]: pgmap v2051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:31 smithi016 conmon[32727]: 2022-03-21T18:20:31.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:32 smithi016 conmon[32727]: 2022-03-21T18:20:32.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:31 smithi016 ceph-mon[22010]: pgmap v2052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:31 smithi079 ceph-mon[23158]: pgmap v2052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:33 smithi016 conmon[32727]: 2022-03-21T18:20:33.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:33 smithi016 ceph-mon[22010]: pgmap v2053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:34 smithi016 conmon[32727]: 2022-03-21T18:20:34.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:33 smithi079 ceph-mon[23158]: pgmap v2053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:35 smithi016 conmon[32727]: 2022-03-21T18:20:35.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:36 smithi016 conmon[32727]: 2022-03-21T18:20:36.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:35 smithi016 ceph-mon[22010]: pgmap v2054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:35 smithi079 ceph-mon[23158]: pgmap v2054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:36.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:37 smithi016 conmon[32727]: 2022-03-21T18:20:37.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:38.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:37 smithi016 ceph-mon[22010]: pgmap v2055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:38.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:37 smithi079 ceph-mon[23158]: pgmap v2055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:38 smithi016 conmon[32727]: 2022-03-21T18:20:38.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:39 smithi016 conmon[32727]: 2022-03-21T18:20:39.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:39 smithi016 ceph-mon[22010]: pgmap v2056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:39 smithi079 ceph-mon[23158]: pgmap v2056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:40 smithi016 conmon[32727]: 2022-03-21T18:20:40.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:41 smithi016 conmon[32727]: 2022-03-21T18:20:41.222+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:41 smithi016 ceph-mon[22010]: pgmap v2057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:41 smithi079 ceph-mon[23158]: pgmap v2057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:42 smithi016 conmon[32727]: 2022-03-21T18:20:42.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:43 smithi016 conmon[32727]: 2022-03-21T18:20:43.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:43 smithi016 ceph-mon[22010]: pgmap v2058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:43 smithi079 ceph-mon[23158]: pgmap v2058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:44 smithi016 conmon[32727]: 2022-03-21T18:20:44.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:45.210 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:45 smithi016 conmon[32727]: 2022-03-21T18:20:45.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:45 smithi079 ceph-mon[23158]: pgmap v2059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:46.226 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:45 smithi016 ceph-mon[22010]: pgmap v2059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:46.227 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:46 smithi016 conmon[32727]: 2022-03-21T18:20:46.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:47 smithi016 conmon[32727]: 2022-03-21T18:20:47.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:47 smithi079 ceph-mon[23158]: pgmap v2060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:47 smithi016 ceph-mon[22010]: pgmap v2060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:48 smithi016 conmon[32727]: 2022-03-21T18:20:48.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:49 smithi016 conmon[32727]: 2022-03-21T18:20:49.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:49 smithi079 ceph-mon[23158]: pgmap v2061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:49 smithi016 ceph-mon[22010]: pgmap v2061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:50 smithi016 conmon[32727]: 2022-03-21T18:20:50.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:51 smithi016 conmon[32727]: 2022-03-21T18:20:51.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:51 smithi079 ceph-mon[23158]: pgmap v2062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:52.230 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:51 smithi016 ceph-mon[22010]: pgmap v2062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:52.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:52 smithi016 conmon[32727]: 2022-03-21T18:20:52.228+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:53.225 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:53 smithi016 conmon[32727]: 2022-03-21T18:20:53.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:53 smithi079 ceph-mon[23158]: pgmap v2063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:54.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:54.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:54.227 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:53 smithi016 ceph-mon[22010]: pgmap v2063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:54.227 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:54.228 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:54 smithi016 conmon[32727]: 2022-03-21T18:20:54.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:55.263 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:55 smithi016 conmon[32727]: 2022-03-21T18:20:55.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:55 smithi079 ceph-mon[23158]: pgmap v2064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:56.264 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:55 smithi016 ceph-mon[22010]: pgmap v2064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:20:56.265 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:56 smithi016 conmon[32727]: 2022-03-21T18:20:56.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:57.282 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:57 smithi016 conmon[32727]: 2022-03-21T18:20:57.280+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:57 smithi079 ceph-mon[23158]: pgmap v2065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:58.269 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:57 smithi016 ceph-mon[22010]: pgmap v2065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:20:58.270 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:58.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:58 smithi016 conmon[32727]: 2022-03-21T18:20:58.301+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:20:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:20:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:20:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:20:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:20:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:20:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:20:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:20:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:20:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:20:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:59.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:20:59.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:20:59.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:20:59.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:20:59.345 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:20:59.346 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:20:59.346 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:20:59.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:20:59.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:20:59.347 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:20:59.348 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:20:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:20:59 smithi016 conmon[32727]: 2022-03-21T18:20:59.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:59 smithi079 ceph-mon[23158]: pgmap v2066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:20:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:00.366 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:59 smithi016 ceph-mon[22010]: pgmap v2066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:00.367 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:20:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:00 smithi016 conmon[32727]: 2022-03-21T18:21:00.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:01.327 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[32727]: 2022-03-21T18:21:01.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:01.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: 2022-03-21T18:21:01.325+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:01.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:21:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: yield 2022-03-21T18:21:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:21:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:21:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:21:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:21:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:21:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:01.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:01 smithi079 ceph-mon[23158]: pgmap v2067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:21:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:21:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:21:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:21:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:21:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:21:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:21:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:02.393 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:01 smithi016 ceph-mon[22010]: pgmap v2067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:02.393 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:02.394 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:21:02.394 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:02.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:21:02.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:02.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:02.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:02.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:21:02.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:02.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:21:02.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:02.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:21:02.398 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:02.398 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:02.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:02.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:21:02.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:02.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:02.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:02.400 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:21:02.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:02 smithi016 conmon[32727]: 2022-03-21T18:21:02.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:03 smithi016 conmon[32727]: 2022-03-21T18:21:03.416+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:04.400 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:03 smithi016 ceph-mon[22010]: pgmap v2068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:04.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:04.402 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:03 smithi079 ceph-mon[23158]: pgmap v2068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[32727]: 2022-03-21T18:21:04.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 2022-03-21T18:21:04.397+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:21:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: yield 2022-03-21T18:21:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:21:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:21:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:21:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:21:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:21:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 2022-03-21T18:21:04.398+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:21:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:21:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: yield 2022-03-21T18:21:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:21:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:21:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:21:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:21:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:21:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 2022-03-21T18:21:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:21:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: 2022-03-21T18:21:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:21:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:21:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:21:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:21:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:21:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:21:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:21:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:21:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:21:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:21:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:21:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:21:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:21:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:21:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:21:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:21:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:21:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:21:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:21:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:21:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:21:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:21:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise 2022-03-21T18:21:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:21:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:21:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:21:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:21:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:21:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.380 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.380 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:05.381 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:21:05.381 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:05.381 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:05.382 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:05.382 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:21:05.383 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:05.383 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:21:05.383 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:05.384 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:21:05.384 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:05.384 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:05.385 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:05.385 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:21:05.385 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:05.386 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:05.386 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.387 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:04 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:21:05.387 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.387 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:05.388 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:21:05.388 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:05.388 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:05.389 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:05.389 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:21:05.389 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:05.390 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:21:05.390 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:05.391 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:21:05.391 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:05.391 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:05.392 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:05.392 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:21:05.392 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:05.393 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:05.393 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.393 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:21:05.394 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:21:05.394 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:21:05.394 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:21:05.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:21:05.395 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:21:05.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:21:05.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:21:05.396 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:21:05.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:21:05.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:21:05.397 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:21:05.398 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:21:05.398 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:21:05.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:21:05.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:21:05.399 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:21:05.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:21:05.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:21:05.400 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:21:05.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:21:05.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:21:05.401 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:21:05.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:21:05.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:21:05.402 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:21:05.403 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:21:05.403 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:21:05.403 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:21:05.404 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:21:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:05.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:05.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:05.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:21:05.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:05.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:21:05.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:05.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:21:05.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:05.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:05.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:05.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:21:05.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:05.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:05.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:04 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:21:05.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:21:05.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:21:05.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:21:05.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:21:05.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:21:05.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:21:05.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:21:05.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:21:05.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:21:05.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:21:05.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:21:05.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:21:05.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:21:05.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:21:05.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:21:05.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:21:05.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:21:05.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:21:05.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:21:05.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:21:05.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:21:05.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:21:05.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:21:05.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:21:05.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:21:05.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:21:05.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:21:05.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:21:05.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:21:05.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:21:05.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:21:05.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:21:05.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:21:05.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:21:05.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:21:05.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:21:05.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:21:05.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:21:05.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:21:05.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:21:05.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:21:05.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:21:05.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:21:05.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:21:05.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:21:05.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:21:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:21:05.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:05 smithi016 conmon[32727]: 2022-03-21T18:21:05.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:06.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:05 smithi016 ceph-mon[22010]: pgmap v2069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:06.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:05 smithi079 ceph-mon[23158]: pgmap v2069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:06 smithi016 conmon[32727]: 2022-03-21T18:21:06.362+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:07.322 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:07 smithi016 conmon[32727]: 2022-03-21T18:21:07.320+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:08.335 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:07 smithi016 ceph-mon[22010]: pgmap v2070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:08.336 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:07 smithi079 ceph-mon[23158]: pgmap v2070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:08 smithi016 conmon[32727]: 2022-03-21T18:21:08.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:09.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:09.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:09 smithi016 conmon[32727]: 2022-03-21T18:21:09.378+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:10.386 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:09 smithi016 ceph-mon[22010]: pgmap v2071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:10.387 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:09 smithi079 ceph-mon[23158]: pgmap v2071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:10 smithi016 conmon[32727]: 2022-03-21T18:21:10.384+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:11.377 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:11 smithi016 conmon[32727]: 2022-03-21T18:21:11.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:12 smithi016 ceph-mon[22010]: pgmap v2072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:12 smithi079 ceph-mon[23158]: pgmap v2072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:12 smithi016 conmon[32727]: 2022-03-21T18:21:12.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:13.396 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:13 smithi016 conmon[32727]: 2022-03-21T18:21:13.395+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:14.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:14 smithi016 ceph-mon[22010]: pgmap v2073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:14.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:14.404 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:14 smithi079 ceph-mon[23158]: pgmap v2073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:14 smithi016 conmon[32727]: 2022-03-21T18:21:14.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:15 smithi016 conmon[32727]: 2022-03-21T18:21:15.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:16.393 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:16 smithi016 ceph-mon[22010]: pgmap v2074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:16 smithi079 ceph-mon[23158]: pgmap v2074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:16 smithi016 conmon[32727]: 2022-03-21T18:21:16.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:17 smithi016 conmon[32727]: 2022-03-21T18:21:17.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:18 smithi016 ceph-mon[22010]: pgmap v2075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:18 smithi079 ceph-mon[23158]: pgmap v2075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:18 smithi016 conmon[32727]: 2022-03-21T18:21:18.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:19.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:19 smithi016 conmon[32727]: 2022-03-21T18:21:19.397+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:20 smithi016 ceph-mon[22010]: pgmap v2076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:20 smithi079 ceph-mon[23158]: pgmap v2076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:20 smithi016 conmon[32727]: 2022-03-21T18:21:20.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:21 smithi016 conmon[32727]: 2022-03-21T18:21:21.407+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:22 smithi016 ceph-mon[22010]: pgmap v2077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:22 smithi079 ceph-mon[23158]: pgmap v2077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:22 smithi016 conmon[32727]: 2022-03-21T18:21:22.425+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:23 smithi016 conmon[32727]: 2022-03-21T18:21:23.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:24 smithi016 ceph-mon[22010]: pgmap v2078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:24 smithi079 ceph-mon[23158]: pgmap v2078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:24 smithi016 conmon[32727]: 2022-03-21T18:21:24.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:25 smithi016 conmon[32727]: 2022-03-21T18:21:25.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:26 smithi016 ceph-mon[22010]: pgmap v2079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:26 smithi079 ceph-mon[23158]: pgmap v2079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:26 smithi016 conmon[32727]: 2022-03-21T18:21:26.482+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:27 smithi016 conmon[32727]: 2022-03-21T18:21:27.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:28 smithi016 ceph-mon[22010]: pgmap v2080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:28 smithi079 ceph-mon[23158]: pgmap v2080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:28 smithi016 conmon[32727]: 2022-03-21T18:21:28.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:29 smithi016 conmon[32727]: 2022-03-21T18:21:29.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:30 smithi016 ceph-mon[22010]: pgmap v2081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:30 smithi079 ceph-mon[23158]: pgmap v2081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:30 smithi016 conmon[32727]: 2022-03-21T18:21:30.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:31 smithi016 conmon[32727]: 2022-03-21T18:21:31.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:32 smithi016 ceph-mon[22010]: pgmap v2082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:32 smithi079 ceph-mon[23158]: pgmap v2082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:32 smithi016 conmon[32727]: 2022-03-21T18:21:32.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:33.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:33 smithi016 conmon[32727]: 2022-03-21T18:21:33.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:34 smithi079 ceph-mon[23158]: pgmap v2083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:34.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:34.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:34 smithi016 ceph-mon[22010]: pgmap v2083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:34.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:34 smithi016 conmon[32727]: 2022-03-21T18:21:34.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:35.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:35 smithi016 conmon[32727]: 2022-03-21T18:21:35.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:36 smithi079 ceph-mon[23158]: pgmap v2084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:36.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:36.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:36 smithi016 ceph-mon[22010]: pgmap v2084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:36 smithi016 conmon[32727]: 2022-03-21T18:21:36.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:37.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:37 smithi016 conmon[32727]: 2022-03-21T18:21:37.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:38.452 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:38.452 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:38 smithi016 ceph-mon[22010]: pgmap v2085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:38 smithi079 ceph-mon[23158]: pgmap v2085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:38 smithi016 conmon[32727]: 2022-03-21T18:21:38.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:39 smithi016 conmon[32727]: 2022-03-21T18:21:39.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:40 smithi079 ceph-mon[23158]: pgmap v2086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:40 smithi016 ceph-mon[22010]: pgmap v2086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:40 smithi016 conmon[32727]: 2022-03-21T18:21:40.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:41.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:41 smithi016 conmon[32727]: 2022-03-21T18:21:41.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:42 smithi079 ceph-mon[23158]: pgmap v2087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:42 smithi016 ceph-mon[22010]: pgmap v2087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:42 smithi016 conmon[32727]: 2022-03-21T18:21:42.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:43.459 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:43.460 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:43 smithi016 conmon[32727]: 2022-03-21T18:21:43.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:44 smithi079 ceph-mon[23158]: pgmap v2088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:44.491 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:44 smithi016 ceph-mon[22010]: pgmap v2088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:44.492 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:44 smithi016 conmon[32727]: 2022-03-21T18:21:44.488+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:45.538 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:45 smithi016 conmon[32727]: 2022-03-21T18:21:45.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:46 smithi079 ceph-mon[23158]: pgmap v2089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:46.529 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:46.529 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:46 smithi016 ceph-mon[22010]: pgmap v2089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:46 smithi016 conmon[32727]: 2022-03-21T18:21:46.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:47.542 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:47 smithi016 conmon[32727]: 2022-03-21T18:21:47.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:48 smithi079 ceph-mon[23158]: pgmap v2090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:48.514 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:48.514 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:48 smithi016 ceph-mon[22010]: pgmap v2090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:48 smithi016 conmon[32727]: 2022-03-21T18:21:48.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:49.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:49.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:49 smithi016 conmon[32727]: 2022-03-21T18:21:49.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:50 smithi079 ceph-mon[23158]: pgmap v2091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:50.566 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:50.566 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:50 smithi016 ceph-mon[22010]: pgmap v2091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:50 smithi016 conmon[32727]: 2022-03-21T18:21:50.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:51.561 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:51 smithi016 conmon[32727]: 2022-03-21T18:21:51.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:52 smithi079 ceph-mon[23158]: pgmap v2092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:52.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:52.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:52 smithi016 ceph-mon[22010]: pgmap v2092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:52 smithi016 conmon[32727]: 2022-03-21T18:21:52.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:53.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:53 smithi016 conmon[32727]: 2022-03-21T18:21:53.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:54.535 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:54.535 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:54 smithi016 ceph-mon[22010]: pgmap v2093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:54.536 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:54 smithi079 ceph-mon[23158]: pgmap v2093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:54 smithi016 conmon[32727]: 2022-03-21T18:21:54.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:55.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:55 smithi016 conmon[32727]: 2022-03-21T18:21:55.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:56.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:56.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:56 smithi016 ceph-mon[22010]: pgmap v2094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:56 smithi079 ceph-mon[23158]: pgmap v2094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:21:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:56 smithi016 conmon[32727]: 2022-03-21T18:21:56.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:57.582 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:57 smithi016 conmon[32727]: 2022-03-21T18:21:57.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:58.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:58.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: pgmap v2095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:58.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:21:58.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:21:58.627 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:21:58.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:21:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: pgmap v2095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:21:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:21:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:21:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:21:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:21:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:58 smithi016 conmon[32727]: 2022-03-21T18:21:58.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:21:59.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:59.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:21:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:21:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:21:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:21:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:21:59 smithi016 conmon[32727]: 2022-03-21T18:21:59.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:00.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:00.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:00 smithi016 ceph-mon[22010]: pgmap v2096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:00 smithi079 ceph-mon[23158]: pgmap v2096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:00 smithi016 conmon[32727]: 2022-03-21T18:22:00.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:01.567 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:01 smithi016 conmon[32727]: 2022-03-21T18:22:01.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:02.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:02.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:02 smithi016 ceph-mon[22010]: pgmap v2097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:02 smithi079 ceph-mon[23158]: pgmap v2097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:02 smithi016 conmon[32727]: 2022-03-21T18:22:02.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:03.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:03 smithi016 conmon[32727]: 2022-03-21T18:22:03.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:04.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:04.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:04 smithi016 ceph-mon[22010]: pgmap v2098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:04.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:04 smithi079 ceph-mon[23158]: pgmap v2098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:04.862 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:04 smithi016 conmon[32727]: 2022-03-21T18:22:04.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:05.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:05.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:22:05.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:22:05.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:22:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:22:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:22:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:22:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:05 smithi016 conmon[32727]: 2022-03-21T18:22:05.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:06.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:06.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:06 smithi016 ceph-mon[22010]: pgmap v2099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:06 smithi079 ceph-mon[23158]: pgmap v2099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:06 smithi016 conmon[32727]: 2022-03-21T18:22:06.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:07.599 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:07.600 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: 2022-03-21T18:22:07.501+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:07.600 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:22:07.600 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:07.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: yield 2022-03-21T18:22:07.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:07.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:07.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:07.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:22:07.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:07.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:22:07.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:07.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:22:07.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:07.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:22:07.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:07.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:22:07.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:07.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:07.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:07 smithi016 conmon[32727]: 2022-03-21T18:22:07.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:08.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:08.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:08 smithi016 ceph-mon[22010]: pgmap v2100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:08.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:08.607 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:22:08.607 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:08.607 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:22:08.608 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:08.608 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:08.609 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:08.609 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:22:08.609 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:08.610 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:22:08.610 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:08.611 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:22:08.611 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:08.611 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:08.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:08.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:22:08.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:08.613 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:08.613 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:08.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:08 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:22:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:08 smithi079 ceph-mon[23158]: pgmap v2100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:22:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:22:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:22:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:22:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:22:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:22:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:08 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:22:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:08 smithi016 conmon[32727]: 2022-03-21T18:22:08.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:09.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:09.654 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:09.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:09.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:09 smithi016 conmon[32727]: 2022-03-21T18:22:09.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:10.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:10.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:10 smithi016 ceph-mon[22010]: pgmap v2101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:10 smithi079 ceph-mon[23158]: pgmap v2101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[32727]: 2022-03-21T18:22:10.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:10.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 2022-03-21T18:22:10.574+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:10.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:22:10.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:10.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: yield 2022-03-21T18:22:10.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:10.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:10.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:10.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:22:10.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:10.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:22:10.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:10.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:22:10.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:10.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:22:10.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:10.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:22:10.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:10.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:10.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:10.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 2022-03-21T18:22:10.575+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:22:10.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:22:10.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:10.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: yield 2022-03-21T18:22:10.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:10.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:10.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:10.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:22:10.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:10.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:22:10.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:10.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:22:10.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:10.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:22:10.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:10.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:22:10.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:10.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:10.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:10.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 2022-03-21T18:22:10.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:22:10.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: 2022-03-21T18:22:10.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:22:10.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:22:10.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:22:10.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:22:10.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:22:10.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:22:10.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:22:10.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:22:10.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:22:10.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:22:10.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:22:10.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:22:10.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:22:10.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:22:10.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:22:10.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:22:10.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:22:10.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:22:10.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:22:10.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:22:10.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:22:10.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:22:10.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise 2022-03-21T18:22:10.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:22:10.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:22:10.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:22:10.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:22:10.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:22:10 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:11.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.637 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.638 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:11.638 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:22:11.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:11.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:11.639 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:11.640 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:22:11.640 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:11.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:22:11.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:11.641 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:22:11.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:11.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:11.642 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:11.643 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:22:11.643 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:11.643 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:11.644 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:11 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:22:11.644 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:11.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:22:11.645 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:11.646 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:11.646 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:11.646 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:22:11.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:11.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:22:11.647 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:11.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:22:11.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:11.648 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:11.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:11.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:22:11.649 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:11.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:11.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.650 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:22:11.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:22:11.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:22:11.651 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:22:11.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:22:11.652 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:22:11.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:22:11.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:22:11.653 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:22:11.654 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:22:11.654 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:22:11.654 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:22:11.655 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:22:11.655 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:22:11.655 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:22:11.656 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:22:11.656 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:22:11.656 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:22:11.657 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:22:11.657 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:22:11.657 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:22:11.658 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:22:11.658 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:22:11.659 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:22:11.659 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:22:11.659 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:22:11.660 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:22:11.660 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:22:11.661 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:22:11.661 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:22:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:22:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:22:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:22:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:22:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:11 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:22:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:22:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:22:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:22:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:22:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:22:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:22:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:22:11.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:22:11.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:22:11.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:22:11.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:22:11.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:22:11.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:22:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:22:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:22:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:22:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:22:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:22:11.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:22:11.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:22:11.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:22:11.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:22:11.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:22:11.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:22:11.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:22:11.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:22:11.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:22:11.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:22:11.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:22:11.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:22:11.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:22:11.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:22:11.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:22:11.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:22:11.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:22:11.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:22:11.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:22:11.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:22:11.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:22:11.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:22:11.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:22:11.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:22:11.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:22:11.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:22:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:22:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:22:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:22:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:11 smithi016 conmon[32727]: 2022-03-21T18:22:11.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:12.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:12.602 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:12 smithi016 ceph-mon[22010]: pgmap v2102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:12 smithi079 ceph-mon[23158]: pgmap v2102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:12 smithi016 conmon[32727]: 2022-03-21T18:22:12.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:13.620 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:13 smithi016 conmon[32727]: 2022-03-21T18:22:13.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:14.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:14.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:14 smithi016 ceph-mon[22010]: pgmap v2103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:14.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:14 smithi079 ceph-mon[23158]: pgmap v2103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:14 smithi016 conmon[32727]: 2022-03-21T18:22:14.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:15 smithi016 conmon[32727]: 2022-03-21T18:22:15.666+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:16.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:16.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:16 smithi016 ceph-mon[22010]: pgmap v2104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:16 smithi079 ceph-mon[23158]: pgmap v2104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:16 smithi016 conmon[32727]: 2022-03-21T18:22:16.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:17 smithi016 conmon[32727]: 2022-03-21T18:22:17.671+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:18 smithi016 ceph-mon[22010]: pgmap v2105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:18 smithi079 ceph-mon[23158]: pgmap v2105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:18 smithi016 conmon[32727]: 2022-03-21T18:22:18.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:19.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:19.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:19 smithi016 conmon[32727]: 2022-03-21T18:22:19.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:20.650 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:20.650 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:20 smithi016 ceph-mon[22010]: pgmap v2106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:20 smithi079 ceph-mon[23158]: pgmap v2106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:20 smithi016 conmon[32727]: 2022-03-21T18:22:20.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:21 smithi016 conmon[32727]: 2022-03-21T18:22:21.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:22.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:22.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:22 smithi016 ceph-mon[22010]: pgmap v2107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:22 smithi079 ceph-mon[23158]: pgmap v2107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:22 smithi016 conmon[32727]: 2022-03-21T18:22:22.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:23.624 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:23 smithi016 conmon[32727]: 2022-03-21T18:22:23.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:24.642 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:24.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:24 smithi016 ceph-mon[22010]: pgmap v2108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:24.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:24 smithi079 ceph-mon[23158]: pgmap v2108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:24 smithi016 conmon[32727]: 2022-03-21T18:22:24.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:25.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:25 smithi016 conmon[32727]: 2022-03-21T18:22:25.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:26.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:26.642 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:26 smithi016 ceph-mon[22010]: pgmap v2109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:26 smithi079 ceph-mon[23158]: pgmap v2109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:26 smithi016 conmon[32727]: 2022-03-21T18:22:26.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:27.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:27 smithi016 conmon[32727]: 2022-03-21T18:22:27.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:28.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:28.642 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:28 smithi016 ceph-mon[22010]: pgmap v2110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:28 smithi079 ceph-mon[23158]: pgmap v2110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:28 smithi016 conmon[32727]: 2022-03-21T18:22:28.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:29.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:29.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:29 smithi016 conmon[32727]: 2022-03-21T18:22:29.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:30 smithi016 ceph-mon[22010]: pgmap v2111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:30 smithi016 conmon[32727]: 2022-03-21T18:22:30.548+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:30 smithi079 ceph-mon[23158]: pgmap v2111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:31.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:31 smithi016 conmon[32727]: 2022-03-21T18:22:31.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:32 smithi016 conmon[32727]: 2022-03-21T18:22:32.568+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:32 smithi016 ceph-mon[22010]: pgmap v2112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:32 smithi079 ceph-mon[23158]: pgmap v2112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:33 smithi016 conmon[32727]: 2022-03-21T18:22:33.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:34 smithi016 conmon[32727]: 2022-03-21T18:22:34.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:34 smithi016 ceph-mon[22010]: pgmap v2113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:34 smithi079 ceph-mon[23158]: pgmap v2113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:35 smithi016 conmon[32727]: 2022-03-21T18:22:35.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:36 smithi016 ceph-mon[22010]: pgmap v2114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:36.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:36 smithi016 conmon[32727]: 2022-03-21T18:22:36.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:36 smithi079 ceph-mon[23158]: pgmap v2114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:37 smithi016 conmon[32727]: 2022-03-21T18:22:37.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:38 smithi016 ceph-mon[22010]: pgmap v2115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:38.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:38 smithi016 conmon[32727]: 2022-03-21T18:22:38.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:38 smithi079 ceph-mon[23158]: pgmap v2115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:39 smithi016 conmon[32727]: 2022-03-21T18:22:39.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:40 smithi016 ceph-mon[22010]: pgmap v2116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:40 smithi016 conmon[32727]: 2022-03-21T18:22:40.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:40 smithi079 ceph-mon[23158]: pgmap v2116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:41 smithi016 conmon[32727]: 2022-03-21T18:22:41.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:42 smithi016 ceph-mon[22010]: pgmap v2117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:42 smithi016 conmon[32727]: 2022-03-21T18:22:42.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:42 smithi079 ceph-mon[23158]: pgmap v2117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:43.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:43 smithi016 conmon[32727]: 2022-03-21T18:22:43.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:44 smithi016 conmon[32727]: 2022-03-21T18:22:44.452+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:44 smithi016 ceph-mon[22010]: pgmap v2118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:44 smithi079 ceph-mon[23158]: pgmap v2118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:45 smithi016 conmon[32727]: 2022-03-21T18:22:45.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:46 smithi016 ceph-mon[22010]: pgmap v2119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:46 smithi016 conmon[32727]: 2022-03-21T18:22:46.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:46 smithi079 ceph-mon[23158]: pgmap v2119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:47.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:47 smithi016 conmon[32727]: 2022-03-21T18:22:47.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:48 smithi016 ceph-mon[22010]: pgmap v2120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:48.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:48 smithi016 conmon[32727]: 2022-03-21T18:22:48.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:48 smithi079 ceph-mon[23158]: pgmap v2120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:49.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:49 smithi016 conmon[32727]: 2022-03-21T18:22:49.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:50 smithi016 ceph-mon[22010]: pgmap v2121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:50.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:50 smithi016 conmon[32727]: 2022-03-21T18:22:50.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:50 smithi079 ceph-mon[23158]: pgmap v2121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:51 smithi016 conmon[32727]: 2022-03-21T18:22:51.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:52 smithi016 ceph-mon[22010]: pgmap v2122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:52.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:52 smithi016 conmon[32727]: 2022-03-21T18:22:52.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:52 smithi079 ceph-mon[23158]: pgmap v2122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:53 smithi016 conmon[32727]: 2022-03-21T18:22:53.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:54 smithi016 ceph-mon[22010]: pgmap v2123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:54 smithi016 conmon[32727]: 2022-03-21T18:22:54.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:54 smithi079 ceph-mon[23158]: pgmap v2123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:55 smithi016 conmon[32727]: 2022-03-21T18:22:55.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:56 smithi016 ceph-mon[22010]: pgmap v2124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:56 smithi016 conmon[32727]: 2022-03-21T18:22:56.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:56 smithi079 ceph-mon[23158]: pgmap v2124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:22:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:57 smithi016 conmon[32727]: 2022-03-21T18:22:57.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:58.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:58.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: pgmap v2125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:58.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:22:58.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:22:58.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:22:58.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:22:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: pgmap v2125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:22:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:22:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:22:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:22:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:22:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:58 smithi016 conmon[32727]: 2022-03-21T18:22:58.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:22:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:22:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:22:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:22:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:22:59.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:22:59 smithi016 conmon[32727]: 2022-03-21T18:22:59.664+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:00.695 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:00.695 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:00 smithi016 ceph-mon[22010]: pgmap v2126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:00 smithi079 ceph-mon[23158]: pgmap v2126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:00 smithi016 conmon[32727]: 2022-03-21T18:23:00.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:01.735 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:01 smithi016 conmon[32727]: 2022-03-21T18:23:01.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:02.713 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:02.714 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:02 smithi016 ceph-mon[22010]: pgmap v2127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:02 smithi079 ceph-mon[23158]: pgmap v2127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:02 smithi016 conmon[32727]: 2022-03-21T18:23:02.712+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:03 smithi016 conmon[32727]: 2022-03-21T18:23:03.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:04.813 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:04 smithi079 ceph-mon[23158]: pgmap v2128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:04.814 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:04 smithi016 ceph-mon[22010]: pgmap v2128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:04 smithi016 conmon[32727]: 2022-03-21T18:23:04.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:05 smithi016 conmon[32727]: 2022-03-21T18:23:05.662+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:06 smithi079 ceph-mon[23158]: pgmap v2129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:06 smithi016 ceph-mon[22010]: pgmap v2129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:06 smithi016 conmon[32727]: 2022-03-21T18:23:06.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:07 smithi016 conmon[32727]: 2022-03-21T18:23:07.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:08 smithi079 ceph-mon[23158]: pgmap v2130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:08 smithi016 ceph-mon[22010]: pgmap v2130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:08 smithi016 conmon[32727]: 2022-03-21T18:23:08.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:09 smithi016 conmon[32727]: 2022-03-21T18:23:09.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:10 smithi079 ceph-mon[23158]: pgmap v2131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:10 smithi016 ceph-mon[22010]: pgmap v2131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:10 smithi016 conmon[32727]: 2022-03-21T18:23:10.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:11.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: 2022-03-21T18:23:11.181+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:23:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:11.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: yield 2022-03-21T18:23:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:23:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:23:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:23:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:23:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:23:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:23:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:23:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:23:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:23:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:23:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:23:11.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:11 smithi016 conmon[32727]: 2022-03-21T18:23:11.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:12 smithi079 ceph-mon[23158]: pgmap v2132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:23:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:23:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:23:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:12.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:23:12.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:12.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:23:12.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:12.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:12.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:12.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:23:12.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:12.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:12.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:12.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:12 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:23:12.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:12 smithi016 conmon[32727]: 2022-03-21T18:23:12.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:12.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:12 smithi016 ceph-mon[22010]: pgmap v2132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:12.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:12.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:23:12.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:12.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:23:12.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:12.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:12.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:12.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:23:12.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:12.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:23:12.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:12.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:23:12.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:12.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:12.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:12.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:23:12.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:12.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:12.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:12.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:12 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:23:12.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:13 smithi016 conmon[32727]: 2022-03-21T18:23:13.615+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:14.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:14 smithi016 ceph-mon[22010]: pgmap v2133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:14.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:14.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:14.640 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 2022-03-21T18:23:14.253+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:14.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:23:14.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:14.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: yield 2022-03-21T18:23:14.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:14.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:14.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:14.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:23:14.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:14.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:23:14.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:14.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:23:14.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:14.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:23:14.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:14.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:23:14.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:14.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:14.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:14.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 2022-03-21T18:23:14.254+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:23:14.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:23:14.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:14.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: yield 2022-03-21T18:23:14.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:14.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:14.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:14.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:23:14.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:14.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:23:14.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:14.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:23:14.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:14.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:23:14.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:14.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:23:14.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:14.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:14.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:14.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 2022-03-21T18:23:14.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:23:14.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: 2022-03-21T18:23:14.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:23:14.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:23:14.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:23:14.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:23:14.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:23:14.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:23:14.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:23:14.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:23:14.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:23:14.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:23:14.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:23:14.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:23:14.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:23:14.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:23:14.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:23:14.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:23:14.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:23:14.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:23:14.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:23:14.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:23:14.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:23:14.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:23:14.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise 2022-03-21T18:23:14.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:23:14.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:23:14.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:23:14.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:23:14.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:14 smithi079 ceph-mon[23158]: pgmap v2133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:14 smithi016 conmon[32727]: 2022-03-21T18:23:14.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:23:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:15.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:15.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:23:15.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:15.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:23:15.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:23:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:23:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:15 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:23:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:23:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:23:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:23:15.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:15.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:23:15.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:15.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:15.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:23:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:23:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:23:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:23:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:23:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:23:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:23:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:23:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:23:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:23:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:23:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:23:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:23:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:23:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:23:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:23:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:23:15.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:23:15.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:23:15.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:23:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:23:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:23:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:23:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:23:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:23:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:23:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:23:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:23:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:23:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:23:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:15 smithi016 conmon[32727]: 2022-03-21T18:23:15.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:23:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:23:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:23:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:23:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:23:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:15 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:23:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:23:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:23:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:23:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:23:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:23:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:23:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:23:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:23:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:23:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:23:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:23:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:23:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:23:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:23:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:23:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:23:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:23:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:23:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:23:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:23:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:23:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:23:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:23:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:23:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:23:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:23:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:23:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:23:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:23:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:23:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:23:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:23:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:23:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:23:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:23:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:23:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:23:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:23:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:23:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:23:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:23:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:23:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:23:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:23:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:23:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:23:15.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:23:15.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:16 smithi079 ceph-mon[23158]: pgmap v2134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:16 smithi016 ceph-mon[22010]: pgmap v2134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:16 smithi016 conmon[32727]: 2022-03-21T18:23:16.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:17 smithi016 conmon[32727]: 2022-03-21T18:23:17.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:18 smithi016 ceph-mon[22010]: pgmap v2135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:18 smithi016 conmon[32727]: 2022-03-21T18:23:18.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:18 smithi079 ceph-mon[23158]: pgmap v2135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:19 smithi016 conmon[32727]: 2022-03-21T18:23:19.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:20 smithi016 ceph-mon[22010]: pgmap v2136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:20 smithi016 conmon[32727]: 2022-03-21T18:23:20.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:20 smithi079 ceph-mon[23158]: pgmap v2136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:21 smithi016 conmon[32727]: 2022-03-21T18:23:21.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:22 smithi016 ceph-mon[22010]: pgmap v2137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:22 smithi016 conmon[32727]: 2022-03-21T18:23:22.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:22 smithi079 ceph-mon[23158]: pgmap v2137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:23 smithi016 conmon[32727]: 2022-03-21T18:23:23.736+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:24.780 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:24 smithi016 ceph-mon[22010]: pgmap v2138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:24.780 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:24.781 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:24 smithi079 ceph-mon[23158]: pgmap v2138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:24 smithi016 conmon[32727]: 2022-03-21T18:23:24.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:25.762 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:25 smithi016 conmon[32727]: 2022-03-21T18:23:25.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:26.798 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:26 smithi016 ceph-mon[22010]: pgmap v2139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:26.799 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:26 smithi079 ceph-mon[23158]: pgmap v2139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:26 smithi016 conmon[32727]: 2022-03-21T18:23:26.796+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:27.760 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:27 smithi016 conmon[32727]: 2022-03-21T18:23:27.757+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:28 smithi016 ceph-mon[22010]: pgmap v2140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:28 smithi016 conmon[32727]: 2022-03-21T18:23:28.742+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:28 smithi079 ceph-mon[23158]: pgmap v2140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:29 smithi016 conmon[32727]: 2022-03-21T18:23:29.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:30 smithi016 ceph-mon[22010]: pgmap v2141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:30 smithi016 conmon[32727]: 2022-03-21T18:23:30.716+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:30 smithi079 ceph-mon[23158]: pgmap v2141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:31 smithi016 conmon[32727]: 2022-03-21T18:23:31.742+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:32 smithi016 ceph-mon[22010]: pgmap v2142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:32 smithi016 conmon[32727]: 2022-03-21T18:23:32.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:32 smithi079 ceph-mon[23158]: pgmap v2142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:33 smithi016 conmon[32727]: 2022-03-21T18:23:33.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:34 smithi016 conmon[32727]: 2022-03-21T18:23:34.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:34 smithi016 ceph-mon[22010]: pgmap v2143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:34 smithi079 ceph-mon[23158]: pgmap v2143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:35 smithi016 conmon[32727]: 2022-03-21T18:23:35.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:36 smithi016 conmon[32727]: 2022-03-21T18:23:36.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:36 smithi016 ceph-mon[22010]: pgmap v2144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:36 smithi079 ceph-mon[23158]: pgmap v2144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:37 smithi016 conmon[32727]: 2022-03-21T18:23:37.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:38 smithi016 conmon[32727]: 2022-03-21T18:23:38.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:38 smithi016 ceph-mon[22010]: pgmap v2145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:38 smithi079 ceph-mon[23158]: pgmap v2145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:39 smithi016 conmon[32727]: 2022-03-21T18:23:39.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:40 smithi016 ceph-mon[22010]: pgmap v2146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:40 smithi016 conmon[32727]: 2022-03-21T18:23:40.523+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:40 smithi079 ceph-mon[23158]: pgmap v2146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:41 smithi016 conmon[32727]: 2022-03-21T18:23:41.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:42 smithi016 ceph-mon[22010]: pgmap v2147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:42 smithi016 conmon[32727]: 2022-03-21T18:23:42.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:42 smithi079 ceph-mon[23158]: pgmap v2147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:43 smithi016 conmon[32727]: 2022-03-21T18:23:43.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:44 smithi016 conmon[32727]: 2022-03-21T18:23:44.520+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:44 smithi016 ceph-mon[22010]: pgmap v2148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:44 smithi079 ceph-mon[23158]: pgmap v2148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:45 smithi016 conmon[32727]: 2022-03-21T18:23:45.471+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:46 smithi016 conmon[32727]: 2022-03-21T18:23:46.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:46 smithi016 ceph-mon[22010]: pgmap v2149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:46 smithi079 ceph-mon[23158]: pgmap v2149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:47 smithi016 conmon[32727]: 2022-03-21T18:23:47.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:47.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:48 smithi016 conmon[32727]: 2022-03-21T18:23:48.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:48 smithi016 ceph-mon[22010]: pgmap v2150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:48 smithi079 ceph-mon[23158]: pgmap v2150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:49 smithi016 conmon[32727]: 2022-03-21T18:23:49.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:49.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:50 smithi016 conmon[32727]: 2022-03-21T18:23:50.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:50 smithi016 ceph-mon[22010]: pgmap v2151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:50 smithi079 ceph-mon[23158]: pgmap v2151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:51 smithi016 conmon[32727]: 2022-03-21T18:23:51.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:52 smithi016 conmon[32727]: 2022-03-21T18:23:52.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:52 smithi016 ceph-mon[22010]: pgmap v2152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:52 smithi079 ceph-mon[23158]: pgmap v2152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:53 smithi016 conmon[32727]: 2022-03-21T18:23:53.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:54 smithi016 ceph-mon[22010]: pgmap v2153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:54 smithi016 conmon[32727]: 2022-03-21T18:23:54.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:54 smithi079 ceph-mon[23158]: pgmap v2153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:55.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:55 smithi016 conmon[32727]: 2022-03-21T18:23:55.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:56 smithi016 ceph-mon[22010]: pgmap v2154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:56 smithi016 conmon[32727]: 2022-03-21T18:23:56.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:56 smithi079 ceph-mon[23158]: pgmap v2154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:23:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:57 smithi016 conmon[32727]: 2022-03-21T18:23:57.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:58 smithi016 conmon[32727]: 2022-03-21T18:23:58.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: pgmap v2155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:23:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:23:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:23:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:23:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: pgmap v2155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:23:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:23:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:23:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:23:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:23:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:23:59 smithi016 conmon[32727]: 2022-03-21T18:23:59.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:23:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:23:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:23:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:23:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:23:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:00 smithi016 ceph-mon[22010]: pgmap v2156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:00.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:00 smithi016 conmon[32727]: 2022-03-21T18:24:00.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:00 smithi079 ceph-mon[23158]: pgmap v2156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:01 smithi016 conmon[32727]: 2022-03-21T18:24:01.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:02 smithi016 ceph-mon[22010]: pgmap v2157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:02 smithi016 conmon[32727]: 2022-03-21T18:24:02.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:02 smithi079 ceph-mon[23158]: pgmap v2157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:03 smithi016 conmon[32727]: 2022-03-21T18:24:03.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:04 smithi016 ceph-mon[22010]: pgmap v2158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:04 smithi016 conmon[32727]: 2022-03-21T18:24:04.742+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:04 smithi079 ceph-mon[23158]: pgmap v2158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:05 smithi016 conmon[32727]: 2022-03-21T18:24:05.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:06 smithi016 ceph-mon[22010]: pgmap v2159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:06 smithi016 conmon[32727]: 2022-03-21T18:24:06.736+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:06 smithi079 ceph-mon[23158]: pgmap v2159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:07 smithi016 conmon[32727]: 2022-03-21T18:24:07.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:08 smithi016 conmon[32727]: 2022-03-21T18:24:08.682+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:08 smithi016 ceph-mon[22010]: pgmap v2160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:08 smithi079 ceph-mon[23158]: pgmap v2160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:09 smithi016 conmon[32727]: 2022-03-21T18:24:09.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:10 smithi016 conmon[32727]: 2022-03-21T18:24:10.703+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:10 smithi016 ceph-mon[22010]: pgmap v2161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:10 smithi079 ceph-mon[23158]: pgmap v2161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:11 smithi016 conmon[32727]: 2022-03-21T18:24:11.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:12 smithi016 conmon[32727]: 2022-03-21T18:24:12.774+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:12 smithi016 ceph-mon[22010]: pgmap v2162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:12 smithi079 ceph-mon[23158]: pgmap v2162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:13 smithi016 conmon[32727]: 2022-03-21T18:24:13.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:14 smithi016 conmon[32727]: 2022-03-21T18:24:14.808+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: pgmap v2163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:24:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:24:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:24:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: pgmap v2163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:24:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:24:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:24:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:15 smithi016 conmon[32727]: 2022-03-21T18:24:15.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:16 smithi016 ceph-mon[22010]: pgmap v2164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:16 smithi016 conmon[32727]: 2022-03-21T18:24:16.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:16 smithi079 ceph-mon[23158]: pgmap v2164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:17.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: 2022-03-21T18:24:17.357+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:17.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:24:17.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:17.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: yield 2022-03-21T18:24:17.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:17.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:17.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:17.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:24:17.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:17.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:24:17.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:17.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:24:17.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:17.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:24:17.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:17.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:24:17.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:17.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:17.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:17 smithi016 conmon[32727]: 2022-03-21T18:24:17.788+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:18 smithi016 conmon[32727]: 2022-03-21T18:24:18.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:18 smithi016 ceph-mon[22010]: pgmap v2165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:24:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:24:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:24:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:24:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:24:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:24:18.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:18.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:18.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:18.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:24:18.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:18 smithi079 ceph-mon[23158]: pgmap v2165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:24:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:24:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:24:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:24:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:18.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:24:18.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:18.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:18.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:18.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:24:18.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:18.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:18.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:18.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:24:18.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:19 smithi016 conmon[32727]: 2022-03-21T18:24:19.759+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:20.792 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:20 smithi016 ceph-mon[22010]: pgmap v2166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:20.792 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:20.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 2022-03-21T18:24:20.429+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:20.794 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:24:20.794 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:20.794 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: yield 2022-03-21T18:24:20.795 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:20.795 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:20.795 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:20.796 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:24:20.796 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:20.796 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:24:20.797 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:20.797 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:24:20.798 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:20.798 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:24:20.799 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:20.799 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:24:20.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:20.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:20.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:20.801 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 2022-03-21T18:24:20.430+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:24:20.801 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:24:20.802 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:20.802 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: yield 2022-03-21T18:24:20.802 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:20.803 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:20.803 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:20.803 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:24:20.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:20.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:24:20.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:20.805 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:24:20.805 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:20.805 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:24:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:24:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 2022-03-21T18:24:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:24:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: 2022-03-21T18:24:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:24:20.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:24:20.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:24:20.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:24:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:24:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:24:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:24:20.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:24:20.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:24:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:24:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:24:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:24:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:24:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:24:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:24:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:24:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:24:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:24:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:24:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:24:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:24:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:24:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise 2022-03-21T18:24:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:24:20.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:24:20.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:24:20.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:24:20.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:20 smithi079 ceph-mon[23158]: pgmap v2166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:20 smithi016 conmon[32727]: 2022-03-21T18:24:20.790+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:21 smithi016 conmon[32727]: 2022-03-21T18:24:21.773+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:24:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:24:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:24:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:24:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:24:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:24:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:24:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:24:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:24:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:24:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:24:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:24:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:24:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:24:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:24:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:24:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:24:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:24:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:24:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:24:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:24:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:24:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:24:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:24:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:24:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:24:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:24:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:24:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:24:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:24:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:24:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:24:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:24:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:24:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:24:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:24:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:24:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:24:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:24:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:24:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:24:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:24:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:24:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:24:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:24:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:24:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:24:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:24:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:24:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:24:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:24:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:24:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:24:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:24:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:24:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:24:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:24:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:24:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:24:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:24:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:24:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:24:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:24:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:24:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:24:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:24:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:24:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:24:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:24:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:24:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:24:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:24:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:24:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:24:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:24:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:24:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:24:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:24:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:24:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:24:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:24:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:24:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:24:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:24:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:24:21.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:24:21.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:24:21.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:24:21.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:24:21.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:24:21.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:24:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:24:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:24:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:22 smithi016 ceph-mon[22010]: pgmap v2167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:22 smithi016 conmon[32727]: 2022-03-21T18:24:22.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:22 smithi079 ceph-mon[23158]: pgmap v2167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:23 smithi016 conmon[32727]: 2022-03-21T18:24:23.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:24 smithi016 ceph-mon[22010]: pgmap v2168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:24 smithi016 conmon[32727]: 2022-03-21T18:24:24.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:24 smithi079 ceph-mon[23158]: pgmap v2168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:25 smithi016 conmon[32727]: 2022-03-21T18:24:25.763+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:26 smithi079 ceph-mon[23158]: pgmap v2169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:26 smithi016 ceph-mon[22010]: pgmap v2169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:27.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:26 smithi016 conmon[32727]: 2022-03-21T18:24:26.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:27 smithi016 conmon[32727]: 2022-03-21T18:24:27.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:28 smithi079 ceph-mon[23158]: pgmap v2170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:28 smithi016 ceph-mon[22010]: pgmap v2170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:28 smithi016 conmon[32727]: 2022-03-21T18:24:28.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:29 smithi016 conmon[32727]: 2022-03-21T18:24:29.808+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:30.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:30 smithi079 ceph-mon[23158]: pgmap v2171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:30 smithi016 conmon[32727]: 2022-03-21T18:24:30.796+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:30 smithi016 ceph-mon[22010]: pgmap v2171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:31 smithi079 ceph-mon[23158]: pgmap v2172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:31 smithi016 ceph-mon[22010]: pgmap v2172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:31 smithi016 conmon[32727]: 2022-03-21T18:24:31.757+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:32 smithi016 conmon[32727]: 2022-03-21T18:24:32.774+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:33 smithi079 ceph-mon[23158]: pgmap v2173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:33 smithi016 ceph-mon[22010]: pgmap v2173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:33 smithi016 conmon[32727]: 2022-03-21T18:24:33.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:34 smithi016 conmon[32727]: 2022-03-21T18:24:34.841+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:35 smithi079 ceph-mon[23158]: pgmap v2174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:35 smithi016 ceph-mon[22010]: pgmap v2174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:35 smithi016 conmon[32727]: 2022-03-21T18:24:35.806+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:36 smithi016 conmon[32727]: 2022-03-21T18:24:36.804+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:37 smithi079 ceph-mon[23158]: pgmap v2175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:37 smithi016 ceph-mon[22010]: pgmap v2175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:38.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:37 smithi016 conmon[32727]: 2022-03-21T18:24:37.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:38 smithi016 conmon[32727]: 2022-03-21T18:24:38.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:39 smithi079 ceph-mon[23158]: pgmap v2176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:39 smithi016 ceph-mon[22010]: pgmap v2176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:39 smithi016 conmon[32727]: 2022-03-21T18:24:39.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:40 smithi016 conmon[32727]: 2022-03-21T18:24:40.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:42.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:42.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:41 smithi016 ceph-mon[22010]: pgmap v2177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:42.170 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:41 smithi016 conmon[32727]: 2022-03-21T18:24:41.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:41 smithi079 ceph-mon[23158]: pgmap v2177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:42 smithi016 conmon[32727]: 2022-03-21T18:24:42.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:43 smithi016 ceph-mon[22010]: pgmap v2178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:44.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:43 smithi016 conmon[32727]: 2022-03-21T18:24:43.821+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:43 smithi079 ceph-mon[23158]: pgmap v2178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:44 smithi016 conmon[32727]: 2022-03-21T18:24:44.846+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:45 smithi016 ceph-mon[22010]: pgmap v2179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:45 smithi016 conmon[32727]: 2022-03-21T18:24:45.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:45 smithi079 ceph-mon[23158]: pgmap v2179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:46 smithi016 conmon[32727]: 2022-03-21T18:24:46.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:47 smithi016 ceph-mon[22010]: pgmap v2180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:47 smithi016 conmon[32727]: 2022-03-21T18:24:47.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:47 smithi079 ceph-mon[23158]: pgmap v2180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:48 smithi016 conmon[32727]: 2022-03-21T18:24:48.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:49 smithi016 ceph-mon[22010]: pgmap v2181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:49 smithi016 conmon[32727]: 2022-03-21T18:24:49.912+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:49 smithi079 ceph-mon[23158]: pgmap v2181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:50 smithi016 conmon[32727]: 2022-03-21T18:24:50.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:51 smithi016 ceph-mon[22010]: pgmap v2182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:51 smithi016 conmon[32727]: 2022-03-21T18:24:51.903+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:51 smithi079 ceph-mon[23158]: pgmap v2182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:52 smithi016 conmon[32727]: 2022-03-21T18:24:52.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:53 smithi016 ceph-mon[22010]: pgmap v2183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:53 smithi016 conmon[32727]: 2022-03-21T18:24:53.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:53 smithi079 ceph-mon[23158]: pgmap v2183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:54 smithi016 conmon[32727]: 2022-03-21T18:24:54.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:55 smithi016 ceph-mon[22010]: pgmap v2184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:55 smithi016 conmon[32727]: 2022-03-21T18:24:55.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:55 smithi079 ceph-mon[23158]: pgmap v2184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:24:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:56 smithi016 conmon[32727]: 2022-03-21T18:24:56.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:57 smithi016 conmon[32727]: 2022-03-21T18:24:57.828+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:58.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:57 smithi016 ceph-mon[22010]: pgmap v2185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:57 smithi079 ceph-mon[23158]: pgmap v2185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:24:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:58 smithi016 conmon[32727]: 2022-03-21T18:24:58.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:24:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:24:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:24:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:24:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:24:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:24:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:24:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:24:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:24:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:24:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:24:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:24:59 smithi016 ceph-mon[22010]: pgmap v2186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:24:59 smithi016 conmon[32727]: 2022-03-21T18:24:59.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:24:59 smithi079 ceph-mon[23158]: pgmap v2186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:00 smithi016 conmon[32727]: 2022-03-21T18:25:00.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:01 smithi016 ceph-mon[22010]: pgmap v2187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:01 smithi016 conmon[32727]: 2022-03-21T18:25:01.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:01 smithi079 ceph-mon[23158]: pgmap v2187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:02 smithi016 conmon[32727]: 2022-03-21T18:25:02.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:03 smithi016 conmon[32727]: 2022-03-21T18:25:03.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:03 smithi016 ceph-mon[22010]: pgmap v2188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:03 smithi079 ceph-mon[23158]: pgmap v2188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:04 smithi016 conmon[32727]: 2022-03-21T18:25:04.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:05 smithi016 conmon[32727]: 2022-03-21T18:25:05.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:05 smithi016 ceph-mon[22010]: pgmap v2189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:05 smithi079 ceph-mon[23158]: pgmap v2189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:06 smithi016 conmon[32727]: 2022-03-21T18:25:06.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:07 smithi016 ceph-mon[22010]: pgmap v2190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:08.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:07 smithi016 conmon[32727]: 2022-03-21T18:25:07.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:07 smithi079 ceph-mon[23158]: pgmap v2190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:08 smithi016 conmon[32727]: 2022-03-21T18:25:08.891+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:09 smithi016 ceph-mon[22010]: pgmap v2191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:10.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:09 smithi016 conmon[32727]: 2022-03-21T18:25:09.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:09 smithi079 ceph-mon[23158]: pgmap v2191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:10 smithi016 conmon[32727]: 2022-03-21T18:25:10.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:11 smithi016 ceph-mon[22010]: pgmap v2192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:11 smithi016 conmon[32727]: 2022-03-21T18:25:11.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:11 smithi079 ceph-mon[23158]: pgmap v2192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:12 smithi016 conmon[32727]: 2022-03-21T18:25:12.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:13 smithi016 ceph-mon[22010]: pgmap v2193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:14.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:13 smithi016 conmon[32727]: 2022-03-21T18:25:13.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:13 smithi079 ceph-mon[23158]: pgmap v2193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:14 smithi016 conmon[32727]: 2022-03-21T18:25:14.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:15 smithi016 conmon[32727]: 2022-03-21T18:25:15.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:15 smithi016 ceph-mon[22010]: pgmap v2194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:15 smithi079 ceph-mon[23158]: pgmap v2194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:16 smithi016 conmon[32727]: 2022-03-21T18:25:16.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:18 smithi016 conmon[32727]: 2022-03-21T18:25:18.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:18.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:17 smithi016 ceph-mon[22010]: pgmap v2195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:17 smithi079 ceph-mon[23158]: pgmap v2195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:19 smithi016 conmon[32727]: 2022-03-21T18:25:19.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:19 smithi016 ceph-mon[22010]: pgmap v2196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[32727]: 2022-03-21T18:25:20.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:19 smithi079 ceph-mon[23158]: pgmap v2196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:21 smithi016 conmon[32727]: 2022-03-21T18:25:21.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:25:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:25:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:25:21.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: 2022-03-21T18:25:20.910+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:25:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: yield 2022-03-21T18:25:21.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:21.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:25:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:25:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:25:21.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:21.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:25:21.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:25:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:21.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:25:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:25:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:22 smithi016 conmon[32727]: 2022-03-21T18:25:22.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:25:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:25:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:25:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:25:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:25:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:25:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:22.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:25:22.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:22.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:21 smithi016 ceph-mon[22010]: pgmap v2197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:25:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:25:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:25:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:25:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:25:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:25:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:25:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:21 smithi079 ceph-mon[23158]: pgmap v2197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[32727]: 2022-03-21T18:25:23.081+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:24 smithi016 conmon[32727]: 2022-03-21T18:25:24.053+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:23 smithi016 ceph-mon[22010]: pgmap v2198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:24.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 2022-03-21T18:25:23.981+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:24.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:25:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: yield 2022-03-21T18:25:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:24.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:24.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:24.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:25:24.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:24.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:25:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:25:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:25:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:25:24.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:24.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:24.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 2022-03-21T18:25:23.982+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:25:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:25:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: yield 2022-03-21T18:25:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:25:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:25:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:25:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:25:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:25:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 2022-03-21T18:25:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:25:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: 2022-03-21T18:25:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:25:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:25:24.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:25:24.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:25:24.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:25:24.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:25:24.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:25:24.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:25:24.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:25:24.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:25:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:25:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:25:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:25:24.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:25:24.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:25:24.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:25:24.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:25:24.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:25:24.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:25:24.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:25:24.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:25:24.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:25:24.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise 2022-03-21T18:25:24.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:25:24.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:25:24.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:25:24.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:25:24.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:25:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:23 smithi079 ceph-mon[23158]: pgmap v2198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:25 smithi016 conmon[32727]: 2022-03-21T18:25:25.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:25:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:25:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:25:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:25:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:25:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:25:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:25:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:25:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:25:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:25:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:25:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:25:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:25:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:25:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:25:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:25:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:25:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:25:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:25:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:25:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:25:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:25:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:25:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:25:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:25:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:25:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:25:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:25:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:25:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:25:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:25:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:25:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:25:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:25:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:25:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:25:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:25:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:25:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:25:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:25:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:25:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:25:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:25:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:25:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:25:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:25:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:25:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:25:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:25:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:25:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:25:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:25:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:25:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:25:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:25:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:25:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:25:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:25:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:25:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:25:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:25:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:25:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:25:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:25:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:25:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:25:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:25:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:25:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:25:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:25:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:25:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:25:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:25:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:25:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:25:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:25:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:25:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:25:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:25:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:25:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:25:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:25:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:25:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:25:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:25:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:25:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:25:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:25:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:25:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:25:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:25:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:25:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:25:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:25 smithi016 ceph-mon[22010]: pgmap v2199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:26.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:26 smithi016 conmon[32727]: 2022-03-21T18:25:26.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:25 smithi079 ceph-mon[23158]: pgmap v2199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:27 smithi016 conmon[32727]: 2022-03-21T18:25:27.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:27 smithi016 ceph-mon[22010]: pgmap v2200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:28 smithi016 conmon[32727]: 2022-03-21T18:25:28.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:27 smithi079 ceph-mon[23158]: pgmap v2200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:29 smithi016 conmon[32727]: 2022-03-21T18:25:29.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:29 smithi016 ceph-mon[22010]: pgmap v2201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:30 smithi016 conmon[32727]: 2022-03-21T18:25:30.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:29 smithi079 ceph-mon[23158]: pgmap v2201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:31 smithi016 conmon[32727]: 2022-03-21T18:25:31.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:31 smithi016 ceph-mon[22010]: pgmap v2202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:32 smithi016 conmon[32727]: 2022-03-21T18:25:32.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:31 smithi079 ceph-mon[23158]: pgmap v2202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:33.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:33 smithi016 conmon[32727]: 2022-03-21T18:25:33.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:34 smithi016 conmon[32727]: 2022-03-21T18:25:34.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:33 smithi016 ceph-mon[22010]: pgmap v2203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:34.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:33 smithi079 ceph-mon[23158]: pgmap v2203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:35 smithi016 conmon[32727]: 2022-03-21T18:25:35.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:35 smithi016 ceph-mon[22010]: pgmap v2204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:36 smithi016 conmon[32727]: 2022-03-21T18:25:36.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:35 smithi079 ceph-mon[23158]: pgmap v2204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:37 smithi016 conmon[32727]: 2022-03-21T18:25:37.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:37 smithi016 ceph-mon[22010]: pgmap v2205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:38 smithi016 conmon[32727]: 2022-03-21T18:25:38.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:37 smithi079 ceph-mon[23158]: pgmap v2205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:39 smithi016 conmon[32727]: 2022-03-21T18:25:39.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:40 smithi016 conmon[32727]: 2022-03-21T18:25:40.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:39 smithi016 ceph-mon[22010]: pgmap v2206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:39 smithi079 ceph-mon[23158]: pgmap v2206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:41 smithi016 conmon[32727]: 2022-03-21T18:25:41.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:41 smithi079 ceph-mon[23158]: pgmap v2207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:42 smithi016 conmon[32727]: 2022-03-21T18:25:42.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:41 smithi016 ceph-mon[22010]: pgmap v2207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:43.187 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:43 smithi016 conmon[32727]: 2022-03-21T18:25:43.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:44.211 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:43 smithi016 ceph-mon[22010]: pgmap v2208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:44.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:44.212 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:43 smithi079 ceph-mon[23158]: pgmap v2208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:44 smithi016 conmon[32727]: 2022-03-21T18:25:44.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:45.213 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:45 smithi016 conmon[32727]: 2022-03-21T18:25:45.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:45 smithi079 ceph-mon[23158]: pgmap v2209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:45 smithi016 ceph-mon[22010]: pgmap v2209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:46 smithi016 conmon[32727]: 2022-03-21T18:25:46.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:47 smithi016 conmon[32727]: 2022-03-21T18:25:47.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:47 smithi079 ceph-mon[23158]: pgmap v2210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:47 smithi016 ceph-mon[22010]: pgmap v2210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:48 smithi016 conmon[32727]: 2022-03-21T18:25:48.132+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:49 smithi016 conmon[32727]: 2022-03-21T18:25:49.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:49 smithi079 ceph-mon[23158]: pgmap v2211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:49 smithi016 ceph-mon[22010]: pgmap v2211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:50 smithi016 conmon[32727]: 2022-03-21T18:25:50.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:51 smithi016 conmon[32727]: 2022-03-21T18:25:51.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:51 smithi079 ceph-mon[23158]: pgmap v2212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:51 smithi016 ceph-mon[22010]: pgmap v2212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:52 smithi016 conmon[32727]: 2022-03-21T18:25:52.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:53 smithi016 conmon[32727]: 2022-03-21T18:25:53.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:53 smithi079 ceph-mon[23158]: pgmap v2213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:53 smithi016 ceph-mon[22010]: pgmap v2213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:54.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:54 smithi016 conmon[32727]: 2022-03-21T18:25:54.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:55 smithi016 conmon[32727]: 2022-03-21T18:25:55.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:55 smithi079 ceph-mon[23158]: pgmap v2214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:55 smithi016 ceph-mon[22010]: pgmap v2214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:25:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:56 smithi016 conmon[32727]: 2022-03-21T18:25:56.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:57 smithi016 conmon[32727]: 2022-03-21T18:25:57.104+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:57 smithi079 ceph-mon[23158]: pgmap v2215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:58 smithi016 conmon[32727]: 2022-03-21T18:25:58.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:57 smithi016 ceph-mon[22010]: pgmap v2215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:25:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:25:59 smithi016 conmon[32727]: 2022-03-21T18:25:59.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:25:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:25:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:25:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:25:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:25:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:25:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:25:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:25:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:25:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:25:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:25:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:25:59 smithi016 ceph-mon[22010]: pgmap v2216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:00 smithi016 conmon[32727]: 2022-03-21T18:26:00.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:25:59 smithi079 ceph-mon[23158]: pgmap v2216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:01 smithi016 conmon[32727]: 2022-03-21T18:26:01.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:01 smithi016 ceph-mon[22010]: pgmap v2217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:02 smithi016 conmon[32727]: 2022-03-21T18:26:02.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:01 smithi079 ceph-mon[23158]: pgmap v2217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:03 smithi016 conmon[32727]: 2022-03-21T18:26:03.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:03 smithi016 ceph-mon[22010]: pgmap v2218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:04 smithi016 conmon[32727]: 2022-03-21T18:26:04.081+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:03 smithi079 ceph-mon[23158]: pgmap v2218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:05 smithi016 conmon[32727]: 2022-03-21T18:26:05.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:05 smithi016 ceph-mon[22010]: pgmap v2219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:06 smithi016 conmon[32727]: 2022-03-21T18:26:06.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:05 smithi079 ceph-mon[23158]: pgmap v2219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:07 smithi016 conmon[32727]: 2022-03-21T18:26:07.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:08 smithi016 ceph-mon[22010]: pgmap v2220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:08 smithi016 conmon[32727]: 2022-03-21T18:26:07.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:08 smithi079 ceph-mon[23158]: pgmap v2220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:09.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:08 smithi016 conmon[32727]: 2022-03-21T18:26:08.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:10 smithi016 ceph-mon[22010]: pgmap v2221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:09 smithi016 conmon[32727]: 2022-03-21T18:26:09.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:10 smithi079 ceph-mon[23158]: pgmap v2221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:10 smithi016 conmon[32727]: 2022-03-21T18:26:10.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:12 smithi016 ceph-mon[22010]: pgmap v2222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:11 smithi016 conmon[32727]: 2022-03-21T18:26:11.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:12 smithi079 ceph-mon[23158]: pgmap v2222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:12 smithi016 conmon[32727]: 2022-03-21T18:26:12.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:13 smithi016 conmon[32727]: 2022-03-21T18:26:13.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:14 smithi016 ceph-mon[22010]: pgmap v2223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:14 smithi079 ceph-mon[23158]: pgmap v2223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:14 smithi016 conmon[32727]: 2022-03-21T18:26:14.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:16 smithi016 ceph-mon[22010]: pgmap v2224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:15 smithi016 conmon[32727]: 2022-03-21T18:26:15.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:16 smithi079 ceph-mon[23158]: pgmap v2224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:16 smithi016 conmon[32727]: 2022-03-21T18:26:16.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:18 smithi016 ceph-mon[22010]: pgmap v2225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:17 smithi016 conmon[32727]: 2022-03-21T18:26:17.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:18 smithi079 ceph-mon[23158]: pgmap v2225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:18 smithi016 conmon[32727]: 2022-03-21T18:26:18.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:20 smithi016 ceph-mon[22010]: pgmap v2226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:20.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:19 smithi016 conmon[32727]: 2022-03-21T18:26:19.943+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:20.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:20 smithi079 ceph-mon[23158]: pgmap v2226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:20 smithi016 conmon[32727]: 2022-03-21T18:26:20.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:22 smithi016 ceph-mon[22010]: pgmap v2227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:21 smithi016 conmon[32727]: 2022-03-21T18:26:21.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:22 smithi079 ceph-mon[23158]: pgmap v2227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:22 smithi016 conmon[32727]: 2022-03-21T18:26:22.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:23 smithi016 conmon[32727]: 2022-03-21T18:26:23.900+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: pgmap v2228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:26:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:26:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:26:24.431 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:24.432 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: pgmap v2228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:24.432 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:24.432 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:26:24.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:26:24.433 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:26:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:24 smithi016 conmon[32727]: 2022-03-21T18:26:24.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:25 smithi016 conmon[32727]: 2022-03-21T18:26:25.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:26 smithi016 ceph-mon[22010]: pgmap v2229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:26 smithi079 ceph-mon[23158]: pgmap v2229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:26 smithi016 conmon[32727]: 2022-03-21T18:26:26.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:27.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: 2022-03-21T18:26:27.085+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:27.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:26:27.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:27.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: yield 2022-03-21T18:26:27.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:27.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:27.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:27.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:26:27.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:27.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:26:27.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:27.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:26:27.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:27.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:26:27.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:27.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:26:27.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:27.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:27.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:27 smithi016 conmon[32727]: 2022-03-21T18:26:27.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:26:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:26:28.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:28.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:26:28.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:28.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:26:28.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:28.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:26:28.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:28.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:28.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:28.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:26:28.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:28.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:28.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:28.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:28 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:26:28.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:28 smithi016 ceph-mon[22010]: pgmap v2230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:26:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:26:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:28.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:26:28.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:28.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:26:28.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:28.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:26:28.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:28.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:28.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:28.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:26:28.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:28.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:28.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:28.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:28 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:26:28.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:28 smithi079 ceph-mon[23158]: pgmap v2230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:28 smithi016 conmon[32727]: 2022-03-21T18:26:28.973+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:30 smithi016 ceph-mon[22010]: pgmap v2231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:29 smithi016 conmon[32727]: 2022-03-21T18:26:29.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:30.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 2022-03-21T18:26:30.157+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:30.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:26:30.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:30.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: yield 2022-03-21T18:26:30.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:30.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:30.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:30.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:26:30.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:30.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:26:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:26:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:26:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:26:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 2022-03-21T18:26:30.158+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:26:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:26:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:30.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: yield 2022-03-21T18:26:30.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:26:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:26:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:30.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:26:30.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:30.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:26:30.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:30.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:26:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 2022-03-21T18:26:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:26:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: 2022-03-21T18:26:30.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:26:30.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:26:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:26:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:26:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:26:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:26:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:26:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:26:30.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:26:30.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:26:30.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:26:30.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:26:30.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:26:30.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:26:30.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:26:30.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:26:30.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:26:30.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:26:30.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:26:30.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:26:30.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:26:30.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:26:30.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise 2022-03-21T18:26:30.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:26:30.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:26:30.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:26:30.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:26:30.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:26:30 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:30 smithi079 ceph-mon[23158]: pgmap v2231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:31 smithi016 conmon[32727]: 2022-03-21T18:26:31.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:26:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:31.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:31.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:26:31.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:31.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:26:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:31.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:26:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:31.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:26:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:31.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:26:31.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:26:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:31.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:26:31.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:26:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:31.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:26:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:31.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:26:31.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:31.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:26:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:26:31.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:26:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:26:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:26:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:26:31.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:26:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:26:31.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:26:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:26:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:26:31.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:26:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:26:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:26:31.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:26:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:26:31.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:26:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:26:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:26:31.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:26:31.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:26:31.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:26:31.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:26:31.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:26:31.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:26:31.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:26:31.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:26:31.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:26:31.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:26:31.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:26:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:31.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:26:31.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:26:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:31.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:26:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:31.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:26:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:31.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:26:31.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:26:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:26:31.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:26:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:26:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:26:31.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:26:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:26:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:26:31.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:26:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:26:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:26:31.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:26:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:26:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:26:31.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:26:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:26:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:31.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:26:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:26:31.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:26:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:26:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:26:31.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:26:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:26:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:26:31.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:26:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:26:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:26:31.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:26:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:26:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:26:31.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:26:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:26:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:26:31.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:26:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:26:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:26:31.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:26:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:26:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:26:31.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:26:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:26:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:26:31.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:26:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:26:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:26:31.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:26:31.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:26:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:32 smithi016 ceph-mon[22010]: pgmap v2232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:32 smithi016 conmon[32727]: 2022-03-21T18:26:32.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:32 smithi079 ceph-mon[23158]: pgmap v2232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:33 smithi016 conmon[32727]: 2022-03-21T18:26:33.065+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:34 smithi016 ceph-mon[22010]: pgmap v2233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:34 smithi016 conmon[32727]: 2022-03-21T18:26:34.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:34 smithi079 ceph-mon[23158]: pgmap v2233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:35 smithi016 conmon[32727]: 2022-03-21T18:26:35.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:36 smithi016 ceph-mon[22010]: pgmap v2234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:36 smithi016 conmon[32727]: 2022-03-21T18:26:36.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:36 smithi079 ceph-mon[23158]: pgmap v2234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:37 smithi016 conmon[32727]: 2022-03-21T18:26:37.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:37 smithi016 conmon[32727]: 2022-03-21T18:26:37.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:38 smithi016 ceph-mon[22010]: pgmap v2235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:38 smithi079 ceph-mon[23158]: pgmap v2235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:38 smithi016 conmon[32727]: 2022-03-21T18:26:38.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:40 smithi016 conmon[32727]: 2022-03-21T18:26:40.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:40 smithi016 ceph-mon[22010]: pgmap v2236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:40 smithi079 ceph-mon[23158]: pgmap v2236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:41 smithi016 conmon[32727]: 2022-03-21T18:26:41.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:42 smithi016 conmon[32727]: 2022-03-21T18:26:42.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:42 smithi016 ceph-mon[22010]: pgmap v2237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:42 smithi079 ceph-mon[23158]: pgmap v2237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:43 smithi016 conmon[32727]: 2022-03-21T18:26:43.122+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:44 smithi016 conmon[32727]: 2022-03-21T18:26:44.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:44 smithi016 ceph-mon[22010]: pgmap v2238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:44 smithi079 ceph-mon[23158]: pgmap v2238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:45 smithi016 conmon[32727]: 2022-03-21T18:26:45.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:46 smithi016 ceph-mon[22010]: pgmap v2239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:46 smithi016 conmon[32727]: 2022-03-21T18:26:46.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:46 smithi079 ceph-mon[23158]: pgmap v2239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:47 smithi016 conmon[32727]: 2022-03-21T18:26:47.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:48 smithi016 ceph-mon[22010]: pgmap v2240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:48 smithi016 conmon[32727]: 2022-03-21T18:26:48.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:48 smithi079 ceph-mon[23158]: pgmap v2240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:49 smithi016 conmon[32727]: 2022-03-21T18:26:49.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:50 smithi016 ceph-mon[22010]: pgmap v2241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:50 smithi016 conmon[32727]: 2022-03-21T18:26:50.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:50 smithi079 ceph-mon[23158]: pgmap v2241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:51 smithi016 conmon[32727]: 2022-03-21T18:26:51.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:52 smithi016 ceph-mon[22010]: pgmap v2242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:52 smithi016 conmon[32727]: 2022-03-21T18:26:52.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:52 smithi079 ceph-mon[23158]: pgmap v2242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:53 smithi016 conmon[32727]: 2022-03-21T18:26:53.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:54 smithi016 ceph-mon[22010]: pgmap v2243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:54 smithi016 conmon[32727]: 2022-03-21T18:26:54.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:54 smithi079 ceph-mon[23158]: pgmap v2243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:55 smithi016 conmon[32727]: 2022-03-21T18:26:55.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:56 smithi016 ceph-mon[22010]: pgmap v2244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:56 smithi016 conmon[32727]: 2022-03-21T18:26:56.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:56 smithi079 ceph-mon[23158]: pgmap v2244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:26:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:57 smithi016 conmon[32727]: 2022-03-21T18:26:57.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:58 smithi079 ceph-mon[23158]: pgmap v2245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:58 smithi016 ceph-mon[22010]: pgmap v2245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:26:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:58 smithi016 conmon[32727]: 2022-03-21T18:26:58.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:26:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:26:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:26:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:26:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:26:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:26:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:26:59 smithi016 conmon[32727]: 2022-03-21T18:26:59.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:26:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:26:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:26:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:26:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:26:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:26:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:26:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:00 smithi079 ceph-mon[23158]: pgmap v2246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:00 smithi016 ceph-mon[22010]: pgmap v2246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:00 smithi016 conmon[32727]: 2022-03-21T18:27:00.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:01 smithi016 conmon[32727]: 2022-03-21T18:27:01.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:02 smithi016 ceph-mon[22010]: pgmap v2247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:02 smithi016 conmon[32727]: 2022-03-21T18:27:02.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:02 smithi079 ceph-mon[23158]: pgmap v2247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:03 smithi016 conmon[32727]: 2022-03-21T18:27:03.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:04 smithi016 ceph-mon[22010]: pgmap v2248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:04 smithi016 conmon[32727]: 2022-03-21T18:27:04.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:04 smithi079 ceph-mon[23158]: pgmap v2248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:05 smithi016 conmon[32727]: 2022-03-21T18:27:05.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:06 smithi079 ceph-mon[23158]: pgmap v2249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:06 smithi016 ceph-mon[22010]: pgmap v2249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:06 smithi016 conmon[32727]: 2022-03-21T18:27:06.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:07 smithi016 conmon[32727]: 2022-03-21T18:27:07.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:08 smithi016 ceph-mon[22010]: pgmap v2250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:08 smithi016 conmon[32727]: 2022-03-21T18:27:08.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:08 smithi079 ceph-mon[23158]: pgmap v2250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:09 smithi016 conmon[32727]: 2022-03-21T18:27:09.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:10 smithi016 conmon[32727]: 2022-03-21T18:27:10.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:10 smithi016 ceph-mon[22010]: pgmap v2251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:10 smithi079 ceph-mon[23158]: pgmap v2251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:11.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:11 smithi016 conmon[32727]: 2022-03-21T18:27:11.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:12 smithi016 ceph-mon[22010]: pgmap v2252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:12 smithi016 conmon[32727]: 2022-03-21T18:27:12.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:12 smithi079 ceph-mon[23158]: pgmap v2252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:13 smithi016 conmon[32727]: 2022-03-21T18:27:13.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:14 smithi016 ceph-mon[22010]: pgmap v2253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:14 smithi016 conmon[32727]: 2022-03-21T18:27:14.065+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:14 smithi079 ceph-mon[23158]: pgmap v2253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:15 smithi016 conmon[32727]: 2022-03-21T18:27:15.053+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:16 smithi016 ceph-mon[22010]: pgmap v2254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:16 smithi016 conmon[32727]: 2022-03-21T18:27:16.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:16 smithi079 ceph-mon[23158]: pgmap v2254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:17 smithi016 conmon[32727]: 2022-03-21T18:27:17.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:18 smithi016 ceph-mon[22010]: pgmap v2255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:18 smithi016 conmon[32727]: 2022-03-21T18:27:18.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:18 smithi079 ceph-mon[23158]: pgmap v2255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:19 smithi016 conmon[32727]: 2022-03-21T18:27:19.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:20 smithi016 ceph-mon[22010]: pgmap v2256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:20 smithi016 conmon[32727]: 2022-03-21T18:27:20.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:20 smithi079 ceph-mon[23158]: pgmap v2256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:21 smithi016 conmon[32727]: 2022-03-21T18:27:21.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:22 smithi016 ceph-mon[22010]: pgmap v2257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:22 smithi016 conmon[32727]: 2022-03-21T18:27:22.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:22 smithi079 ceph-mon[23158]: pgmap v2257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:23 smithi016 conmon[32727]: 2022-03-21T18:27:23.051+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:24 smithi016 ceph-mon[22010]: pgmap v2258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:24 smithi016 conmon[32727]: 2022-03-21T18:27:24.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:24 smithi079 ceph-mon[23158]: pgmap v2258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:25 smithi016 conmon[32727]: 2022-03-21T18:27:25.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:26 smithi016 ceph-mon[22010]: pgmap v2259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:26 smithi016 conmon[32727]: 2022-03-21T18:27:26.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:26 smithi079 ceph-mon[23158]: pgmap v2259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:27 smithi016 conmon[32727]: 2022-03-21T18:27:27.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:28 smithi016 ceph-mon[22010]: pgmap v2260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:28 smithi016 conmon[32727]: 2022-03-21T18:27:28.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:28 smithi079 ceph-mon[23158]: pgmap v2260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:29 smithi016 conmon[32727]: 2022-03-21T18:27:29.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:30 smithi016 conmon[32727]: 2022-03-21T18:27:30.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:30 smithi016 ceph-mon[22010]: pgmap v2261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:27:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:27:30.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:27:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:30 smithi079 ceph-mon[23158]: pgmap v2261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:27:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:27:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:27:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:31 smithi016 conmon[32727]: 2022-03-21T18:27:31.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:32 smithi016 ceph-mon[22010]: pgmap v2262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:32 smithi016 conmon[32727]: 2022-03-21T18:27:32.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:32 smithi079 ceph-mon[23158]: pgmap v2262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[32727]: 2022-03-21T18:27:33.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:33.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: 2022-03-21T18:27:33.260+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:27:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: yield 2022-03-21T18:27:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:27:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:27:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:27:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:27:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:27:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:34 smithi016 conmon[32727]: 2022-03-21T18:27:34.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:34 smithi016 ceph-mon[22010]: pgmap v2263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:34 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:27:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:27:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:34.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:27:34.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:34.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:27:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:27:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:27:34.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:34.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:34 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:27:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:34 smithi079 ceph-mon[23158]: pgmap v2263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:34 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:27:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:27:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:27:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:27:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:27:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:27:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:34.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:34 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:27:34.734 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:35 smithi016 conmon[32727]: 2022-03-21T18:27:35.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[32727]: 2022-03-21T18:27:36.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:36 smithi016 ceph-mon[22010]: pgmap v2264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 2022-03-21T18:27:36.332+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:27:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: yield 2022-03-21T18:27:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:27:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:27:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:27:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:27:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:27:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 2022-03-21T18:27:36.333+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:27:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:27:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: yield 2022-03-21T18:27:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:27:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:27:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:27:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:27:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:27:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 2022-03-21T18:27:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:27:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: 2022-03-21T18:27:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:27:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:27:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:27:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:27:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:27:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:27:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:27:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:27:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:27:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:27:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:27:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:27:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:27:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:27:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:27:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:27:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:27:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:27:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:27:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:27:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:27:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:27:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise 2022-03-21T18:27:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:27:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:27:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:27:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:27:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:27:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:36 smithi079 ceph-mon[23158]: pgmap v2264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:37 smithi016 conmon[32727]: 2022-03-21T18:27:37.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:37.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:27:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:37.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:27:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:27:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:27:37.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:37.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:37.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:37.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:27:37.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:37.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:37.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:37 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:27:37.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:37.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:27:37.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:37.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:37.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:37.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:27:37.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:37.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:27:37.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:37.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:27:37.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:37.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:37.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:37.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:27:37.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:37.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:37.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:27:37.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:27:37.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:27:37.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:27:37.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:27:37.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:27:37.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:27:37.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:27:37.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:27:37.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:27:37.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:27:37.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:27:37.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:27:37.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:27:37.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:27:37.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:27:37.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:27:37.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:27:37.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:27:37.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:27:37.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:27:37.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:27:37.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:27:37.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:27:37.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:27:37.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:27:37.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:27:37.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:27:37.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:27:37.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:27:37.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:37.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:37.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:37.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:27:37.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:37.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:27:37.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:37.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:27:37.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:37.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:37.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:37.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:27:37.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:37.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:37.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:37 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:27:37.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:27:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:27:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:27:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:27:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:27:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:27:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:27:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:27:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:27:37.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:27:37.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:27:37.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:27:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:27:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:27:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:27:37.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:27:37.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:27:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:27:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:27:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:27:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:27:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:27:37.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:27:37.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:27:37.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:27:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:27:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:27:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:27:37.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:27:37.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:27:37.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:27:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:27:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:27:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:27:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:27:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:27:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:27:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:27:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:27:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:27:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:27:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:27:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:27:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:27:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:27:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:27:37.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:27:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:38 smithi016 conmon[32727]: 2022-03-21T18:27:38.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:38 smithi016 ceph-mon[22010]: pgmap v2265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:38 smithi079 ceph-mon[23158]: pgmap v2265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:39 smithi016 conmon[32727]: 2022-03-21T18:27:39.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:40 smithi016 conmon[32727]: 2022-03-21T18:27:40.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:40 smithi016 ceph-mon[22010]: pgmap v2266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:40 smithi079 ceph-mon[23158]: pgmap v2266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:41 smithi016 conmon[32727]: 2022-03-21T18:27:41.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:42 smithi016 ceph-mon[22010]: pgmap v2267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:42.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:42 smithi016 conmon[32727]: 2022-03-21T18:27:42.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:42 smithi079 ceph-mon[23158]: pgmap v2267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:43.296 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:43 smithi016 conmon[32727]: 2022-03-21T18:27:43.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:44.298 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:44 smithi016 conmon[32727]: 2022-03-21T18:27:44.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:44 smithi016 ceph-mon[22010]: pgmap v2268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:44 smithi079 ceph-mon[23158]: pgmap v2268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:45.303 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:45 smithi016 conmon[32727]: 2022-03-21T18:27:45.048+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:46.304 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:46 smithi016 conmon[32727]: 2022-03-21T18:27:46.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:46 smithi016 ceph-mon[22010]: pgmap v2269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:46 smithi079 ceph-mon[23158]: pgmap v2269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:47.309 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:46 smithi016 conmon[32727]: 2022-03-21T18:27:46.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:48.311 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:47 smithi016 conmon[32727]: 2022-03-21T18:27:47.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:48 smithi016 ceph-mon[22010]: pgmap v2270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:48 smithi079 ceph-mon[23158]: pgmap v2270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:48 smithi016 conmon[32727]: 2022-03-21T18:27:48.905+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:49 smithi016 conmon[32727]: 2022-03-21T18:27:49.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:50 smithi016 ceph-mon[22010]: pgmap v2271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:50 smithi079 ceph-mon[23158]: pgmap v2271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:50 smithi016 conmon[32727]: 2022-03-21T18:27:50.834+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:51 smithi016 conmon[32727]: 2022-03-21T18:27:51.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:52 smithi016 ceph-mon[22010]: pgmap v2272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:52 smithi079 ceph-mon[23158]: pgmap v2272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:52 smithi016 conmon[32727]: 2022-03-21T18:27:52.821+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:53 smithi016 conmon[32727]: 2022-03-21T18:27:53.836+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:54 smithi016 ceph-mon[22010]: pgmap v2273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:54 smithi079 ceph-mon[23158]: pgmap v2273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:54 smithi016 conmon[32727]: 2022-03-21T18:27:54.834+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:55 smithi016 conmon[32727]: 2022-03-21T18:27:55.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:56 smithi016 ceph-mon[22010]: pgmap v2274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:56 smithi079 ceph-mon[23158]: pgmap v2274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:27:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:56 smithi016 conmon[32727]: 2022-03-21T18:27:56.793+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:57 smithi016 conmon[32727]: 2022-03-21T18:27:57.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: pgmap v2275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:27:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:27:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:27:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:27:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: pgmap v2275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:27:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:27:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:27:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:27:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:27:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:58 smithi016 conmon[32727]: 2022-03-21T18:27:58.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:27:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:27:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:27:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:27:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:27:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:27:59 smithi016 conmon[32727]: 2022-03-21T18:27:59.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:00 smithi016 ceph-mon[22010]: pgmap v2276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:00 smithi079 ceph-mon[23158]: pgmap v2276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:00 smithi016 conmon[32727]: 2022-03-21T18:28:00.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:01 smithi016 conmon[32727]: 2022-03-21T18:28:01.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:02 smithi016 ceph-mon[22010]: pgmap v2277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:02 smithi079 ceph-mon[23158]: pgmap v2277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:02 smithi016 conmon[32727]: 2022-03-21T18:28:02.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:03 smithi016 conmon[32727]: 2022-03-21T18:28:03.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:04 smithi016 ceph-mon[22010]: pgmap v2278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:04.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:04.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:04 smithi079 ceph-mon[23158]: pgmap v2278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:04.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:04 smithi016 conmon[32727]: 2022-03-21T18:28:04.764+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:05 smithi016 conmon[32727]: 2022-03-21T18:28:05.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:06 smithi016 ceph-mon[22010]: pgmap v2279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:06 smithi079 ceph-mon[23158]: pgmap v2279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:06 smithi016 conmon[32727]: 2022-03-21T18:28:06.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:07 smithi016 conmon[32727]: 2022-03-21T18:28:07.774+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:08 smithi016 ceph-mon[22010]: pgmap v2280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:08 smithi079 ceph-mon[23158]: pgmap v2280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:08 smithi016 conmon[32727]: 2022-03-21T18:28:08.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:09 smithi016 conmon[32727]: 2022-03-21T18:28:09.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:10 smithi016 ceph-mon[22010]: pgmap v2281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:10 smithi079 ceph-mon[23158]: pgmap v2281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:10 smithi016 conmon[32727]: 2022-03-21T18:28:10.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:11 smithi016 conmon[32727]: 2022-03-21T18:28:11.803+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:12 smithi016 ceph-mon[22010]: pgmap v2282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:12 smithi079 ceph-mon[23158]: pgmap v2282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:12 smithi016 conmon[32727]: 2022-03-21T18:28:12.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:13 smithi016 conmon[32727]: 2022-03-21T18:28:13.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:14 smithi016 ceph-mon[22010]: pgmap v2283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:14 smithi079 ceph-mon[23158]: pgmap v2283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:14 smithi016 conmon[32727]: 2022-03-21T18:28:14.739+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:15 smithi016 conmon[32727]: 2022-03-21T18:28:15.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:16 smithi016 ceph-mon[22010]: pgmap v2284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:16 smithi079 ceph-mon[23158]: pgmap v2284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:16 smithi016 conmon[32727]: 2022-03-21T18:28:16.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:17 smithi016 conmon[32727]: 2022-03-21T18:28:17.766+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:18 smithi016 ceph-mon[22010]: pgmap v2285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:18 smithi079 ceph-mon[23158]: pgmap v2285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:18 smithi016 conmon[32727]: 2022-03-21T18:28:18.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:19 smithi016 conmon[32727]: 2022-03-21T18:28:19.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:20 smithi016 ceph-mon[22010]: pgmap v2286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:20 smithi079 ceph-mon[23158]: pgmap v2286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:20 smithi016 conmon[32727]: 2022-03-21T18:28:20.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:21 smithi016 conmon[32727]: 2022-03-21T18:28:21.882+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:22 smithi016 ceph-mon[22010]: pgmap v2287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:22 smithi079 ceph-mon[23158]: pgmap v2287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:22 smithi016 conmon[32727]: 2022-03-21T18:28:22.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:23 smithi016 conmon[32727]: 2022-03-21T18:28:23.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:24 smithi016 ceph-mon[22010]: pgmap v2288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:24 smithi079 ceph-mon[23158]: pgmap v2288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:24 smithi016 conmon[32727]: 2022-03-21T18:28:24.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:25 smithi016 conmon[32727]: 2022-03-21T18:28:25.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:26 smithi079 ceph-mon[23158]: pgmap v2289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:26 smithi016 ceph-mon[22010]: pgmap v2289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:26 smithi016 conmon[32727]: 2022-03-21T18:28:26.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:27 smithi016 conmon[32727]: 2022-03-21T18:28:27.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:28 smithi079 ceph-mon[23158]: pgmap v2290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:28 smithi016 ceph-mon[22010]: pgmap v2290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:28 smithi016 conmon[32727]: 2022-03-21T18:28:28.923+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:29 smithi016 conmon[32727]: 2022-03-21T18:28:29.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:30 smithi079 ceph-mon[23158]: pgmap v2291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:30 smithi016 ceph-mon[22010]: pgmap v2291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:31 smithi016 conmon[32727]: 2022-03-21T18:28:31.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:31 smithi016 conmon[32727]: 2022-03-21T18:28:31.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:32 smithi079 ceph-mon[23158]: pgmap v2292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:32 smithi016 ceph-mon[22010]: pgmap v2292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:33 smithi016 conmon[32727]: 2022-03-21T18:28:33.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:33 smithi016 conmon[32727]: 2022-03-21T18:28:33.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:34 smithi079 ceph-mon[23158]: pgmap v2293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:34 smithi016 ceph-mon[22010]: pgmap v2293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:34 smithi016 conmon[32727]: 2022-03-21T18:28:34.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:36.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[32727]: 2022-03-21T18:28:36.007+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:36 smithi016 ceph-mon[22010]: pgmap v2294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:28:36.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:28:36.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:28:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:36 smithi079 ceph-mon[23158]: pgmap v2294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:28:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:28:36.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:28:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:37 smithi016 conmon[32727]: 2022-03-21T18:28:37.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:37.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: 2022-03-21T18:28:36.940+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:37.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:28:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: yield 2022-03-21T18:28:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:28:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:28:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:37.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:28:37.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:37.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:28:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:28:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:37.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:37.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:38 smithi016 conmon[32727]: 2022-03-21T18:28:38.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:28:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:38.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:38.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:28:38.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:28:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:28:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:28:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:38.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:38.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:38 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:28:38.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:38.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:38 smithi079 ceph-mon[23158]: pgmap v2295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:38.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:38.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:28:38.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:38.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:28:38.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:38.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:38.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:38.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:28:38.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:38.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:28:38.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:38.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:28:38.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:38.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:38.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:38.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:28:38.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:38.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:38.911 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:38.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:38 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:28:38.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:38.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:38 smithi016 ceph-mon[22010]: pgmap v2295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:38 smithi016 conmon[32727]: 2022-03-21T18:28:38.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:39.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:39.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:28:39.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:28:39.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:39 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:39.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:28:39.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:28:39.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:39.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:39 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:28:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[32727]: 2022-03-21T18:28:40.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:40.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 2022-03-21T18:28:40.012+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:40.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:28:40.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:40.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: yield 2022-03-21T18:28:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:40.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:28:40.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:40.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:28:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:28:40.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:40.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:28:40.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:28:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:40.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:40.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:40.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 2022-03-21T18:28:40.013+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:28:40.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:28:40.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:40.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: yield 2022-03-21T18:28:40.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:40.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:40.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:40.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:28:40.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:40.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:28:40.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:40.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:28:40.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:40.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:28:40.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:40.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:28:40.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:40.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:40.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:40.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 2022-03-21T18:28:40.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:28:40.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: 2022-03-21T18:28:40.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:28:40.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:28:40.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:28:40.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:28:40.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:28:40.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:28:40.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:28:40.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:28:40.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:28:40.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:28:40.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:28:40.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:28:40.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:28:40.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:28:40.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:28:40.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:28:40.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:28:40.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:28:40.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:28:40.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:28:40.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:28:40.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:28:40.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise 2022-03-21T18:28:40.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:28:40.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:28:40.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:28:40.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:28:40.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: pgmap v2296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:40.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:28:40.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:28:40.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:28:40.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:40 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:28:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: pgmap v2296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:28:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:28:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:28:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:40 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:28:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:40 smithi016 conmon[32727]: 2022-03-21T18:28:40.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:41 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:28:41.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:41.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:41.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:41.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:28:41.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:41.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:28:41.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:41.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:28:41.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:41.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:41.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:41.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:28:41.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:41.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:41.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:41 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:28:41.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:41.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:28:41.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:41.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:41.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:41.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:28:41.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:41.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:28:41.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:41.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:28:41.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:41.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:41.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:41.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:28:41.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:41.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:41.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:28:41.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:28:41.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:28:41.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:28:41.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:28:41.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:28:41.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:28:41.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:28:41.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:28:41.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:28:41.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:28:41.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:28:41.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:28:41.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:28:41.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:28:41.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:28:41.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:28:41.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:28:41.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:28:41.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:28:41.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:28:41.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:28:41.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:28:41.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:28:41.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:28:41.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:28:41.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:28:41.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:28:41.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:28:41.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:41 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:28:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:28:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:28:41.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:41.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:28:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:41.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:28:41.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:41.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:41.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:41 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:28:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:28:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:28:41.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:28:41.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:28:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:28:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:28:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:28:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:28:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:28:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:28:41.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:28:41.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:28:41.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:28:41.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:28:41.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:28:41.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:28:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:28:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:28:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:28:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:28:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:28:41.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:28:41.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:28:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:28:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:28:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:28:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:28:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:28:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:28:41.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:28:41.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:28:41.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:28:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:28:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:28:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:28:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:28:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:28:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:28:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:28:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:28:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:28:41.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:28:41.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:28:41.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:28:41.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:28:41.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:28:41.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:28:41.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:42 smithi016 conmon[32727]: 2022-03-21T18:28:42.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:42 smithi079 ceph-mon[23158]: pgmap v2297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:42 smithi016 ceph-mon[22010]: pgmap v2297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:43 smithi016 conmon[32727]: 2022-03-21T18:28:43.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:44 smithi016 conmon[32727]: 2022-03-21T18:28:44.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:44 smithi079 ceph-mon[23158]: pgmap v2298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:44.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:44 smithi016 ceph-mon[22010]: pgmap v2298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:44.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:45.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:45 smithi016 conmon[32727]: 2022-03-21T18:28:45.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:46 smithi016 conmon[32727]: 2022-03-21T18:28:46.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:46 smithi016 ceph-mon[22010]: pgmap v2299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:46 smithi079 ceph-mon[23158]: pgmap v2299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:47.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:47 smithi016 conmon[32727]: 2022-03-21T18:28:47.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:48 smithi016 conmon[32727]: 2022-03-21T18:28:48.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:48 smithi016 ceph-mon[22010]: pgmap v2300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:48 smithi079 ceph-mon[23158]: pgmap v2300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:49 smithi016 conmon[32727]: 2022-03-21T18:28:49.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:50 smithi016 conmon[32727]: 2022-03-21T18:28:50.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:50 smithi016 ceph-mon[22010]: pgmap v2301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:50 smithi079 ceph-mon[23158]: pgmap v2301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:51 smithi016 conmon[32727]: 2022-03-21T18:28:51.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:52 smithi016 conmon[32727]: 2022-03-21T18:28:52.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:52 smithi016 ceph-mon[22010]: pgmap v2302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:52 smithi079 ceph-mon[23158]: pgmap v2302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:52 smithi016 conmon[32727]: 2022-03-21T18:28:52.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:53 smithi016 conmon[32727]: 2022-03-21T18:28:53.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:54 smithi016 ceph-mon[22010]: pgmap v2303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:54 smithi079 ceph-mon[23158]: pgmap v2303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:28:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:55.420 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:54 smithi016 conmon[32727]: 2022-03-21T18:28:54.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:55 smithi016 conmon[32727]: 2022-03-21T18:28:55.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:56 smithi016 ceph-mon[22010]: pgmap v2304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:56 smithi079 ceph-mon[23158]: pgmap v2304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:28:57.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:56 smithi016 conmon[32727]: 2022-03-21T18:28:56.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:57 smithi016 conmon[32727]: 2022-03-21T18:28:57.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: pgmap v2305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:28:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:28:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:28:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:28:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:28:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: pgmap v2305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:28:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:28:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:28:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:28:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:28:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:58 smithi016 conmon[32727]: 2022-03-21T18:28:58.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:28:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:28:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:28:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:28:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:28:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:28:59 smithi016 conmon[32727]: 2022-03-21T18:28:59.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:00 smithi016 ceph-mon[22010]: pgmap v2306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:00 smithi079 ceph-mon[23158]: pgmap v2306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:00 smithi016 conmon[32727]: 2022-03-21T18:29:00.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:02.424 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:01 smithi016 conmon[32727]: 2022-03-21T18:29:01.973+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:02 smithi016 ceph-mon[22010]: pgmap v2307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:29:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:02 smithi079 ceph-mon[23158]: pgmap v2307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:29:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:03 smithi016 conmon[32727]: 2022-03-21T18:29:03.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:04 smithi016 conmon[32727]: 2022-03-21T18:29:04.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:04 smithi016 ceph-mon[22010]: pgmap v2308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:29:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:04 smithi079 ceph-mon[23158]: pgmap v2308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:29:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:05 smithi016 conmon[32727]: 2022-03-21T18:29:05.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:06 smithi016 conmon[32727]: 2022-03-21T18:29:06.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:06 smithi016 ceph-mon[22010]: pgmap v2309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:06 smithi079 ceph-mon[23158]: pgmap v2309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:06 smithi016 conmon[32727]: 2022-03-21T18:29:06.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:07 smithi016 conmon[32727]: 2022-03-21T18:29:07.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:08 smithi016 ceph-mon[22010]: pgmap v2310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:08 smithi079 ceph-mon[23158]: pgmap v2310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:09 smithi016 conmon[32727]: 2022-03-21T18:29:09.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:10 smithi016 conmon[32727]: 2022-03-21T18:29:10.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:10 smithi016 ceph-mon[22010]: pgmap v2311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:10 smithi079 ceph-mon[23158]: pgmap v2311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:10 smithi016 conmon[32727]: 2022-03-21T18:29:10.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:11 smithi016 conmon[32727]: 2022-03-21T18:29:11.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:12 smithi016 ceph-mon[22010]: pgmap v2312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:12 smithi079 ceph-mon[23158]: pgmap v2312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:12 smithi016 conmon[32727]: 2022-03-21T18:29:12.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:13 smithi016 conmon[32727]: 2022-03-21T18:29:13.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:14 smithi016 ceph-mon[22010]: pgmap v2313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 3995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:14 smithi079 ceph-mon[23158]: pgmap v2313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 3995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:14 smithi016 conmon[32727]: 2022-03-21T18:29:14.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:16 smithi016 conmon[32727]: 2022-03-21T18:29:16.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:16 smithi016 ceph-mon[22010]: pgmap v2314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:16 smithi079 ceph-mon[23158]: pgmap v2314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:17 smithi016 conmon[32727]: 2022-03-21T18:29:17.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:18 smithi016 conmon[32727]: 2022-03-21T18:29:18.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:18 smithi016 ceph-mon[22010]: pgmap v2315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:18 smithi079 ceph-mon[23158]: pgmap v2315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:18 smithi016 conmon[32727]: 2022-03-21T18:29:18.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:20 smithi016 conmon[32727]: 2022-03-21T18:29:20.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:20 smithi016 ceph-mon[22010]: pgmap v2316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:20 smithi079 ceph-mon[23158]: pgmap v2316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:21 smithi016 conmon[32727]: 2022-03-21T18:29:21.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:22 smithi016 conmon[32727]: 2022-03-21T18:29:22.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:22 smithi016 ceph-mon[22010]: pgmap v2317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:22 smithi079 ceph-mon[23158]: pgmap v2317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:23 smithi016 conmon[32727]: 2022-03-21T18:29:23.068+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:24 smithi016 conmon[32727]: 2022-03-21T18:29:24.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:24 smithi016 ceph-mon[22010]: pgmap v2318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:24 smithi079 ceph-mon[23158]: pgmap v2318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:25.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:25 smithi016 conmon[32727]: 2022-03-21T18:29:25.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:26.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:26 smithi016 conmon[32727]: 2022-03-21T18:29:26.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:26 smithi016 ceph-mon[22010]: pgmap v2319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:26 smithi079 ceph-mon[23158]: pgmap v2319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:27 smithi016 conmon[32727]: 2022-03-21T18:29:27.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:28 smithi016 conmon[32727]: 2022-03-21T18:29:28.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:28 smithi016 ceph-mon[22010]: pgmap v2320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:28 smithi079 ceph-mon[23158]: pgmap v2320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:29 smithi016 conmon[32727]: 2022-03-21T18:29:29.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:30 smithi016 conmon[32727]: 2022-03-21T18:29:30.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:30 smithi016 ceph-mon[22010]: pgmap v2321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:30 smithi079 ceph-mon[23158]: pgmap v2321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:31.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:31 smithi016 conmon[32727]: 2022-03-21T18:29:31.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:32 smithi016 conmon[32727]: 2022-03-21T18:29:32.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:32 smithi016 ceph-mon[22010]: pgmap v2322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:32 smithi079 ceph-mon[23158]: pgmap v2322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:33 smithi016 conmon[32727]: 2022-03-21T18:29:33.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:34 smithi016 conmon[32727]: 2022-03-21T18:29:34.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:34 smithi016 ceph-mon[22010]: pgmap v2323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:34 smithi079 ceph-mon[23158]: pgmap v2323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:35 smithi016 conmon[32727]: 2022-03-21T18:29:35.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:36 smithi016 conmon[32727]: 2022-03-21T18:29:36.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:36 smithi016 ceph-mon[22010]: pgmap v2324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:36 smithi079 ceph-mon[23158]: pgmap v2324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:37 smithi016 conmon[32727]: 2022-03-21T18:29:37.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:38 smithi016 conmon[32727]: 2022-03-21T18:29:38.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:38 smithi016 ceph-mon[22010]: pgmap v2325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:38 smithi079 ceph-mon[23158]: pgmap v2325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:39 smithi016 conmon[32727]: 2022-03-21T18:29:39.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:40 smithi016 conmon[32727]: 2022-03-21T18:29:40.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:40 smithi079 ceph-mon[23158]: pgmap v2326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:29:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:29:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:29:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:40 smithi016 ceph-mon[22010]: pgmap v2326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:29:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:29:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:29:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:41 smithi016 conmon[32727]: 2022-03-21T18:29:41.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:42 smithi016 conmon[32727]: 2022-03-21T18:29:42.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:42 smithi016 ceph-mon[22010]: pgmap v2327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:42 smithi079 ceph-mon[23158]: pgmap v2327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[32727]: 2022-03-21T18:29:43.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:43.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: 2022-03-21T18:29:43.116+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:43.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:29:43.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:43.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: yield 2022-03-21T18:29:43.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:43.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:43.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:43.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:29:43.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:43.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:29:43.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:43.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:29:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:29:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:43.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:29:43.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:43.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:43.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:43 smithi016 conmon[32727]: 2022-03-21T18:29:43.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:44 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:29:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:29:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:29:44.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:44.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:29:44.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:44.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:29:44.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:44.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:44.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:44.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:29:44.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:44.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:44.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:44.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:44 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:29:44.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:44 smithi016 ceph-mon[22010]: pgmap v2328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:44.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:44.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:44 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:29:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:29:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:29:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:29:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:29:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:29:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:44 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:29:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:44 smithi079 ceph-mon[23158]: pgmap v2328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:44.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:44 smithi016 conmon[32727]: 2022-03-21T18:29:44.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:45 smithi016 conmon[32727]: 2022-03-21T18:29:45.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:46.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 2022-03-21T18:29:46.188+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:46.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:29:46.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: yield 2022-03-21T18:29:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:46.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:46.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:29:46.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:46.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:29:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:29:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:29:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:29:46.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:46.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:46.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 2022-03-21T18:29:46.189+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:29:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:29:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: yield 2022-03-21T18:29:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:29:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:46.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:29:46.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:46.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:29:46.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:46.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:29:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:29:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:46.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:46.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:46.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 2022-03-21T18:29:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:29:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: 2022-03-21T18:29:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:29:46.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:29:46.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:29:46.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:29:46.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:29:46.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:29:46.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:29:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:29:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:29:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:29:46.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:29:46.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:29:46.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:29:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:29:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:29:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:29:46.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:29:46.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:29:46.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:29:46.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:29:46.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:29:46.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:29:46.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise 2022-03-21T18:29:46.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:29:46.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:29:46.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:29:46.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:29:46.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:29:46 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:46 smithi016 ceph-mon[22010]: pgmap v2329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:46 smithi079 ceph-mon[23158]: pgmap v2329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:47 smithi016 conmon[32727]: 2022-03-21T18:29:47.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:29:47.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:47.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:47.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:47.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:29:47.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:47.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:29:47.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:47.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:29:47.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:47.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:47.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:47.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:29:47.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:47.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:47.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:47 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:29:47.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:47.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:29:47.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:47.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:47.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:47.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:29:47.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:47.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:29:47.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:47.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:29:47.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:47.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:47.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:47.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:29:47.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:47.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:47.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:29:47.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:29:47.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:29:47.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:29:47.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:29:47.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:29:47.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:29:47.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:29:47.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:29:47.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:29:47.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:29:47.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:29:47.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:29:47.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:29:47.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:29:47.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:29:47.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:29:47.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:29:47.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:29:47.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:29:47.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:29:47.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:29:47.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:29:47.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:29:47.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:29:47.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:29:47.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:29:47.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:29:47.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:29:47.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:29:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:29:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:29:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:29:47.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:47.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:29:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:47 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:29:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:29:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:29:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:29:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:29:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:29:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:29:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:29:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:29:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:29:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:29:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:29:47.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:29:47.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:29:47.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:29:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:29:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:29:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:29:47.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:29:47.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:29:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:29:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:29:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:29:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:29:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:29:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:29:47.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:29:47.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:29:47.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:29:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:29:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:29:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:29:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:29:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:29:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:29:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:29:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:29:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:29:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:29:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:29:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:29:47.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:29:47.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:29:47.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:29:47.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:29:47.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:29:47.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:29:47.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:29:47.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:48 smithi016 conmon[32727]: 2022-03-21T18:29:48.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:48 smithi016 ceph-mon[22010]: pgmap v2330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:48 smithi079 ceph-mon[23158]: pgmap v2330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:49 smithi016 conmon[32727]: 2022-03-21T18:29:49.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:50 smithi016 conmon[32727]: 2022-03-21T18:29:50.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:50 smithi016 ceph-mon[22010]: pgmap v2331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:50 smithi079 ceph-mon[23158]: pgmap v2331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:51 smithi016 conmon[32727]: 2022-03-21T18:29:51.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:51 smithi016 conmon[32727]: 2022-03-21T18:29:51.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:52 smithi016 ceph-mon[22010]: pgmap v2332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:52 smithi079 ceph-mon[23158]: pgmap v2332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:52 smithi016 conmon[32727]: 2022-03-21T18:29:52.983+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:53.944 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:53 smithi016 conmon[32727]: 2022-03-21T18:29:53.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:54 smithi079 ceph-mon[23158]: pgmap v2333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:54 smithi016 conmon[32727]: 2022-03-21T18:29:54.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:54 smithi016 ceph-mon[22010]: pgmap v2333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:55 smithi016 conmon[32727]: 2022-03-21T18:29:55.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:56 smithi079 ceph-mon[23158]: pgmap v2334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:56 smithi016 conmon[32727]: 2022-03-21T18:29:56.905+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:56 smithi016 ceph-mon[22010]: pgmap v2334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:29:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:57.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:57 smithi016 conmon[32727]: 2022-03-21T18:29:57.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: pgmap v2335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:29:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:29:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:29:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:29:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: pgmap v2335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:29:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:29:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:29:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:29:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:29:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:29:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:58 smithi016 conmon[32727]: 2022-03-21T18:29:58.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:29:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:29:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:29:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:29:59 smithi016 conmon[32727]: 2022-03-21T18:29:59.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:29:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: pgmap v2336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops 2022-03-21T18:30:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:30:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:30:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:30:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:30:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:30:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops 2022-03-21T18:30:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:00 smithi016 conmon[32727]: 2022-03-21T18:30:00.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: pgmap v2336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops 2022-03-21T18:30:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:30:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:30:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:30:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:30:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:30:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops 2022-03-21T18:30:01.959 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:01 smithi016 conmon[32727]: 2022-03-21T18:30:01.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:02 smithi079 ceph-mon[23158]: pgmap v2337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:02 smithi016 conmon[32727]: 2022-03-21T18:30:02.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:02 smithi016 ceph-mon[22010]: pgmap v2337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:03 smithi016 conmon[32727]: 2022-03-21T18:30:03.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:04 smithi079 ceph-mon[23158]: pgmap v2338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:04 smithi016 conmon[32727]: 2022-03-21T18:30:04.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:04 smithi016 ceph-mon[22010]: pgmap v2338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:05 smithi016 conmon[32727]: 2022-03-21T18:30:05.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:06 smithi079 ceph-mon[23158]: pgmap v2339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:06 smithi016 conmon[32727]: 2022-03-21T18:30:06.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:06 smithi016 ceph-mon[22010]: pgmap v2339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:07 smithi016 conmon[32727]: 2022-03-21T18:30:07.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:08 smithi079 ceph-mon[23158]: pgmap v2340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:08 smithi016 conmon[32727]: 2022-03-21T18:30:08.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:08 smithi016 ceph-mon[22010]: pgmap v2340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:09 smithi016 conmon[32727]: 2022-03-21T18:30:09.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:10.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:10 smithi016 ceph-mon[22010]: pgmap v2341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:10 smithi016 conmon[32727]: 2022-03-21T18:30:10.876+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:10 smithi079 ceph-mon[23158]: pgmap v2341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:11 smithi016 conmon[32727]: 2022-03-21T18:30:11.905+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:12 smithi016 ceph-mon[22010]: pgmap v2342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:12 smithi016 conmon[32727]: 2022-03-21T18:30:12.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:12 smithi079 ceph-mon[23158]: pgmap v2342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:13 smithi016 conmon[32727]: 2022-03-21T18:30:13.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:14 smithi016 ceph-mon[22010]: pgmap v2343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:14 smithi016 conmon[32727]: 2022-03-21T18:30:14.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:14 smithi079 ceph-mon[23158]: pgmap v2343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:16.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:15 smithi016 conmon[32727]: 2022-03-21T18:30:15.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:16.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:16 smithi016 ceph-mon[22010]: pgmap v2344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:16 smithi016 conmon[32727]: 2022-03-21T18:30:16.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:16 smithi079 ceph-mon[23158]: pgmap v2344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:17 smithi016 ceph-mon[22010]: pgmap v2345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:17 smithi016 conmon[32727]: 2022-03-21T18:30:17.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:17 smithi079 ceph-mon[23158]: pgmap v2345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:18 smithi016 conmon[32727]: 2022-03-21T18:30:18.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:19 smithi016 ceph-mon[22010]: pgmap v2346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:20.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:19 smithi016 conmon[32727]: 2022-03-21T18:30:19.983+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:19 smithi079 ceph-mon[23158]: pgmap v2346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:21 smithi016 conmon[32727]: 2022-03-21T18:30:21.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:22.074 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:22.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:21 smithi016 ceph-mon[22010]: pgmap v2347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:21 smithi079 ceph-mon[23158]: pgmap v2347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:22 smithi016 conmon[32727]: 2022-03-21T18:30:22.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:23.104 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:23 smithi016 conmon[32727]: 2022-03-21T18:30:23.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:24.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:24.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:23 smithi016 ceph-mon[22010]: pgmap v2348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:24.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:23 smithi079 ceph-mon[23158]: pgmap v2348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:24 smithi016 conmon[32727]: 2022-03-21T18:30:24.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:25.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:25 smithi016 conmon[32727]: 2022-03-21T18:30:25.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:26.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:26.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:25 smithi016 ceph-mon[22010]: pgmap v2349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:25 smithi079 ceph-mon[23158]: pgmap v2349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:26 smithi016 conmon[32727]: 2022-03-21T18:30:26.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:27 smithi016 conmon[32727]: 2022-03-21T18:30:27.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:28.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:28.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:27 smithi016 ceph-mon[22010]: pgmap v2350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:27 smithi079 ceph-mon[23158]: pgmap v2350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:28 smithi016 conmon[32727]: 2022-03-21T18:30:28.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:29.104 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:29.105 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:28 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:28 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:29 smithi016 conmon[32727]: 2022-03-21T18:30:29.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:30.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:30.102 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:29 smithi016 ceph-mon[22010]: pgmap v2351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:29 smithi079 ceph-mon[23158]: pgmap v2351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:30 smithi016 conmon[32727]: 2022-03-21T18:30:30.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:31.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:31.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:31 smithi016 conmon[32727]: 2022-03-21T18:30:31.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:32.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:32.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:31 smithi016 ceph-mon[22010]: pgmap v2352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:31 smithi079 ceph-mon[23158]: pgmap v2352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:32 smithi016 conmon[32727]: 2022-03-21T18:30:32.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:33.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:33 smithi016 conmon[32727]: 2022-03-21T18:30:33.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:33 smithi016 ceph-mon[22010]: pgmap v2353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:34 smithi016 conmon[32727]: 2022-03-21T18:30:34.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:34.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:33 smithi079 ceph-mon[23158]: pgmap v2353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:34.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:35 smithi016 conmon[32727]: 2022-03-21T18:30:35.015+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:35 smithi016 ceph-mon[22010]: pgmap v2354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:36 smithi016 conmon[32727]: 2022-03-21T18:30:36.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:35 smithi079 ceph-mon[23158]: pgmap v2354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:37 smithi016 conmon[32727]: 2022-03-21T18:30:37.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:37 smithi016 ceph-mon[22010]: pgmap v2355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:38 smithi016 conmon[32727]: 2022-03-21T18:30:38.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:37 smithi079 ceph-mon[23158]: pgmap v2355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:38 smithi016 conmon[32727]: 2022-03-21T18:30:38.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:39 smithi016 ceph-mon[22010]: pgmap v2356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:39 smithi016 conmon[32727]: 2022-03-21T18:30:39.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:39 smithi079 ceph-mon[23158]: pgmap v2356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:41 smithi016 conmon[32727]: 2022-03-21T18:30:41.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:41 smithi016 ceph-mon[22010]: pgmap v2357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:42 smithi016 conmon[32727]: 2022-03-21T18:30:42.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:41 smithi079 ceph-mon[23158]: pgmap v2357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:43 smithi016 conmon[32727]: 2022-03-21T18:30:43.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:43 smithi016 ceph-mon[22010]: pgmap v2358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:44.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:44 smithi016 conmon[32727]: 2022-03-21T18:30:44.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:43 smithi079 ceph-mon[23158]: pgmap v2358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:45 smithi016 conmon[32727]: 2022-03-21T18:30:45.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[32727]: 2022-03-21T18:30:46.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:45 smithi016 ceph-mon[22010]: pgmap v2359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:45 smithi079 ceph-mon[23158]: pgmap v2359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:30:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:30:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:30:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:30:46.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:46.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:30:46.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:46.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:46.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:46.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:30:46.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:46.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:30:46.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:46.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:30:46.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:46.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:46.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:46.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:30:46.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:46.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:46.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:46.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:30:47.027 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:47.027 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:30:47.028 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:30:47.028 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:30:47.028 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:47.029 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:30:47.029 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:47.029 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:30:47.030 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:47.030 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:47.031 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:47.031 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:30:47.031 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:47.032 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:30:47.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:47.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:30:47.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:47.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:47.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:47.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:30:47.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:47.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:47.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:47.040 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:30:47.041 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: 2022-03-21T18:30:46.669+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:47.042 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:30:47.042 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:47.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: yield 2022-03-21T18:30:47.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:47.044 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:47.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:47.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:30:47.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:47.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:30:47.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:47.049 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:30:47.050 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:47.050 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:30:47.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:47.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:30:47.052 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:47.053 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:47.053 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:47.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:47 smithi016 conmon[32727]: 2022-03-21T18:30:47.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:47 smithi016 ceph-mon[22010]: pgmap v2360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:47 smithi016 conmon[32727]: 2022-03-21T18:30:47.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:47 smithi079 ceph-mon[23158]: pgmap v2360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[32727]: 2022-03-21T18:30:49.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:50.012 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:50.013 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:49 smithi016 ceph-mon[22010]: pgmap v2361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:50.013 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.013 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:30:50.014 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:50.014 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:30:50.014 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:50.015 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:50.015 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:50.015 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:30:50.016 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:50.016 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:30:50.016 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:50.017 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:30:50.017 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:50.018 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:50.018 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:50.018 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:30:50.019 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:50.019 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:50.019 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.020 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 2022-03-21T18:30:49.740+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:30:50.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:50.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: yield 2022-03-21T18:30:50.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:50.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:50.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:50.023 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:30:50.023 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:50.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:30:50.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:50.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:30:50.025 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:50.025 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:30:50.025 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:50.026 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:30:50.026 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:50.026 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:50.027 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.027 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 2022-03-21T18:30:49.741+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:30:50.027 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:30:50.028 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:50.028 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: yield 2022-03-21T18:30:50.028 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:50.029 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:50.029 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:50.030 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:30:50.030 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:50.030 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:30:50.031 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:50.031 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:30:50.031 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:50.032 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:30:50.032 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:50.033 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:30:50.033 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:50.033 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:50.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 2022-03-21T18:30:50.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:30:50.035 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: 2022-03-21T18:30:50.035 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:30:50.035 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:30:50.036 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:30:50.036 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:30:50.036 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:30:50.037 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:30:50.037 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:30:50.038 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:30:50.038 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:30:50.038 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:30:50.039 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:30:50.039 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:30:50.039 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:30:50.040 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:30:50.040 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:30:50.040 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:30:50.041 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:30:50.041 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:30:50.042 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:30:50.042 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:30:50.042 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:30:50.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:30:50.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise 2022-03-21T18:30:50.044 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:30:50.044 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:30:50.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:30:50.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:30:50.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:30:49 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:49 smithi079 ceph-mon[23158]: pgmap v2361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:30:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:30:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:30:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:30:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:30:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:30:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:50 smithi016 conmon[32727]: 2022-03-21T18:30:50.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:50 smithi016 conmon[32727]: 2022-03-21T18:30:50.974+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:50 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:30:51.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:30:51.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:51.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:30:51.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:51.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:51.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:51.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:30:51.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:51.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:30:51.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:51.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:30:51.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:51.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:51.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:30:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:51.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:51.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:30:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:30:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:30:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:30:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:30:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:30:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:30:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:30:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:30:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:30:51.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:30:51.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:30:51.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:30:51.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:30:51.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:30:51.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:30:51.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:30:51.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:30:51.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:30:51.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:30:51.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:30:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:30:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:30:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:30:51.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:30:51.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:30:51.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:30:51.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:30:51.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:30:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:30:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:50 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:30:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:30:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:30:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:30:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:30:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:30:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:30:51.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:30:51.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:30:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:30:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:30:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:30:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:30:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:30:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:30:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:30:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:30:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:30:51.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:51.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:30:51.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:30:51.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:30:51.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:30:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:30:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:30:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:30:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:30:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:30:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:30:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:30:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:30:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:30:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:30:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:30:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:30:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:30:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:30:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:30:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:30:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:30:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:30:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:30:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:30:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:30:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:30:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:30:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:30:51.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:30:51.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:30:51.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:30:51.241 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:51 smithi016 conmon[32727]: 2022-03-21T18:30:51.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:51 smithi016 ceph-mon[22010]: pgmap v2362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:51 smithi079 ceph-mon[23158]: pgmap v2362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:52 smithi016 conmon[32727]: 2022-03-21T18:30:52.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:53 smithi016 ceph-mon[22010]: pgmap v2363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:53 smithi016 conmon[32727]: 2022-03-21T18:30:53.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:53 smithi079 ceph-mon[23158]: pgmap v2363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:54 smithi016 conmon[32727]: 2022-03-21T18:30:54.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:55 smithi016 ceph-mon[22010]: pgmap v2364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:55 smithi016 conmon[32727]: 2022-03-21T18:30:55.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:55 smithi079 ceph-mon[23158]: pgmap v2364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:30:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:56 smithi016 conmon[32727]: 2022-03-21T18:30:56.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:57 smithi016 ceph-mon[22010]: pgmap v2365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:57 smithi016 conmon[32727]: 2022-03-21T18:30:57.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:57 smithi079 ceph-mon[23158]: pgmap v2365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:30:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:30:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:30:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:30:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:30:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:30:59.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:30:59 smithi016 conmon[32727]: 2022-03-21T18:30:59.028+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:30:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:30:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:30:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:30:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:30:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:30:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:30:59 smithi079 ceph-mon[23158]: pgmap v2366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:30:59 smithi016 ceph-mon[22010]: pgmap v2366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:00 smithi016 conmon[32727]: 2022-03-21T18:31:00.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:01 smithi016 conmon[32727]: 2022-03-21T18:31:01.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:01 smithi079 ceph-mon[23158]: pgmap v2367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:01 smithi016 ceph-mon[22010]: pgmap v2367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:02 smithi016 conmon[32727]: 2022-03-21T18:31:02.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:03 smithi016 conmon[32727]: 2022-03-21T18:31:03.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:03 smithi079 ceph-mon[23158]: pgmap v2368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:03 smithi016 ceph-mon[22010]: pgmap v2368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:04 smithi016 conmon[32727]: 2022-03-21T18:31:04.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:05 smithi016 conmon[32727]: 2022-03-21T18:31:05.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:05 smithi079 ceph-mon[23158]: pgmap v2369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:05 smithi016 ceph-mon[22010]: pgmap v2369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:06 smithi016 conmon[32727]: 2022-03-21T18:31:06.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:07 smithi016 conmon[32727]: 2022-03-21T18:31:07.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:07 smithi079 ceph-mon[23158]: pgmap v2370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:07 smithi016 ceph-mon[22010]: pgmap v2370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:08 smithi016 conmon[32727]: 2022-03-21T18:31:08.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:09 smithi016 conmon[32727]: 2022-03-21T18:31:09.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:09 smithi079 ceph-mon[23158]: pgmap v2371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:09 smithi016 ceph-mon[22010]: pgmap v2371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:10 smithi016 conmon[32727]: 2022-03-21T18:31:10.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:11 smithi016 conmon[32727]: 2022-03-21T18:31:11.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:11 smithi079 ceph-mon[23158]: pgmap v2372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:11 smithi016 ceph-mon[22010]: pgmap v2372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:12 smithi016 conmon[32727]: 2022-03-21T18:31:12.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:13 smithi016 conmon[32727]: 2022-03-21T18:31:13.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:13 smithi079 ceph-mon[23158]: pgmap v2373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:14 smithi016 conmon[32727]: 2022-03-21T18:31:14.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:13 smithi016 ceph-mon[22010]: pgmap v2373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:15 smithi016 conmon[32727]: 2022-03-21T18:31:15.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:15 smithi079 ceph-mon[23158]: pgmap v2374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:15 smithi016 ceph-mon[22010]: pgmap v2374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:16 smithi016 conmon[32727]: 2022-03-21T18:31:16.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:17.233 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:17.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:17 smithi016 conmon[32727]: 2022-03-21T18:31:17.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:17 smithi016 ceph-mon[22010]: pgmap v2375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:18 smithi016 conmon[32727]: 2022-03-21T18:31:18.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:17 smithi079 ceph-mon[23158]: pgmap v2375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:18 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:19 smithi016 conmon[32727]: 2022-03-21T18:31:19.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:18 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:20 smithi016 conmon[32727]: 2022-03-21T18:31:20.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:19 smithi016 ceph-mon[22010]: pgmap v2376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:19 smithi079 ceph-mon[23158]: pgmap v2376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:21 smithi016 conmon[32727]: 2022-03-21T18:31:21.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:21 smithi016 ceph-mon[22010]: pgmap v2377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:22.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:22 smithi016 conmon[32727]: 2022-03-21T18:31:22.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:21 smithi079 ceph-mon[23158]: pgmap v2377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:23 smithi016 conmon[32727]: 2022-03-21T18:31:23.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:23 smithi016 ceph-mon[22010]: pgmap v2378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:23 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:24 smithi016 conmon[32727]: 2022-03-21T18:31:24.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:23 smithi079 ceph-mon[23158]: pgmap v2378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:23 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:25 smithi016 conmon[32727]: 2022-03-21T18:31:25.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:26 smithi016 ceph-mon[22010]: pgmap v2379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:26 smithi016 conmon[32727]: 2022-03-21T18:31:26.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:26 smithi079 ceph-mon[23158]: pgmap v2379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:27 smithi016 conmon[32727]: 2022-03-21T18:31:27.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:28 smithi016 ceph-mon[22010]: pgmap v2380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:28 smithi016 conmon[32727]: 2022-03-21T18:31:28.230+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:28.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:28 smithi079 ceph-mon[23158]: pgmap v2380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:29.273 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:29.274 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:29 smithi016 conmon[32727]: 2022-03-21T18:31:29.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:30.266 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:30 smithi016 ceph-mon[22010]: pgmap v2381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:30.267 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:30 smithi079 ceph-mon[23158]: pgmap v2381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:30 smithi016 conmon[32727]: 2022-03-21T18:31:30.264+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:31 smithi016 conmon[32727]: 2022-03-21T18:31:31.229+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:32 smithi016 ceph-mon[22010]: pgmap v2382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:32 smithi016 conmon[32727]: 2022-03-21T18:31:32.240+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:32 smithi079 ceph-mon[23158]: pgmap v2382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:33 smithi016 conmon[32727]: 2022-03-21T18:31:33.233+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:34 smithi016 ceph-mon[22010]: pgmap v2383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:34.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:34 smithi016 conmon[32727]: 2022-03-21T18:31:34.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:34 smithi079 ceph-mon[23158]: pgmap v2383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:35 smithi016 conmon[32727]: 2022-03-21T18:31:35.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:36 smithi016 ceph-mon[22010]: pgmap v2384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:36 smithi016 conmon[32727]: 2022-03-21T18:31:36.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:36 smithi079 ceph-mon[23158]: pgmap v2384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:37 smithi016 conmon[32727]: 2022-03-21T18:31:37.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:38 smithi016 ceph-mon[22010]: pgmap v2385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:38 smithi016 conmon[32727]: 2022-03-21T18:31:38.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:38 smithi079 ceph-mon[23158]: pgmap v2385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:39 smithi016 conmon[32727]: 2022-03-21T18:31:39.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:40 smithi016 ceph-mon[22010]: pgmap v2386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:40 smithi016 conmon[32727]: 2022-03-21T18:31:40.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:40 smithi079 ceph-mon[23158]: pgmap v2386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:41 smithi016 conmon[32727]: 2022-03-21T18:31:41.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:42 smithi016 ceph-mon[22010]: pgmap v2387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:42 smithi016 conmon[32727]: 2022-03-21T18:31:42.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:42 smithi079 ceph-mon[23158]: pgmap v2387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:43 smithi016 conmon[32727]: 2022-03-21T18:31:43.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:44 smithi016 conmon[32727]: 2022-03-21T18:31:44.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:44 smithi016 ceph-mon[22010]: pgmap v2388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:44 smithi079 ceph-mon[23158]: pgmap v2388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:45 smithi016 conmon[32727]: 2022-03-21T18:31:45.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:46 smithi016 ceph-mon[22010]: pgmap v2389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:46 smithi016 conmon[32727]: 2022-03-21T18:31:46.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:46 smithi079 ceph-mon[23158]: pgmap v2389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:47 smithi016 conmon[32727]: 2022-03-21T18:31:47.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:48 smithi016 ceph-mon[22010]: pgmap v2390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:48 smithi016 conmon[32727]: 2022-03-21T18:31:48.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:48 smithi079 ceph-mon[23158]: pgmap v2390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:49 smithi016 conmon[32727]: 2022-03-21T18:31:49.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:50 smithi016 conmon[32727]: 2022-03-21T18:31:50.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:50 smithi016 ceph-mon[22010]: pgmap v2391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:31:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:31:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:31:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:50 smithi079 ceph-mon[23158]: pgmap v2391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:31:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:31:50.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:31:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:51 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:31:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:51 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:31:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:51 smithi016 conmon[32727]: 2022-03-21T18:31:51.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:51 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:31:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:51 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:31:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:52 smithi016 ceph-mon[22010]: pgmap v2392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[32727]: 2022-03-21T18:31:52.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:52 smithi079 ceph-mon[23158]: pgmap v2392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:53.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: 2022-03-21T18:31:52.844+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:53.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:31:53.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:53.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: yield 2022-03-21T18:31:53.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:53.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:53.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:53.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:31:53.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:31:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:31:53.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:53.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:31:53.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:53.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:31:53.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:53.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:53.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:52 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:53 smithi016 conmon[32727]: 2022-03-21T18:31:53.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:54.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:54.171 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:31:54.171 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:54.172 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:31:54.172 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:54.172 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:54.173 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:54.173 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:31:54.173 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:54.174 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:31:54.174 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:54.174 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:31:54.175 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:54.175 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:54.175 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:54.176 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:31:54.176 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:54.177 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:54.177 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:54.177 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:31:54.178 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:54.178 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:54 smithi079 ceph-mon[23158]: pgmap v2393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:54.178 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:54 smithi016 conmon[32727]: 2022-03-21T18:31:54.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:31:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:31:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:31:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:31:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:31:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:31:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:31:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:54 smithi016 ceph-mon[22010]: pgmap v2393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:54.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[32727]: 2022-03-21T18:31:55.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:56.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 2022-03-21T18:31:55.916+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:31:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: yield 2022-03-21T18:31:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:31:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:31:56.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:56.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:31:56.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:56.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:31:56.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:56.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:31:56.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:56.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:56.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:56.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 2022-03-21T18:31:55.918+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:31:56.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:31:56.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:56.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: yield 2022-03-21T18:31:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:56.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:31:56.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:56.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:31:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:31:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:31:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:31:56.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:56.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:56.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:56.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 2022-03-21T18:31:56.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:31:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: 2022-03-21T18:31:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:31:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:31:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:31:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:31:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:31:56.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:31:56.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:31:56.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:31:56.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:31:56.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:31:56.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:31:56.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:31:56.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:31:56.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:31:56.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:31:56.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:31:56.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:31:56.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:31:56.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:31:56.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:31:56.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:31:56.197 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:31:56.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise 2022-03-21T18:31:56.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:31:56.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:31:56.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:31:56.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:31:56.199 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:31:55 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:56 smithi079 ceph-mon[23158]: pgmap v2394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:56 smithi016 conmon[32727]: 2022-03-21T18:31:56.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:56 smithi016 ceph-mon[22010]: pgmap v2394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:31:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:31:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:57.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:57.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:57.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:31:57.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:57.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:31:57.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:57.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:31:57.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:57.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:57.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:57.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:31:57.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:57.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:57.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:31:57.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:57.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:31:57.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:57.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:57.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:57.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:31:57.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:57.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:31:57.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:57.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:31:57.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:57.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:57.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:57.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:31:57.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:57.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:57.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:31:57.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:31:57.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:31:57.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:31:57.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:31:57.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:31:57.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:31:57.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:31:57.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:31:57.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:31:57.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:31:57.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:31:57.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:31:57.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:31:57.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:31:57.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:31:57.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:31:57.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:31:57.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:31:57.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:31:57.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:31:57.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:31:57.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:31:57.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:31:57.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:31:57.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:31:57.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:31:57.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:31:57.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:31:57.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:57 smithi016 conmon[32727]: 2022-03-21T18:31:57.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:31:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:31:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:31:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:31:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:31:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:31:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:31:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:31:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:31:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:31:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:31:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:31:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:31:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:31:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:31:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:31:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:31:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:31:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:31:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:31:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:31:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:31:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:31:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:31:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:31:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:31:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:31:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:31:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:31:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:31:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:31:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:31:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:31:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:31:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:31:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:31:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:31:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:31:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:31:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:31:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:31:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:31:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:31:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:31:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:31:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:31:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:31:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:31:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:31:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:31:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:31:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:31:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:31:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:58 smithi079 ceph-mon[23158]: pgmap v2395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:58 smithi016 ceph-mon[22010]: pgmap v2395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:31:58.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:58 smithi016 conmon[32727]: 2022-03-21T18:31:58.219+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:31:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:31:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:31:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:31:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:31:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:31:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:31:59 smithi016 conmon[32727]: 2022-03-21T18:31:59.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:31:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:31:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:31:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:31:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:00 smithi079 ceph-mon[23158]: pgmap v2396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:00 smithi016 ceph-mon[22010]: pgmap v2396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:00 smithi016 conmon[32727]: 2022-03-21T18:32:00.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:01 smithi016 conmon[32727]: 2022-03-21T18:32:01.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:02 smithi016 ceph-mon[22010]: pgmap v2397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:02.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:02 smithi016 conmon[32727]: 2022-03-21T18:32:02.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:02 smithi079 ceph-mon[23158]: pgmap v2397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:03 smithi016 conmon[32727]: 2022-03-21T18:32:03.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:04 smithi079 ceph-mon[23158]: pgmap v2398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:04 smithi016 conmon[32727]: 2022-03-21T18:32:04.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:04 smithi016 ceph-mon[22010]: pgmap v2398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:05 smithi016 conmon[32727]: 2022-03-21T18:32:05.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:06 smithi079 ceph-mon[23158]: pgmap v2399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:06 smithi016 ceph-mon[22010]: pgmap v2399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:06 smithi016 conmon[32727]: 2022-03-21T18:32:06.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:07 smithi016 conmon[32727]: 2022-03-21T18:32:07.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:08 smithi079 ceph-mon[23158]: pgmap v2400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:08 smithi016 ceph-mon[22010]: pgmap v2400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:08 smithi016 conmon[32727]: 2022-03-21T18:32:08.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:09 smithi016 conmon[32727]: 2022-03-21T18:32:09.206+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:10 smithi016 ceph-mon[22010]: pgmap v2401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:10 smithi016 conmon[32727]: 2022-03-21T18:32:10.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:10 smithi079 ceph-mon[23158]: pgmap v2401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:11 smithi016 conmon[32727]: 2022-03-21T18:32:11.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:12 smithi016 ceph-mon[22010]: pgmap v2402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:12 smithi016 conmon[32727]: 2022-03-21T18:32:12.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:12 smithi079 ceph-mon[23158]: pgmap v2402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:13 smithi016 conmon[32727]: 2022-03-21T18:32:13.285+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:14 smithi016 conmon[32727]: 2022-03-21T18:32:14.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:14 smithi016 ceph-mon[22010]: pgmap v2403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:14 smithi079 ceph-mon[23158]: pgmap v2403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:15 smithi016 conmon[32727]: 2022-03-21T18:32:15.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:16 smithi016 ceph-mon[22010]: pgmap v2404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:16 smithi016 conmon[32727]: 2022-03-21T18:32:16.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:16 smithi079 ceph-mon[23158]: pgmap v2404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:17 smithi016 conmon[32727]: 2022-03-21T18:32:17.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:18 smithi016 ceph-mon[22010]: pgmap v2405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:18 smithi016 conmon[32727]: 2022-03-21T18:32:18.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:18 smithi079 ceph-mon[23158]: pgmap v2405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:19 smithi016 conmon[32727]: 2022-03-21T18:32:19.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:20 smithi016 ceph-mon[22010]: pgmap v2406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:20 smithi016 conmon[32727]: 2022-03-21T18:32:20.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:20 smithi079 ceph-mon[23158]: pgmap v2406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:21 smithi016 conmon[32727]: 2022-03-21T18:32:21.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:22 smithi016 ceph-mon[22010]: pgmap v2407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:22 smithi016 conmon[32727]: 2022-03-21T18:32:22.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:22 smithi079 ceph-mon[23158]: pgmap v2407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:23 smithi016 conmon[32727]: 2022-03-21T18:32:23.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:24 smithi016 conmon[32727]: 2022-03-21T18:32:24.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:24 smithi016 ceph-mon[22010]: pgmap v2408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:24 smithi079 ceph-mon[23158]: pgmap v2408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:25 smithi016 conmon[32727]: 2022-03-21T18:32:25.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:26 smithi016 ceph-mon[22010]: pgmap v2409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:26.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:26 smithi016 conmon[32727]: 2022-03-21T18:32:26.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:26 smithi079 ceph-mon[23158]: pgmap v2409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:27.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:27 smithi016 conmon[32727]: 2022-03-21T18:32:27.182+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:28 smithi016 conmon[32727]: 2022-03-21T18:32:28.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:28 smithi016 ceph-mon[22010]: pgmap v2410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:28 smithi079 ceph-mon[23158]: pgmap v2410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:29.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:29 smithi016 conmon[32727]: 2022-03-21T18:32:29.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:30 smithi016 ceph-mon[22010]: pgmap v2411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:30 smithi016 conmon[32727]: 2022-03-21T18:32:30.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:30 smithi079 ceph-mon[23158]: pgmap v2411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:31 smithi016 conmon[32727]: 2022-03-21T18:32:31.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:32 smithi016 ceph-mon[22010]: pgmap v2412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:32 smithi016 conmon[32727]: 2022-03-21T18:32:32.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:32 smithi079 ceph-mon[23158]: pgmap v2412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:33 smithi016 conmon[32727]: 2022-03-21T18:32:33.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:34 smithi016 ceph-mon[22010]: pgmap v2413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:34 smithi016 conmon[32727]: 2022-03-21T18:32:34.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:34 smithi079 ceph-mon[23158]: pgmap v2413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:35 smithi016 conmon[32727]: 2022-03-21T18:32:35.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:36 smithi016 ceph-mon[22010]: pgmap v2414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:36 smithi016 conmon[32727]: 2022-03-21T18:32:36.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:36 smithi079 ceph-mon[23158]: pgmap v2414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:37 smithi016 conmon[32727]: 2022-03-21T18:32:37.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:38 smithi016 ceph-mon[22010]: pgmap v2415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:38 smithi016 conmon[32727]: 2022-03-21T18:32:38.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:38 smithi079 ceph-mon[23158]: pgmap v2415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:39 smithi016 conmon[32727]: 2022-03-21T18:32:39.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:40 smithi016 ceph-mon[22010]: pgmap v2416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:40 smithi016 conmon[32727]: 2022-03-21T18:32:40.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:40 smithi079 ceph-mon[23158]: pgmap v2416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:41 smithi016 conmon[32727]: 2022-03-21T18:32:41.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:42 smithi016 ceph-mon[22010]: pgmap v2417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:42 smithi016 conmon[32727]: 2022-03-21T18:32:42.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:42 smithi079 ceph-mon[23158]: pgmap v2417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:43 smithi016 conmon[32727]: 2022-03-21T18:32:43.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:44 smithi016 ceph-mon[22010]: pgmap v2418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:44 smithi016 conmon[32727]: 2022-03-21T18:32:44.181+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:44 smithi079 ceph-mon[23158]: pgmap v2418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:45 smithi016 conmon[32727]: 2022-03-21T18:32:45.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:46 smithi016 ceph-mon[22010]: pgmap v2419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:46 smithi016 conmon[32727]: 2022-03-21T18:32:46.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:46 smithi079 ceph-mon[23158]: pgmap v2419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:47.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:47 smithi016 conmon[32727]: 2022-03-21T18:32:47.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:48 smithi016 conmon[32727]: 2022-03-21T18:32:48.138+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:48 smithi016 ceph-mon[22010]: pgmap v2420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:48 smithi079 ceph-mon[23158]: pgmap v2420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:49.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:49 smithi016 conmon[32727]: 2022-03-21T18:32:49.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:50 smithi016 ceph-mon[22010]: pgmap v2421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:50 smithi016 conmon[32727]: 2022-03-21T18:32:50.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:50 smithi079 ceph-mon[23158]: pgmap v2421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:51 smithi016 conmon[32727]: 2022-03-21T18:32:51.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:52 smithi016 conmon[32727]: 2022-03-21T18:32:52.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:52 smithi016 ceph-mon[22010]: pgmap v2422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:52 smithi079 ceph-mon[23158]: pgmap v2422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:53.359 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:53 smithi016 conmon[32727]: 2022-03-21T18:32:53.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:54.360 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:54 smithi016 conmon[32727]: 2022-03-21T18:32:54.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:54 smithi016 ceph-mon[22010]: pgmap v2423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:54 smithi079 ceph-mon[23158]: pgmap v2423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:55.365 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:55 smithi016 conmon[32727]: 2022-03-21T18:32:55.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:56.366 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:56 smithi016 conmon[32727]: 2022-03-21T18:32:56.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:56 smithi016 ceph-mon[22010]: pgmap v2424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:32:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:32:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:32:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:56 smithi079 ceph-mon[23158]: pgmap v2424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:32:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:32:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:32:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:32:57.371 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:56 smithi016 conmon[32727]: 2022-03-21T18:32:56.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:58.373 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:57 smithi016 conmon[32727]: 2022-03-21T18:32:57.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: pgmap v2425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:32:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:32:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:32:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:32:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: pgmap v2425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:32:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:32:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:32:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:32:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:32:59.374 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:32:58 smithi016 conmon[32727]: 2022-03-21T18:32:58.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:32:59.375 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: 2022-03-21T18:32:59.020+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:32:59.375 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:32:59.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:32:59.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: yield 2022-03-21T18:32:59.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:32:59.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:32:59.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:32:59.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:32:59.378 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:32:59.378 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:32:59.378 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:32:59.379 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:32:59.379 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:32:59.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:32:59.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:32:59.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:32:59.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:32:59.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:32:59.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:32:59 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:32:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:32:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:32:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:32:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:32:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:00.376 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:00 smithi016 conmon[32727]: 2022-03-21T18:33:00.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:00 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:33:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:33:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:33:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:33:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:33:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:33:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:00 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:33:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:00 smithi016 ceph-mon[22010]: pgmap v2426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:00 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:33:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:33:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:33:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:33:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:00.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:33:00.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:00.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:00.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:00.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:33:00.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:00.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:00.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:00.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:00 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:33:00.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:00 smithi079 ceph-mon[23158]: pgmap v2426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:01.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:01 smithi016 conmon[32727]: 2022-03-21T18:33:01.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:02.382 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[32727]: 2022-03-21T18:33:02.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:02.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 2022-03-21T18:33:02.092+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:02.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:33:02.384 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:02.384 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: yield 2022-03-21T18:33:02.384 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:02.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:02.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:02.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:33:02.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:02.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:33:02.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:02.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:33:02.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:02.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:33:02.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:02.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:33:02.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:02.389 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:02.389 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:02.389 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 2022-03-21T18:33:02.093+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:33:02.390 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:33:02.390 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:02.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: yield 2022-03-21T18:33:02.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:02.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:02.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:02.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:33:02.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:02.393 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:33:02.393 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:02.393 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:33:02.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:02.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:33:02.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:02.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:33:02.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:02.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:02.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:02.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 2022-03-21T18:33:02.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:33:02.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: 2022-03-21T18:33:02.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:33:02.397 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:33:02.398 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:33:02.398 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:33:02.398 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:33:02.399 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:33:02.399 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:33:02.400 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:33:02.400 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:33:02.400 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:33:02.401 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:33:02.401 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:33:02.401 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:33:02.402 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:33:02.402 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:33:02.402 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:33:02.403 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:33:02.403 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:33:02.403 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:33:02.404 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:33:02.404 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:33:02.404 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:33:02.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise 2022-03-21T18:33:02.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:33:02.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:33:02.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:33:02.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:33:02.407 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:33:02 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:02 smithi016 ceph-mon[22010]: pgmap v2427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:02 smithi079 ceph-mon[23158]: pgmap v2427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:03.387 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:03 smithi016 conmon[32727]: 2022-03-21T18:33:03.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:33:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:33:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:33:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:33:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:33:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:03 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:33:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:33:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:03.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:03.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:33:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:33:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:33:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:03.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:03.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:03.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:33:03.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:03.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:03.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:33:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:33:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:33:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:33:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:33:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:33:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:33:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:33:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:33:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:33:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:33:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:33:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:33:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:33:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:33:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:33:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:33:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:33:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:33:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:33:03.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:33:03.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:33:03.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:33:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:33:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:33:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:33:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:33:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:33:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:33:03.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:33:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:33:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:33:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:33:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:33:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:03 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:33:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:33:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:33:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:33:03.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:33:03.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:33:03.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:33:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:33:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:33:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:33:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:33:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:33:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:33:03.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:33:03.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:33:03.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:33:03.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:33:03.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:33:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:33:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:33:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:33:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:33:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:33:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:33:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:33:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:33:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:33:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:33:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:33:03.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:33:03.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:33:03.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:33:03.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:33:03.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:33:03.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:33:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:33:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:33:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:33:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:33:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:33:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:33:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:33:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:33:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:33:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:33:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:33:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:33:03.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:33:04.388 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:04 smithi016 conmon[32727]: 2022-03-21T18:33:04.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:04 smithi016 ceph-mon[22010]: pgmap v2428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:04 smithi079 ceph-mon[23158]: pgmap v2428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:05.393 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:05 smithi016 conmon[32727]: 2022-03-21T18:33:05.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:06.395 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:06 smithi016 conmon[32727]: 2022-03-21T18:33:06.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:06 smithi016 ceph-mon[22010]: pgmap v2429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:06 smithi079 ceph-mon[23158]: pgmap v2429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:07.400 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:07 smithi016 conmon[32727]: 2022-03-21T18:33:07.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:08.401 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:08 smithi016 conmon[32727]: 2022-03-21T18:33:08.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:08 smithi016 ceph-mon[22010]: pgmap v2430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:08 smithi079 ceph-mon[23158]: pgmap v2430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:09.403 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:08 smithi016 conmon[32727]: 2022-03-21T18:33:08.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:10.404 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:09 smithi016 conmon[32727]: 2022-03-21T18:33:09.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:10 smithi016 ceph-mon[22010]: pgmap v2431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:10 smithi079 ceph-mon[23158]: pgmap v2431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:11.409 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:10 smithi016 conmon[32727]: 2022-03-21T18:33:10.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:12.411 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:11 smithi016 conmon[32727]: 2022-03-21T18:33:11.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:12 smithi016 ceph-mon[22010]: pgmap v2432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:12 smithi079 ceph-mon[23158]: pgmap v2432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:12 smithi016 conmon[32727]: 2022-03-21T18:33:12.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:13.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:13 smithi016 conmon[32727]: 2022-03-21T18:33:13.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:14 smithi079 ceph-mon[23158]: pgmap v2433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:14 smithi016 ceph-mon[22010]: pgmap v2433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:14 smithi016 conmon[32727]: 2022-03-21T18:33:14.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:15 smithi016 conmon[32727]: 2022-03-21T18:33:15.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:16 smithi079 ceph-mon[23158]: pgmap v2434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:16 smithi016 ceph-mon[22010]: pgmap v2434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:16 smithi016 conmon[32727]: 2022-03-21T18:33:16.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:17 smithi016 conmon[32727]: 2022-03-21T18:33:17.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:18 smithi079 ceph-mon[23158]: pgmap v2435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:18 smithi016 ceph-mon[22010]: pgmap v2435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:19 smithi016 conmon[32727]: 2022-03-21T18:33:19.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4241 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4241 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:20 smithi016 conmon[32727]: 2022-03-21T18:33:20.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:20 smithi079 ceph-mon[23158]: pgmap v2436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:20 smithi016 ceph-mon[22010]: pgmap v2436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:21 smithi016 conmon[32727]: 2022-03-21T18:33:21.099+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:22 smithi016 conmon[32727]: 2022-03-21T18:33:22.099+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:22 smithi079 ceph-mon[23158]: pgmap v2437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:22 smithi016 ceph-mon[22010]: pgmap v2437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:23 smithi016 conmon[32727]: 2022-03-21T18:33:23.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:24 smithi016 conmon[32727]: 2022-03-21T18:33:24.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:24 smithi079 ceph-mon[23158]: pgmap v2438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:24 smithi016 ceph-mon[22010]: pgmap v2438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:25 smithi016 conmon[32727]: 2022-03-21T18:33:25.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:26 smithi016 conmon[32727]: 2022-03-21T18:33:26.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:26 smithi079 ceph-mon[23158]: pgmap v2439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:26 smithi016 ceph-mon[22010]: pgmap v2439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:27 smithi016 conmon[32727]: 2022-03-21T18:33:27.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:28 smithi016 conmon[32727]: 2022-03-21T18:33:28.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:28 smithi079 ceph-mon[23158]: pgmap v2440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:28 smithi016 ceph-mon[22010]: pgmap v2440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:29 smithi016 conmon[32727]: 2022-03-21T18:33:29.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:30 smithi016 conmon[32727]: 2022-03-21T18:33:30.065+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:30 smithi079 ceph-mon[23158]: pgmap v2441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:30 smithi016 ceph-mon[22010]: pgmap v2441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:31 smithi016 conmon[32727]: 2022-03-21T18:33:31.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:32 smithi016 conmon[32727]: 2022-03-21T18:33:32.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:32 smithi079 ceph-mon[23158]: pgmap v2442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:32 smithi016 ceph-mon[22010]: pgmap v2442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:33.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:33 smithi016 conmon[32727]: 2022-03-21T18:33:33.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:34 smithi016 conmon[32727]: 2022-03-21T18:33:34.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:34 smithi016 ceph-mon[22010]: pgmap v2443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:34 smithi079 ceph-mon[23158]: pgmap v2443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:35 smithi016 conmon[32727]: 2022-03-21T18:33:35.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:36 smithi016 conmon[32727]: 2022-03-21T18:33:36.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:36 smithi016 ceph-mon[22010]: pgmap v2444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:36 smithi079 ceph-mon[23158]: pgmap v2444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:37.488 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:37 smithi016 conmon[32727]: 2022-03-21T18:33:37.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:38.489 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:38 smithi016 conmon[32727]: 2022-03-21T18:33:38.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:38 smithi016 ceph-mon[22010]: pgmap v2445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:38 smithi079 ceph-mon[23158]: pgmap v2445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:39.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:39 smithi016 conmon[32727]: 2022-03-21T18:33:39.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:40.492 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:40 smithi016 conmon[32727]: 2022-03-21T18:33:40.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:40 smithi016 ceph-mon[22010]: pgmap v2446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:40 smithi079 ceph-mon[23158]: pgmap v2446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:41.497 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:41 smithi016 conmon[32727]: 2022-03-21T18:33:41.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:42.498 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:42 smithi016 conmon[32727]: 2022-03-21T18:33:42.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:42 smithi016 ceph-mon[22010]: pgmap v2447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:42 smithi079 ceph-mon[23158]: pgmap v2447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:43 smithi016 conmon[32727]: 2022-03-21T18:33:43.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:44 smithi016 ceph-mon[22010]: pgmap v2448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:44 smithi016 conmon[32727]: 2022-03-21T18:33:44.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:44 smithi079 ceph-mon[23158]: pgmap v2448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:45.510 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:45 smithi016 conmon[32727]: 2022-03-21T18:33:45.246+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:46.511 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:46 smithi016 conmon[32727]: 2022-03-21T18:33:46.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:46 smithi016 ceph-mon[22010]: pgmap v2449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:46 smithi079 ceph-mon[23158]: pgmap v2449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:47.516 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:47 smithi016 conmon[32727]: 2022-03-21T18:33:47.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:48.518 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:48 smithi016 conmon[32727]: 2022-03-21T18:33:48.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:48 smithi016 ceph-mon[22010]: pgmap v2450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:48 smithi079 ceph-mon[23158]: pgmap v2450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:49.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:49 smithi016 conmon[32727]: 2022-03-21T18:33:49.277+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:50.520 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:50 smithi016 conmon[32727]: 2022-03-21T18:33:50.267+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:50 smithi016 ceph-mon[22010]: pgmap v2451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:50 smithi079 ceph-mon[23158]: pgmap v2451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:51 smithi016 conmon[32727]: 2022-03-21T18:33:51.292+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:52.527 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:52 smithi016 conmon[32727]: 2022-03-21T18:33:52.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:52 smithi016 ceph-mon[22010]: pgmap v2452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:52 smithi079 ceph-mon[23158]: pgmap v2452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:53 smithi016 conmon[32727]: 2022-03-21T18:33:53.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:54 smithi016 conmon[32727]: 2022-03-21T18:33:54.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:54 smithi016 ceph-mon[22010]: pgmap v2453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:54 smithi079 ceph-mon[23158]: pgmap v2453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:55 smithi016 conmon[32727]: 2022-03-21T18:33:55.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:56 smithi016 ceph-mon[22010]: pgmap v2454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:56 smithi016 conmon[32727]: 2022-03-21T18:33:56.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:56 smithi079 ceph-mon[23158]: pgmap v2454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:33:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:57 smithi016 conmon[32727]: 2022-03-21T18:33:57.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:58 smithi016 conmon[32727]: 2022-03-21T18:33:58.320+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: pgmap v2455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:33:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:33:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:33:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:33:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: pgmap v2455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:33:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:33:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:33:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:33:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:33:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:33:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:33:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:33:59 smithi016 conmon[32727]: 2022-03-21T18:33:59.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:33:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:33:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:33:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:00 smithi016 conmon[32727]: 2022-03-21T18:34:00.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:00 smithi016 ceph-mon[22010]: pgmap v2456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:00 smithi079 ceph-mon[23158]: pgmap v2456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:01 smithi016 conmon[32727]: 2022-03-21T18:34:01.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[32727]: 2022-03-21T18:34:02.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:02 smithi016 ceph-mon[22010]: pgmap v2457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:34:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:34:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:34:02.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: 2022-03-21T18:34:02.700+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:02.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:34:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: yield 2022-03-21T18:34:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:34:02.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:02.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:34:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:34:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:34:02.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:02.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:34:02.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:02.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:02.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:02 smithi079 ceph-mon[23158]: pgmap v2457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:34:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:34:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:34:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:03 smithi016 conmon[32727]: 2022-03-21T18:34:03.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:34:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:34:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:34:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:34:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:34:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:34:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:03 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:34:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:34:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:34:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:03.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:03.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:03.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:34:03.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:03.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:34:03.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:03.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:34:03.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:03.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:03.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:03.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:34:03.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:03.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:03.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:03.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:03 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:34:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:04 smithi016 ceph-mon[22010]: pgmap v2458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:04 smithi016 conmon[32727]: 2022-03-21T18:34:04.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:04 smithi079 ceph-mon[23158]: pgmap v2458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:05.565 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[32727]: 2022-03-21T18:34:05.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:05.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 2022-03-21T18:34:05.773+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:05.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:34:05.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:05.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: yield 2022-03-21T18:34:05.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:05.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:05.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:05.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:34:05.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:05.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:34:05.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:05.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:34:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:34:05.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:05.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:34:05.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:05.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 2022-03-21T18:34:05.774+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:34:05.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:34:05.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:05.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: yield 2022-03-21T18:34:05.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:05.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:34:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:34:05.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:05.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:34:05.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:34:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:05.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:34:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:05.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:05.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 2022-03-21T18:34:05.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:34:05.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: 2022-03-21T18:34:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:34:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:34:05.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:34:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:34:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:34:05.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:34:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:34:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:34:05.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:34:05.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:34:05.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:34:05.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:34:05.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:34:05.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:34:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:34:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:34:05.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:34:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:34:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:34:05.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:34:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:34:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:34:05.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise 2022-03-21T18:34:05.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:34:05.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:34:05.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:34:05.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:34:05.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:34:05 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:06.567 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:06 smithi016 conmon[32727]: 2022-03-21T18:34:06.308+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:06 smithi016 ceph-mon[22010]: pgmap v2459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:06 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:34:06.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:06.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:06.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:06.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:34:06.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:06.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:34:06.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:06.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:34:06.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:06.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:06.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:06.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:34:06.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:06.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:06.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:06 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:34:06.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:06.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:34:06.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:06.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:06.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:06.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:34:06.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:06.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:34:06.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:06.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:34:06.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:06.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:06.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:06.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:34:06.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:06.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:06.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:34:06.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:34:06.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:34:06.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:34:06.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:34:06.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:34:06.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:34:06.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:34:06.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:34:06.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:34:06.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:34:06.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:34:06.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:34:06.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:34:06.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:34:06.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:34:06.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:34:06.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:34:06.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:34:06.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:34:06.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:34:06.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:34:06.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:34:06.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:34:06.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:34:06.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:34:06.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:34:06.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:34:06.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:34:06.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:06 smithi079 ceph-mon[23158]: pgmap v2459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:06 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:06.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:34:06.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:06.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:06.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:06.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:34:06.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:06.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:34:06.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:06.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:34:06.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:06.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:06.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:06.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:34:06.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:06.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:06.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:06 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:34:06.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:34:06.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:34:06.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:34:06.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:34:06.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:34:06.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:34:06.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:34:06.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:34:06.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:34:06.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:34:06.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:34:06.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:34:06.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:34:06.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:34:06.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:34:06.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:34:06.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:06.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:34:06.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:34:06.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:34:06.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:34:06.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:34:06.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:34:06.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:34:06.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:34:06.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:34:06.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:34:06.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:34:06.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:34:06.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:34:06.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:34:06.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:34:06.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:34:06.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:34:06.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:34:06.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:34:06.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:34:06.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:34:06.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:34:06.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:34:06.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:34:06.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:34:06.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:34:06.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:34:06.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:34:06.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:34:06.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:34:06.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:34:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:07 smithi016 conmon[32727]: 2022-03-21T18:34:07.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:08.574 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:08 smithi016 conmon[32727]: 2022-03-21T18:34:08.297+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:08 smithi016 ceph-mon[22010]: pgmap v2460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:08 smithi079 ceph-mon[23158]: pgmap v2460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:09.575 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:09 smithi016 conmon[32727]: 2022-03-21T18:34:09.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:10 smithi016 conmon[32727]: 2022-03-21T18:34:10.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:10 smithi016 ceph-mon[22010]: pgmap v2461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:10 smithi079 ceph-mon[23158]: pgmap v2461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:11.581 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:11 smithi016 conmon[32727]: 2022-03-21T18:34:11.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:12 smithi016 ceph-mon[22010]: pgmap v2462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:12 smithi016 conmon[32727]: 2022-03-21T18:34:12.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:12 smithi079 ceph-mon[23158]: pgmap v2462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:12.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:13 smithi016 conmon[32727]: 2022-03-21T18:34:13.340+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:14.589 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:14 smithi016 conmon[32727]: 2022-03-21T18:34:14.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:14 smithi016 ceph-mon[22010]: pgmap v2463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:14 smithi079 ceph-mon[23158]: pgmap v2463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:15 smithi016 conmon[32727]: 2022-03-21T18:34:15.376+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:16 smithi016 ceph-mon[22010]: pgmap v2464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:16 smithi016 conmon[32727]: 2022-03-21T18:34:16.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:16 smithi079 ceph-mon[23158]: pgmap v2464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:17 smithi016 conmon[32727]: 2022-03-21T18:34:17.403+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:18 smithi016 conmon[32727]: 2022-03-21T18:34:18.440+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:18 smithi016 ceph-mon[22010]: pgmap v2465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:18 smithi079 ceph-mon[23158]: pgmap v2465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:19 smithi016 conmon[32727]: 2022-03-21T18:34:19.488+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:20 smithi016 ceph-mon[22010]: pgmap v2466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:20 smithi016 conmon[32727]: 2022-03-21T18:34:20.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:20 smithi079 ceph-mon[23158]: pgmap v2466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:21 smithi016 conmon[32727]: 2022-03-21T18:34:21.455+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:22 smithi016 ceph-mon[22010]: pgmap v2467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:22 smithi016 conmon[32727]: 2022-03-21T18:34:22.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:22 smithi079 ceph-mon[23158]: pgmap v2467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:23 smithi016 conmon[32727]: 2022-03-21T18:34:23.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:24 smithi016 conmon[32727]: 2022-03-21T18:34:24.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:24 smithi016 ceph-mon[22010]: pgmap v2468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:24 smithi079 ceph-mon[23158]: pgmap v2468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:25 smithi016 conmon[32727]: 2022-03-21T18:34:25.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:26 smithi016 ceph-mon[22010]: pgmap v2469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:26 smithi016 conmon[32727]: 2022-03-21T18:34:26.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:26 smithi079 ceph-mon[23158]: pgmap v2469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:27 smithi016 conmon[32727]: 2022-03-21T18:34:27.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:28 smithi016 ceph-mon[22010]: pgmap v2470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:28 smithi016 conmon[32727]: 2022-03-21T18:34:28.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:28 smithi079 ceph-mon[23158]: pgmap v2470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:29 smithi016 conmon[32727]: 2022-03-21T18:34:29.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:30 smithi016 ceph-mon[22010]: pgmap v2471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:30 smithi016 conmon[32727]: 2022-03-21T18:34:30.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:30 smithi079 ceph-mon[23158]: pgmap v2471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:31 smithi016 conmon[32727]: 2022-03-21T18:34:31.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:32 smithi016 ceph-mon[22010]: pgmap v2472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:32 smithi016 conmon[32727]: 2022-03-21T18:34:32.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:32 smithi079 ceph-mon[23158]: pgmap v2472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:33.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:33 smithi016 conmon[32727]: 2022-03-21T18:34:33.348+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:34.647 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:34 smithi016 conmon[32727]: 2022-03-21T18:34:34.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:34 smithi016 ceph-mon[22010]: pgmap v2473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:34 smithi079 ceph-mon[23158]: pgmap v2473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:35.652 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:35 smithi016 conmon[32727]: 2022-03-21T18:34:35.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:36.653 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:36 smithi016 conmon[32727]: 2022-03-21T18:34:36.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:36 smithi016 ceph-mon[22010]: pgmap v2474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:36 smithi079 ceph-mon[23158]: pgmap v2474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:37.658 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:37 smithi016 conmon[32727]: 2022-03-21T18:34:37.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:37.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:38.659 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:38 smithi016 conmon[32727]: 2022-03-21T18:34:38.286+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:38 smithi016 ceph-mon[22010]: pgmap v2475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:38 smithi079 ceph-mon[23158]: pgmap v2475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:39.660 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:39 smithi016 conmon[32727]: 2022-03-21T18:34:39.326+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:40.662 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:40 smithi016 conmon[32727]: 2022-03-21T18:34:40.315+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:40 smithi016 ceph-mon[22010]: pgmap v2476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:40 smithi079 ceph-mon[23158]: pgmap v2476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:41 smithi016 conmon[32727]: 2022-03-21T18:34:41.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:42 smithi016 conmon[32727]: 2022-03-21T18:34:42.318+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:42 smithi079 ceph-mon[23158]: pgmap v2477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:42 smithi016 ceph-mon[22010]: pgmap v2477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:43 smithi016 conmon[32727]: 2022-03-21T18:34:43.289+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:44 smithi016 conmon[32727]: 2022-03-21T18:34:44.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:44 smithi079 ceph-mon[23158]: pgmap v2478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:44 smithi016 ceph-mon[22010]: pgmap v2478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:45 smithi016 conmon[32727]: 2022-03-21T18:34:45.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:46 smithi016 conmon[32727]: 2022-03-21T18:34:46.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:46 smithi079 ceph-mon[23158]: pgmap v2479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:46 smithi016 ceph-mon[22010]: pgmap v2479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:47 smithi016 conmon[32727]: 2022-03-21T18:34:47.378+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:48.686 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:48 smithi016 conmon[32727]: 2022-03-21T18:34:48.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:48 smithi079 ceph-mon[23158]: pgmap v2480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:48 smithi016 ceph-mon[22010]: pgmap v2480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:49 smithi016 conmon[32727]: 2022-03-21T18:34:49.455+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:50 smithi016 ceph-mon[22010]: pgmap v2481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:50 smithi016 conmon[32727]: 2022-03-21T18:34:50.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:50 smithi079 ceph-mon[23158]: pgmap v2481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:51 smithi016 conmon[32727]: 2022-03-21T18:34:51.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:52 smithi016 ceph-mon[22010]: pgmap v2482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:52.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:52 smithi016 conmon[32727]: 2022-03-21T18:34:52.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:52 smithi079 ceph-mon[23158]: pgmap v2482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:53 smithi016 conmon[32727]: 2022-03-21T18:34:53.453+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:54.702 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:54 smithi016 conmon[32727]: 2022-03-21T18:34:54.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:54 smithi079 ceph-mon[23158]: pgmap v2483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:54 smithi016 ceph-mon[22010]: pgmap v2483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:55.707 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:55 smithi016 conmon[32727]: 2022-03-21T18:34:55.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:56 smithi016 conmon[32727]: 2022-03-21T18:34:56.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:56 smithi079 ceph-mon[23158]: pgmap v2484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:56 smithi016 ceph-mon[22010]: pgmap v2484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:34:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:57 smithi016 conmon[32727]: 2022-03-21T18:34:57.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:58.715 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:58 smithi016 conmon[32727]: 2022-03-21T18:34:58.447+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: pgmap v2485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:34:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:34:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:34:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:34:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: pgmap v2485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:34:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:34:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:34:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:34:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:34:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:34:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:34:59 smithi016 conmon[32727]: 2022-03-21T18:34:59.403+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:34:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:34:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:34:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:34:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:00 smithi016 conmon[32727]: 2022-03-21T18:35:00.384+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:00 smithi079 ceph-mon[23158]: pgmap v2486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:00 smithi016 ceph-mon[22010]: pgmap v2486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:01.724 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:01 smithi016 conmon[32727]: 2022-03-21T18:35:01.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:02.725 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:02 smithi016 conmon[32727]: 2022-03-21T18:35:02.452+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:02 smithi016 ceph-mon[22010]: pgmap v2487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:02 smithi079 ceph-mon[23158]: pgmap v2487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:03.730 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:03 smithi016 conmon[32727]: 2022-03-21T18:35:03.417+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:04.732 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:04 smithi016 conmon[32727]: 2022-03-21T18:35:04.441+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:04 smithi016 ceph-mon[22010]: pgmap v2488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:04 smithi079 ceph-mon[23158]: pgmap v2488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:05.737 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:05 smithi016 conmon[32727]: 2022-03-21T18:35:05.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:06 smithi016 conmon[32727]: 2022-03-21T18:35:06.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:06 smithi016 ceph-mon[22010]: pgmap v2489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:35:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:35:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:35:07.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:06 smithi079 ceph-mon[23158]: pgmap v2489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:35:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:35:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:35:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:07.744 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:07 smithi016 conmon[32727]: 2022-03-21T18:35:07.436+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:08.745 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[32727]: 2022-03-21T18:35:08.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:08 smithi016 ceph-mon[22010]: pgmap v2490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: 2022-03-21T18:35:08.877+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:35:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: yield 2022-03-21T18:35:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:09.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:09.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:35:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:35:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:35:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:35:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:35:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:09.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:09.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:08 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:08 smithi079 ceph-mon[23158]: pgmap v2490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:09.747 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:09 smithi016 conmon[32727]: 2022-03-21T18:35:09.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:10.748 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:10 smithi016 conmon[32727]: 2022-03-21T18:35:10.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:35:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:35:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:11.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:35:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:35:11.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:35:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:11.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:11.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:35:11.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:11.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:10 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:35:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:10 smithi016 ceph-mon[22010]: pgmap v2491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:11.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:35:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:35:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:11.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:35:11.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:35:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:11.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:35:11.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:11.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:11.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:11.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:35:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:11.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:11.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:10 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:35:11.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:10 smithi079 ceph-mon[23158]: pgmap v2491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:11.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[32727]: 2022-03-21T18:35:11.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:12.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 2022-03-21T18:35:11.949+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:35:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: yield 2022-03-21T18:35:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:35:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:35:12.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:12.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:35:12.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:35:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:35:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 2022-03-21T18:35:11.950+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:35:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:35:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:12.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: yield 2022-03-21T18:35:12.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:35:12.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:12.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:35:12.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:35:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:35:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:35:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 2022-03-21T18:35:12.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:35:12.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: 2022-03-21T18:35:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:35:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:35:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:35:12.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:35:12.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:35:12.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:35:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:35:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:35:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:35:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:35:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:35:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:35:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:35:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:35:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:35:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:35:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:35:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:35:12.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:35:12.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:35:12.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:35:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:35:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise 2022-03-21T18:35:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:35:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:35:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:35:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:35:12.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:35:11 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:12 smithi016 conmon[32727]: 2022-03-21T18:35:12.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:12 smithi016 ceph-mon[22010]: pgmap v2492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:12 smithi079 ceph-mon[23158]: pgmap v2492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:13 smithi016 conmon[32727]: 2022-03-21T18:35:13.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:35:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:14.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:35:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:14.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:35:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:35:14.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:14.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:35:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:14.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:13 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:35:14.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:35:14.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:14.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:35:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:35:14.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:14.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:35:14.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:14.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:14.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:14.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:35:14.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:35:14.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:35:14.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:35:14.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:35:14.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:35:14.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:35:14.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:35:14.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:35:14.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:35:14.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:35:14.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:35:14.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:35:14.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:35:14.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:35:14.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:35:14.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:35:14.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:35:14.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:35:14.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:35:14.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:35:14.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:35:14.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:35:14.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:35:14.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:35:14.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:35:14.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:35:14.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:35:14.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:35:14.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:35:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:35:14.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:35:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:14.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:35:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:14.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:35:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:14.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:13 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:35:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:35:14.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:35:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:35:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:35:14.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:35:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:35:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:35:14.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:35:14.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:35:14.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:35:14.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:35:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:35:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:35:14.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:35:14.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:35:14.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:35:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:35:14.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:35:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:35:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:35:14.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:35:14.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:35:14.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:35:14.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:35:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:35:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:35:14.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:35:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:35:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:35:14.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:35:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:35:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:35:14.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:35:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:35:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:35:14.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:35:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:35:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:35:14.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:35:14.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:35:14.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:35:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:35:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:35:14.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:35:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:35:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:35:14.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:35:14.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:14.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:14 smithi016 conmon[32727]: 2022-03-21T18:35:14.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:14 smithi016 ceph-mon[22010]: pgmap v2493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:14 smithi079 ceph-mon[23158]: pgmap v2493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:15.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:15 smithi016 conmon[32727]: 2022-03-21T18:35:15.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:16.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:16 smithi016 conmon[32727]: 2022-03-21T18:35:16.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:16 smithi016 ceph-mon[22010]: pgmap v2494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:16 smithi079 ceph-mon[23158]: pgmap v2494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:17 smithi016 conmon[32727]: 2022-03-21T18:35:17.384+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:18 smithi016 conmon[32727]: 2022-03-21T18:35:18.361+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:18 smithi016 ceph-mon[22010]: pgmap v2495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:18 smithi079 ceph-mon[23158]: pgmap v2495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:19 smithi016 conmon[32727]: 2022-03-21T18:35:19.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:20 smithi016 conmon[32727]: 2022-03-21T18:35:20.399+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:20 smithi016 ceph-mon[22010]: pgmap v2496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:20 smithi079 ceph-mon[23158]: pgmap v2496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:21 smithi016 conmon[32727]: 2022-03-21T18:35:21.376+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:22.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:22 smithi016 conmon[32727]: 2022-03-21T18:35:22.369+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:22 smithi016 ceph-mon[22010]: pgmap v2497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:22 smithi079 ceph-mon[23158]: pgmap v2497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:23.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:23.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:23 smithi016 conmon[32727]: 2022-03-21T18:35:23.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:24 smithi016 conmon[32727]: 2022-03-21T18:35:24.332+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:24 smithi016 ceph-mon[22010]: pgmap v2498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:24 smithi079 ceph-mon[23158]: pgmap v2498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:25 smithi016 conmon[32727]: 2022-03-21T18:35:25.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:26 smithi016 conmon[32727]: 2022-03-21T18:35:26.355+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:26 smithi016 ceph-mon[22010]: pgmap v2499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:26 smithi079 ceph-mon[23158]: pgmap v2499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:27 smithi016 conmon[32727]: 2022-03-21T18:35:27.356+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:27 smithi016 ceph-mon[22010]: pgmap v2500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:27 smithi079 ceph-mon[23158]: pgmap v2500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:28 smithi016 conmon[32727]: 2022-03-21T18:35:28.332+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:29 smithi016 conmon[32727]: 2022-03-21T18:35:29.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:29 smithi016 ceph-mon[22010]: pgmap v2501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:30.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:29 smithi079 ceph-mon[23158]: pgmap v2501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:30.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:30 smithi016 conmon[32727]: 2022-03-21T18:35:30.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:31 smithi016 conmon[32727]: 2022-03-21T18:35:31.246+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:31 smithi016 ceph-mon[22010]: pgmap v2502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:31 smithi079 ceph-mon[23158]: pgmap v2502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:32 smithi016 conmon[32727]: 2022-03-21T18:35:32.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:33 smithi016 conmon[32727]: 2022-03-21T18:35:33.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:33 smithi016 ceph-mon[22010]: pgmap v2503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:33 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:33 smithi079 ceph-mon[23158]: pgmap v2503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:33 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:34 smithi016 conmon[32727]: 2022-03-21T18:35:34.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:35.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:35.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:35 smithi016 conmon[32727]: 2022-03-21T18:35:35.162+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:36.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:36.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:35 smithi016 ceph-mon[22010]: pgmap v2504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:35 smithi079 ceph-mon[23158]: pgmap v2504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:36 smithi016 conmon[32727]: 2022-03-21T18:35:36.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:37 smithi016 conmon[32727]: 2022-03-21T18:35:37.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:37 smithi016 ceph-mon[22010]: pgmap v2505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:38 smithi016 conmon[32727]: 2022-03-21T18:35:38.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:37 smithi079 ceph-mon[23158]: pgmap v2505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:39 smithi016 conmon[32727]: 2022-03-21T18:35:39.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:38 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:38 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:39 smithi016 ceph-mon[22010]: pgmap v2506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:40 smithi016 conmon[32727]: 2022-03-21T18:35:40.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:39 smithi079 ceph-mon[23158]: pgmap v2506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:41 smithi016 conmon[32727]: 2022-03-21T18:35:41.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:42 smithi016 conmon[32727]: 2022-03-21T18:35:42.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:41 smithi016 ceph-mon[22010]: pgmap v2507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:41 smithi079 ceph-mon[23158]: pgmap v2507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:43 smithi016 conmon[32727]: 2022-03-21T18:35:43.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:43 smithi079 ceph-mon[23158]: pgmap v2508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:43 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:43 smithi016 ceph-mon[22010]: pgmap v2508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:43 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:44 smithi016 conmon[32727]: 2022-03-21T18:35:44.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:45 smithi016 conmon[32727]: 2022-03-21T18:35:45.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:45 smithi079 ceph-mon[23158]: pgmap v2509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:45 smithi016 ceph-mon[22010]: pgmap v2509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:46 smithi016 conmon[32727]: 2022-03-21T18:35:46.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:47 smithi016 conmon[32727]: 2022-03-21T18:35:47.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:47 smithi079 ceph-mon[23158]: pgmap v2510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:47 smithi016 conmon[32727]: 2022-03-21T18:35:47.973+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:47 smithi016 ceph-mon[22010]: pgmap v2510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:48 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:49.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:49 smithi016 conmon[32727]: 2022-03-21T18:35:49.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:48 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:49 smithi079 ceph-mon[23158]: pgmap v2511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:49 smithi016 conmon[32727]: 2022-03-21T18:35:49.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:49 smithi016 ceph-mon[22010]: pgmap v2511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:50 smithi016 conmon[32727]: 2022-03-21T18:35:50.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:51 smithi079 ceph-mon[23158]: pgmap v2512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:51 smithi016 ceph-mon[22010]: pgmap v2512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:51 smithi016 conmon[32727]: 2022-03-21T18:35:51.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:52 smithi016 conmon[32727]: 2022-03-21T18:35:52.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:53 smithi079 ceph-mon[23158]: pgmap v2513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:53 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:53 smithi016 ceph-mon[22010]: pgmap v2513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:53 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:54 smithi016 conmon[32727]: 2022-03-21T18:35:54.004+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:55 smithi016 conmon[32727]: 2022-03-21T18:35:55.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:55 smithi079 ceph-mon[23158]: pgmap v2514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:55 smithi016 ceph-mon[22010]: pgmap v2514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:35:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:56 smithi016 conmon[32727]: 2022-03-21T18:35:56.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:57 smithi016 conmon[32727]: 2022-03-21T18:35:57.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:57 smithi079 ceph-mon[23158]: pgmap v2515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:57 smithi016 ceph-mon[22010]: pgmap v2515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:35:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:58 smithi016 conmon[32727]: 2022-03-21T18:35:58.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:35:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:35:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:58 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:35:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:35:59 smithi016 conmon[32727]: 2022-03-21T18:35:59.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:35:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:35:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:35:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:58 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:35:59 smithi079 ceph-mon[23158]: pgmap v2516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:35:59 smithi016 ceph-mon[22010]: pgmap v2516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:00 smithi016 conmon[32727]: 2022-03-21T18:36:00.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:01.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:01 smithi016 conmon[32727]: 2022-03-21T18:36:01.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:02 smithi016 conmon[32727]: 2022-03-21T18:36:02.047+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:01 smithi016 ceph-mon[22010]: pgmap v2517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:01 smithi079 ceph-mon[23158]: pgmap v2517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:03 smithi016 conmon[32727]: 2022-03-21T18:36:03.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:04 smithi016 conmon[32727]: 2022-03-21T18:36:04.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:03 smithi016 ceph-mon[22010]: pgmap v2518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:03 smithi079 ceph-mon[23158]: pgmap v2518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:05 smithi016 conmon[32727]: 2022-03-21T18:36:05.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:05 smithi016 ceph-mon[22010]: pgmap v2519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:06 smithi016 conmon[32727]: 2022-03-21T18:36:06.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:05 smithi079 ceph-mon[23158]: pgmap v2519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:07 smithi016 conmon[32727]: 2022-03-21T18:36:07.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:07 smithi016 ceph-mon[22010]: pgmap v2520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:08 smithi016 conmon[32727]: 2022-03-21T18:36:08.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:07 smithi079 ceph-mon[23158]: pgmap v2520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:09 smithi016 conmon[32727]: 2022-03-21T18:36:09.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:10 smithi016 ceph-mon[22010]: pgmap v2521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:10 smithi016 conmon[32727]: 2022-03-21T18:36:10.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:10 smithi079 ceph-mon[23158]: pgmap v2521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:11 smithi016 conmon[32727]: 2022-03-21T18:36:11.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:12 smithi079 ceph-mon[23158]: pgmap v2522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:36:12.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:36:12.408 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:36:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[32727]: 2022-03-21T18:36:12.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:12 smithi016 ceph-mon[22010]: pgmap v2522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:36:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:36:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:36:12.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: 2022-03-21T18:36:12.429+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:12.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:36:12.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:12.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: yield 2022-03-21T18:36:12.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:12.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:12.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:12.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:36:12.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:12.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:36:12.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:12.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:36:12.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:12.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:36:12.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:12.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:36:12.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:12.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:12.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:13 smithi016 conmon[32727]: 2022-03-21T18:36:13.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:36:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:36:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:13.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:36:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:13.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:36:13.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:13.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:36:13.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:13.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:13.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:36:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:13.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:13.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:13 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:36:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:36:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:36:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:36:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:36:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:36:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:13 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:36:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:14 smithi016 conmon[32727]: 2022-03-21T18:36:14.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:14 smithi016 ceph-mon[22010]: pgmap v2523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:14 smithi079 ceph-mon[23158]: pgmap v2523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[32727]: 2022-03-21T18:36:15.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:15.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 2022-03-21T18:36:15.501+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:15.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:36:15.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:15.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: yield 2022-03-21T18:36:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:15.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:36:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:15.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:36:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:36:15.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:15.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:36:15.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:36:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:15.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 2022-03-21T18:36:15.502+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:36:15.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:36:15.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:15.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: yield 2022-03-21T18:36:15.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:15.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:15.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:15.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:36:15.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:15.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:36:15.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:15.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:36:15.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:15.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:36:15.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:15.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:36:15.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:15.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:15.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:15.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 2022-03-21T18:36:15.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:36:15.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: 2022-03-21T18:36:15.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:36:15.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:36:15.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:36:15.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:36:15.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:36:15.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:36:15.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:36:15.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:36:15.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:36:15.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:36:15.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:36:15.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:36:15.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:36:15.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:36:15.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:36:15.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:36:15.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:36:15.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:36:15.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:36:15.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:36:15.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:36:15.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:36:15.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise 2022-03-21T18:36:15.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:36:15.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:36:15.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:36:15.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:36:15.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:36:15 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:16 smithi016 conmon[32727]: 2022-03-21T18:36:16.086+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:16 smithi016 ceph-mon[22010]: pgmap v2524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:16 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:16.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:36:16.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:16.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:16.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:16.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:36:16.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:16.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:36:16.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:16.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:36:16.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:16.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:16.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:16.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:36:16.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:16.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:16.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:16 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:36:16.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:16.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:36:16.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:16.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:16.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:16.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:36:16.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:16.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:36:16.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:16.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:36:16.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:16.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:16.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:16.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:36:16.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:16.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:16.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:36:16.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:36:16.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:36:16.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:36:16.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:36:16.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:36:16.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:36:16.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:36:16.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:36:16.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:36:16.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:36:16.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:36:16.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:36:16.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:36:16.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:36:16.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:36:16.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:36:16.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:36:16.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:36:16.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:36:16.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:36:16.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:36:16.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:36:16.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:36:16.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:36:16.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:36:16.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:36:16.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:36:16.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:36:16.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:16.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:16 smithi079 ceph-mon[23158]: pgmap v2524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:16.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:16 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:16.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:36:16.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:16.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:16.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:16.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:36:16.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:16.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:36:16.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:16.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:36:16.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:16.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:16.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:16.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:36:16.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:16.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:16.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:16 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:36:16.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:36:16.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:36:16.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:36:16.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:36:16.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:36:16.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:36:16.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:36:16.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:36:16.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:36:16.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:36:16.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:36:16.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:36:16.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:36:16.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:36:16.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:36:16.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:36:16.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:16.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:36:16.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:36:16.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:36:16.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:36:16.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:36:16.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:36:16.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:36:16.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:36:16.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:36:16.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:36:16.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:36:16.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:36:16.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:36:16.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:36:16.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:36:16.505 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:36:16.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:36:16.506 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:36:16.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:36:16.507 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:36:16.508 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:36:16.509 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:36:16.509 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:36:16.510 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:36:16.510 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:36:16.511 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:36:16.512 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:36:16.512 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:36:16.513 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:36:16.514 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:36:16.514 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:36:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:17 smithi016 conmon[32727]: 2022-03-21T18:36:17.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:18 smithi016 ceph-mon[22010]: pgmap v2525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:18 smithi016 conmon[32727]: 2022-03-21T18:36:18.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:18 smithi079 ceph-mon[23158]: pgmap v2525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:19 smithi016 conmon[32727]: 2022-03-21T18:36:19.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:20 smithi016 ceph-mon[22010]: pgmap v2526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:20 smithi016 conmon[32727]: 2022-03-21T18:36:20.148+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:20 smithi079 ceph-mon[23158]: pgmap v2526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:21 smithi016 conmon[32727]: 2022-03-21T18:36:21.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:22 smithi016 ceph-mon[22010]: pgmap v2527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:22 smithi016 conmon[32727]: 2022-03-21T18:36:22.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:22 smithi079 ceph-mon[23158]: pgmap v2527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:23 smithi016 conmon[32727]: 2022-03-21T18:36:23.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:24 smithi016 ceph-mon[22010]: pgmap v2528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:24 smithi016 conmon[32727]: 2022-03-21T18:36:24.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:24 smithi079 ceph-mon[23158]: pgmap v2528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:25 smithi016 conmon[32727]: 2022-03-21T18:36:25.039+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:26 smithi016 ceph-mon[22010]: pgmap v2529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:26 smithi016 conmon[32727]: 2022-03-21T18:36:26.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:26 smithi079 ceph-mon[23158]: pgmap v2529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:27 smithi016 conmon[32727]: 2022-03-21T18:36:27.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:28 smithi016 ceph-mon[22010]: pgmap v2530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:28 smithi016 conmon[32727]: 2022-03-21T18:36:28.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:28 smithi079 ceph-mon[23158]: pgmap v2530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:29 smithi016 conmon[32727]: 2022-03-21T18:36:29.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:30 smithi016 ceph-mon[22010]: pgmap v2531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:30 smithi016 conmon[32727]: 2022-03-21T18:36:30.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:30 smithi079 ceph-mon[23158]: pgmap v2531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:31 smithi016 conmon[32727]: 2022-03-21T18:36:31.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:32 smithi016 ceph-mon[22010]: pgmap v2532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:31 smithi016 conmon[32727]: 2022-03-21T18:36:31.973+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:32 smithi079 ceph-mon[23158]: pgmap v2532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:32 smithi016 conmon[32727]: 2022-03-21T18:36:32.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:34 smithi016 ceph-mon[22010]: pgmap v2533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:33 smithi016 conmon[32727]: 2022-03-21T18:36:33.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:34 smithi079 ceph-mon[23158]: pgmap v2533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:34 smithi016 conmon[32727]: 2022-03-21T18:36:34.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:36 smithi016 ceph-mon[22010]: pgmap v2534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:35 smithi016 conmon[32727]: 2022-03-21T18:36:35.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:36 smithi079 ceph-mon[23158]: pgmap v2534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:36 smithi016 conmon[32727]: 2022-03-21T18:36:36.882+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:38 smithi016 ceph-mon[22010]: pgmap v2535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:37 smithi016 conmon[32727]: 2022-03-21T18:36:37.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:38 smithi079 ceph-mon[23158]: pgmap v2535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4441 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:38 smithi016 conmon[32727]: 2022-03-21T18:36:38.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4441 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:40 smithi016 ceph-mon[22010]: pgmap v2536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:39 smithi016 conmon[32727]: 2022-03-21T18:36:39.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:40 smithi079 ceph-mon[23158]: pgmap v2536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:40 smithi016 conmon[32727]: 2022-03-21T18:36:40.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:42 smithi016 ceph-mon[22010]: pgmap v2537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:41 smithi016 conmon[32727]: 2022-03-21T18:36:41.943+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:42 smithi079 ceph-mon[23158]: pgmap v2537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:42 smithi016 conmon[32727]: 2022-03-21T18:36:42.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:44 smithi016 ceph-mon[22010]: pgmap v2538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4446 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:43 smithi016 conmon[32727]: 2022-03-21T18:36:43.920+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:44 smithi079 ceph-mon[23158]: pgmap v2538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4446 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:44 smithi016 conmon[32727]: 2022-03-21T18:36:44.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:46 smithi016 ceph-mon[22010]: pgmap v2539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:45 smithi016 conmon[32727]: 2022-03-21T18:36:45.904+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:46 smithi079 ceph-mon[23158]: pgmap v2539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:46 smithi016 conmon[32727]: 2022-03-21T18:36:46.893+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:48 smithi016 ceph-mon[22010]: pgmap v2540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:47 smithi016 conmon[32727]: 2022-03-21T18:36:47.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:48 smithi079 ceph-mon[23158]: pgmap v2540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4451 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:48 smithi016 conmon[32727]: 2022-03-21T18:36:48.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4451 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:50 smithi016 ceph-mon[22010]: pgmap v2541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:49 smithi016 conmon[32727]: 2022-03-21T18:36:49.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:50 smithi079 ceph-mon[23158]: pgmap v2541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:50 smithi016 conmon[32727]: 2022-03-21T18:36:50.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:52 smithi016 ceph-mon[22010]: pgmap v2542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:51 smithi016 conmon[32727]: 2022-03-21T18:36:51.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:52 smithi079 ceph-mon[23158]: pgmap v2542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:52 smithi016 conmon[32727]: 2022-03-21T18:36:52.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:54 smithi016 ceph-mon[22010]: pgmap v2543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4456 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:54 smithi016 conmon[32727]: 2022-03-21T18:36:54.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:54 smithi079 ceph-mon[23158]: pgmap v2543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4456 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:55.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:55.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:55 smithi016 conmon[32727]: 2022-03-21T18:36:55.051+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:56 smithi016 ceph-mon[22010]: pgmap v2544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:56 smithi016 conmon[32727]: 2022-03-21T18:36:56.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:56 smithi079 ceph-mon[23158]: pgmap v2544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:36:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:56 smithi016 conmon[32727]: 2022-03-21T18:36:56.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:58 smithi016 ceph-mon[22010]: pgmap v2545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:57 smithi016 conmon[32727]: 2022-03-21T18:36:57.974+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:58 smithi079 ceph-mon[23158]: pgmap v2545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:36:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:58 smithi016 conmon[32727]: 2022-03-21T18:36:58.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:36:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:36:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:36:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:36:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:36:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:36:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:36:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:36:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:36:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:36:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:36:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:36:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:36:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:36:59 smithi016 conmon[32727]: 2022-03-21T18:36:59.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:00 smithi016 ceph-mon[22010]: pgmap v2546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:00 smithi079 ceph-mon[23158]: pgmap v2546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:01 smithi016 conmon[32727]: 2022-03-21T18:37:01.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:02 smithi016 ceph-mon[22010]: pgmap v2547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:01 smithi016 conmon[32727]: 2022-03-21T18:37:01.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:02 smithi079 ceph-mon[23158]: pgmap v2547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:02 smithi016 conmon[32727]: 2022-03-21T18:37:02.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:04 smithi016 ceph-mon[22010]: pgmap v2548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:03 smithi016 conmon[32727]: 2022-03-21T18:37:03.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:04 smithi079 ceph-mon[23158]: pgmap v2548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:04 smithi016 conmon[32727]: 2022-03-21T18:37:04.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:06.184 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:05 smithi016 conmon[32727]: 2022-03-21T18:37:05.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:06 smithi079 ceph-mon[23158]: pgmap v2549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:06 smithi016 ceph-mon[22010]: pgmap v2549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:06 smithi016 conmon[32727]: 2022-03-21T18:37:06.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:07 smithi016 conmon[32727]: 2022-03-21T18:37:07.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:08 smithi079 ceph-mon[23158]: pgmap v2550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:08 smithi016 ceph-mon[22010]: pgmap v2550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:08 smithi016 conmon[32727]: 2022-03-21T18:37:08.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:09 smithi016 conmon[32727]: 2022-03-21T18:37:09.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:10 smithi079 ceph-mon[23158]: pgmap v2551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:10 smithi016 ceph-mon[22010]: pgmap v2551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:10 smithi016 conmon[32727]: 2022-03-21T18:37:10.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:12.205 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:11 smithi016 conmon[32727]: 2022-03-21T18:37:11.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:12 smithi079 ceph-mon[23158]: pgmap v2552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:12 smithi016 ceph-mon[22010]: pgmap v2552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:13.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:12 smithi016 conmon[32727]: 2022-03-21T18:37:12.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:13 smithi016 conmon[32727]: 2022-03-21T18:37:13.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:14 smithi079 ceph-mon[23158]: pgmap v2553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4476 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:14 smithi016 ceph-mon[22010]: pgmap v2553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4476 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:15.210 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:14 smithi016 conmon[32727]: 2022-03-21T18:37:14.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:15.549 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:16.214 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:15 smithi016 conmon[32727]: 2022-03-21T18:37:15.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:16 smithi079 ceph-mon[23158]: pgmap v2554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:16 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:37:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:16 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:37:16.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:16 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:37:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:16.795 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:16 smithi016 ceph-mon[22010]: pgmap v2554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:16.795 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:16 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:37:16.796 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:16 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:37:16.796 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:16 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:37:17.216 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:16 smithi016 conmon[32727]: 2022-03-21T18:37:16.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:17 smithi016 conmon[32727]: 2022-03-21T18:37:17.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:18 smithi079 ceph-mon[23158]: pgmap v2555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:18.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:18.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:18 smithi016 ceph-mon[22010]: pgmap v2555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:18.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: 2022-03-21T18:37:18.605+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:18.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:37:18.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:18.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: yield 2022-03-21T18:37:18.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:18.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:18.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:18.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:37:18.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:18.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:37:18.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:18.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:37:18.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:18.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:37:18.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:18.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:37:18.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:18.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:18.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:19.223 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:18 smithi016 conmon[32727]: 2022-03-21T18:37:18.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:19 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:37:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:37:19.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:19.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:19.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:19.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:37:19.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:19.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:37:19.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:19.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:37:19.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:19.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:19.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:19.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:37:19.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:19.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:19.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:19.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:19 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:37:19.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4481 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:19 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:37:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:19.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:37:19.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:19.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:19.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:19.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:37:19.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:19.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:37:19.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:19.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:37:19.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:19.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:19.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:19.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:37:19.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:19.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:19.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:19.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:19 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:37:19.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4481 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:19 smithi016 conmon[32727]: 2022-03-21T18:37:19.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:20 smithi016 ceph-mon[22010]: pgmap v2556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:20 smithi079 ceph-mon[23158]: pgmap v2556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:20 smithi016 conmon[32727]: 2022-03-21T18:37:20.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:22.001 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 2022-03-21T18:37:21.677+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.001 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:37:22.002 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.002 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: yield 2022-03-21T18:37:22.002 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.003 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.003 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.003 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:37:22.004 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.004 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:37:22.004 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:37:22.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.005 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.006 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.006 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:37:22.006 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.007 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 2022-03-21T18:37:21.678+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:37:22.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:37:22.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.008 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: yield 2022-03-21T18:37:22.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.009 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:37:22.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.011 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:37:22.011 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.011 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:37:22.012 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.012 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.012 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.013 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:37:22.013 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.013 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.014 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.014 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 2022-03-21T18:37:22.014 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:37:22.015 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: 2022-03-21T18:37:22.015 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:37:22.015 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:37:22.016 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:37:22.016 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:37:22.016 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:37:22.017 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:37:22.017 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:37:22.017 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:37:22.018 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:37:22.018 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:37:22.018 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:37:22.019 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:37:22.019 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:37:22.020 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:37:22.020 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:37:22.020 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:37:22.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:37:22.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:37:22.021 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:37:22.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:37:22.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:37:22.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:37:22.023 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise 2022-03-21T18:37:22.023 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:37:22.023 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:37:22.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:37:22.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:37:22.024 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:21 smithi016 conmon[32727]: 2022-03-21T18:37:21.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:22 smithi016 ceph-mon[22010]: pgmap v2557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:22 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:37:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:37:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:37:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:37:22.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:37:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:22 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:37:22.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:37:22.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:37:22.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:37:22.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:37:22.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:37:22.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:37:22.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:37:22.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:37:22.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:37:22.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:37:22.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:37:22.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:37:22.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:37:22.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:37:22.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:37:22.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:37:22.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:37:22.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:37:22.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:37:22.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:37:22.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:37:22.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:37:22.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:37:22.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:37:22.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:37:22.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:37:22.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:37:22.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:37:22.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:37:22.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:37:22.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:37:22.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:37:22.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:37:22.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:37:22.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:22 smithi079 ceph-mon[23158]: pgmap v2557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:22 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:37:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:37:22.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:37:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:37:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:37:22.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:22 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:37:22.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:37:22.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:37:22.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:37:22.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:37:22.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:37:22.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:37:22.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:37:22.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:37:22.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:37:22.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:37:22.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:37:22.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:37:22.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:37:22.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:37:22.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:37:22.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:37:22.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:22.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:37:22.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:37:22.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:37:22.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:37:22.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:37:22.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:37:22.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:37:22.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:37:22.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:37:22.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:37:22.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:37:22.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:37:22.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:37:22.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:37:22.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:37:22.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:37:22.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:37:22.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:37:22.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:37:22.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:37:22.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:37:22.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:37:22.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:37:22.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:37:22.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:37:22.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:37:22.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:37:22.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:37:22.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:37:22.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:37:22.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:37:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:23 smithi016 conmon[32727]: 2022-03-21T18:37:23.048+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:24 smithi016 conmon[32727]: 2022-03-21T18:37:24.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:24 smithi016 ceph-mon[22010]: pgmap v2558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:24 smithi079 ceph-mon[23158]: pgmap v2558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:25.244 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:24 smithi016 conmon[32727]: 2022-03-21T18:37:24.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:26.249 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:25 smithi016 conmon[32727]: 2022-03-21T18:37:25.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:26 smithi016 ceph-mon[22010]: pgmap v2559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:26 smithi079 ceph-mon[23158]: pgmap v2559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:27.250 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:26 smithi016 conmon[32727]: 2022-03-21T18:37:26.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:28.257 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:28 smithi016 conmon[32727]: 2022-03-21T18:37:28.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:28 smithi016 ceph-mon[22010]: pgmap v2560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:28 smithi079 ceph-mon[23158]: pgmap v2560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:29 smithi016 conmon[32727]: 2022-03-21T18:37:29.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:30 smithi016 conmon[32727]: 2022-03-21T18:37:30.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:30 smithi016 ceph-mon[22010]: pgmap v2561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:30 smithi079 ceph-mon[23158]: pgmap v2561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:31 smithi016 conmon[32727]: 2022-03-21T18:37:31.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:32.269 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:31 smithi016 conmon[32727]: 2022-03-21T18:37:31.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:32 smithi016 ceph-mon[22010]: pgmap v2562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:32 smithi079 ceph-mon[23158]: pgmap v2562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:33.270 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:32 smithi016 conmon[32727]: 2022-03-21T18:37:32.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:34.277 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:33 smithi016 conmon[32727]: 2022-03-21T18:37:33.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:34 smithi016 ceph-mon[22010]: pgmap v2563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:34 smithi079 ceph-mon[23158]: pgmap v2563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:35.279 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:34 smithi016 conmon[32727]: 2022-03-21T18:37:34.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:36.284 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:35 smithi016 conmon[32727]: 2022-03-21T18:37:35.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:36 smithi016 ceph-mon[22010]: pgmap v2564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:36 smithi079 ceph-mon[23158]: pgmap v2564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:37.286 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:36 smithi016 conmon[32727]: 2022-03-21T18:37:36.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:38.291 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:37 smithi016 conmon[32727]: 2022-03-21T18:37:37.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:38 smithi016 ceph-mon[22010]: pgmap v2565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:38 smithi079 ceph-mon[23158]: pgmap v2565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:39.292 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:38 smithi016 conmon[32727]: 2022-03-21T18:37:38.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:40.297 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:39 smithi016 conmon[32727]: 2022-03-21T18:37:39.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:40 smithi016 ceph-mon[22010]: pgmap v2566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:40 smithi079 ceph-mon[23158]: pgmap v2566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:41.299 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:40 smithi016 conmon[32727]: 2022-03-21T18:37:40.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:42.304 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:41 smithi016 conmon[32727]: 2022-03-21T18:37:41.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:42 smithi016 ceph-mon[22010]: pgmap v2567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:42 smithi079 ceph-mon[23158]: pgmap v2567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:43.305 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:42 smithi016 conmon[32727]: 2022-03-21T18:37:42.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:44.312 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:43 smithi016 conmon[32727]: 2022-03-21T18:37:43.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:44 smithi016 ceph-mon[22010]: pgmap v2568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:44 smithi079 ceph-mon[23158]: pgmap v2568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:45.314 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:44 smithi016 conmon[32727]: 2022-03-21T18:37:44.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:46.319 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:45 smithi016 conmon[32727]: 2022-03-21T18:37:45.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:46 smithi016 ceph-mon[22010]: pgmap v2569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:46 smithi079 ceph-mon[23158]: pgmap v2569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:47.320 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:46 smithi016 conmon[32727]: 2022-03-21T18:37:46.950+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:48.325 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:47 smithi016 conmon[32727]: 2022-03-21T18:37:47.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:48 smithi016 ceph-mon[22010]: pgmap v2570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:48 smithi079 ceph-mon[23158]: pgmap v2570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:49.326 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:48 smithi016 conmon[32727]: 2022-03-21T18:37:48.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:49 smithi016 conmon[32727]: 2022-03-21T18:37:49.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:50 smithi016 ceph-mon[22010]: pgmap v2571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:50 smithi079 ceph-mon[23158]: pgmap v2571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:51.333 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:50 smithi016 conmon[32727]: 2022-03-21T18:37:50.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:52.338 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:51 smithi016 conmon[32727]: 2022-03-21T18:37:51.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:52 smithi016 ceph-mon[22010]: pgmap v2572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:52 smithi079 ceph-mon[23158]: pgmap v2572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:52 smithi016 conmon[32727]: 2022-03-21T18:37:52.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:53 smithi016 conmon[32727]: 2022-03-21T18:37:53.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:54 smithi016 ceph-mon[22010]: pgmap v2573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:54 smithi079 ceph-mon[23158]: pgmap v2573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:54 smithi016 conmon[32727]: 2022-03-21T18:37:54.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:55 smithi016 conmon[32727]: 2022-03-21T18:37:55.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:56 smithi016 ceph-mon[22010]: pgmap v2574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:56 smithi079 ceph-mon[23158]: pgmap v2574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:37:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:56 smithi016 conmon[32727]: 2022-03-21T18:37:56.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:57 smithi016 conmon[32727]: 2022-03-21T18:37:57.788+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: pgmap v2575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:37:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:37:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:37:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:37:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: pgmap v2575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:37:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:37:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:37:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:37:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:37:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:58 smithi016 conmon[32727]: 2022-03-21T18:37:58.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:37:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:37:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:37:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:37:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:37:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:37:59 smithi016 conmon[32727]: 2022-03-21T18:37:59.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:00 smithi016 ceph-mon[22010]: pgmap v2576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:00 smithi079 ceph-mon[23158]: pgmap v2576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:00 smithi016 conmon[32727]: 2022-03-21T18:38:00.765+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:01 smithi016 conmon[32727]: 2022-03-21T18:38:01.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:02 smithi016 ceph-mon[22010]: pgmap v2577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:02 smithi079 ceph-mon[23158]: pgmap v2577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:02 smithi016 conmon[32727]: 2022-03-21T18:38:02.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:03 smithi016 conmon[32727]: 2022-03-21T18:38:03.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:04 smithi016 ceph-mon[22010]: pgmap v2578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:04 smithi079 ceph-mon[23158]: pgmap v2578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:04 smithi016 conmon[32727]: 2022-03-21T18:38:04.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:05 smithi016 conmon[32727]: 2022-03-21T18:38:05.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:06 smithi016 ceph-mon[22010]: pgmap v2579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:06 smithi079 ceph-mon[23158]: pgmap v2579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:06 smithi016 conmon[32727]: 2022-03-21T18:38:06.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:07 smithi016 conmon[32727]: 2022-03-21T18:38:07.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:08 smithi016 ceph-mon[22010]: pgmap v2580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:08 smithi079 ceph-mon[23158]: pgmap v2580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:08 smithi016 conmon[32727]: 2022-03-21T18:38:08.759+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:09 smithi016 conmon[32727]: 2022-03-21T18:38:09.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:10 smithi016 ceph-mon[22010]: pgmap v2581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:10 smithi079 ceph-mon[23158]: pgmap v2581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:10 smithi016 conmon[32727]: 2022-03-21T18:38:10.722+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:12.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:11 smithi016 conmon[32727]: 2022-03-21T18:38:11.677+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:12 smithi016 ceph-mon[22010]: pgmap v2582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:12 smithi079 ceph-mon[23158]: pgmap v2582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:12 smithi016 conmon[32727]: 2022-03-21T18:38:12.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:13 smithi016 conmon[32727]: 2022-03-21T18:38:13.726+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:14 smithi016 ceph-mon[22010]: pgmap v2583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:14 smithi079 ceph-mon[23158]: pgmap v2583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:14 smithi016 conmon[32727]: 2022-03-21T18:38:14.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:15 smithi016 conmon[32727]: 2022-03-21T18:38:15.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:16.717 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:16.718 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:16 smithi016 ceph-mon[22010]: pgmap v2584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:16 smithi079 ceph-mon[23158]: pgmap v2584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:16 smithi016 conmon[32727]: 2022-03-21T18:38:16.716+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:17.701 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:17 smithi016 conmon[32727]: 2022-03-21T18:38:17.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:18 smithi079 ceph-mon[23158]: pgmap v2585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:18.733 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:18.734 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:18 smithi016 ceph-mon[22010]: pgmap v2585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:18 smithi016 conmon[32727]: 2022-03-21T18:38:18.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:19.739 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:19.739 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:19 smithi016 conmon[32727]: 2022-03-21T18:38:19.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:20.700 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:20.701 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:20 smithi016 ceph-mon[22010]: pgmap v2586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:20 smithi079 ceph-mon[23158]: pgmap v2586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:20 smithi016 conmon[32727]: 2022-03-21T18:38:20.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:21.724 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:21 smithi016 conmon[32727]: 2022-03-21T18:38:21.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:21.724 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:22 smithi079 ceph-mon[23158]: pgmap v2587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:38:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:38:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:38:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:22 smithi016 conmon[32727]: 2022-03-21T18:38:22.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:22 smithi016 ceph-mon[22010]: pgmap v2587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:38:22.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:38:22.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:38:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:23 smithi016 conmon[32727]: 2022-03-21T18:38:23.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:24 smithi079 ceph-mon[23158]: pgmap v2588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:24.783 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[32727]: 2022-03-21T18:38:24.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:24.784 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:24.784 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:24 smithi016 ceph-mon[22010]: pgmap v2588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:24.784 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:24.785 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:25.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: 2022-03-21T18:38:24.781+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:38:25.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:25.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: yield 2022-03-21T18:38:25.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:25.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:25.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:25.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:38:25.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:25.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:38:25.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:25.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:38:25.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:25.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:38:25.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:25.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:38:25.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:25.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:25.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:25 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:38:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:38:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:38:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:38:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:25.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:38:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:25.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:38:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:25 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:38:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:25 smithi016 conmon[32727]: 2022-03-21T18:38:25.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:25 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:38:25.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:25.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:38:25.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:25.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:25.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:25.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:38:25.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:25.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:38:25.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:25.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:38:25.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:25.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:25.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:25.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:38:25.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:25.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:25.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:25.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:25 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:38:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:26 smithi079 ceph-mon[23158]: pgmap v2589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:26 smithi016 ceph-mon[22010]: pgmap v2589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:26 smithi016 conmon[32727]: 2022-03-21T18:38:26.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:27.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:27.856 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[32727]: 2022-03-21T18:38:27.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:28.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 2022-03-21T18:38:27.853+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:28.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:38:28.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:28.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: yield 2022-03-21T18:38:28.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:28.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:28.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:28.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:38:28.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:28.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:38:28.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:28.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:38:28.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:28.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:38:28.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:28.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:38:28.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:28.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:28.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:28.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 2022-03-21T18:38:27.854+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:38:28.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:38:28.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:28.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: yield 2022-03-21T18:38:28.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:28.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:28.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:28.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:38:28.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:28.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:38:28.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:28.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:38:28.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:28.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:38:28.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:28.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:38:28.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:28.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:28.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:28.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 2022-03-21T18:38:28.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:38:28.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: 2022-03-21T18:38:28.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:38:28.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:38:28.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:38:28.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:38:28.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:38:28.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:38:28.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:38:28.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:38:28.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:38:28.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:38:28.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:38:28.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:38:28.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:38:28.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:38:28.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:38:28.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:38:28.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:38:28.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:38:28.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:38:28.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:38:28.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:38:28.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:38:28.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise 2022-03-21T18:38:28.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:38:28.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:38:28.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:38:28.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:38:28.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:38:27 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:28 smithi079 ceph-mon[23158]: pgmap v2590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:28 smithi016 ceph-mon[22010]: pgmap v2590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:28 smithi016 conmon[32727]: 2022-03-21T18:38:28.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:38:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:38:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:38:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:38:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:29.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:38:29.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:29.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:29.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:29 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:38:29.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:29.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:38:29.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:29.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:29.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:29.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:38:29.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:29.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:38:29.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:29.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:38:29.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:29.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:29.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:29.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:38:29.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:29.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:29.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:38:29.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:38:29.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:38:29.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:38:29.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:38:29.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:38:29.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:38:29.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:38:29.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:38:29.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:38:29.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:38:29.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:38:29.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:38:29.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:38:29.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:38:29.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:38:29.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:38:29.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:38:29.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:38:29.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:38:29.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:38:29.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:38:29.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:38:29.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:38:29.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:38:29.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:38:29.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:38:29.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:38:29.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:38:29.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:29.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:29 smithi016 conmon[32727]: 2022-03-21T18:38:29.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:38:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:38:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:38:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:38:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:38:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:29 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:38:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:38:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:38:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:38:29.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:38:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:38:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:38:29.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:38:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:38:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:38:29.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:38:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:38:29.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:38:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:38:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:38:29.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:38:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:38:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:38:29.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:38:29.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:38:29.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:38:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:38:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:38:29.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:38:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:38:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:38:29.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:38:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:38:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:38:29.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:38:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:38:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:38:29.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:38:29.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:38:29.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:38:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:38:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:38:29.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:38:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:38:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:38:29.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:38:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:38:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:38:29.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:38:29.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:38:29.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:38:29.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:38:29.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:38:29.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:29.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:30 smithi079 ceph-mon[23158]: pgmap v2591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:30 smithi016 conmon[32727]: 2022-03-21T18:38:30.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:30 smithi016 ceph-mon[22010]: pgmap v2591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:31 smithi016 conmon[32727]: 2022-03-21T18:38:31.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:32 smithi079 ceph-mon[23158]: pgmap v2592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:32 smithi016 conmon[32727]: 2022-03-21T18:38:32.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:32 smithi016 ceph-mon[22010]: pgmap v2592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:33 smithi016 conmon[32727]: 2022-03-21T18:38:33.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:34 smithi079 ceph-mon[23158]: pgmap v2593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:34 smithi016 conmon[32727]: 2022-03-21T18:38:34.522+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:34 smithi016 ceph-mon[22010]: pgmap v2593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:35 smithi016 conmon[32727]: 2022-03-21T18:38:35.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:36 smithi016 ceph-mon[22010]: pgmap v2594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:36 smithi016 conmon[32727]: 2022-03-21T18:38:36.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:36 smithi079 ceph-mon[23158]: pgmap v2594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:37 smithi016 conmon[32727]: 2022-03-21T18:38:37.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:38 smithi016 ceph-mon[22010]: pgmap v2595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:38 smithi016 conmon[32727]: 2022-03-21T18:38:38.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:38 smithi079 ceph-mon[23158]: pgmap v2595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:39 smithi016 conmon[32727]: 2022-03-21T18:38:39.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:40 smithi016 ceph-mon[22010]: pgmap v2596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:40 smithi016 conmon[32727]: 2022-03-21T18:38:40.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:40 smithi079 ceph-mon[23158]: pgmap v2596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:41 smithi016 conmon[32727]: 2022-03-21T18:38:41.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:42 smithi016 conmon[32727]: 2022-03-21T18:38:42.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:42 smithi016 ceph-mon[22010]: pgmap v2597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:42 smithi079 ceph-mon[23158]: pgmap v2597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:43 smithi016 conmon[32727]: 2022-03-21T18:38:43.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:44 smithi016 ceph-mon[22010]: pgmap v2598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:44 smithi016 conmon[32727]: 2022-03-21T18:38:44.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:44 smithi079 ceph-mon[23158]: pgmap v2598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:45 smithi016 conmon[32727]: 2022-03-21T18:38:45.494+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:46 smithi016 ceph-mon[22010]: pgmap v2599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:46 smithi016 conmon[32727]: 2022-03-21T18:38:46.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:46 smithi079 ceph-mon[23158]: pgmap v2599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:47 smithi016 conmon[32727]: 2022-03-21T18:38:47.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:48 smithi016 ceph-mon[22010]: pgmap v2600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:48 smithi016 conmon[32727]: 2022-03-21T18:38:48.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:48 smithi079 ceph-mon[23158]: pgmap v2600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:49 smithi016 conmon[32727]: 2022-03-21T18:38:49.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:50 smithi016 ceph-mon[22010]: pgmap v2601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:50 smithi016 conmon[32727]: 2022-03-21T18:38:50.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:50 smithi079 ceph-mon[23158]: pgmap v2601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:51 smithi016 conmon[32727]: 2022-03-21T18:38:51.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:52 smithi016 ceph-mon[22010]: pgmap v2602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:52 smithi016 conmon[32727]: 2022-03-21T18:38:52.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:52 smithi079 ceph-mon[23158]: pgmap v2602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:53 smithi016 conmon[32727]: 2022-03-21T18:38:53.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:54 smithi016 ceph-mon[22010]: pgmap v2603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:54 smithi016 conmon[32727]: 2022-03-21T18:38:54.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:54 smithi079 ceph-mon[23158]: pgmap v2603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:55 smithi016 conmon[32727]: 2022-03-21T18:38:55.670+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:56 smithi016 ceph-mon[22010]: pgmap v2604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:56 smithi016 conmon[32727]: 2022-03-21T18:38:56.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:56 smithi079 ceph-mon[23158]: pgmap v2604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:38:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:57 smithi016 conmon[32727]: 2022-03-21T18:38:57.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:58 smithi016 conmon[32727]: 2022-03-21T18:38:58.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: pgmap v2605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:38:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:38:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:38:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:38:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: pgmap v2605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:38:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:38:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:38:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:38:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:38:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:38:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:38:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:38:59.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:38:59 smithi016 conmon[32727]: 2022-03-21T18:38:59.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:38:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:38:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:00 smithi016 ceph-mon[22010]: pgmap v2606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:00 smithi016 conmon[32727]: 2022-03-21T18:39:00.622+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:00 smithi079 ceph-mon[23158]: pgmap v2606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:01 smithi016 conmon[32727]: 2022-03-21T18:39:01.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:02 smithi016 ceph-mon[22010]: pgmap v2607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:02 smithi016 conmon[32727]: 2022-03-21T18:39:02.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:02 smithi079 ceph-mon[23158]: pgmap v2607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:03 smithi016 conmon[32727]: 2022-03-21T18:39:03.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:04 smithi016 ceph-mon[22010]: pgmap v2608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:04 smithi016 conmon[32727]: 2022-03-21T18:39:04.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:04 smithi079 ceph-mon[23158]: pgmap v2608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:05 smithi016 conmon[32727]: 2022-03-21T18:39:05.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:06 smithi016 ceph-mon[22010]: pgmap v2609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:06 smithi016 conmon[32727]: 2022-03-21T18:39:06.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:06 smithi079 ceph-mon[23158]: pgmap v2609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:07 smithi016 conmon[32727]: 2022-03-21T18:39:07.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:08 smithi016 ceph-mon[22010]: pgmap v2610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:08 smithi016 conmon[32727]: 2022-03-21T18:39:08.614+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:08 smithi079 ceph-mon[23158]: pgmap v2610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:09 smithi016 conmon[32727]: 2022-03-21T18:39:09.614+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:10 smithi016 ceph-mon[22010]: pgmap v2611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:10 smithi016 conmon[32727]: 2022-03-21T18:39:10.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:10 smithi079 ceph-mon[23158]: pgmap v2611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:11 smithi016 conmon[32727]: 2022-03-21T18:39:11.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:12 smithi016 ceph-mon[22010]: pgmap v2612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:12 smithi016 conmon[32727]: 2022-03-21T18:39:12.671+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:12 smithi079 ceph-mon[23158]: pgmap v2612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:13 smithi016 conmon[32727]: 2022-03-21T18:39:13.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:14 smithi016 ceph-mon[22010]: pgmap v2613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:14 smithi016 conmon[32727]: 2022-03-21T18:39:14.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:14 smithi079 ceph-mon[23158]: pgmap v2613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:15 smithi016 conmon[32727]: 2022-03-21T18:39:15.687+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:16 smithi016 ceph-mon[22010]: pgmap v2614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:16 smithi016 conmon[32727]: 2022-03-21T18:39:16.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:16 smithi079 ceph-mon[23158]: pgmap v2614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:17 smithi016 conmon[32727]: 2022-03-21T18:39:17.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:18 smithi016 ceph-mon[22010]: pgmap v2615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:18 smithi016 conmon[32727]: 2022-03-21T18:39:18.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:18 smithi079 ceph-mon[23158]: pgmap v2615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:19 smithi016 conmon[32727]: 2022-03-21T18:39:19.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:20 smithi016 ceph-mon[22010]: pgmap v2616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:20 smithi016 conmon[32727]: 2022-03-21T18:39:20.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:20 smithi079 ceph-mon[23158]: pgmap v2616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:21 smithi016 conmon[32727]: 2022-03-21T18:39:21.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:22 smithi016 ceph-mon[22010]: pgmap v2617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:22 smithi016 conmon[32727]: 2022-03-21T18:39:22.773+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:22 smithi079 ceph-mon[23158]: pgmap v2617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:23 smithi016 conmon[32727]: 2022-03-21T18:39:23.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:24 smithi016 ceph-mon[22010]: pgmap v2618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:24 smithi016 conmon[32727]: 2022-03-21T18:39:24.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:24 smithi079 ceph-mon[23158]: pgmap v2618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:25 smithi016 conmon[32727]: 2022-03-21T18:39:25.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:26 smithi016 conmon[32727]: 2022-03-21T18:39:26.772+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:26 smithi016 ceph-mon[22010]: pgmap v2619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:26 smithi079 ceph-mon[23158]: pgmap v2619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:27.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:27.901 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:27 smithi016 conmon[32727]: 2022-03-21T18:39:27.736+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:27.902 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[32727]: 2022-03-21T18:39:28.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:28 smithi016 ceph-mon[22010]: pgmap v2620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:39:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:39:28.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:39:28.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: 2022-03-21T18:39:28.461+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:28.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:39:28.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:28.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: yield 2022-03-21T18:39:28.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:28.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:28.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:28.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:39:28.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:28.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:39:28.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:28.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:39:28.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:28.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:39:28.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:28.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:39:28.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:28.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:28.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:28 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:28 smithi079 ceph-mon[23158]: pgmap v2620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:39:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:39:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:39:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:39:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:29 smithi016 conmon[32727]: 2022-03-21T18:39:29.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:39:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:39:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:29.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:39:29.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:39:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:29.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:39:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:29.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:39:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:29.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:29.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:29 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:39:29.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:39:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:39:29.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:29.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:39:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:39:29.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:39:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:29.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:39:29.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:29.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:29 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:39:29.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:30.747 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:30.748 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:30 smithi016 ceph-mon[22010]: pgmap v2621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:30.748 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:30 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:39:30.749 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:30 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:39:30.905 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:30.905 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:30 smithi079 ceph-mon[23158]: pgmap v2621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:30.905 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:30 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:39:30.906 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:30 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:39:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:30 smithi016 conmon[32727]: 2022-03-21T18:39:30.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[32727]: 2022-03-21T18:39:31.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.920 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.921 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 2022-03-21T18:39:31.533+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:31.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:39:31.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:31.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: yield 2022-03-21T18:39:31.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:31.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:31.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:31.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:39:31.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:31.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:39:31.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:31.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:39:31.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:31.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:39:31.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:31.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:39:31.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:31.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:31.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:31.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 2022-03-21T18:39:31.534+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:39:31.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:39:31.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:31.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: yield 2022-03-21T18:39:31.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:31.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:31.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:31.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:39:31.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:31.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:39:31.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:31.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:39:31.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:31.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:39:31.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:31.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:39:31.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:31.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:31.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:31.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 2022-03-21T18:39:31.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:39:31.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: 2022-03-21T18:39:31.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:39:31.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:39:31.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:39:31.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:39:31.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:39:31.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:39:31.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:39:31.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:39:31.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:39:31.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:39:31.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:39:31.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:39:31.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:39:31.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:39:31.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:39:31.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:39:31.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:39:31.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:39:31.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:39:31.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:39:31.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:39:31.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:39:31.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise 2022-03-21T18:39:31.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:39:31.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:39:31.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:39:31.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:39:31.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:39:31 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:39:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:32 smithi016 conmon[32727]: 2022-03-21T18:39:32.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:39:32.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:39:32.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:39:32.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:39:32.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: pgmap v2622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:32.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:32.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:39:32.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:32.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:32.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:32.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:39:32.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:39:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:32.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:39:32.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:32.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:32.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:32.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:39:32.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:32 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:39:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:32.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:39:32.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:32.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:39:32.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:39:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:32.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:39:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:32.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:32.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:39:32.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:32.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:32.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:39:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:39:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:39:32.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:39:32.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:39:32.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:39:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:39:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:39:32.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:39:32.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:39:32.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:39:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:39:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:39:32.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:39:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:39:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:39:32.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:39:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:39:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:39:32.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:39:32.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:39:32.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:39:32.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:39:32.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:39:32.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:39:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:39:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:39:32.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:39:32.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:39:32.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:39:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:39:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:39:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:39:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: pgmap v2622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:39:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:32.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:39:32.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:32.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:39:32.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:39:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:32.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:39:32.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:32 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:39:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:39:32.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:39:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:39:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:39:32.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:39:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:39:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:39:32.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:39:32.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:39:32.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:39:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:39:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:39:32.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:39:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:39:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:39:32.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:39:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:39:32.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:39:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:39:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:39:32.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:39:32.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:39:32.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:39:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:39:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:39:32.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:39:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:39:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:39:32.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:39:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:39:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:39:32.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:39:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:39:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:39:32.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:39:32.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:39:32.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:39:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:39:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:39:32.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:39:32.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:39:32.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:39:32.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:39:32.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:39:32.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:39:32.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:39:33.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:39:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:33 smithi016 conmon[32727]: 2022-03-21T18:39:33.777+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:34 smithi016 conmon[32727]: 2022-03-21T18:39:34.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:34 smithi016 ceph-mon[22010]: pgmap v2623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:34 smithi079 ceph-mon[23158]: pgmap v2623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:35 smithi016 conmon[32727]: 2022-03-21T18:39:35.828+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:36 smithi016 ceph-mon[22010]: pgmap v2624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:36 smithi016 conmon[32727]: 2022-03-21T18:39:36.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:36 smithi079 ceph-mon[23158]: pgmap v2624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:37 smithi016 conmon[32727]: 2022-03-21T18:39:37.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:38 smithi016 conmon[32727]: 2022-03-21T18:39:38.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:38 smithi016 ceph-mon[22010]: pgmap v2625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:38 smithi079 ceph-mon[23158]: pgmap v2625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:39 smithi016 conmon[32727]: 2022-03-21T18:39:39.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:40 smithi016 ceph-mon[22010]: pgmap v2626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:40 smithi016 conmon[32727]: 2022-03-21T18:39:40.863+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:40 smithi079 ceph-mon[23158]: pgmap v2626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:41 smithi016 conmon[32727]: 2022-03-21T18:39:41.857+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:42 smithi016 conmon[32727]: 2022-03-21T18:39:42.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:42 smithi016 ceph-mon[22010]: pgmap v2627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:42.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:42 smithi079 ceph-mon[23158]: pgmap v2627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:43 smithi016 conmon[32727]: 2022-03-21T18:39:43.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:44 smithi016 ceph-mon[22010]: pgmap v2628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4626 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:44 smithi016 conmon[32727]: 2022-03-21T18:39:44.805+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:44 smithi079 ceph-mon[23158]: pgmap v2628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4626 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:45.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:45 smithi016 conmon[32727]: 2022-03-21T18:39:45.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:46 smithi016 ceph-mon[22010]: pgmap v2629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:46.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:46 smithi016 conmon[32727]: 2022-03-21T18:39:46.805+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:46 smithi079 ceph-mon[23158]: pgmap v2629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:47 smithi016 conmon[32727]: 2022-03-21T18:39:47.785+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:48 smithi079 ceph-mon[23158]: pgmap v2630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:48 smithi016 ceph-mon[22010]: pgmap v2630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:48 smithi016 conmon[32727]: 2022-03-21T18:39:48.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:49 smithi016 conmon[32727]: 2022-03-21T18:39:49.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:50 smithi079 ceph-mon[23158]: pgmap v2631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:50 smithi016 ceph-mon[22010]: pgmap v2631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:50 smithi016 conmon[32727]: 2022-03-21T18:39:50.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:51 smithi016 conmon[32727]: 2022-03-21T18:39:51.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:52 smithi079 ceph-mon[23158]: pgmap v2632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:52 smithi016 ceph-mon[22010]: pgmap v2632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:52 smithi016 conmon[32727]: 2022-03-21T18:39:52.747+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:53 smithi016 conmon[32727]: 2022-03-21T18:39:53.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:54 smithi079 ceph-mon[23158]: pgmap v2633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:54 smithi016 ceph-mon[22010]: pgmap v2633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:39:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:54 smithi016 conmon[32727]: 2022-03-21T18:39:54.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:55 smithi016 conmon[32727]: 2022-03-21T18:39:55.727+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:56 smithi079 ceph-mon[23158]: pgmap v2634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:56 smithi016 ceph-mon[22010]: pgmap v2634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:39:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:56 smithi016 conmon[32727]: 2022-03-21T18:39:56.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:57 smithi016 conmon[32727]: 2022-03-21T18:39:57.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: pgmap v2635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:39:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:39:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:39:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:39:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:58 smithi016 conmon[32727]: 2022-03-21T18:39:58.708+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:39:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: pgmap v2635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:39:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:39:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:39:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:39:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:39:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:39:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:39:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:39:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:39:59 smithi016 conmon[32727]: 2022-03-21T18:39:59.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: pgmap v2636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops 2022-03-21T18:40:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:40:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:40:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:40:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:40:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:40:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops 2022-03-21T18:40:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:00 smithi016 conmon[32727]: 2022-03-21T18:40:00.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: pgmap v2636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops 2022-03-21T18:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:40:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:40:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:40:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:40:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops 2022-03-21T18:40:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:01 smithi016 conmon[32727]: 2022-03-21T18:40:01.756+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:02 smithi079 ceph-mon[23158]: pgmap v2637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:02 smithi016 conmon[32727]: 2022-03-21T18:40:02.765+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:02 smithi016 ceph-mon[22010]: pgmap v2637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:03 smithi016 conmon[32727]: 2022-03-21T18:40:03.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:04 smithi079 ceph-mon[23158]: pgmap v2638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:04 smithi016 conmon[32727]: 2022-03-21T18:40:04.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:04 smithi016 ceph-mon[22010]: pgmap v2638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:05 smithi016 conmon[32727]: 2022-03-21T18:40:05.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:06 smithi079 ceph-mon[23158]: pgmap v2639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:06 smithi016 ceph-mon[22010]: pgmap v2639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:06 smithi016 conmon[32727]: 2022-03-21T18:40:06.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:07 smithi016 conmon[32727]: 2022-03-21T18:40:07.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:08 smithi016 ceph-mon[22010]: pgmap v2640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:08 smithi016 conmon[32727]: 2022-03-21T18:40:08.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:08 smithi079 ceph-mon[23158]: pgmap v2640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:09 smithi016 conmon[32727]: 2022-03-21T18:40:09.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:10 smithi016 ceph-mon[22010]: pgmap v2641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:10 smithi016 conmon[32727]: 2022-03-21T18:40:10.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:10 smithi079 ceph-mon[23158]: pgmap v2641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:11 smithi016 conmon[32727]: 2022-03-21T18:40:11.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:12 smithi016 ceph-mon[22010]: pgmap v2642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:12 smithi016 conmon[32727]: 2022-03-21T18:40:12.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:12 smithi079 ceph-mon[23158]: pgmap v2642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:13 smithi016 conmon[32727]: 2022-03-21T18:40:13.739+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:14 smithi016 ceph-mon[22010]: pgmap v2643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:14 smithi016 conmon[32727]: 2022-03-21T18:40:14.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:14 smithi079 ceph-mon[23158]: pgmap v2643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:15 smithi016 conmon[32727]: 2022-03-21T18:40:15.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:16 smithi016 ceph-mon[22010]: pgmap v2644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:16 smithi016 conmon[32727]: 2022-03-21T18:40:16.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:16 smithi079 ceph-mon[23158]: pgmap v2644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:17 smithi016 conmon[32727]: 2022-03-21T18:40:17.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:18 smithi016 ceph-mon[22010]: pgmap v2645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:18 smithi016 conmon[32727]: 2022-03-21T18:40:18.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:18 smithi079 ceph-mon[23158]: pgmap v2645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:19 smithi016 conmon[32727]: 2022-03-21T18:40:19.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:20 smithi016 conmon[32727]: 2022-03-21T18:40:20.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:20 smithi016 ceph-mon[22010]: pgmap v2646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:20 smithi079 ceph-mon[23158]: pgmap v2646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:21 smithi016 conmon[32727]: 2022-03-21T18:40:21.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:22 smithi016 conmon[32727]: 2022-03-21T18:40:22.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:22 smithi016 ceph-mon[22010]: pgmap v2647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:22 smithi079 ceph-mon[23158]: pgmap v2647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:23 smithi016 conmon[32727]: 2022-03-21T18:40:23.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:24 smithi016 conmon[32727]: 2022-03-21T18:40:24.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:24 smithi016 ceph-mon[22010]: pgmap v2648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4666 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:24 smithi079 ceph-mon[23158]: pgmap v2648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4666 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:25 smithi016 conmon[32727]: 2022-03-21T18:40:25.836+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:26 smithi016 ceph-mon[22010]: pgmap v2649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:26 smithi016 conmon[32727]: 2022-03-21T18:40:26.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:26 smithi079 ceph-mon[23158]: pgmap v2649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:27 smithi016 conmon[32727]: 2022-03-21T18:40:27.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:28 smithi016 conmon[32727]: 2022-03-21T18:40:28.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:28 smithi016 ceph-mon[22010]: pgmap v2650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:28 smithi079 ceph-mon[23158]: pgmap v2650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:29 smithi016 conmon[32727]: 2022-03-21T18:40:29.724+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4671 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4671 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:30 smithi016 ceph-mon[22010]: pgmap v2651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:30 smithi016 conmon[32727]: 2022-03-21T18:40:30.738+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:30 smithi079 ceph-mon[23158]: pgmap v2651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:40:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:40:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:40:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:31 smithi016 conmon[32727]: 2022-03-21T18:40:31.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:40:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:40:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:40:33.020 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:32 smithi016 ceph-mon[22010]: pgmap v2652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:33.020 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:33.021 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:32 smithi016 conmon[32727]: 2022-03-21T18:40:32.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:32 smithi079 ceph-mon[23158]: pgmap v2652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:33 smithi016 conmon[32727]: 2022-03-21T18:40:33.773+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[32727]: 2022-03-21T18:40:34.817+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:34 smithi016 ceph-mon[22010]: pgmap v2653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:34.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:34 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:40:34.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: 2022-03-21T18:40:34.637+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:34.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:40:34.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:34.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: yield 2022-03-21T18:40:34.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:34.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:34.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:34.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:40:34.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:34.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:40:34.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:34.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:40:34.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:34.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:40:34.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:34.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:40:34.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:34.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:34.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:34 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:34 smithi079 ceph-mon[23158]: pgmap v2653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:34 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:40:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:35 smithi016 conmon[32727]: 2022-03-21T18:40:35.784+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:35 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T18:40:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:35 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:40:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:40:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:36.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:36.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:36.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:40:36.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:36.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:40:36.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:36.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:40:36.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:36.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:36.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:36.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:40:36.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:36.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:36.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:36.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:35 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:40:36.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:35 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T18:40:36.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:35 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:36.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:40:36.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:36.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:40:36.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:36.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:36.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:36.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:40:36.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:36.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:40:36.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:36.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:40:36.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:36.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:36.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:36.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:40:36.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:36.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:36.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:36.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:35 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:40:36.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:36 smithi016 ceph-mon[22010]: pgmap v2654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:36 smithi016 conmon[32727]: 2022-03-21T18:40:36.804+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:36 smithi079 ceph-mon[23158]: pgmap v2654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[32727]: 2022-03-21T18:40:37.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:38.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 2022-03-21T18:40:37.709+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:38.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:40:38.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:38.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: yield 2022-03-21T18:40:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:38.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:40:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:38.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:40:38.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:38.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:40:38.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:38.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:40:38.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:40:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:38.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 2022-03-21T18:40:37.710+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:40:38.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:40:38.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:38.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: yield 2022-03-21T18:40:38.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:38.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:38.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:38.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:40:38.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:38.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:40:38.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:38.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:40:38.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:38.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:40:38.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:38.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:40:38.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:38.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:38.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:38.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 2022-03-21T18:40:38.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:40:38.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: 2022-03-21T18:40:38.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:40:38.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:40:38.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:40:38.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:40:38.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:40:38.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:40:38.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:40:38.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:40:38.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:40:38.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:40:38.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:40:38.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:40:38.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:40:38.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:40:38.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:40:38.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:40:38.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:40:38.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:40:38.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:40:38.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:40:38.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:40:38.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:40:38.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise 2022-03-21T18:40:38.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:40:38.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:40:38.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:40:38.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:40:38.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:40:37 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:38 smithi016 conmon[32727]: 2022-03-21T18:40:38.790+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:38 smithi016 ceph-mon[22010]: pgmap v2655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:40:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:40:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:40:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:40:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:40:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:38 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:40:39.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:39.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:40:39.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:39.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:39.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:39.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:40:39.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:39.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:40:39.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:39.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:40:39.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:39.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:39.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:39.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:40:39.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:39.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:39.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:40:39.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:40:39.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:40:39.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:40:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:40:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:40:39.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:40:39.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:40:39.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:40:39.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:40:39.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:40:39.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:40:39.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:40:39.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:40:39.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:40:39.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:40:39.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:40:39.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:40:39.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:40:39.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:40:39.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:40:39.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:40:39.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:40:39.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:40:39.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:40:39.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:40:39.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:40:39.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:40:39.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:40:39.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:38 smithi079 ceph-mon[23158]: pgmap v2655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:40:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:40:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:40:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:40:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:40:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:38 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:40:39.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:40:39.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:40:39.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:40:39.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:40:39.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:40:39.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:40:39.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:40:39.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:40:39.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:40:39.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:40:39.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:40:39.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:40:39.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:40:39.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:40:39.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:40:39.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:40:39.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:40:39.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:40:39.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:40:39.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:40:39.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:40:39.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:40:39.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:40:39.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:40:39.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:40:39.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:40:39.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:40:39.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:40:39.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:40:39.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:40:39.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:40:39.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:40:39.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:40:39.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:40:39.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:40:39.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:40:39.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:40:39.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:40:39.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:40:39.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:40:39.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:40:39.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:40:39.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:40:39.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:40:39.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:40:39.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:40:39.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:40:39.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:39 smithi016 conmon[32727]: 2022-03-21T18:40:39.819+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:40 smithi016 ceph-mon[22010]: pgmap v2656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:40 smithi016 conmon[32727]: 2022-03-21T18:40:40.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:40 smithi079 ceph-mon[23158]: pgmap v2656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:41 smithi016 conmon[32727]: 2022-03-21T18:40:41.832+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:42 smithi016 ceph-mon[22010]: pgmap v2657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:42 smithi016 conmon[32727]: 2022-03-21T18:40:42.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:42 smithi079 ceph-mon[23158]: pgmap v2657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:43 smithi016 conmon[32727]: 2022-03-21T18:40:43.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:44 smithi016 ceph-mon[22010]: pgmap v2658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:44 smithi016 conmon[32727]: 2022-03-21T18:40:44.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:44 smithi079 ceph-mon[23158]: pgmap v2658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:45 smithi016 conmon[32727]: 2022-03-21T18:40:45.872+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:46 smithi016 ceph-mon[22010]: pgmap v2659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:46 smithi016 conmon[32727]: 2022-03-21T18:40:46.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:46 smithi079 ceph-mon[23158]: pgmap v2659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:47 smithi016 conmon[32727]: 2022-03-21T18:40:47.863+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:48 smithi016 conmon[32727]: 2022-03-21T18:40:48.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:48 smithi016 ceph-mon[22010]: pgmap v2660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:48 smithi079 ceph-mon[23158]: pgmap v2660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:49 smithi016 conmon[32727]: 2022-03-21T18:40:49.876+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:50 smithi016 ceph-mon[22010]: pgmap v2661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:50 smithi016 conmon[32727]: 2022-03-21T18:40:50.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:50 smithi079 ceph-mon[23158]: pgmap v2661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:51 smithi016 conmon[32727]: 2022-03-21T18:40:51.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:53 smithi016 conmon[32727]: 2022-03-21T18:40:53.007+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:52 smithi016 ceph-mon[22010]: pgmap v2662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:52 smithi079 ceph-mon[23158]: pgmap v2662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:54 smithi016 conmon[32727]: 2022-03-21T18:40:54.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:54 smithi016 ceph-mon[22010]: pgmap v2663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4696 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:55 smithi016 conmon[32727]: 2022-03-21T18:40:55.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:54 smithi079 ceph-mon[23158]: pgmap v2663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4696 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:40:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:55 smithi016 ceph-mon[22010]: pgmap v2664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:55 smithi016 conmon[32727]: 2022-03-21T18:40:55.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:55 smithi079 ceph-mon[23158]: pgmap v2664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:40:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:56 smithi016 conmon[32727]: 2022-03-21T18:40:56.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:57 smithi016 conmon[32727]: 2022-03-21T18:40:57.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:57 smithi016 ceph-mon[22010]: pgmap v2665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:57 smithi079 ceph-mon[23158]: pgmap v2665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:40:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:58 smithi016 conmon[32727]: 2022-03-21T18:40:58.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:40:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:40:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:40:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:40:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:40:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:40:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:40:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:40:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:40:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:41:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:40:59 smithi016 conmon[32727]: 2022-03-21T18:40:59.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:40:59 smithi016 ceph-mon[22010]: pgmap v2666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:40:59 smithi079 ceph-mon[23158]: pgmap v2666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:01 smithi016 conmon[32727]: 2022-03-21T18:41:01.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:01 smithi079 ceph-mon[23158]: pgmap v2667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:02 smithi016 conmon[32727]: 2022-03-21T18:41:02.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:01 smithi016 ceph-mon[22010]: pgmap v2667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:03 smithi016 conmon[32727]: 2022-03-21T18:41:03.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:03 smithi079 ceph-mon[23158]: pgmap v2668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:03 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:03 smithi016 ceph-mon[22010]: pgmap v2668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:03 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:04 smithi016 conmon[32727]: 2022-03-21T18:41:04.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:05 smithi016 conmon[32727]: 2022-03-21T18:41:05.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:05 smithi016 ceph-mon[22010]: pgmap v2669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:06 smithi016 conmon[32727]: 2022-03-21T18:41:06.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:05 smithi079 ceph-mon[23158]: pgmap v2669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:07 smithi016 conmon[32727]: 2022-03-21T18:41:07.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:08 smithi016 conmon[32727]: 2022-03-21T18:41:08.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:07 smithi016 ceph-mon[22010]: pgmap v2670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:07 smithi079 ceph-mon[23158]: pgmap v2670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:09 smithi016 conmon[32727]: 2022-03-21T18:41:09.047+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:08 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:08 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:10 smithi016 conmon[32727]: 2022-03-21T18:41:10.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:09 smithi016 ceph-mon[22010]: pgmap v2671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:09 smithi079 ceph-mon[23158]: pgmap v2671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:11 smithi016 conmon[32727]: 2022-03-21T18:41:11.026+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:12 smithi016 conmon[32727]: 2022-03-21T18:41:12.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:11 smithi016 ceph-mon[22010]: pgmap v2672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:11 smithi079 ceph-mon[23158]: pgmap v2672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:13 smithi016 conmon[32727]: 2022-03-21T18:41:13.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:14 smithi016 conmon[32727]: 2022-03-21T18:41:14.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:13 smithi016 ceph-mon[22010]: pgmap v2673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:13 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:13 smithi079 ceph-mon[23158]: pgmap v2673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:13 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:15.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:15 smithi016 conmon[32727]: 2022-03-21T18:41:15.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:16 smithi016 conmon[32727]: 2022-03-21T18:41:16.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:16 smithi016 ceph-mon[22010]: pgmap v2674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:16 smithi079 ceph-mon[23158]: pgmap v2674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:17 smithi016 conmon[32727]: 2022-03-21T18:41:17.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:18 smithi016 conmon[32727]: 2022-03-21T18:41:18.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:18 smithi016 ceph-mon[22010]: pgmap v2675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:18 smithi079 ceph-mon[23158]: pgmap v2675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:19 smithi016 conmon[32727]: 2022-03-21T18:41:19.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:20 smithi016 ceph-mon[22010]: pgmap v2676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:20 smithi016 conmon[32727]: 2022-03-21T18:41:20.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:20 smithi079 ceph-mon[23158]: pgmap v2676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:21 smithi016 conmon[32727]: 2022-03-21T18:41:21.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:22 smithi016 ceph-mon[22010]: pgmap v2677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:22 smithi016 conmon[32727]: 2022-03-21T18:41:22.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:22 smithi079 ceph-mon[23158]: pgmap v2677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:22 smithi016 conmon[32727]: 2022-03-21T18:41:22.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:24 smithi016 ceph-mon[22010]: pgmap v2678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:23 smithi016 conmon[32727]: 2022-03-21T18:41:23.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:24 smithi079 ceph-mon[23158]: pgmap v2678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:24 smithi016 conmon[32727]: 2022-03-21T18:41:24.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:26 smithi016 ceph-mon[22010]: pgmap v2679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:25 smithi016 conmon[32727]: 2022-03-21T18:41:25.892+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:26 smithi079 ceph-mon[23158]: pgmap v2679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:26 smithi016 conmon[32727]: 2022-03-21T18:41:26.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:28 smithi016 ceph-mon[22010]: pgmap v2680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:27 smithi016 conmon[32727]: 2022-03-21T18:41:27.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:28 smithi079 ceph-mon[23158]: pgmap v2680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:28 smithi016 conmon[32727]: 2022-03-21T18:41:28.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:30 smithi016 ceph-mon[22010]: pgmap v2681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:29 smithi016 conmon[32727]: 2022-03-21T18:41:29.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:30 smithi079 ceph-mon[23158]: pgmap v2681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:30 smithi016 conmon[32727]: 2022-03-21T18:41:30.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:32.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:32 smithi016 ceph-mon[22010]: pgmap v2682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:32.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:31 smithi016 conmon[32727]: 2022-03-21T18:41:31.974+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:32 smithi079 ceph-mon[23158]: pgmap v2682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:32 smithi016 conmon[32727]: 2022-03-21T18:41:32.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:33 smithi016 conmon[32727]: 2022-03-21T18:41:33.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:34 smithi016 ceph-mon[22010]: pgmap v2683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:34 smithi079 ceph-mon[23158]: pgmap v2683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:34 smithi016 conmon[32727]: 2022-03-21T18:41:34.919+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:36 smithi016 ceph-mon[22010]: pgmap v2684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:35 smithi016 conmon[32727]: 2022-03-21T18:41:35.912+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:36 smithi079 ceph-mon[23158]: pgmap v2684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:36 smithi016 conmon[32727]: 2022-03-21T18:41:36.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:38.076 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:37 smithi016 conmon[32727]: 2022-03-21T18:41:37.974+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:38 smithi016 ceph-mon[22010]: pgmap v2685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:41:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:41:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:41:38.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: 2022-03-21T18:41:38.189+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:38.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:41:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: yield 2022-03-21T18:41:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:38.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:38.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:38.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:41:38.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:38.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:41:38.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:38.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:41:38.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:38.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:41:38.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:38.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:41:38.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:38.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:38.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:38 smithi079 ceph-mon[23158]: pgmap v2685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:41:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:41:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:41:39.201 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:39.202 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:39.202 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:41:39.203 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:39.203 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:41:39.203 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:39.204 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:39.204 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:39.205 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:41:39.205 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:39.205 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:41:39.206 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:39.206 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:41:39.206 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:39.207 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:39.207 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:39.208 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:41:39.208 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:39.209 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:39.209 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:39.209 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:39 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:41:39.210 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:39.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:38 smithi016 conmon[32727]: 2022-03-21T18:41:38.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:41:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:41:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:39.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:39.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:39.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:41:39.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:39.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:41:39.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:39.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:41:39.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:39.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:39.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:39.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:41:39.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:39.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:39.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:39.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:39 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:41:39.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:40.393 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:40.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:40 smithi079 ceph-mon[23158]: pgmap v2686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:40 smithi016 ceph-mon[22010]: pgmap v2686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:39 smithi016 conmon[32727]: 2022-03-21T18:41:39.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:41.263 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:40 smithi016 conmon[32727]: 2022-03-21T18:41:40.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:41.264 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:41.264 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:41 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T18:41:41.264 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:41 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:41:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:41 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T18:41:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:41 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:41:41.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 2022-03-21T18:41:41.261+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:41.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:41:41.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:41.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: yield 2022-03-21T18:41:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:41.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:41:41.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:41.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:41:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:41:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:41:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:41:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:41.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 2022-03-21T18:41:41.262+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:41:41.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:41:41.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: yield 2022-03-21T18:41:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:41.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:41.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:41:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:41:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:41.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:41:41.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:41.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:41:41.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:41.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:41:41.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:41.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 2022-03-21T18:41:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:41:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: 2022-03-21T18:41:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:41:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:41:41.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:41:41.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:41:41.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:41:41.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:41:41.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:41:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:41:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:41:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:41:41.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:41:41.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:41:41.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:41:41.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:41:41.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:41:41.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:41:41.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:41:41.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:41:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:41:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:41:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:41:41.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:41:41.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise 2022-03-21T18:41:41.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:41:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:41:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:41:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:41:41.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:41 smithi016 conmon[32727]: 2022-03-21T18:41:41.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:42 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:42.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:41:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:42.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:42.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:41:42.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:41:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:42.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:41:42.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:42.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:42.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:42.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:41:42.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:42 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:41:42.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:41:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:42.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:41:42.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:41:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:42.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:41:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:42.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:42.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:41:42.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:42.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:42.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:41:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:41:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:41:42.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:41:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:41:42.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:41:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:41:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:41:42.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:41:42.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:41:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:41:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:41:42.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:41:42.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:41:42.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:41:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:41:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:41:42.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:41:42.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:41:42.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:41:42.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:41:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:41:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:41:42.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:41:42.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:41:42.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:41:42.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:41:42.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:41:42.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:41:42.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:42 smithi016 ceph-mon[22010]: pgmap v2687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:42 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:41:42.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:42.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:42.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:42.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:41:42.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:42.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:41:42.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:42.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:41:42.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:42.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:42.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:42.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:41:42.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:42.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:42.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:42 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:41:42.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:41:42.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:41:42.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:41:42.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:41:42.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:41:42.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:41:42.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:41:42.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:41:42.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:41:42.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:41:42.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:41:42.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:41:42.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:41:42.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:41:42.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:41:42.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:41:42.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:41:42.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:41:42.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:41:42.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:41:42.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:41:42.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:41:42.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:41:42.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:41:42.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:41:42.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:41:42.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:41:42.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:41:42.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:41:42.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:41:42.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:41:42.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:41:42.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:41:42.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:41:42.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:41:42.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:41:42.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:41:42.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:41:42.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:41:42.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:41:42.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:41:42.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:41:42.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:41:42.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:41:42.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:41:42.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:41:42.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:41:42.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:42 smithi079 ceph-mon[23158]: pgmap v2687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:42 smithi016 conmon[32727]: 2022-03-21T18:41:42.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:44 smithi016 conmon[32727]: 2022-03-21T18:41:44.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:44 smithi016 ceph-mon[22010]: pgmap v2688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4746 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:44 smithi079 ceph-mon[23158]: pgmap v2688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4746 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:45 smithi016 conmon[32727]: 2022-03-21T18:41:45.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:46 smithi016 ceph-mon[22010]: pgmap v2689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:46 smithi016 conmon[32727]: 2022-03-21T18:41:46.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:46 smithi079 ceph-mon[23158]: pgmap v2689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:46 smithi016 conmon[32727]: 2022-03-21T18:41:46.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:48 smithi016 ceph-mon[22010]: pgmap v2690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:48 smithi016 conmon[32727]: 2022-03-21T18:41:48.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:48 smithi079 ceph-mon[23158]: pgmap v2690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:49 smithi016 conmon[32727]: 2022-03-21T18:41:49.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:50 smithi016 ceph-mon[22010]: pgmap v2691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:49 smithi016 conmon[32727]: 2022-03-21T18:41:49.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:50 smithi079 ceph-mon[23158]: pgmap v2691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:51 smithi016 conmon[32727]: 2022-03-21T18:41:51.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:52 smithi016 ceph-mon[22010]: pgmap v2692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:52 smithi016 conmon[32727]: 2022-03-21T18:41:52.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:52 smithi079 ceph-mon[23158]: pgmap v2692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:53 smithi016 conmon[32727]: 2022-03-21T18:41:53.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:54 smithi016 ceph-mon[22010]: pgmap v2693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:54 smithi016 conmon[32727]: 2022-03-21T18:41:54.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:54 smithi079 ceph-mon[23158]: pgmap v2693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:55 smithi016 conmon[32727]: 2022-03-21T18:41:55.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:56 smithi016 ceph-mon[22010]: pgmap v2694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:56 smithi016 conmon[32727]: 2022-03-21T18:41:56.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:56 smithi079 ceph-mon[23158]: pgmap v2694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:41:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:57 smithi016 conmon[32727]: 2022-03-21T18:41:57.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:58 smithi016 ceph-mon[22010]: pgmap v2695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:58 smithi016 conmon[32727]: 2022-03-21T18:41:58.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:58 smithi079 ceph-mon[23158]: pgmap v2695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:41:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:41:59 smithi016 conmon[32727]: 2022-03-21T18:41:59.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:41:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:41:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:41:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:41:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:41:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:41:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:41:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:41:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:41:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:41:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:41:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:41:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:41:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:00 smithi016 ceph-mon[22010]: pgmap v2696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:00 smithi016 conmon[32727]: 2022-03-21T18:42:00.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:00 smithi079 ceph-mon[23158]: pgmap v2696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:01 smithi016 conmon[32727]: 2022-03-21T18:42:01.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:02 smithi016 ceph-mon[22010]: pgmap v2697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:02 smithi016 conmon[32727]: 2022-03-21T18:42:02.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:02 smithi079 ceph-mon[23158]: pgmap v2697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:03 smithi016 conmon[32727]: 2022-03-21T18:42:03.149+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:04 smithi016 ceph-mon[22010]: pgmap v2698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:04 smithi016 conmon[32727]: 2022-03-21T18:42:04.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:04 smithi079 ceph-mon[23158]: pgmap v2698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:05 smithi016 conmon[32727]: 2022-03-21T18:42:05.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:06 smithi079 ceph-mon[23158]: pgmap v2699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:06 smithi016 ceph-mon[22010]: pgmap v2699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:06 smithi016 conmon[32727]: 2022-03-21T18:42:06.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:07 smithi016 conmon[32727]: 2022-03-21T18:42:07.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:08 smithi016 ceph-mon[22010]: pgmap v2700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:08 smithi016 conmon[32727]: 2022-03-21T18:42:08.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:08 smithi079 ceph-mon[23158]: pgmap v2700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:09 smithi016 conmon[32727]: 2022-03-21T18:42:09.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:10 smithi016 ceph-mon[22010]: pgmap v2701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:10 smithi016 conmon[32727]: 2022-03-21T18:42:10.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:10 smithi079 ceph-mon[23158]: pgmap v2701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:11 smithi016 conmon[32727]: 2022-03-21T18:42:11.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:12 smithi016 conmon[32727]: 2022-03-21T18:42:12.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:12.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:12 smithi016 ceph-mon[22010]: pgmap v2702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:12 smithi079 ceph-mon[23158]: pgmap v2702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:13.779 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:13 smithi016 conmon[32727]: 2022-03-21T18:42:13.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:14 smithi079 ceph-mon[23158]: pgmap v2703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:14 smithi016 ceph-mon[22010]: pgmap v2703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:14.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:14.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:14 smithi016 conmon[32727]: 2022-03-21T18:42:14.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:15 smithi016 conmon[32727]: 2022-03-21T18:42:15.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:16 smithi079 ceph-mon[23158]: pgmap v2704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:16 smithi016 ceph-mon[22010]: pgmap v2704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:16 smithi016 conmon[32727]: 2022-03-21T18:42:16.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:17 smithi016 conmon[32727]: 2022-03-21T18:42:17.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:18 smithi079 ceph-mon[23158]: pgmap v2705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:18 smithi016 ceph-mon[22010]: pgmap v2705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:18 smithi016 conmon[32727]: 2022-03-21T18:42:18.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:19 smithi016 conmon[32727]: 2022-03-21T18:42:19.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:20 smithi016 ceph-mon[22010]: pgmap v2706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:20 smithi016 conmon[32727]: 2022-03-21T18:42:20.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:20 smithi079 ceph-mon[23158]: pgmap v2706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:21 smithi016 conmon[32727]: 2022-03-21T18:42:21.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:22 smithi016 ceph-mon[22010]: pgmap v2707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:22 smithi016 conmon[32727]: 2022-03-21T18:42:22.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:22 smithi079 ceph-mon[23158]: pgmap v2707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:23 smithi016 conmon[32727]: 2022-03-21T18:42:23.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:24 smithi016 ceph-mon[22010]: pgmap v2708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:24 smithi016 conmon[32727]: 2022-03-21T18:42:24.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:24 smithi079 ceph-mon[23158]: pgmap v2708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:25 smithi016 conmon[32727]: 2022-03-21T18:42:25.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:26 smithi016 ceph-mon[22010]: pgmap v2709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:26 smithi016 conmon[32727]: 2022-03-21T18:42:26.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:26 smithi079 ceph-mon[23158]: pgmap v2709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:27 smithi016 conmon[32727]: 2022-03-21T18:42:27.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:28 smithi016 ceph-mon[22010]: pgmap v2710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:28 smithi016 conmon[32727]: 2022-03-21T18:42:28.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:28 smithi079 ceph-mon[23158]: pgmap v2710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:29 smithi016 conmon[32727]: 2022-03-21T18:42:29.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:30 smithi016 ceph-mon[22010]: pgmap v2711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:30 smithi016 conmon[32727]: 2022-03-21T18:42:30.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:30 smithi079 ceph-mon[23158]: pgmap v2711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:31 smithi016 conmon[32727]: 2022-03-21T18:42:31.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:32 smithi016 ceph-mon[22010]: pgmap v2712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:32 smithi016 conmon[32727]: 2022-03-21T18:42:32.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:32 smithi079 ceph-mon[23158]: pgmap v2712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:33 smithi016 conmon[32727]: 2022-03-21T18:42:33.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:34.256 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:34 smithi016 conmon[32727]: 2022-03-21T18:42:34.004+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:34 smithi016 ceph-mon[22010]: pgmap v2713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:34 smithi079 ceph-mon[23158]: pgmap v2713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:35 smithi016 conmon[32727]: 2022-03-21T18:42:35.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:36 smithi016 conmon[32727]: 2022-03-21T18:42:36.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:36 smithi016 ceph-mon[22010]: pgmap v2714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:36 smithi079 ceph-mon[23158]: pgmap v2714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:37.264 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:36 smithi016 conmon[32727]: 2022-03-21T18:42:36.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:38.269 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:37 smithi016 conmon[32727]: 2022-03-21T18:42:37.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:38 smithi016 ceph-mon[22010]: pgmap v2715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:38 smithi079 ceph-mon[23158]: pgmap v2715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:39.270 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:38 smithi016 conmon[32727]: 2022-03-21T18:42:38.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:40.275 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:39 smithi016 conmon[32727]: 2022-03-21T18:42:39.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:40 smithi016 ceph-mon[22010]: pgmap v2716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:40 smithi079 ceph-mon[23158]: pgmap v2716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:41.276 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:40 smithi016 conmon[32727]: 2022-03-21T18:42:40.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:42:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:42:41.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:41 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:42:41.673 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:41.673 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:42:41.674 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:42:41.674 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:41 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:42:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:41 smithi016 conmon[32727]: 2022-03-21T18:42:41.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:42 smithi079 ceph-mon[23158]: pgmap v2717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:42 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:42:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:42 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:42:42.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:42.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:42 smithi016 ceph-mon[22010]: pgmap v2717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:42.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:42 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:42:42.983 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:42 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:42:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:42 smithi016 conmon[32727]: 2022-03-21T18:42:42.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:43.971 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:44.367 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:43 smithi016 conmon[32727]: 2022-03-21T18:42:43.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:44.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: 2022-03-21T18:42:44.365+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:44.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:42:44.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:44.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: yield 2022-03-21T18:42:44.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:44.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:44.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:44.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:42:44.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:44.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:42:44.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:44.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:42:44.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:44.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:42:44.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:44.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:42:44.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:44.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:44.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:44 smithi079 ceph-mon[23158]: pgmap v2718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:44.981 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:44.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:44 smithi016 ceph-mon[22010]: pgmap v2718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:44.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:44.982 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:45.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:44 smithi016 conmon[32727]: 2022-03-21T18:42:44.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:45 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:42:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:42:45.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:45.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:45.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:45.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:42:45.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:45.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:42:45.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:45.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:42:45.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:45.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:45.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:45.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:42:45.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:45.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:45.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:45.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:45 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:42:45.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:46.002 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:45 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:46.002 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:42:46.003 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:46.003 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:42:46.003 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:46.004 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:46.004 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:46.004 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:42:46.005 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:46.005 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:42:46.005 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:46.006 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:42:46.006 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:46.007 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:46.007 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:46.007 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:42:46.008 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:46.008 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:46.008 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:46.009 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:45 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:42:46.009 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:46.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:46 smithi016 conmon[32727]: 2022-03-21T18:42:46.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:46 smithi079 ceph-mon[23158]: pgmap v2719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:47.011 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:46 smithi016 ceph-mon[22010]: pgmap v2719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:47.011 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:47.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[32727]: 2022-03-21T18:42:47.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:47.701 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 2022-03-21T18:42:47.437+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:47.701 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:42:47.702 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:47.702 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: yield 2022-03-21T18:42:47.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:47.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:47.703 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:47.704 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:42:47.704 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:47.704 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:42:47.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:47.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:42:47.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:47.705 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:42:47.706 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:47.706 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:42:47.706 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:47.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:47.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:47.707 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 2022-03-21T18:42:47.438+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:42:47.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:42:47.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:47.708 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: yield 2022-03-21T18:42:47.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:47.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:47.709 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:47.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:42:47.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:47.710 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:42:47.711 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:47.711 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:42:47.711 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:47.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:42:47.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:47.712 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:42:47.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:47.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:47.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:47.714 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 2022-03-21T18:42:47.714 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:42:47.714 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: 2022-03-21T18:42:47.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:42:47.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:42:47.715 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:42:47.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:42:47.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:42:47.716 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:42:47.717 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:42:47.717 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:42:47.717 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:42:47.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:42:47.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:42:47.718 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:42:47.719 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:42:47.719 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:42:47.719 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:42:47.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:42:47.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:42:47.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:42:47.721 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:42:47.721 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:42:47.722 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:42:47.722 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:42:47.723 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise 2022-03-21T18:42:47.723 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:42:47.724 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:42:47.725 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:42:47.725 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:42:47.726 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:47.996 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:48.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:47 smithi016 conmon[32727]: 2022-03-21T18:42:47.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:48.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:48.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:42:48.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:48.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:42:48.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:48.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:48.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:48.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:42:48.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:48.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:42:48.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:48.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:42:48.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:48.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:48.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:48.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:42:48.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:48.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:48.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:48.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:48 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:42:48.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:42:48.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:48.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:42:48.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:48.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:48.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:48.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:42:48.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:48.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:42:48.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:48.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:42:48.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:48.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:48.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:48.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:42:48.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:48.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:48.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:48.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:42:48.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:42:48.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:42:48.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:42:48.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:42:48.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:42:48.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:42:48.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:42:48.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:42:48.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:42:48.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:42:48.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:42:48.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:42:48.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:42:49.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:42:49.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:42:49.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:42:49.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:42:49.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:42:49.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:42:49.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:42:49.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:42:49.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:42:49.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:42:49.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:42:49.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:42:49.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:42:49.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:42:49.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:42:49.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:42:49.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:49.006 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:48 smithi079 ceph-mon[23158]: pgmap v2720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:49.006 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:49.034 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:49.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:42:49.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:49.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:42:49.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:49.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:49.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:49.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:42:49.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:49.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:42:49.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:49.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:42:49.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:49.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:49.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:49.040 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:42:49.040 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:49.040 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:49.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:49.041 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:48 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:42:49.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:42:49.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:42:49.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:42:49.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:42:49.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:42:49.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:42:49.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:42:49.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:42:49.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:42:49.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:42:49.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:42:49.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:42:49.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:42:49.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:42:49.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:42:49.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:42:49.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:42:49.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:49.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:42:49.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:42:49.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:42:49.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:42:49.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:42:49.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:42:49.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:42:49.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:42:49.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:42:49.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:42:49.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:42:49.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:42:49.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:42:49.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:42:49.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:42:49.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:42:49.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:42:49.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:42:49.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:42:49.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:42:49.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:42:49.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:42:49.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:42:49.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:42:49.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:42:49.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:42:49.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:42:49.059 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:42:49.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:48 smithi016 ceph-mon[22010]: pgmap v2720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:49.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:49.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:49 smithi016 conmon[32727]: 2022-03-21T18:42:49.033+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:50.039 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:50.040 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:50 smithi016 conmon[32727]: 2022-03-21T18:42:50.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:50 smithi079 ceph-mon[23158]: pgmap v2721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:51.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:50 smithi016 ceph-mon[22010]: pgmap v2721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:51.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:51.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:51 smithi016 conmon[32727]: 2022-03-21T18:42:51.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:52.084 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:52 smithi016 conmon[32727]: 2022-03-21T18:42:52.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:52 smithi079 ceph-mon[23158]: pgmap v2722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:53.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:52 smithi016 ceph-mon[22010]: pgmap v2722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:53.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:53 smithi016 conmon[32727]: 2022-03-21T18:42:53.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:54.111 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:54 smithi016 conmon[32727]: 2022-03-21T18:42:54.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:54 smithi079 ceph-mon[23158]: pgmap v2723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:55.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:54 smithi016 ceph-mon[22010]: pgmap v2723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:55.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:42:55.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:55 smithi016 conmon[32727]: 2022-03-21T18:42:55.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:56.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:56 smithi016 conmon[32727]: 2022-03-21T18:42:56.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:57.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:56 smithi016 ceph-mon[22010]: pgmap v2724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:57.063 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:56 smithi079 ceph-mon[23158]: pgmap v2724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:42:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:57.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:57 smithi016 conmon[32727]: 2022-03-21T18:42:57.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:58.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:58 smithi016 conmon[32727]: 2022-03-21T18:42:58.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:42:59.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: pgmap v2725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:59.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:59.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:42:59.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:42:59.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:42:59.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:42:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: pgmap v2725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:42:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:42:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:42:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:42:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:42:59 smithi016 conmon[32727]: 2022-03-21T18:42:59.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:00.090 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:00.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:42:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:42:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:00 smithi016 conmon[32727]: 2022-03-21T18:43:00.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:01.113 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:00 smithi016 ceph-mon[22010]: pgmap v2726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:01.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:00 smithi079 ceph-mon[23158]: pgmap v2726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:01 smithi016 conmon[32727]: 2022-03-21T18:43:01.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:02.102 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:02 smithi016 conmon[32727]: 2022-03-21T18:43:02.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:03.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:02 smithi016 ceph-mon[22010]: pgmap v2727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:03.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:02 smithi079 ceph-mon[23158]: pgmap v2727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:03 smithi016 conmon[32727]: 2022-03-21T18:43:03.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:04.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:04 smithi016 conmon[32727]: 2022-03-21T18:43:04.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:05.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:04 smithi016 ceph-mon[22010]: pgmap v2728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:05.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:05.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:04 smithi079 ceph-mon[23158]: pgmap v2728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:05 smithi016 conmon[32727]: 2022-03-21T18:43:05.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:06.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:06 smithi016 conmon[32727]: 2022-03-21T18:43:06.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:07.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:06 smithi016 ceph-mon[22010]: pgmap v2729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:07.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:06 smithi079 ceph-mon[23158]: pgmap v2729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:07.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:07 smithi016 conmon[32727]: 2022-03-21T18:43:07.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:08.103 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:08 smithi016 conmon[32727]: 2022-03-21T18:43:08.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:09.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:08 smithi016 ceph-mon[22010]: pgmap v2730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:09.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:08 smithi079 ceph-mon[23158]: pgmap v2730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:09 smithi016 conmon[32727]: 2022-03-21T18:43:09.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:10.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:10.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:10 smithi016 conmon[32727]: 2022-03-21T18:43:10.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:11.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:10 smithi016 ceph-mon[22010]: pgmap v2731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:11.063 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:10 smithi079 ceph-mon[23158]: pgmap v2731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:11 smithi016 conmon[32727]: 2022-03-21T18:43:11.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:12.027 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:12 smithi016 conmon[32727]: 2022-03-21T18:43:12.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:13.065 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:12 smithi016 ceph-mon[22010]: pgmap v2732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:13.065 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:12 smithi079 ceph-mon[23158]: pgmap v2732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:13 smithi016 conmon[32727]: 2022-03-21T18:43:13.062+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:14.072 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:14 smithi016 conmon[32727]: 2022-03-21T18:43:14.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:15.036 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:14 smithi016 ceph-mon[22010]: pgmap v2733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:15.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:15.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:14 smithi079 ceph-mon[23158]: pgmap v2733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:15 smithi016 conmon[32727]: 2022-03-21T18:43:15.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:16.073 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:16 smithi016 conmon[32727]: 2022-03-21T18:43:16.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:17.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:16 smithi016 ceph-mon[22010]: pgmap v2734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:17.034 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:16 smithi079 ceph-mon[23158]: pgmap v2734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:17 smithi016 conmon[32727]: 2022-03-21T18:43:17.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:17 smithi016 conmon[32727]: 2022-03-21T18:43:17.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:18 smithi016 ceph-mon[22010]: pgmap v2735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:18 smithi016 conmon[32727]: 2022-03-21T18:43:18.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:18 smithi079 ceph-mon[23158]: pgmap v2735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:19 smithi016 conmon[32727]: 2022-03-21T18:43:19.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:20 smithi016 ceph-mon[22010]: pgmap v2736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:20 smithi016 conmon[32727]: 2022-03-21T18:43:20.950+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:20 smithi079 ceph-mon[23158]: pgmap v2736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:21 smithi016 conmon[32727]: 2022-03-21T18:43:21.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:22 smithi016 ceph-mon[22010]: pgmap v2737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:22 smithi016 conmon[32727]: 2022-03-21T18:43:22.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:22 smithi079 ceph-mon[23158]: pgmap v2737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:24.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:24.098 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:23 smithi016 conmon[32727]: 2022-03-21T18:43:23.943+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:25.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:24 smithi016 ceph-mon[22010]: pgmap v2738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:25.081 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:25.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:25.082 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:24 smithi016 conmon[32727]: 2022-03-21T18:43:24.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:24 smithi079 ceph-mon[23158]: pgmap v2738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:26.068 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:26.069 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:25 smithi016 conmon[32727]: 2022-03-21T18:43:25.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:26 smithi016 ceph-mon[22010]: pgmap v2739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:26 smithi079 ceph-mon[23158]: pgmap v2739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:26 smithi016 conmon[32727]: 2022-03-21T18:43:26.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:27 smithi016 conmon[32727]: 2022-03-21T18:43:27.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:28 smithi016 ceph-mon[22010]: pgmap v2740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:28 smithi016 conmon[32727]: 2022-03-21T18:43:28.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:28 smithi079 ceph-mon[23158]: pgmap v2740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:29 smithi016 conmon[32727]: 2022-03-21T18:43:29.998+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:30 smithi016 ceph-mon[22010]: pgmap v2741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:30 smithi016 conmon[32727]: 2022-03-21T18:43:30.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:30 smithi079 ceph-mon[23158]: pgmap v2741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:31 smithi016 conmon[32727]: 2022-03-21T18:43:31.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:32 smithi016 ceph-mon[22010]: pgmap v2742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:32 smithi016 conmon[32727]: 2022-03-21T18:43:32.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:32 smithi079 ceph-mon[23158]: pgmap v2742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:33 smithi016 conmon[32727]: 2022-03-21T18:43:33.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:34 smithi016 conmon[32727]: 2022-03-21T18:43:34.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:34 smithi016 ceph-mon[22010]: pgmap v2743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:34 smithi079 ceph-mon[23158]: pgmap v2743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:35 smithi016 conmon[32727]: 2022-03-21T18:43:35.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:37.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:36 smithi016 conmon[32727]: 2022-03-21T18:43:36.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:36 smithi016 ceph-mon[22010]: pgmap v2744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:36 smithi079 ceph-mon[23158]: pgmap v2744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:37 smithi016 conmon[32727]: 2022-03-21T18:43:37.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:39 smithi016 conmon[32727]: 2022-03-21T18:43:39.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:38 smithi016 ceph-mon[22010]: pgmap v2745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:38 smithi079 ceph-mon[23158]: pgmap v2745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:40 smithi016 conmon[32727]: 2022-03-21T18:43:40.012+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:40 smithi016 ceph-mon[22010]: pgmap v2746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:40 smithi016 conmon[32727]: 2022-03-21T18:43:40.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:40 smithi079 ceph-mon[23158]: pgmap v2746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:41 smithi016 conmon[32727]: 2022-03-21T18:43:41.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:42 smithi016 ceph-mon[22010]: pgmap v2747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:42 smithi016 conmon[32727]: 2022-03-21T18:43:42.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:42 smithi079 ceph-mon[23158]: pgmap v2747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:44 smithi016 conmon[32727]: 2022-03-21T18:43:44.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:44 smithi016 ceph-mon[22010]: pgmap v2748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:44 smithi016 conmon[32727]: 2022-03-21T18:43:44.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:44 smithi079 ceph-mon[23158]: pgmap v2748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:46 smithi016 conmon[32727]: 2022-03-21T18:43:46.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:46 smithi016 ceph-mon[22010]: pgmap v2749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:47 smithi016 conmon[32727]: 2022-03-21T18:43:47.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:46 smithi079 ceph-mon[23158]: pgmap v2749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:43:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:43:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:47 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:43:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:48 smithi016 conmon[32727]: 2022-03-21T18:43:48.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:43:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:43:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:47 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:43:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:48 smithi016 ceph-mon[22010]: pgmap v2750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:48 smithi016 conmon[32727]: 2022-03-21T18:43:48.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:48 smithi079 ceph-mon[23158]: pgmap v2750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:49 smithi016 conmon[32727]: 2022-03-21T18:43:49.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:50.871 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: 2022-03-21T18:43:50.541+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:50.871 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:43:50.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:50.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: yield 2022-03-21T18:43:50.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:50.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:50.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:50.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:43:50.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:50.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:43:50.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:50.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:43:50.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:50.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:43:50.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:50.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:43:50.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:50.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:50.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:50 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:50 smithi016 ceph-mon[22010]: pgmap v2751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:51 smithi016 conmon[32727]: 2022-03-21T18:43:51.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:50 smithi079 ceph-mon[23158]: pgmap v2751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:52 smithi016 conmon[32727]: 2022-03-21T18:43:52.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:43:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:43:52.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:52.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:52.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:52.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:43:52.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:52.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:43:52.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:52.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:43:52.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:52.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:52.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:52.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:43:52.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:52.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:52.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:52.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:51 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:43:52.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:43:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:43:52.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:52.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:52.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:52.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:43:52.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:52.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:43:52.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:52.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:43:52.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:52.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:52.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:52.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:43:52.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:52.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:52.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:52.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:51 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:43:52.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:52 smithi016 ceph-mon[22010]: pgmap v2752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:52 smithi016 conmon[32727]: 2022-03-21T18:43:52.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:52 smithi079 ceph-mon[23158]: pgmap v2752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:53.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 2022-03-21T18:43:53.613+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:53.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:43:53.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:53.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: yield 2022-03-21T18:43:53.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:53.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:53.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:53.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:43:53.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:53.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:43:53.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:53.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:43:53.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:53.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:43:53.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:53.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:43:53.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:53.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:53.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:53.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 2022-03-21T18:43:53.614+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:43:53.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:43:53.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:53.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: yield 2022-03-21T18:43:53.891 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:53.891 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:53.891 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:53.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:43:53.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:53.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:43:53.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:53.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:43:53.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:53.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:43:53.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:53.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:43:53.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:53.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:53.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:53.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 2022-03-21T18:43:53.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:43:53.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: 2022-03-21T18:43:53.897 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:43:53.897 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:43:53.897 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:43:53.898 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:43:53.898 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:43:53.899 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:43:53.899 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:43:53.899 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:43:53.899 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:43:53.900 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:43:53.900 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:43:53.900 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:43:53.901 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:43:53.901 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:43:53.901 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:43:53.902 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:43:53.902 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:43:53.902 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:43:53.903 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:43:53.903 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:43:53.903 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:43:53.904 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:43:53.904 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise 2022-03-21T18:43:53.905 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:43:53.905 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:43:53.905 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:43:53.906 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:43:53.906 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:43:53 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:54 smithi016 conmon[32727]: 2022-03-21T18:43:54.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:54 smithi016 conmon[32727]: 2022-03-21T18:43:54.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:54 smithi016 ceph-mon[22010]: pgmap v2753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:43:55.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:55.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:55.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:43:55.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:55.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:43:55.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:55.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:43:55.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:55.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:55.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:55.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:43:55.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:55.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:55.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:54 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:43:55.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:55.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:43:55.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:55.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:55.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:55.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:43:55.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:55.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:43:55.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:55.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:43:55.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:55.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:55.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:55.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:43:55.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:55.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:55.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:43:55.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:43:55.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:43:55.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:43:55.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:43:55.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:43:55.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:43:55.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:43:55.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:43:55.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:43:55.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:43:55.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:43:55.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:43:55.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:43:55.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:43:55.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:43:55.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:43:55.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:43:55.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:43:55.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:43:55.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:43:55.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:43:55.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:43:55.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:43:55.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:43:55.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:43:55.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:43:55.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:43:55.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:43:55.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:55.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:54 smithi079 ceph-mon[23158]: pgmap v2753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:43:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:55.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:55.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:55.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:43:55.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:55.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:43:55.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:55.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:43:55.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:55.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:55.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:55.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:43:55.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:55.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:55.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:54 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:43:55.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:43:55.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:43:55.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:43:55.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:43:55.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:43:55.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:43:55.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:43:55.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:43:55.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:43:55.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:43:55.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:43:55.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:43:55.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:43:55.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:43:55.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:43:55.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:43:55.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:43:55.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:43:55.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:43:55.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:43:55.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:43:55.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:43:55.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:43:55.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:43:55.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:43:55.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:43:55.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:43:55.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:43:55.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:43:55.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:43:55.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:43:55.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:43:55.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:43:55.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:43:55.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:43:55.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:43:55.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:43:55.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:43:55.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:43:55.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:43:55.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:43:55.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:43:55.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:43:55.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:43:55.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:43:55.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:43:55.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:43:55.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:43:55.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:55 smithi016 ceph-mon[22010]: pgmap v2754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:56 smithi016 conmon[32727]: 2022-03-21T18:43:56.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:55 smithi079 ceph-mon[23158]: pgmap v2754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:43:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:57 smithi016 conmon[32727]: 2022-03-21T18:43:57.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:58 smithi016 conmon[32727]: 2022-03-21T18:43:58.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:57 smithi016 ceph-mon[22010]: pgmap v2755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:57 smithi079 ceph-mon[23158]: pgmap v2755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:43:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:43:59 smithi016 conmon[32727]: 2022-03-21T18:43:59.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:43:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:43:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:43:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:43:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:43:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:43:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:43:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:43:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:43:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:44:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:43:59 smithi016 ceph-mon[22010]: pgmap v2756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:00 smithi016 conmon[32727]: 2022-03-21T18:44:00.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:43:59 smithi079 ceph-mon[23158]: pgmap v2756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:01.423 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:01 smithi016 conmon[32727]: 2022-03-21T18:44:01.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:02 smithi016 ceph-mon[22010]: pgmap v2757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:02 smithi016 conmon[32727]: 2022-03-21T18:44:02.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:02 smithi079 ceph-mon[23158]: pgmap v2757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:03 smithi016 conmon[32727]: 2022-03-21T18:44:03.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:04 smithi016 ceph-mon[22010]: pgmap v2758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:04 smithi016 conmon[32727]: 2022-03-21T18:44:04.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:04 smithi079 ceph-mon[23158]: pgmap v2758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:05 smithi016 conmon[32727]: 2022-03-21T18:44:05.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:06 smithi016 ceph-mon[22010]: pgmap v2759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:06 smithi016 conmon[32727]: 2022-03-21T18:44:06.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:06 smithi079 ceph-mon[23158]: pgmap v2759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:07 smithi016 conmon[32727]: 2022-03-21T18:44:07.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:08 smithi016 ceph-mon[22010]: pgmap v2760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:08 smithi016 conmon[32727]: 2022-03-21T18:44:08.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:08 smithi079 ceph-mon[23158]: pgmap v2760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:09 smithi016 conmon[32727]: 2022-03-21T18:44:09.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:10 smithi016 ceph-mon[22010]: pgmap v2761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:10 smithi016 conmon[32727]: 2022-03-21T18:44:10.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:10 smithi079 ceph-mon[23158]: pgmap v2761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:11 smithi016 conmon[32727]: 2022-03-21T18:44:11.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:12 smithi016 ceph-mon[22010]: pgmap v2762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:12 smithi016 conmon[32727]: 2022-03-21T18:44:12.182+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:12 smithi079 ceph-mon[23158]: pgmap v2762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:13 smithi016 conmon[32727]: 2022-03-21T18:44:13.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:14 smithi016 conmon[32727]: 2022-03-21T18:44:14.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:14 smithi016 ceph-mon[22010]: pgmap v2763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:14 smithi079 ceph-mon[23158]: pgmap v2763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:15 smithi016 conmon[32727]: 2022-03-21T18:44:15.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:16 smithi016 conmon[32727]: 2022-03-21T18:44:16.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:16 smithi016 ceph-mon[22010]: pgmap v2764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:16 smithi079 ceph-mon[23158]: pgmap v2764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:17 smithi016 conmon[32727]: 2022-03-21T18:44:17.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:18 smithi016 conmon[32727]: 2022-03-21T18:44:18.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:18 smithi016 ceph-mon[22010]: pgmap v2765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:18 smithi079 ceph-mon[23158]: pgmap v2765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:19.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:19 smithi016 conmon[32727]: 2022-03-21T18:44:19.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:20 smithi016 conmon[32727]: 2022-03-21T18:44:20.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:20 smithi016 ceph-mon[22010]: pgmap v2766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:20 smithi079 ceph-mon[23158]: pgmap v2766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:21 smithi016 conmon[32727]: 2022-03-21T18:44:21.179+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:22 smithi016 conmon[32727]: 2022-03-21T18:44:22.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:22 smithi016 ceph-mon[22010]: pgmap v2767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:22 smithi079 ceph-mon[23158]: pgmap v2767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:23 smithi016 conmon[32727]: 2022-03-21T18:44:23.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:24 smithi016 conmon[32727]: 2022-03-21T18:44:24.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:24 smithi016 ceph-mon[22010]: pgmap v2768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:24 smithi079 ceph-mon[23158]: pgmap v2768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:25.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:25 smithi016 conmon[32727]: 2022-03-21T18:44:25.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:26.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:26 smithi016 conmon[32727]: 2022-03-21T18:44:26.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:26 smithi016 ceph-mon[22010]: pgmap v2769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:26 smithi079 ceph-mon[23158]: pgmap v2769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:27 smithi016 conmon[32727]: 2022-03-21T18:44:27.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:28 smithi016 conmon[32727]: 2022-03-21T18:44:28.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:28 smithi016 ceph-mon[22010]: pgmap v2770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:28 smithi079 ceph-mon[23158]: pgmap v2770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:29 smithi016 conmon[32727]: 2022-03-21T18:44:29.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:30 smithi016 ceph-mon[22010]: pgmap v2771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:30 smithi016 conmon[32727]: 2022-03-21T18:44:30.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:30 smithi079 ceph-mon[23158]: pgmap v2771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:31 smithi016 conmon[32727]: 2022-03-21T18:44:31.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:32 smithi016 ceph-mon[22010]: pgmap v2772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:32 smithi016 conmon[32727]: 2022-03-21T18:44:32.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:32 smithi079 ceph-mon[23158]: pgmap v2772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:33 smithi016 conmon[32727]: 2022-03-21T18:44:33.048+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:34 smithi016 ceph-mon[22010]: pgmap v2773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:34 smithi016 conmon[32727]: 2022-03-21T18:44:34.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:34 smithi079 ceph-mon[23158]: pgmap v2773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:34 smithi016 conmon[32727]: 2022-03-21T18:44:34.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:36 smithi016 ceph-mon[22010]: pgmap v2774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:35 smithi016 conmon[32727]: 2022-03-21T18:44:35.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:36 smithi079 ceph-mon[23158]: pgmap v2774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:36 smithi016 conmon[32727]: 2022-03-21T18:44:36.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:38 smithi016 ceph-mon[22010]: pgmap v2775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:37 smithi016 conmon[32727]: 2022-03-21T18:44:37.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:38 smithi079 ceph-mon[23158]: pgmap v2775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:39 smithi016 conmon[32727]: 2022-03-21T18:44:39.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:40 smithi016 ceph-mon[22010]: pgmap v2776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:39 smithi016 conmon[32727]: 2022-03-21T18:44:39.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:40 smithi079 ceph-mon[23158]: pgmap v2776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:40 smithi016 conmon[32727]: 2022-03-21T18:44:40.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:42 smithi016 ceph-mon[22010]: pgmap v2777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:41 smithi016 conmon[32727]: 2022-03-21T18:44:41.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:42 smithi079 ceph-mon[23158]: pgmap v2777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:43 smithi016 conmon[32727]: 2022-03-21T18:44:43.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:44 smithi016 ceph-mon[22010]: pgmap v2778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:44.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:43 smithi016 conmon[32727]: 2022-03-21T18:44:43.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:44 smithi079 ceph-mon[23158]: pgmap v2778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:44 smithi016 conmon[32727]: 2022-03-21T18:44:44.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:46 smithi016 ceph-mon[22010]: pgmap v2779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:45 smithi016 conmon[32727]: 2022-03-21T18:44:45.917+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:46 smithi079 ceph-mon[23158]: pgmap v2779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:46 smithi016 conmon[32727]: 2022-03-21T18:44:46.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:48 smithi016 ceph-mon[22010]: pgmap v2780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:47 smithi016 conmon[32727]: 2022-03-21T18:44:47.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:48 smithi079 ceph-mon[23158]: pgmap v2780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:49 smithi016 conmon[32727]: 2022-03-21T18:44:49.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:50 smithi016 ceph-mon[22010]: pgmap v2781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:50 smithi016 conmon[32727]: 2022-03-21T18:44:50.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:50 smithi079 ceph-mon[23158]: pgmap v2781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:50 smithi016 conmon[32727]: 2022-03-21T18:44:50.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:52 smithi016 ceph-mon[22010]: pgmap v2782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:52 smithi016 conmon[32727]: 2022-03-21T18:44:52.026+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:52.423 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:52.423 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:52 smithi079 ceph-mon[23158]: pgmap v2782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:53 smithi016 conmon[32727]: 2022-03-21T18:44:53.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:54.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:54.336 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: pgmap v2783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:54.337 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:44:54.337 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:44:54.337 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:44:54.338 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[32727]: 2022-03-21T18:44:54.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: pgmap v2783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:44:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:44:54.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:44:54.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:54.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: 2022-03-21T18:44:54.221+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:54.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:44:54.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: yield 2022-03-21T18:44:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:54.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:54.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:44:54.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:54.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:44:54.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:54.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:44:54.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:54.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:44:54.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:54.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:44:54.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:54.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:54.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:55 smithi016 conmon[32727]: 2022-03-21T18:44:55.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:44:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:44:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:44:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:55.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:44:55.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:44:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:44:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:55.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:55.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:55 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:44:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:55 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:44:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:55 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:44:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:44:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:44:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:44:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:44:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:44:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:44:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:55 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:44:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:55 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:44:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:55 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:44:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:56 smithi016 conmon[32727]: 2022-03-21T18:44:56.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:56 smithi079 ceph-mon[23158]: pgmap v2784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:56 smithi016 ceph-mon[22010]: pgmap v2784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:44:57.295 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[32727]: 2022-03-21T18:44:57.039+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:57.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 2022-03-21T18:44:57.293+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:44:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:57.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: yield 2022-03-21T18:44:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:57.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:44:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:57.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:44:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:44:57.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:57.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:44:57.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:57.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:44:57.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:57.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:57.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:57.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 2022-03-21T18:44:57.294+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:44:57.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:44:57.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:57.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: yield 2022-03-21T18:44:57.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:57.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:57.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:57.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:44:57.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:57.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:44:57.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:57.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:44:57.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:57.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:44:57.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:57.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:44:57.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:57.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:57.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:57.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 2022-03-21T18:44:57.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:44:57.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: 2022-03-21T18:44:57.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:44:57.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:44:57.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:44:57.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:44:57.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:44:57.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:44:57.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:44:57.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:44:57.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:44:57.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:44:57.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:44:57.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:44:57.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:44:57.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:44:57.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:44:57.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:44:57.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:44:57.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:44:57.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:44:57.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:44:57.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:44:57.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:44:57.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise 2022-03-21T18:44:57.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:44:57.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:44:57.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:44:57.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:44:57.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:44:57 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:58 smithi016 conmon[32727]: 2022-03-21T18:44:58.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:44:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:44:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:58.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:44:58.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:58.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:44:58.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:58.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:58.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:58.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:44:58.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:58.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:58.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:44:58.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:58.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:44:58.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:58.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:58.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:58.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:44:58.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:58.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:44:58.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:58.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:44:58.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:58.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:58.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:58.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:44:58.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:58.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:58.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:44:58.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:44:58.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:44:58.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:44:58.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:44:58.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:44:58.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:44:58.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:44:58.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:44:58.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:44:58.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:44:58.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:44:58.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:44:58.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:44:58.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:44:58.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:44:58.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:44:58.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:44:58.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:44:58.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:44:58.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:44:58.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:44:58.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:44:58.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:44:58.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:44:58.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:44:58.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:44:58.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:44:58.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:44:58.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: pgmap v2785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:58.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:44:58.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:44:58.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:44:58.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:44:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:44:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:44:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:58.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:44:58.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:44:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:44:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:44:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:44:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:44:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:44:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:44:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:44:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:44:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:44:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:44:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:44:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:44:58.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:44:58.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:44:58.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:44:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:44:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:44:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:44:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:44:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:44:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:44:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:44:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:44:58.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:44:58.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:44:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:44:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:44:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:44:58.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:44:58.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:44:58.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:44:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:44:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:44:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:44:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:44:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:44:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:44:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:44:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:44:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:44:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:44:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:44:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:44:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:44:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:44:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:44:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:44:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:44:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:44:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: pgmap v2785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:44:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:44:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:44:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:44:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:44:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:44:59 smithi016 conmon[32727]: 2022-03-21T18:44:59.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:44:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:44:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:44:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:44:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:44:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:00 smithi016 conmon[32727]: 2022-03-21T18:45:00.004+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:00 smithi079 ceph-mon[23158]: pgmap v2786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:00 smithi016 ceph-mon[22010]: pgmap v2786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:01 smithi016 conmon[32727]: 2022-03-21T18:45:01.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:02 smithi016 conmon[32727]: 2022-03-21T18:45:02.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:02 smithi079 ceph-mon[23158]: pgmap v2787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:02 smithi016 ceph-mon[22010]: pgmap v2787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:03 smithi016 conmon[32727]: 2022-03-21T18:45:03.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:04 smithi016 conmon[32727]: 2022-03-21T18:45:04.047+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:04.866 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:04.867 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:04 smithi079 ceph-mon[23158]: pgmap v2788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:04.867 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:04 smithi016 ceph-mon[22010]: pgmap v2788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:05 smithi016 conmon[32727]: 2022-03-21T18:45:05.085+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:06 smithi016 conmon[32727]: 2022-03-21T18:45:06.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:06 smithi016 ceph-mon[22010]: pgmap v2789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:06 smithi079 ceph-mon[23158]: pgmap v2789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:07.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:07 smithi016 conmon[32727]: 2022-03-21T18:45:07.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:08 smithi016 conmon[32727]: 2022-03-21T18:45:08.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:08 smithi016 ceph-mon[22010]: pgmap v2790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:08 smithi079 ceph-mon[23158]: pgmap v2790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:09 smithi016 conmon[32727]: 2022-03-21T18:45:09.081+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:10 smithi016 conmon[32727]: 2022-03-21T18:45:10.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:10 smithi016 ceph-mon[22010]: pgmap v2791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:10 smithi079 ceph-mon[23158]: pgmap v2791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:11 smithi016 conmon[32727]: 2022-03-21T18:45:11.138+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:12 smithi016 conmon[32727]: 2022-03-21T18:45:12.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:12 smithi016 ceph-mon[22010]: pgmap v2792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:12 smithi079 ceph-mon[23158]: pgmap v2792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:13.502 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:13 smithi016 conmon[32727]: 2022-03-21T18:45:13.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:14.507 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:14 smithi016 conmon[32727]: 2022-03-21T18:45:14.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:14 smithi016 ceph-mon[22010]: pgmap v2793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:14 smithi079 ceph-mon[23158]: pgmap v2793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:15.509 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:15 smithi016 conmon[32727]: 2022-03-21T18:45:15.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:16.514 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:16 smithi016 conmon[32727]: 2022-03-21T18:45:16.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:16 smithi016 ceph-mon[22010]: pgmap v2794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:16 smithi079 ceph-mon[23158]: pgmap v2794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:17.515 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:17 smithi016 conmon[32727]: 2022-03-21T18:45:17.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:18.520 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:18 smithi016 conmon[32727]: 2022-03-21T18:45:18.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:18 smithi016 ceph-mon[22010]: pgmap v2795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:18 smithi079 ceph-mon[23158]: pgmap v2795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:19.521 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:19 smithi016 conmon[32727]: 2022-03-21T18:45:19.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:20.527 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:20 smithi016 conmon[32727]: 2022-03-21T18:45:20.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:20 smithi016 ceph-mon[22010]: pgmap v2796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:20 smithi079 ceph-mon[23158]: pgmap v2796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:21.531 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:21 smithi016 conmon[32727]: 2022-03-21T18:45:21.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:22.533 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:22 smithi016 conmon[32727]: 2022-03-21T18:45:22.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:22 smithi016 ceph-mon[22010]: pgmap v2797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:22 smithi079 ceph-mon[23158]: pgmap v2797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:23.539 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:23 smithi016 conmon[32727]: 2022-03-21T18:45:23.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:24 smithi016 conmon[32727]: 2022-03-21T18:45:24.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:24 smithi016 ceph-mon[22010]: pgmap v2798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:24 smithi079 ceph-mon[23158]: pgmap v2798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:25.544 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:25 smithi016 conmon[32727]: 2022-03-21T18:45:25.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:26.546 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:26 smithi016 conmon[32727]: 2022-03-21T18:45:26.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:26 smithi016 ceph-mon[22010]: pgmap v2799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:26 smithi079 ceph-mon[23158]: pgmap v2799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:27.551 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:27 smithi016 conmon[32727]: 2022-03-21T18:45:27.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:28.553 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:28 smithi016 conmon[32727]: 2022-03-21T18:45:28.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:28 smithi016 ceph-mon[22010]: pgmap v2800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:28 smithi079 ceph-mon[23158]: pgmap v2800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:29.554 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:29 smithi016 conmon[32727]: 2022-03-21T18:45:29.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:30.556 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:30 smithi016 conmon[32727]: 2022-03-21T18:45:30.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:30 smithi016 ceph-mon[22010]: pgmap v2801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:30 smithi079 ceph-mon[23158]: pgmap v2801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:31.561 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:31 smithi016 conmon[32727]: 2022-03-21T18:45:31.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:32.562 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:32 smithi016 conmon[32727]: 2022-03-21T18:45:32.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:32 smithi016 ceph-mon[22010]: pgmap v2802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:32 smithi079 ceph-mon[23158]: pgmap v2802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:33.567 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:33 smithi016 conmon[32727]: 2022-03-21T18:45:33.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:34.569 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:34 smithi016 conmon[32727]: 2022-03-21T18:45:34.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:34 smithi016 ceph-mon[22010]: pgmap v2803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:34 smithi079 ceph-mon[23158]: pgmap v2803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:35.574 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:35 smithi016 conmon[32727]: 2022-03-21T18:45:35.271+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:36.575 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:36 smithi016 conmon[32727]: 2022-03-21T18:45:36.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:36 smithi016 ceph-mon[22010]: pgmap v2804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:36 smithi079 ceph-mon[23158]: pgmap v2804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:37.581 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:37 smithi016 conmon[32727]: 2022-03-21T18:45:37.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:38.582 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:38 smithi016 conmon[32727]: 2022-03-21T18:45:38.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:38 smithi016 ceph-mon[22010]: pgmap v2805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:38 smithi079 ceph-mon[23158]: pgmap v2805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:39.583 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:39 smithi016 conmon[32727]: 2022-03-21T18:45:39.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:40.584 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:40 smithi016 conmon[32727]: 2022-03-21T18:45:40.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:40 smithi016 ceph-mon[22010]: pgmap v2806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:40 smithi079 ceph-mon[23158]: pgmap v2806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:41.589 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:41 smithi016 conmon[32727]: 2022-03-21T18:45:41.206+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:42.591 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:42 smithi016 conmon[32727]: 2022-03-21T18:45:42.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:42 smithi016 ceph-mon[22010]: pgmap v2807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:42 smithi079 ceph-mon[23158]: pgmap v2807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:43.596 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:43 smithi016 conmon[32727]: 2022-03-21T18:45:43.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:44.598 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:44 smithi016 conmon[32727]: 2022-03-21T18:45:44.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:44 smithi016 ceph-mon[22010]: pgmap v2808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:44 smithi079 ceph-mon[23158]: pgmap v2808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:45.603 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:45 smithi016 conmon[32727]: 2022-03-21T18:45:45.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:46.604 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:46 smithi016 conmon[32727]: 2022-03-21T18:45:46.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:46 smithi016 ceph-mon[22010]: pgmap v2809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:46 smithi079 ceph-mon[23158]: pgmap v2809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:47.609 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:47 smithi016 conmon[32727]: 2022-03-21T18:45:47.234+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:48.611 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:48 smithi016 conmon[32727]: 2022-03-21T18:45:48.206+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:48 smithi016 ceph-mon[22010]: pgmap v2810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:48 smithi079 ceph-mon[23158]: pgmap v2810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:49.613 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:49 smithi016 conmon[32727]: 2022-03-21T18:45:49.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:50.614 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:50 smithi016 conmon[32727]: 2022-03-21T18:45:50.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:50 smithi016 ceph-mon[22010]: pgmap v2811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:50.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:50 smithi079 ceph-mon[23158]: pgmap v2811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:51.619 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:51 smithi016 conmon[32727]: 2022-03-21T18:45:51.222+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:52.620 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:52 smithi016 conmon[32727]: 2022-03-21T18:45:52.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:52 smithi016 ceph-mon[22010]: pgmap v2812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:52 smithi079 ceph-mon[23158]: pgmap v2812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:53.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:53 smithi016 conmon[32727]: 2022-03-21T18:45:53.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:54 smithi016 conmon[32727]: 2022-03-21T18:45:54.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:54 smithi016 ceph-mon[22010]: pgmap v2813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 4995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:54 smithi079 ceph-mon[23158]: pgmap v2813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 4995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:55 smithi016 conmon[32727]: 2022-03-21T18:45:55.149+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:56.634 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:56 smithi016 conmon[32727]: 2022-03-21T18:45:56.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:56 smithi016 ceph-mon[22010]: pgmap v2814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:56 smithi079 ceph-mon[23158]: pgmap v2814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:45:57.637 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:57 smithi016 conmon[32727]: 2022-03-21T18:45:57.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:45:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:45:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:57 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:45:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:45:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:45:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:57 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:45:58.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:58 smithi016 conmon[32727]: 2022-03-21T18:45:58.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: pgmap v2815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:45:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:45:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:45:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:45:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: pgmap v2815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:45:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:45:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:45:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:45:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:45:59.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:45:59 smithi016 conmon[32727]: 2022-03-21T18:45:59.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:45:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:45:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:45:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:45:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:45:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:00.642 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[32727]: 2022-03-21T18:46:00.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:00.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: 2022-03-21T18:46:00.397+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:00.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:46:00.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:00.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: yield 2022-03-21T18:46:00.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:00.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:00.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:00.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:46:00.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:00.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:46:00.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:00.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:46:00.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:00.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:46:00.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:00.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:46:00.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:00.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:00.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:00 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:00 smithi016 ceph-mon[22010]: pgmap v2816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:00 smithi079 ceph-mon[23158]: pgmap v2816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:01.647 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:01 smithi016 conmon[32727]: 2022-03-21T18:46:01.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:46:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:01.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:46:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:01.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:01.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:46:01.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:01.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:46:01.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:01.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:46:01.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:01.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:01.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:01.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:46:01.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:01.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:01.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:01.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:46:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:01.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:46:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:01.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:46:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:01.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:01.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:46:01.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:01.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:46:01.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:01.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:46:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:01.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:01.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:46:01.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:01.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:01.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:01.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:46:02.648 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:02 smithi016 conmon[32727]: 2022-03-21T18:46:02.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:02.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:02.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:02 smithi016 ceph-mon[22010]: pgmap v2817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:02 smithi079 ceph-mon[23158]: pgmap v2817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:03.471 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[32727]: 2022-03-21T18:46:03.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:03.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 2022-03-21T18:46:03.468+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:03.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:46:03.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:03.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: yield 2022-03-21T18:46:03.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:03.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:03.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:03.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:46:03.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:03.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:46:03.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:03.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:46:03.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:03.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:46:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:46:03.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:03.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 2022-03-21T18:46:03.469+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:46:03.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:46:03.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:03.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: yield 2022-03-21T18:46:03.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:03.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:03.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:03.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:46:03.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:03.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:46:03.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:03.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:46:03.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:03.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:46:03.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:03.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:46:03.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:03.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:03.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:03.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 2022-03-21T18:46:03.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:46:03.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: 2022-03-21T18:46:03.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:46:03.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:46:03.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:46:03.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:46:03.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:46:03.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:46:03.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:46:03.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:46:03.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:46:03.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:46:03.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:46:03.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:46:03.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:46:03.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:46:03.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:46:03.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:46:03.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:46:03.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:46:03.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:46:03.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:46:03.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:46:03.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:46:03.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise 2022-03-21T18:46:03.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:46:03.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:46:03.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:46:03.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:46:03.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:46:03 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:04.654 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:04 smithi016 conmon[32727]: 2022-03-21T18:46:04.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:46:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:46:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:46:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:46:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:04.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:04.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:04.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:46:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:04 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:46:04.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:04.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:46:04.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:04.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:04.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:04.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:46:04.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:04.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:46:04.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:04.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:46:04.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:04.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:04.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:04.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:46:04.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:04.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:04.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:46:04.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:46:04.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:46:04.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:46:04.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:46:04.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:46:04.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:46:04.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:46:04.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:46:04.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:46:04.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:46:04.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:46:04.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:46:04.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:46:04.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:46:04.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:46:04.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:46:04.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:46:04.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:46:04.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:46:04.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:46:04.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:46:04.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:46:04.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:46:04.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:46:04.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:46:04.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:46:04.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:46:04.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:46:04.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:04 smithi016 ceph-mon[22010]: pgmap v2818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:04.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:46:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:46:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:46:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:04.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:46:04.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:04.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:04.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:04.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:46:04.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:04 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:46:04.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:46:04.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:46:04.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:46:04.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:46:04.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:46:04.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:46:04.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:46:04.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:46:04.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:46:04.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:46:04.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:46:04.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:46:04.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:46:04.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:46:04.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:46:04.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:46:04.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:46:04.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:46:04.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:46:04.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:46:04.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:46:04.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:46:04.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:46:04.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:46:04.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:46:04.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:46:04.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:46:04.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:46:04.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:46:04.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:46:04.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:46:04.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:46:04.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:46:04.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:46:04.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:46:04.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:46:04.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:46:04.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:46:04.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:46:04.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:46:04.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:46:04.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:46:04.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:46:04.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:46:04.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:46:04.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:46:04.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:46:04.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:04 smithi079 ceph-mon[23158]: pgmap v2818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:04.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:05 smithi016 conmon[32727]: 2022-03-21T18:46:05.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:06 smithi016 conmon[32727]: 2022-03-21T18:46:06.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:06 smithi016 ceph-mon[22010]: pgmap v2819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:06 smithi079 ceph-mon[23158]: pgmap v2819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:07 smithi016 conmon[32727]: 2022-03-21T18:46:07.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:08 smithi016 conmon[32727]: 2022-03-21T18:46:08.229+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:08 smithi079 ceph-mon[23158]: pgmap v2820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:08 smithi016 ceph-mon[22010]: pgmap v2820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:09.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:09 smithi016 conmon[32727]: 2022-03-21T18:46:09.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:10 smithi016 conmon[32727]: 2022-03-21T18:46:10.239+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:10 smithi079 ceph-mon[23158]: pgmap v2821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:10 smithi016 ceph-mon[22010]: pgmap v2821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:11 smithi016 conmon[32727]: 2022-03-21T18:46:11.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:12 smithi016 conmon[32727]: 2022-03-21T18:46:12.297+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:12 smithi079 ceph-mon[23158]: pgmap v2822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:12 smithi016 ceph-mon[22010]: pgmap v2822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:13 smithi016 conmon[32727]: 2022-03-21T18:46:13.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:14.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:14 smithi016 conmon[32727]: 2022-03-21T18:46:14.320+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:14 smithi079 ceph-mon[23158]: pgmap v2823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:14 smithi016 ceph-mon[22010]: pgmap v2823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:15.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:15 smithi016 conmon[32727]: 2022-03-21T18:46:15.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:16.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:16 smithi016 conmon[32727]: 2022-03-21T18:46:16.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:16 smithi079 ceph-mon[23158]: pgmap v2824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:16 smithi016 ceph-mon[22010]: pgmap v2824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:17.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:17 smithi016 conmon[32727]: 2022-03-21T18:46:17.243+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:18.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:18 smithi016 conmon[32727]: 2022-03-21T18:46:18.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:18 smithi079 ceph-mon[23158]: pgmap v2825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:18 smithi016 ceph-mon[22010]: pgmap v2825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:19.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:19 smithi016 conmon[32727]: 2022-03-21T18:46:19.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:20 smithi016 conmon[32727]: 2022-03-21T18:46:20.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:20 smithi079 ceph-mon[23158]: pgmap v2826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:20 smithi016 ceph-mon[22010]: pgmap v2826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:21.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:21 smithi016 conmon[32727]: 2022-03-21T18:46:21.239+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:22.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:22 smithi016 conmon[32727]: 2022-03-21T18:46:22.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:22 smithi079 ceph-mon[23158]: pgmap v2827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:22 smithi016 ceph-mon[22010]: pgmap v2827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:23.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:23 smithi016 conmon[32727]: 2022-03-21T18:46:23.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:24 smithi016 conmon[32727]: 2022-03-21T18:46:24.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:24 smithi079 ceph-mon[23158]: pgmap v2828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:24 smithi016 ceph-mon[22010]: pgmap v2828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:25.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:25 smithi016 conmon[32727]: 2022-03-21T18:46:25.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:26 smithi016 conmon[32727]: 2022-03-21T18:46:26.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:26 smithi079 ceph-mon[23158]: pgmap v2829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:26 smithi016 ceph-mon[22010]: pgmap v2829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:27 smithi016 conmon[32727]: 2022-03-21T18:46:27.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:28 smithi016 conmon[32727]: 2022-03-21T18:46:28.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:28 smithi016 ceph-mon[22010]: pgmap v2830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:28 smithi079 ceph-mon[23158]: pgmap v2830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:29 smithi016 conmon[32727]: 2022-03-21T18:46:29.179+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:30 smithi016 conmon[32727]: 2022-03-21T18:46:30.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:30 smithi016 ceph-mon[22010]: pgmap v2831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:30 smithi079 ceph-mon[23158]: pgmap v2831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:31 smithi016 conmon[32727]: 2022-03-21T18:46:31.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:32 smithi016 conmon[32727]: 2022-03-21T18:46:32.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:33.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:33.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:32 smithi016 ceph-mon[22010]: pgmap v2832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:33.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:32 smithi079 ceph-mon[23158]: pgmap v2832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:33 smithi016 conmon[32727]: 2022-03-21T18:46:33.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:34.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:34 smithi016 conmon[32727]: 2022-03-21T18:46:34.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:35.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:34 smithi016 ceph-mon[22010]: pgmap v2833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:35.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:35.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:34 smithi079 ceph-mon[23158]: pgmap v2833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:35.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:35 smithi016 conmon[32727]: 2022-03-21T18:46:35.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:36.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:36 smithi016 conmon[32727]: 2022-03-21T18:46:36.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:37.052 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:36 smithi016 ceph-mon[22010]: pgmap v2834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:37.052 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:36 smithi079 ceph-mon[23158]: pgmap v2834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:37.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:37 smithi016 conmon[32727]: 2022-03-21T18:46:37.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:38.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:38.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:38 smithi016 conmon[32727]: 2022-03-21T18:46:38.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:39.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:38 smithi016 ceph-mon[22010]: pgmap v2835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:39.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:38 smithi079 ceph-mon[23158]: pgmap v2835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:39.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:39 smithi016 conmon[32727]: 2022-03-21T18:46:39.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:40.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:40.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:40.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:40 smithi016 conmon[32727]: 2022-03-21T18:46:40.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:40 smithi016 ceph-mon[22010]: pgmap v2836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:40 smithi079 ceph-mon[23158]: pgmap v2836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:41 smithi016 conmon[32727]: 2022-03-21T18:46:41.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:42 smithi016 conmon[32727]: 2022-03-21T18:46:42.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:42 smithi016 ceph-mon[22010]: pgmap v2837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:42 smithi079 ceph-mon[23158]: pgmap v2837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:43 smithi016 conmon[32727]: 2022-03-21T18:46:43.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:44 smithi016 conmon[32727]: 2022-03-21T18:46:44.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:44 smithi016 ceph-mon[22010]: pgmap v2838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:44 smithi079 ceph-mon[23158]: pgmap v2838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:45 smithi016 conmon[32727]: 2022-03-21T18:46:45.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:46 smithi016 conmon[32727]: 2022-03-21T18:46:46.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:46 smithi016 ceph-mon[22010]: pgmap v2839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:46 smithi079 ceph-mon[23158]: pgmap v2839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:47.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:47 smithi016 conmon[32727]: 2022-03-21T18:46:47.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:48 smithi016 conmon[32727]: 2022-03-21T18:46:48.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:48 smithi016 ceph-mon[22010]: pgmap v2840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:48 smithi079 ceph-mon[23158]: pgmap v2840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:49 smithi016 conmon[32727]: 2022-03-21T18:46:49.234+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:50 smithi016 conmon[32727]: 2022-03-21T18:46:50.279+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:50 smithi016 ceph-mon[22010]: pgmap v2841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:50 smithi079 ceph-mon[23158]: pgmap v2841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:51 smithi016 conmon[32727]: 2022-03-21T18:46:51.277+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:52.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:52 smithi016 conmon[32727]: 2022-03-21T18:46:52.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:52 smithi016 ceph-mon[22010]: pgmap v2842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:52 smithi079 ceph-mon[23158]: pgmap v2842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:53.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:53 smithi016 conmon[32727]: 2022-03-21T18:46:53.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:54 smithi016 conmon[32727]: 2022-03-21T18:46:54.265+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:54 smithi016 ceph-mon[22010]: pgmap v2843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:54 smithi079 ceph-mon[23158]: pgmap v2843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:46:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:55.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:55 smithi016 conmon[32727]: 2022-03-21T18:46:55.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:56.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:56 smithi016 conmon[32727]: 2022-03-21T18:46:56.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:56 smithi016 ceph-mon[22010]: pgmap v2844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:56 smithi079 ceph-mon[23158]: pgmap v2844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:46:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:57.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:57 smithi016 conmon[32727]: 2022-03-21T18:46:57.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:58 smithi016 conmon[32727]: 2022-03-21T18:46:58.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:46:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: pgmap v2845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:46:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:46:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:46:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:46:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: pgmap v2845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:46:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:46:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:46:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:46:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:46:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:46:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:46:59 smithi016 conmon[32727]: 2022-03-21T18:46:59.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:46:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:46:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:00 smithi016 conmon[32727]: 2022-03-21T18:47:00.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:00 smithi016 ceph-mon[22010]: pgmap v2846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:00 smithi079 ceph-mon[23158]: pgmap v2846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:01.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:01 smithi016 conmon[32727]: 2022-03-21T18:47:01.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:02 smithi016 conmon[32727]: 2022-03-21T18:47:02.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:02 smithi016 ceph-mon[22010]: pgmap v2847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:02 smithi079 ceph-mon[23158]: pgmap v2847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:03.524 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[32727]: 2022-03-21T18:47:03.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:04.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:04.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:47:04.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:47:04.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:03 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:47:04.138 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: 2022-03-21T18:47:03.949+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:04.138 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:47:04.139 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:04.139 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: yield 2022-03-21T18:47:04.139 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:04.140 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:04.140 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:04.140 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:47:04.141 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:04.141 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:47:04.142 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:04.142 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:47:04.142 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:04.143 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:47:04.143 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:04.143 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:47:04.144 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:04.144 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:04.144 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:47:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:47:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:03 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:47:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:04 smithi016 conmon[32727]: 2022-03-21T18:47:04.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:04 smithi016 ceph-mon[22010]: pgmap v2848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:04 smithi079 ceph-mon[23158]: pgmap v2848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:05.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:05 smithi016 conmon[32727]: 2022-03-21T18:47:05.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:47:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:47:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:47:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:47:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:47:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:47:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:05 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:47:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:47:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:47:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:47:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:47:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:47:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:47:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:05 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:47:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:06 smithi016 conmon[32727]: 2022-03-21T18:47:06.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:06 smithi016 ceph-mon[22010]: pgmap v2849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 2022-03-21T18:47:07.021+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:47:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:07.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: yield 2022-03-21T18:47:07.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:07.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:47:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:07.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:47:07.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:47:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:47:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:47:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 2022-03-21T18:47:07.021+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:47:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:47:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: yield 2022-03-21T18:47:07.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:07.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:47:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:47:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:47:07.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:07.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:47:07.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:07.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:47:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 2022-03-21T18:47:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:47:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: 2022-03-21T18:47:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:47:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:47:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:47:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:47:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:47:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:47:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:47:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:47:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:47:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:47:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:47:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:47:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:47:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:47:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:47:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:47:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:47:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:47:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:47:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:47:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:47:07.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:47:07.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise 2022-03-21T18:47:07.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:47:07.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:47:07.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:47:07.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:47:07.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:06 smithi079 ceph-mon[23158]: pgmap v2849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:07 smithi016 conmon[32727]: 2022-03-21T18:47:07.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:08 smithi016 conmon[32727]: 2022-03-21T18:47:08.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:47:09.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:09.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:47:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:09.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:47:09.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:09.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:47:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:09.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:47:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:09.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:08 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:47:09.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:47:09.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:09.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:09.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:09.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:47:09.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:09.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:47:09.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:09.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:47:09.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:09.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:09.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:09.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:47:09.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:09.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:09.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:47:09.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:47:09.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:47:09.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:47:09.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:47:09.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:47:09.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:47:09.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:47:09.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:47:09.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:47:09.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:47:09.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:47:09.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:47:09.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:47:09.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:47:09.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:47:09.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:47:09.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:47:09.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:47:09.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:47:09.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:47:09.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:47:09.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:47:09.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:47:09.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:47:09.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:47:09.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:47:09.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:47:09.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:47:09.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:08 smithi016 ceph-mon[22010]: pgmap v2850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:09.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:47:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:09.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:47:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:09.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:47:09.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:09.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:47:09.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:09.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:47:09.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:09.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:09.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:08 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:47:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:47:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:47:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:47:09.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:47:09.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:47:09.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:47:09.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:47:09.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:47:09.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:47:09.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:47:09.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:47:09.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:47:09.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:47:09.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:47:09.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:47:09.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:47:09.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:47:09.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:47:09.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:47:09.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:47:09.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:47:09.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:47:09.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:47:09.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:47:09.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:47:09.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:47:09.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:47:09.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:47:09.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:47:09.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:47:09.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:47:09.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:47:09.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:47:09.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:47:09.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:47:09.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:47:09.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:47:09.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:47:09.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:47:09.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:47:09.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:47:09.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:47:09.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:47:09.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:47:09.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:47:09.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:47:09.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:47:09.247 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:08 smithi079 ceph-mon[23158]: pgmap v2850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:09.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:09 smithi016 conmon[32727]: 2022-03-21T18:47:09.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:10 smithi016 conmon[32727]: 2022-03-21T18:47:10.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:10 smithi016 ceph-mon[22010]: pgmap v2851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:10 smithi079 ceph-mon[23158]: pgmap v2851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:11 smithi016 conmon[32727]: 2022-03-21T18:47:11.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:12 smithi016 conmon[32727]: 2022-03-21T18:47:12.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:12 smithi016 ceph-mon[22010]: pgmap v2852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:12 smithi079 ceph-mon[23158]: pgmap v2852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:13 smithi016 conmon[32727]: 2022-03-21T18:47:13.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:14.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:14 smithi016 conmon[32727]: 2022-03-21T18:47:14.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:14 smithi016 ceph-mon[22010]: pgmap v2853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:14 smithi079 ceph-mon[23158]: pgmap v2853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:15.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:15 smithi016 conmon[32727]: 2022-03-21T18:47:15.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:16.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:16 smithi016 conmon[32727]: 2022-03-21T18:47:16.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:16 smithi016 ceph-mon[22010]: pgmap v2854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:16 smithi079 ceph-mon[23158]: pgmap v2854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:17.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:17 smithi016 conmon[32727]: 2022-03-21T18:47:17.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:18.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:18 smithi016 conmon[32727]: 2022-03-21T18:47:18.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:18 smithi016 ceph-mon[22010]: pgmap v2855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:18 smithi079 ceph-mon[23158]: pgmap v2855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:19.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:19 smithi016 conmon[32727]: 2022-03-21T18:47:19.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:20.865 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:20 smithi016 conmon[32727]: 2022-03-21T18:47:20.432+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:20 smithi016 ceph-mon[22010]: pgmap v2856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:20 smithi079 ceph-mon[23158]: pgmap v2856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:21.870 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:21 smithi016 conmon[32727]: 2022-03-21T18:47:21.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:22.872 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:22 smithi016 conmon[32727]: 2022-03-21T18:47:22.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:22 smithi016 ceph-mon[22010]: pgmap v2857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:22 smithi079 ceph-mon[23158]: pgmap v2857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:23.877 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:23 smithi016 conmon[32727]: 2022-03-21T18:47:23.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:24 smithi016 conmon[32727]: 2022-03-21T18:47:24.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:24 smithi016 ceph-mon[22010]: pgmap v2858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:24 smithi079 ceph-mon[23158]: pgmap v2858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:25.883 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:25 smithi016 conmon[32727]: 2022-03-21T18:47:25.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:26 smithi016 conmon[32727]: 2022-03-21T18:47:26.390+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:26 smithi016 ceph-mon[22010]: pgmap v2859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:26 smithi079 ceph-mon[23158]: pgmap v2859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:27 smithi016 conmon[32727]: 2022-03-21T18:47:27.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:28.891 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:28 smithi016 conmon[32727]: 2022-03-21T18:47:28.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:28 smithi016 ceph-mon[22010]: pgmap v2860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:28 smithi079 ceph-mon[23158]: pgmap v2860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:29 smithi016 conmon[32727]: 2022-03-21T18:47:29.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:30 smithi016 conmon[32727]: 2022-03-21T18:47:30.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:30 smithi016 ceph-mon[22010]: pgmap v2861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:30 smithi079 ceph-mon[23158]: pgmap v2861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:31 smithi016 conmon[32727]: 2022-03-21T18:47:31.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:32 smithi016 conmon[32727]: 2022-03-21T18:47:32.319+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:32 smithi016 ceph-mon[22010]: pgmap v2862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:32 smithi079 ceph-mon[23158]: pgmap v2862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:33 smithi016 conmon[32727]: 2022-03-21T18:47:33.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:34 smithi016 conmon[32727]: 2022-03-21T18:47:34.322+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:34 smithi016 ceph-mon[22010]: pgmap v2863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:34 smithi079 ceph-mon[23158]: pgmap v2863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:35 smithi016 conmon[32727]: 2022-03-21T18:47:35.292+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:36.714 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:36.715 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:36 smithi016 conmon[32727]: 2022-03-21T18:47:36.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:36 smithi016 ceph-mon[22010]: pgmap v2864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:36 smithi079 ceph-mon[23158]: pgmap v2864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:37 smithi016 conmon[32727]: 2022-03-21T18:47:37.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:38.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:38 smithi016 conmon[32727]: 2022-03-21T18:47:38.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:39.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:38 smithi016 ceph-mon[22010]: pgmap v2865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:39.220 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:38 smithi079 ceph-mon[23158]: pgmap v2865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:39 smithi016 conmon[32727]: 2022-03-21T18:47:39.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:40.249 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:40.249 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:40 smithi016 conmon[32727]: 2022-03-21T18:47:40.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:40 smithi079 ceph-mon[23158]: pgmap v2866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:41.262 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:40 smithi016 ceph-mon[22010]: pgmap v2866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:41.263 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:41 smithi016 conmon[32727]: 2022-03-21T18:47:41.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:41 smithi079 ceph-mon[23158]: pgmap v2867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:42.237 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:42.237 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:41 smithi016 ceph-mon[22010]: pgmap v2867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:42 smithi016 conmon[32727]: 2022-03-21T18:47:42.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:43.263 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:43 smithi016 conmon[32727]: 2022-03-21T18:47:43.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:43 smithi079 ceph-mon[23158]: pgmap v2868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:44.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:44.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:43 smithi016 ceph-mon[22010]: pgmap v2868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:44 smithi016 conmon[32727]: 2022-03-21T18:47:44.301+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:45.258 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:45.259 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:45 smithi016 conmon[32727]: 2022-03-21T18:47:45.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:45 smithi079 ceph-mon[23158]: pgmap v2869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:46.284 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:46.285 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:45 smithi016 ceph-mon[22010]: pgmap v2869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:46 smithi016 conmon[32727]: 2022-03-21T18:47:46.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:47.289 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:47.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:47 smithi016 conmon[32727]: 2022-03-21T18:47:47.288+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:47 smithi079 ceph-mon[23158]: pgmap v2870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:48.289 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:48.289 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:47 smithi016 ceph-mon[22010]: pgmap v2870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:48 smithi016 conmon[32727]: 2022-03-21T18:47:48.288+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:49.316 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:49 smithi016 conmon[32727]: 2022-03-21T18:47:49.315+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:49 smithi079 ceph-mon[23158]: pgmap v2871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:50.356 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:50.357 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:50.357 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:49 smithi016 ceph-mon[22010]: pgmap v2871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:50 smithi016 conmon[32727]: 2022-03-21T18:47:50.355+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:51.405 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:51 smithi016 conmon[32727]: 2022-03-21T18:47:51.404+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:52.361 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:52.361 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:52 smithi016 ceph-mon[22010]: pgmap v2872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:52 smithi079 ceph-mon[23158]: pgmap v2872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:52.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:52 smithi016 conmon[32727]: 2022-03-21T18:47:52.359+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:53.405 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:53 smithi016 conmon[32727]: 2022-03-21T18:47:53.404+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:54 smithi016 ceph-mon[22010]: pgmap v2873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:54 smithi079 ceph-mon[23158]: pgmap v2873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:54 smithi016 conmon[32727]: 2022-03-21T18:47:54.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:55 smithi016 conmon[32727]: 2022-03-21T18:47:55.432+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:56 smithi016 ceph-mon[22010]: pgmap v2874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:56 smithi079 ceph-mon[23158]: pgmap v2874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:47:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:56 smithi016 conmon[32727]: 2022-03-21T18:47:56.468+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:57 smithi016 conmon[32727]: 2022-03-21T18:47:57.466+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:58 smithi016 ceph-mon[22010]: pgmap v2875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:58 smithi079 ceph-mon[23158]: pgmap v2875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:47:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:58 smithi016 conmon[32727]: 2022-03-21T18:47:58.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:47:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:47:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:47:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:47:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:47:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:47:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:47:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:47:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:47:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:47:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:47:59 smithi016 conmon[32727]: 2022-03-21T18:47:59.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:00 smithi016 ceph-mon[22010]: pgmap v2876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:00 smithi079 ceph-mon[23158]: pgmap v2876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:00 smithi016 conmon[32727]: 2022-03-21T18:48:00.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:01 smithi016 conmon[32727]: 2022-03-21T18:48:01.508+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:02 smithi016 ceph-mon[22010]: pgmap v2877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:02 smithi079 ceph-mon[23158]: pgmap v2877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:02.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:02 smithi016 conmon[32727]: 2022-03-21T18:48:02.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:03 smithi016 conmon[32727]: 2022-03-21T18:48:03.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:04.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:04 smithi016 ceph-mon[22010]: pgmap v2878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:04 smithi079 ceph-mon[23158]: pgmap v2878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:04.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:04 smithi016 conmon[32727]: 2022-03-21T18:48:04.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:05 smithi016 conmon[32727]: 2022-03-21T18:48:05.521+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:06 smithi016 ceph-mon[22010]: pgmap v2879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:06 smithi079 ceph-mon[23158]: pgmap v2879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:06 smithi016 conmon[32727]: 2022-03-21T18:48:06.520+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:48:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:48:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:48:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:48:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:48:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:48:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:07 smithi016 conmon[32727]: 2022-03-21T18:48:07.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:08 smithi016 ceph-mon[22010]: pgmap v2880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:08 smithi079 ceph-mon[23158]: pgmap v2880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:08 smithi016 conmon[32727]: 2022-03-21T18:48:08.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:09 smithi016 conmon[32727]: 2022-03-21T18:48:09.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:10 smithi016 ceph-mon[22010]: pgmap v2881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: 2022-03-21T18:48:10.124+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:48:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:10.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: yield 2022-03-21T18:48:10.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:10.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:10.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:10.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:48:10.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:10.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:48:10.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:10.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:48:10.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:10.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:48:10.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:10.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:48:10.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:10.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:10.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:10 smithi079 ceph-mon[23158]: pgmap v2881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:10 smithi016 conmon[32727]: 2022-03-21T18:48:10.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:48:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:48:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:11.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:11.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:11.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:48:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:48:11.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:11.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:48:11.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:11.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:11.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:11.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:48:11.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:11.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:11.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:11.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:11 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:48:11.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:48:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:48:11.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:11.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:11.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:11.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:48:11.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:11.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:48:11.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:11.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:48:11.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:11.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:11.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:11.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:48:11.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:11.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:11.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:11.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:11 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:48:11.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:11 smithi016 conmon[32727]: 2022-03-21T18:48:11.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:12 smithi016 ceph-mon[22010]: pgmap v2882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:12 smithi079 ceph-mon[23158]: pgmap v2882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:12 smithi016 conmon[32727]: 2022-03-21T18:48:12.661+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:13.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 2022-03-21T18:48:13.196+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:13.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:48:13.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: yield 2022-03-21T18:48:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:13.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:13.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:48:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:48:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:48:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:48:13.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:13.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:48:13.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:13.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:13.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 2022-03-21T18:48:13.197+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:48:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:48:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:13.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: yield 2022-03-21T18:48:13.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:48:13.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:13.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:48:13.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:48:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:48:13.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:13.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:48:13.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 2022-03-21T18:48:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:48:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: 2022-03-21T18:48:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:48:13.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:48:13.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:48:13.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:48:13.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:48:13.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:48:13.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:48:13.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:48:13.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:48:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:48:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:48:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:48:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:48:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:48:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:48:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:48:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:48:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:48:13.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:48:13.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:48:13.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:48:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:48:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise 2022-03-21T18:48:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:48:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:48:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:48:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:48:13.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:13 smithi016 conmon[32727]: 2022-03-21T18:48:13.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:14 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:14.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:48:14.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:14.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:14.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:14.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:48:14.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:14.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:48:14.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:14.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:48:14.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:14.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:14.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:14.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:48:14.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:14.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:14.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:14 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:48:14.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:14.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:48:14.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:14.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:14.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:14.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:48:14.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:14.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:48:14.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:14.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:48:14.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:14.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:14.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:14.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:48:14.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:14.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:14.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:48:14.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:48:14.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:48:14.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:48:14.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:48:14.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:48:14.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:48:14.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:48:14.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:48:14.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:48:14.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:48:14.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:48:14.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:48:14.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:48:14.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:48:14.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:48:14.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:48:14.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:48:14.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:48:14.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:48:14.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:48:14.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:48:14.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:48:14.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:48:14.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:48:14.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:48:14.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:48:14.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:48:14.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:48:14.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:14 smithi016 ceph-mon[22010]: pgmap v2883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:14.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:14.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:14 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:48:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:14.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:14.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:48:14.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:14.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:48:14.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:14.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:48:14.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:14.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:14.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:14.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:48:14.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:14.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:14.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:14 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:48:14.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:48:14.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:48:14.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:48:14.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:48:14.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:48:14.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:48:14.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:48:14.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:48:14.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:48:14.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:48:14.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:48:14.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:48:14.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:48:14.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:48:14.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:48:14.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:48:14.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:48:14.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:48:14.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:48:14.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:48:14.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:48:14.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:48:14.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:48:14.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:48:14.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:48:14.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:48:14.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:48:14.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:48:14.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:48:14.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:48:14.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:48:14.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:48:14.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:48:14.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:48:14.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:48:14.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:48:14.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:48:14.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:48:14.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:48:14.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:48:14.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:48:14.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:48:14.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:48:14.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:48:14.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:48:14.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:48:14.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:48:14.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:14 smithi079 ceph-mon[23158]: pgmap v2883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:14.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:14.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:14 smithi016 conmon[32727]: 2022-03-21T18:48:14.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:15 smithi016 conmon[32727]: 2022-03-21T18:48:15.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:16 smithi016 ceph-mon[22010]: pgmap v2884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:16 smithi079 ceph-mon[23158]: pgmap v2884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:16 smithi016 conmon[32727]: 2022-03-21T18:48:16.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:18.140 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:17 smithi016 conmon[32727]: 2022-03-21T18:48:17.696+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:18 smithi016 ceph-mon[22010]: pgmap v2885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:18 smithi079 ceph-mon[23158]: pgmap v2885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:19.141 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:18 smithi016 conmon[32727]: 2022-03-21T18:48:18.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:20.146 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:19 smithi016 conmon[32727]: 2022-03-21T18:48:19.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:20 smithi016 ceph-mon[22010]: pgmap v2886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:20 smithi079 ceph-mon[23158]: pgmap v2886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:21.147 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:20 smithi016 conmon[32727]: 2022-03-21T18:48:20.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:22.152 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:21 smithi016 conmon[32727]: 2022-03-21T18:48:21.686+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:22 smithi016 ceph-mon[22010]: pgmap v2887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:22 smithi079 ceph-mon[23158]: pgmap v2887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:22 smithi016 conmon[32727]: 2022-03-21T18:48:22.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:23 smithi016 conmon[32727]: 2022-03-21T18:48:23.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:24 smithi016 ceph-mon[22010]: pgmap v2888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:24 smithi079 ceph-mon[23158]: pgmap v2888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:24 smithi016 conmon[32727]: 2022-03-21T18:48:24.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:25 smithi016 conmon[32727]: 2022-03-21T18:48:25.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:26 smithi079 ceph-mon[23158]: pgmap v2889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:26.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:26 smithi016 ceph-mon[22010]: pgmap v2889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:26.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:26 smithi016 conmon[32727]: 2022-03-21T18:48:26.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:27.660 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:27 smithi016 conmon[32727]: 2022-03-21T18:48:27.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:28 smithi079 ceph-mon[23158]: pgmap v2890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:28 smithi016 ceph-mon[22010]: pgmap v2890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:29.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:28 smithi016 conmon[32727]: 2022-03-21T18:48:28.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:29 smithi016 conmon[32727]: 2022-03-21T18:48:29.727+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:30 smithi079 ceph-mon[23158]: pgmap v2891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:30 smithi016 ceph-mon[22010]: pgmap v2891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:30 smithi016 conmon[32727]: 2022-03-21T18:48:30.762+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:31 smithi016 conmon[32727]: 2022-03-21T18:48:31.762+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:32 smithi016 conmon[32727]: 2022-03-21T18:48:32.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:32 smithi016 ceph-mon[22010]: pgmap v2892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:32 smithi079 ceph-mon[23158]: pgmap v2892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:33 smithi016 conmon[32727]: 2022-03-21T18:48:33.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:33.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:34 smithi016 conmon[32727]: 2022-03-21T18:48:34.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:34 smithi016 ceph-mon[22010]: pgmap v2893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:34 smithi079 ceph-mon[23158]: pgmap v2893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:35 smithi016 conmon[32727]: 2022-03-21T18:48:35.762+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:36 smithi016 conmon[32727]: 2022-03-21T18:48:36.789+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:36 smithi016 ceph-mon[22010]: pgmap v2894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:36 smithi079 ceph-mon[23158]: pgmap v2894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:37 smithi016 conmon[32727]: 2022-03-21T18:48:37.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:38 smithi016 conmon[32727]: 2022-03-21T18:48:38.716+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:38 smithi016 ceph-mon[22010]: pgmap v2895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:38 smithi079 ceph-mon[23158]: pgmap v2895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:39 smithi016 conmon[32727]: 2022-03-21T18:48:39.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:40 smithi016 ceph-mon[22010]: pgmap v2896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:40 smithi016 conmon[32727]: 2022-03-21T18:48:40.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:40 smithi079 ceph-mon[23158]: pgmap v2896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:41 smithi016 conmon[32727]: 2022-03-21T18:48:41.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:42 smithi016 ceph-mon[22010]: pgmap v2897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:42 smithi016 conmon[32727]: 2022-03-21T18:48:42.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:42 smithi079 ceph-mon[23158]: pgmap v2897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:43 smithi016 conmon[32727]: 2022-03-21T18:48:43.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:44 smithi079 ceph-mon[23158]: pgmap v2898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:44 smithi016 ceph-mon[22010]: pgmap v2898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:44 smithi016 conmon[32727]: 2022-03-21T18:48:44.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:45 smithi016 conmon[32727]: 2022-03-21T18:48:45.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:46 smithi079 ceph-mon[23158]: pgmap v2899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:46 smithi016 ceph-mon[22010]: pgmap v2899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:46 smithi016 conmon[32727]: 2022-03-21T18:48:46.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:47 smithi016 conmon[32727]: 2022-03-21T18:48:47.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:48 smithi079 ceph-mon[23158]: pgmap v2900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:48 smithi016 conmon[32727]: 2022-03-21T18:48:48.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:48 smithi016 ceph-mon[22010]: pgmap v2900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:49 smithi016 conmon[32727]: 2022-03-21T18:48:49.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:50 smithi079 ceph-mon[23158]: pgmap v2901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:50 smithi016 ceph-mon[22010]: pgmap v2901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:50 smithi016 conmon[32727]: 2022-03-21T18:48:50.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:51 smithi016 conmon[32727]: 2022-03-21T18:48:51.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:52 smithi079 ceph-mon[23158]: pgmap v2902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:52 smithi016 ceph-mon[22010]: pgmap v2902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:52 smithi016 conmon[32727]: 2022-03-21T18:48:52.696+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:53 smithi016 conmon[32727]: 2022-03-21T18:48:53.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:54 smithi016 conmon[32727]: 2022-03-21T18:48:54.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:54 smithi016 ceph-mon[22010]: pgmap v2903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:54 smithi079 ceph-mon[23158]: pgmap v2903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:55 smithi016 conmon[32727]: 2022-03-21T18:48:55.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:56 smithi016 conmon[32727]: 2022-03-21T18:48:56.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:56 smithi016 ceph-mon[22010]: pgmap v2904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:56 smithi079 ceph-mon[23158]: pgmap v2904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:48:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:57 smithi016 conmon[32727]: 2022-03-21T18:48:57.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:58 smithi016 conmon[32727]: 2022-03-21T18:48:58.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: pgmap v2905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:48:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:48:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:48:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:48:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: pgmap v2905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:48:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:48:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:48:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:48:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:48:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:48:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:48:59.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:48:59 smithi016 conmon[32727]: 2022-03-21T18:48:59.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:48:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:48:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:48:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:00 smithi016 ceph-mon[22010]: pgmap v2906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:00 smithi016 conmon[32727]: 2022-03-21T18:49:00.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:00 smithi079 ceph-mon[23158]: pgmap v2906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:01 smithi016 conmon[32727]: 2022-03-21T18:49:01.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:02 smithi016 ceph-mon[22010]: pgmap v2907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:02 smithi016 conmon[32727]: 2022-03-21T18:49:02.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:02 smithi079 ceph-mon[23158]: pgmap v2907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:03 smithi016 conmon[32727]: 2022-03-21T18:49:03.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:04 smithi016 ceph-mon[22010]: pgmap v2908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:04.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:04 smithi016 conmon[32727]: 2022-03-21T18:49:04.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:04 smithi079 ceph-mon[23158]: pgmap v2908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:05 smithi016 conmon[32727]: 2022-03-21T18:49:05.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:06 smithi016 ceph-mon[22010]: pgmap v2909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:06 smithi016 conmon[32727]: 2022-03-21T18:49:06.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:06 smithi079 ceph-mon[23158]: pgmap v2909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:07 smithi016 conmon[32727]: 2022-03-21T18:49:07.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:08 smithi016 conmon[32727]: 2022-03-21T18:49:08.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:08 smithi016 ceph-mon[22010]: pgmap v2910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:08 smithi079 ceph-mon[23158]: pgmap v2910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:09 smithi016 conmon[32727]: 2022-03-21T18:49:09.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:10.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:10 smithi016 ceph-mon[22010]: pgmap v2911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:10 smithi016 conmon[32727]: 2022-03-21T18:49:10.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:10 smithi079 ceph-mon[23158]: pgmap v2911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:11 smithi016 conmon[32727]: 2022-03-21T18:49:11.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:12 smithi016 ceph-mon[22010]: pgmap v2912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:12 smithi016 conmon[32727]: 2022-03-21T18:49:12.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:12 smithi079 ceph-mon[23158]: pgmap v2912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:13 smithi016 conmon[32727]: 2022-03-21T18:49:13.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:49:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:49:13.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:49:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:49:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:49:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:49:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:14 smithi016 ceph-mon[22010]: pgmap v2913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:14 smithi016 conmon[32727]: 2022-03-21T18:49:14.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:14 smithi079 ceph-mon[23158]: pgmap v2913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:15 smithi016 conmon[32727]: 2022-03-21T18:49:15.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:16.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: 2022-03-21T18:49:16.300+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:16.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:49:16.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:16.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: yield 2022-03-21T18:49:16.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:16.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:16.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:16.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:49:16.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:16.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:49:16.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:16.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:49:16.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:16.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:49:16.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:16.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:49:16.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:16.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:16.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:16 smithi016 ceph-mon[22010]: pgmap v2914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:16 smithi016 conmon[32727]: 2022-03-21T18:49:16.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:16 smithi079 ceph-mon[23158]: pgmap v2914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:17 smithi016 conmon[32727]: 2022-03-21T18:49:17.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:17 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:49:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:49:17.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:17.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:17.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:17.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:49:17.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:17.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:49:17.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:17.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:49:17.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:17.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:17.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:17.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:49:17.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:17.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:17.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:17.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:17 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:49:17.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:17 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:49:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:49:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:18.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:18.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:49:18.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:18.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:49:18.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:18.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:49:18.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:18.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:18.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:18.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:49:18.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:18.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:18.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:18.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:17 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:49:18.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:18 smithi016 ceph-mon[22010]: pgmap v2915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:18.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:18 smithi016 conmon[32727]: 2022-03-21T18:49:18.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:18 smithi079 ceph-mon[23158]: pgmap v2915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:19.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 2022-03-21T18:49:19.372+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:19.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:49:19.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:19.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: yield 2022-03-21T18:49:19.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:19.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:19.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:19.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:49:19.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:19.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:49:19.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:19.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:49:19.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:19.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:49:19.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:19.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:49:19.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:19.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:19.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:19.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 2022-03-21T18:49:19.373+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:49:19.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:49:19.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:19.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: yield 2022-03-21T18:49:19.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:19.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:19.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:19.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:49:19.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:19.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:49:19.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:19.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:49:19.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:19.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:49:19.640 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:19.640 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:49:19.640 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:19.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:19.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:19.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 2022-03-21T18:49:19.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:49:19.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: 2022-03-21T18:49:19.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:49:19.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:49:19.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:49:19.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:49:19.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:49:19.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:49:19.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:49:19.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:49:19.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:49:19.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:49:19.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:49:19.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:49:19.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:49:19.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:49:19.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:49:19.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:49:19.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:49:19.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:49:19.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:49:19.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:49:19.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:49:19.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:49:19.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise 2022-03-21T18:49:19.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:49:19.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:49:19.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:49:19.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:49:19.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:19 smithi016 conmon[32727]: 2022-03-21T18:49:19.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:20 smithi016 conmon[32727]: 2022-03-21T18:49:20.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:20 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:49:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:49:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:49:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:20.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:49:20.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:20.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:49:20.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:20.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:20.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:20.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:49:20.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:20.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:20.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:20.925 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:20 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:49:20.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:49:20.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:20.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:49:20.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:20.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:20.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:20.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:49:20.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:20.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:49:20.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:20.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:49:20.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:20.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:20.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:20.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:49:20.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:20.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:20.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:20.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:49:20.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:49:20.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:49:20.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:49:20.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:49:20.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:49:20.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:49:20.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:49:20.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:49:20.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:49:20.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:49:20.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:49:20.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:49:20.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:49:20.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:49:20.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:49:20.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:49:20.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:49:20.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:49:20.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:49:20.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:49:20.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:49:20.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:49:20.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:49:20.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:49:20.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:49:20.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:49:20.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:20.948 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:20 smithi016 ceph-mon[22010]: pgmap v2916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:20.948 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:20 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:49:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:49:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:21.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:21.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:49:21.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:21.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:49:21.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:21.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:49:21.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:21.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:21.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:21.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:49:21.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:21.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:21.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:21.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:20 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:49:21.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:49:21.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:49:21.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:49:21.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:49:21.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:49:21.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:49:21.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:49:21.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:49:21.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:49:21.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:49:21.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:49:21.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:49:21.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:49:21.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:49:21.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:49:21.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:49:21.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:49:21.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:21.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:49:21.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:49:21.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:49:21.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:49:21.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:49:21.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:49:21.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:49:21.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:49:21.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:49:21.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:49:21.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:49:21.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:49:21.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:49:21.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:49:21.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:49:21.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:49:21.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:49:21.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:49:21.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:49:21.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:49:21.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:49:21.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:49:21.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:49:21.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:49:21.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:49:21.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:49:21.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:49:21.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:49:21.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:49:21.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:49:21.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:49:21.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:20 smithi079 ceph-mon[23158]: pgmap v2916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:21.250 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:21 smithi016 conmon[32727]: 2022-03-21T18:49:21.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:22 smithi016 conmon[32727]: 2022-03-21T18:49:22.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:22 smithi016 ceph-mon[22010]: pgmap v2917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:22 smithi079 ceph-mon[23158]: pgmap v2917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:23 smithi016 conmon[32727]: 2022-03-21T18:49:23.682+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:24 smithi016 conmon[32727]: 2022-03-21T18:49:24.712+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:24 smithi016 ceph-mon[22010]: pgmap v2918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:24 smithi079 ceph-mon[23158]: pgmap v2918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:25 smithi016 conmon[32727]: 2022-03-21T18:49:25.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:26 smithi016 conmon[32727]: 2022-03-21T18:49:26.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:26 smithi016 ceph-mon[22010]: pgmap v2919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:26 smithi079 ceph-mon[23158]: pgmap v2919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:27 smithi016 conmon[32727]: 2022-03-21T18:49:27.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:28 smithi016 ceph-mon[22010]: pgmap v2920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:28 smithi016 conmon[32727]: 2022-03-21T18:49:28.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:28 smithi079 ceph-mon[23158]: pgmap v2920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:29 smithi016 conmon[32727]: 2022-03-21T18:49:29.729+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:30 smithi016 conmon[32727]: 2022-03-21T18:49:30.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:30 smithi016 ceph-mon[22010]: pgmap v2921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:30 smithi079 ceph-mon[23158]: pgmap v2921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:31 smithi016 conmon[32727]: 2022-03-21T18:49:31.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:32 smithi016 ceph-mon[22010]: pgmap v2922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:32 smithi016 conmon[32727]: 2022-03-21T18:49:32.697+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:32 smithi079 ceph-mon[23158]: pgmap v2922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:33 smithi016 conmon[32727]: 2022-03-21T18:49:33.739+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:34 smithi016 ceph-mon[22010]: pgmap v2923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:35.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:34 smithi016 conmon[32727]: 2022-03-21T18:49:34.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:34 smithi079 ceph-mon[23158]: pgmap v2923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:35 smithi016 conmon[32727]: 2022-03-21T18:49:35.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:36 smithi016 ceph-mon[22010]: pgmap v2924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:36 smithi016 conmon[32727]: 2022-03-21T18:49:36.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:36 smithi079 ceph-mon[23158]: pgmap v2924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:37 smithi016 conmon[32727]: 2022-03-21T18:49:37.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:38 smithi016 ceph-mon[22010]: pgmap v2925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:38 smithi016 conmon[32727]: 2022-03-21T18:49:38.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:38 smithi079 ceph-mon[23158]: pgmap v2925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:39 smithi016 conmon[32727]: 2022-03-21T18:49:39.687+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:40 smithi016 ceph-mon[22010]: pgmap v2926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:40 smithi016 conmon[32727]: 2022-03-21T18:49:40.706+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:40 smithi079 ceph-mon[23158]: pgmap v2926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:41 smithi016 conmon[32727]: 2022-03-21T18:49:41.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:42 smithi016 ceph-mon[22010]: pgmap v2927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:42 smithi016 conmon[32727]: 2022-03-21T18:49:42.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:42 smithi079 ceph-mon[23158]: pgmap v2927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:43 smithi016 conmon[32727]: 2022-03-21T18:49:43.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:44 smithi016 ceph-mon[22010]: pgmap v2928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:44 smithi016 conmon[32727]: 2022-03-21T18:49:44.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:44 smithi079 ceph-mon[23158]: pgmap v2928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:45 smithi016 conmon[32727]: 2022-03-21T18:49:45.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:46 smithi016 ceph-mon[22010]: pgmap v2929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:46 smithi016 conmon[32727]: 2022-03-21T18:49:46.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:46 smithi079 ceph-mon[23158]: pgmap v2929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:47 smithi016 conmon[32727]: 2022-03-21T18:49:47.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:48 smithi016 ceph-mon[22010]: pgmap v2930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:48 smithi016 conmon[32727]: 2022-03-21T18:49:48.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:48 smithi079 ceph-mon[23158]: pgmap v2930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:49 smithi016 conmon[32727]: 2022-03-21T18:49:49.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:50.856 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:50 smithi016 conmon[32727]: 2022-03-21T18:49:50.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:50 smithi016 ceph-mon[22010]: pgmap v2931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:50 smithi079 ceph-mon[23158]: pgmap v2931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:51.861 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:51 smithi016 conmon[32727]: 2022-03-21T18:49:51.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:51.862 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:52.863 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:52 smithi016 conmon[32727]: 2022-03-21T18:49:52.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:52 smithi016 ceph-mon[22010]: pgmap v2932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:52 smithi079 ceph-mon[23158]: pgmap v2932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:53.868 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:53 smithi016 conmon[32727]: 2022-03-21T18:49:53.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:54.869 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:54 smithi016 conmon[32727]: 2022-03-21T18:49:54.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:54 smithi016 ceph-mon[22010]: pgmap v2933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:54 smithi079 ceph-mon[23158]: pgmap v2933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:49:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:49:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:55.874 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:55 smithi016 conmon[32727]: 2022-03-21T18:49:55.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:56.876 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:56 smithi016 conmon[32727]: 2022-03-21T18:49:56.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:56 smithi016 ceph-mon[22010]: pgmap v2934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:56 smithi079 ceph-mon[23158]: pgmap v2934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:49:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:57.880 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:57 smithi016 conmon[32727]: 2022-03-21T18:49:57.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:58 smithi016 conmon[32727]: 2022-03-21T18:49:58.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:49:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: pgmap v2935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:49:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:49:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:49:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:49:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:49:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:49:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: pgmap v2935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:49:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:49:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:49:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:49:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:49:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:49:59 smithi016 conmon[32727]: 2022-03-21T18:49:59.696+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:49:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:49:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:00 smithi016 conmon[32727]: 2022-03-21T18:50:00.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: pgmap v2936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops 2022-03-21T18:50:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:50:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:50:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:50:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:50:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:50:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops 2022-03-21T18:50:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: pgmap v2936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops 2022-03-21T18:50:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T18:50:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T18:50:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T18:50:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:50:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T18:50:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5240 sec, osd.4 has slow ops 2022-03-21T18:50:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:01 smithi016 conmon[32727]: 2022-03-21T18:50:01.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:02 smithi016 conmon[32727]: 2022-03-21T18:50:02.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:02 smithi016 ceph-mon[22010]: pgmap v2937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:50:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:02 smithi079 ceph-mon[23158]: pgmap v2937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:50:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:03 smithi016 conmon[32727]: 2022-03-21T18:50:03.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:04 smithi016 conmon[32727]: 2022-03-21T18:50:04.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:04 smithi016 ceph-mon[22010]: pgmap v2938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:50:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:04 smithi079 ceph-mon[23158]: pgmap v2938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:50:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:05 smithi016 conmon[32727]: 2022-03-21T18:50:05.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:06 smithi016 ceph-mon[22010]: pgmap v2939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:06 smithi016 conmon[32727]: 2022-03-21T18:50:06.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:06 smithi079 ceph-mon[23158]: pgmap v2939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:07 smithi016 conmon[32727]: 2022-03-21T18:50:07.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:08 smithi016 ceph-mon[22010]: pgmap v2940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:08 smithi016 conmon[32727]: 2022-03-21T18:50:08.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:08 smithi079 ceph-mon[23158]: pgmap v2940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:09 smithi016 conmon[32727]: 2022-03-21T18:50:09.797+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:10 smithi016 conmon[32727]: 2022-03-21T18:50:10.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:10 smithi016 ceph-mon[22010]: pgmap v2941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:10 smithi079 ceph-mon[23158]: pgmap v2941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:12.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:11 smithi016 conmon[32727]: 2022-03-21T18:50:11.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:13.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:12 smithi016 conmon[32727]: 2022-03-21T18:50:12.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:12 smithi016 ceph-mon[22010]: pgmap v2942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:12 smithi079 ceph-mon[23158]: pgmap v2942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:13 smithi016 conmon[32727]: 2022-03-21T18:50:13.804+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:15.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:14 smithi016 ceph-mon[22010]: pgmap v2943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:15.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:15.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:15.171 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:14 smithi016 conmon[32727]: 2022-03-21T18:50:14.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:14 smithi079 ceph-mon[23158]: pgmap v2943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:15 smithi016 conmon[32727]: 2022-03-21T18:50:15.832+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:16 smithi016 ceph-mon[22010]: pgmap v2944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:17.222 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:17.223 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:16 smithi016 conmon[32727]: 2022-03-21T18:50:16.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:16 smithi079 ceph-mon[23158]: pgmap v2944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:17 smithi016 conmon[32727]: 2022-03-21T18:50:17.790+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:18 smithi016 ceph-mon[22010]: pgmap v2945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:18 smithi016 conmon[32727]: 2022-03-21T18:50:18.799+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:18 smithi079 ceph-mon[23158]: pgmap v2945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[32727]: 2022-03-21T18:50:19.756+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:50:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:50:20.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:50:20.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:20.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: 2022-03-21T18:50:19.980+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:20.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:50:20.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:20.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: yield 2022-03-21T18:50:20.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:20.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:20.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:20.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:50:20.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:20.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:50:20.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:20.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:50:20.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:20.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:50:20.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:20.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:50:20.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:20.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:20.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:50:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:50:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:50:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:20 smithi016 ceph-mon[22010]: pgmap v2946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:21.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:20 smithi016 conmon[32727]: 2022-03-21T18:50:20.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:20 smithi079 ceph-mon[23158]: pgmap v2946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:21 smithi016 conmon[32727]: 2022-03-21T18:50:21.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:50:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:50:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:50:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:50:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:50:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:50:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:50:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:21 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:50:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:50:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:50:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:50:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:50:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:50:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:50:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:50:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:21 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:50:23.055 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:22 smithi016 conmon[32727]: 2022-03-21T18:50:22.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:23.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: pgmap v2947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:23.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:23.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:50:23.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.061 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.063 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.063 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: pgmap v2947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:50:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T18:50:23.235 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:23.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 2022-03-21T18:50:23.052+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:23.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:50:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: yield 2022-03-21T18:50:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:50:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:50:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:50:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:50:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:50:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 2022-03-21T18:50:23.053+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:50:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:50:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: yield 2022-03-21T18:50:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:50:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:50:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:50:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:50:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:50:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 2022-03-21T18:50:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:50:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: 2022-03-21T18:50:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:50:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:50:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:50:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:50:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:50:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:50:23.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:50:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:50:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:50:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:50:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:50:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:50:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:50:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:50:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:50:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:50:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:50:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:50:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:50:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:50:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:50:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:50:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise 2022-03-21T18:50:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:50:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:50:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:50:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:50:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:23 smithi016 conmon[32727]: 2022-03-21T18:50:23.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:50:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:50:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:23 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:50:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:23 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:50:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T18:50:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T18:50:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:23 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T18:50:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:23 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T18:50:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:24 smithi016 conmon[32727]: 2022-03-21T18:50:24.817+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:50:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:50:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:50:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:50:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:50:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:50:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:50:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:50:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:50:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:50:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:50:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:50:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:50:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:50:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:50:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:50:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:50:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:50:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:50:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:50:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:50:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:50:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:50:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:50:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:50:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:50:25.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:50:25.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:50:25.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:50:25.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:50:25.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:50:25.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:50:25.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:50:25.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:50:25.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:50:25.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:50:25.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:50:25.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:50:25.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:50:25.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:50:25.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:24 smithi016 ceph-mon[22010]: pgmap v2948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:25.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:25.196 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:50:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:50:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:50:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:50:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:50:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:50:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:50:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:50:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:50:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:50:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:50:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:50:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:50:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:50:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:50:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:50:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:50:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:50:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:50:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:50:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:50:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:50:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:50:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:50:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:50:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:50:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:50:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:50:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:50:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:50:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:50:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:50:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:50:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:50:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:50:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:50:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:50:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:50:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:50:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:50:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:50:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:50:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:50:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:50:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:50:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:50:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:50:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:50:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:50:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:50:25.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:50:25.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:50:25.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:50:25.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:24 smithi079 ceph-mon[23158]: pgmap v2948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:25.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:25.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:25 smithi016 conmon[32727]: 2022-03-21T18:50:25.799+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:26 smithi016 ceph-mon[22010]: pgmap v2949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:26 smithi016 conmon[32727]: 2022-03-21T18:50:26.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:26 smithi079 ceph-mon[23158]: pgmap v2949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:27 smithi016 ceph-mon[22010]: pgmap v2950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:28.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:27 smithi016 conmon[32727]: 2022-03-21T18:50:27.849+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:28.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:27 smithi079 ceph-mon[23158]: pgmap v2950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:28 smithi016 conmon[32727]: 2022-03-21T18:50:28.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:29 smithi016 ceph-mon[22010]: pgmap v2951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:29 smithi016 conmon[32727]: 2022-03-21T18:50:29.876+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:29 smithi079 ceph-mon[23158]: pgmap v2951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:31.094 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:30 smithi016 conmon[32727]: 2022-03-21T18:50:30.832+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:31 smithi016 conmon[32727]: 2022-03-21T18:50:31.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:32 smithi016 ceph-mon[22010]: pgmap v2952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:32 smithi079 ceph-mon[23158]: pgmap v2952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:32 smithi016 conmon[32727]: 2022-03-21T18:50:32.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:34.106 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:33 smithi016 conmon[32727]: 2022-03-21T18:50:33.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:34 smithi016 ceph-mon[22010]: pgmap v2953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:34 smithi079 ceph-mon[23158]: pgmap v2953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:35.108 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:34 smithi016 conmon[32727]: 2022-03-21T18:50:34.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:36 smithi016 ceph-mon[22010]: pgmap v2954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:35 smithi016 conmon[32727]: 2022-03-21T18:50:35.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:36 smithi079 ceph-mon[23158]: pgmap v2954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:37.115 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:36 smithi016 conmon[32727]: 2022-03-21T18:50:36.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:38.119 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:37 smithi016 conmon[32727]: 2022-03-21T18:50:37.838+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:38 smithi016 ceph-mon[22010]: pgmap v2955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:38 smithi079 ceph-mon[23158]: pgmap v2955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:39.120 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:38 smithi016 conmon[32727]: 2022-03-21T18:50:38.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:40.125 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:39 smithi016 conmon[32727]: 2022-03-21T18:50:39.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:40 smithi016 ceph-mon[22010]: pgmap v2956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:40 smithi079 ceph-mon[23158]: pgmap v2956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:41.127 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:40 smithi016 conmon[32727]: 2022-03-21T18:50:40.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:42.132 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:41 smithi016 conmon[32727]: 2022-03-21T18:50:41.809+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:42 smithi016 ceph-mon[22010]: pgmap v2957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:42 smithi079 ceph-mon[23158]: pgmap v2957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:43.133 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:42 smithi016 conmon[32727]: 2022-03-21T18:50:42.808+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:44.137 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:43 smithi016 conmon[32727]: 2022-03-21T18:50:43.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:44 smithi016 ceph-mon[22010]: pgmap v2958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:44 smithi079 ceph-mon[23158]: pgmap v2958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:45.139 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:44 smithi016 conmon[32727]: 2022-03-21T18:50:44.812+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:46.144 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:45 smithi016 conmon[32727]: 2022-03-21T18:50:45.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:46 smithi016 ceph-mon[22010]: pgmap v2959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:46 smithi079 ceph-mon[23158]: pgmap v2959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:47.145 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:46 smithi016 conmon[32727]: 2022-03-21T18:50:46.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:48.150 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:47 smithi016 conmon[32727]: 2022-03-21T18:50:47.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:48.460 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:48 smithi016 ceph-mon[22010]: pgmap v2960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:48 smithi079 ceph-mon[23158]: pgmap v2960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:49.152 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:48 smithi016 conmon[32727]: 2022-03-21T18:50:48.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5289 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5289 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:50.157 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:49 smithi016 conmon[32727]: 2022-03-21T18:50:49.797+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:50 smithi016 ceph-mon[22010]: pgmap v2961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:50 smithi079 ceph-mon[23158]: pgmap v2961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:51.158 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:50 smithi016 conmon[32727]: 2022-03-21T18:50:50.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:52.163 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:51 smithi016 conmon[32727]: 2022-03-21T18:50:51.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:52 smithi016 ceph-mon[22010]: pgmap v2962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:52 smithi079 ceph-mon[23158]: pgmap v2962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:52 smithi016 conmon[32727]: 2022-03-21T18:50:52.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:53 smithi016 conmon[32727]: 2022-03-21T18:50:53.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:54 smithi079 ceph-mon[23158]: pgmap v2963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:54 smithi016 ceph-mon[22010]: pgmap v2963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:54 smithi016 conmon[32727]: 2022-03-21T18:50:54.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:55 smithi016 conmon[32727]: 2022-03-21T18:50:55.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:56 smithi079 ceph-mon[23158]: pgmap v2964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:56 smithi016 ceph-mon[22010]: pgmap v2964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:50:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:56 smithi016 conmon[32727]: 2022-03-21T18:50:56.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:57 smithi016 conmon[32727]: 2022-03-21T18:50:57.828+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:58 smithi079 ceph-mon[23158]: pgmap v2965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:58 smithi016 ceph-mon[22010]: pgmap v2965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:50:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:58 smithi016 conmon[32727]: 2022-03-21T18:50:58.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:50:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:50:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:50:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:50:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:50:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:50:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:50:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:50:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:50:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:50:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:50:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:50:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:50:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:50:59 smithi016 conmon[32727]: 2022-03-21T18:50:59.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:00 smithi079 ceph-mon[23158]: pgmap v2966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:00 smithi016 ceph-mon[22010]: pgmap v2966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:00 smithi016 conmon[32727]: 2022-03-21T18:51:00.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:01 smithi016 conmon[32727]: 2022-03-21T18:51:01.727+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:02 smithi079 ceph-mon[23158]: pgmap v2967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:02 smithi016 ceph-mon[22010]: pgmap v2967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:02 smithi016 conmon[32727]: 2022-03-21T18:51:02.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:03 smithi016 conmon[32727]: 2022-03-21T18:51:03.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:04 smithi079 ceph-mon[23158]: pgmap v2968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:04 smithi016 ceph-mon[22010]: pgmap v2968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:04 smithi016 conmon[32727]: 2022-03-21T18:51:04.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:05 smithi016 conmon[32727]: 2022-03-21T18:51:05.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:06 smithi079 ceph-mon[23158]: pgmap v2969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:06 smithi016 ceph-mon[22010]: pgmap v2969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:06 smithi016 conmon[32727]: 2022-03-21T18:51:06.882+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:07 smithi016 conmon[32727]: 2022-03-21T18:51:07.912+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:08 smithi079 ceph-mon[23158]: pgmap v2970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:08 smithi016 ceph-mon[22010]: pgmap v2970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:09.215 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:08 smithi016 conmon[32727]: 2022-03-21T18:51:08.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:10 smithi016 ceph-mon[22010]: pgmap v2971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:09 smithi016 conmon[32727]: 2022-03-21T18:51:09.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:10 smithi079 ceph-mon[23158]: pgmap v2971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:10 smithi016 conmon[32727]: 2022-03-21T18:51:10.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:12.226 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:11 smithi016 conmon[32727]: 2022-03-21T18:51:11.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:12 smithi016 ceph-mon[22010]: pgmap v2972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:12 smithi079 ceph-mon[23158]: pgmap v2972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:12 smithi016 conmon[32727]: 2022-03-21T18:51:12.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:14 smithi016 ceph-mon[22010]: pgmap v2973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:14 smithi016 conmon[32727]: 2022-03-21T18:51:14.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:14 smithi079 ceph-mon[23158]: pgmap v2973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:15 smithi016 conmon[32727]: 2022-03-21T18:51:15.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:16.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:16 smithi016 ceph-mon[22010]: pgmap v2974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:16 smithi016 conmon[32727]: 2022-03-21T18:51:16.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:16 smithi079 ceph-mon[23158]: pgmap v2974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:17 smithi016 conmon[32727]: 2022-03-21T18:51:17.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:18 smithi016 ceph-mon[22010]: pgmap v2975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:18 smithi016 conmon[32727]: 2022-03-21T18:51:18.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:18 smithi079 ceph-mon[23158]: pgmap v2975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:19 smithi016 conmon[32727]: 2022-03-21T18:51:19.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:19.810 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:20 smithi016 ceph-mon[22010]: pgmap v2976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:20 smithi016 conmon[32727]: 2022-03-21T18:51:20.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:20 smithi079 ceph-mon[23158]: pgmap v2976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:21 smithi016 conmon[32727]: 2022-03-21T18:51:21.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:22.259 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:21 smithi016 conmon[32727]: 2022-03-21T18:51:21.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:22 smithi016 ceph-mon[22010]: pgmap v2977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:22 smithi079 ceph-mon[23158]: pgmap v2977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:23.260 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:22 smithi016 conmon[32727]: 2022-03-21T18:51:22.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:51:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:51:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:51:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:51:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:51:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:51:24.264 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:23 smithi016 conmon[32727]: 2022-03-21T18:51:23.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:24 smithi016 ceph-mon[22010]: pgmap v2978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:24 smithi079 ceph-mon[23158]: pgmap v2978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:25.265 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:24 smithi016 conmon[32727]: 2022-03-21T18:51:24.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:26.270 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:25 smithi016 conmon[32727]: 2022-03-21T18:51:25.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:26.271 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: 2022-03-21T18:51:26.156+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:26.271 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:51:26.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:26.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: yield 2022-03-21T18:51:26.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:26.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:26.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:26.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:51:26.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:26.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:51:26.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:26.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:51:26.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:26.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:51:26.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:26.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:51:26.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:26.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:26.278 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:26 smithi016 ceph-mon[22010]: pgmap v2979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:26 smithi079 ceph-mon[23158]: pgmap v2979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:27.272 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:27 smithi016 conmon[32727]: 2022-03-21T18:51:27.007+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:27 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:51:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:51:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:51:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:51:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:51:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:51:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:27 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:51:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:27 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:51:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:51:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:51:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:51:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:51:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:51:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:27 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:51:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:28 smithi016 conmon[32727]: 2022-03-21T18:51:28.028+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:28 smithi016 ceph-mon[22010]: pgmap v2980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:28 smithi079 ceph-mon[23158]: pgmap v2980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:29.278 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:28 smithi016 conmon[32727]: 2022-03-21T18:51:28.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:29.279 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 2022-03-21T18:51:29.228+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:29.279 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:51:29.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:29.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: yield 2022-03-21T18:51:29.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:29.281 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:29.281 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:29.281 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:51:29.282 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:29.282 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:51:29.283 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:29.283 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:51:29.283 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:29.284 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:51:29.284 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:29.284 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:51:29.285 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:29.285 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:29.285 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:29.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 2022-03-21T18:51:29.229+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:51:29.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:51:29.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:29.287 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: yield 2022-03-21T18:51:29.287 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:29.287 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:29.288 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:29.288 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:51:29.288 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:29.289 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:51:29.289 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:29.289 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:51:29.290 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:29.290 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:51:29.290 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:29.291 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:51:29.291 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:29.291 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:29.292 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:29.292 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 2022-03-21T18:51:29.292 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:51:29.293 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: 2022-03-21T18:51:29.293 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:51:29.293 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:51:29.294 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:51:29.294 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:51:29.294 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:51:29.295 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:51:29.295 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:51:29.295 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:51:29.296 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:51:29.296 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:51:29.296 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:51:29.297 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:51:29.297 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:51:29.297 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:51:29.298 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:51:29.298 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:51:29.298 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:51:29.299 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:51:29.299 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:51:29.299 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:51:29.300 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:51:29.300 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:51:29.300 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise 2022-03-21T18:51:29.301 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:51:29.301 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:51:29.301 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:51:29.302 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:51:29.302 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:30.284 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:29 smithi016 conmon[32727]: 2022-03-21T18:51:29.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:51:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:51:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:51:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:51:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:51:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:30 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:51:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:51:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:30.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:51:30.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:30.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:51:30.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:30.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:51:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:30.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:51:30.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:30.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:51:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:51:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:51:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:51:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:51:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:51:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:51:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:51:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:51:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:51:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:51:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:51:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:51:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:51:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:51:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:51:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:51:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:51:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:51:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:51:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:51:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:51:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:51:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:51:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:51:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:51:30.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:51:30.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:51:30.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:51:30.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:30 smithi016 ceph-mon[22010]: pgmap v2981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:51:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:51:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:51:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:51:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:51:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:30 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:51:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:51:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:51:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:51:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:51:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:51:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:51:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:51:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:51:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:51:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:51:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:51:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:51:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:51:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:51:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:51:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:51:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:51:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:51:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:51:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:51:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:51:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:51:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:51:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:51:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:51:30.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:51:30.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:51:30.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:51:30.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:51:30.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:51:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:51:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:51:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:51:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:51:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:51:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:51:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:51:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:51:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:51:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:51:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:51:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:51:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:51:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:51:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:51:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:51:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:51:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:30 smithi079 ceph-mon[23158]: pgmap v2981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:31.285 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:30 smithi016 conmon[32727]: 2022-03-21T18:51:30.983+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:32.290 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:31 smithi016 conmon[32727]: 2022-03-21T18:51:31.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:32 smithi016 ceph-mon[22010]: pgmap v2982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:32 smithi079 ceph-mon[23158]: pgmap v2982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:33.291 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:32 smithi016 conmon[32727]: 2022-03-21T18:51:32.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:34.295 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:33 smithi016 conmon[32727]: 2022-03-21T18:51:33.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:34 smithi016 ceph-mon[22010]: pgmap v2983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:34 smithi079 ceph-mon[23158]: pgmap v2983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:35.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:34 smithi016 conmon[32727]: 2022-03-21T18:51:34.912+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:36.302 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:35 smithi016 conmon[32727]: 2022-03-21T18:51:35.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:36 smithi016 ceph-mon[22010]: pgmap v2984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:36 smithi079 ceph-mon[23158]: pgmap v2984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:37.303 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:36 smithi016 conmon[32727]: 2022-03-21T18:51:36.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:38.308 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:37 smithi016 conmon[32727]: 2022-03-21T18:51:37.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:38 smithi016 ceph-mon[22010]: pgmap v2985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:38 smithi079 ceph-mon[23158]: pgmap v2985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:39.310 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:38 smithi016 conmon[32727]: 2022-03-21T18:51:38.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:40.315 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:39 smithi016 conmon[32727]: 2022-03-21T18:51:39.983+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:40 smithi016 ceph-mon[22010]: pgmap v2986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:40 smithi079 ceph-mon[23158]: pgmap v2986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:41.317 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:41 smithi016 conmon[32727]: 2022-03-21T18:51:41.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:42.322 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:42 smithi016 conmon[32727]: 2022-03-21T18:51:42.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:42 smithi016 ceph-mon[22010]: pgmap v2987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:42 smithi079 ceph-mon[23158]: pgmap v2987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:43.323 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:43 smithi016 conmon[32727]: 2022-03-21T18:51:43.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:44.329 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:44 smithi016 conmon[32727]: 2022-03-21T18:51:44.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:44 smithi016 ceph-mon[22010]: pgmap v2988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:44 smithi079 ceph-mon[23158]: pgmap v2988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:45.330 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:45 smithi016 conmon[32727]: 2022-03-21T18:51:45.047+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:46.336 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:46 smithi016 conmon[32727]: 2022-03-21T18:51:46.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:46 smithi016 ceph-mon[22010]: pgmap v2989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:46 smithi079 ceph-mon[23158]: pgmap v2989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:47.337 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:47 smithi016 conmon[32727]: 2022-03-21T18:51:47.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:48.343 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:48 smithi016 conmon[32727]: 2022-03-21T18:51:48.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:48 smithi016 ceph-mon[22010]: pgmap v2990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:48 smithi079 ceph-mon[23158]: pgmap v2990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:49.345 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:49 smithi016 conmon[32727]: 2022-03-21T18:51:49.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:50.349 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:49 smithi016 conmon[32727]: 2022-03-21T18:51:49.973+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:50 smithi016 ceph-mon[22010]: pgmap v2991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:50 smithi079 ceph-mon[23158]: pgmap v2991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:51.351 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:50 smithi016 conmon[32727]: 2022-03-21T18:51:50.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:52.356 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:51 smithi016 conmon[32727]: 2022-03-21T18:51:51.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:52 smithi016 ceph-mon[22010]: pgmap v2992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:52 smithi079 ceph-mon[23158]: pgmap v2992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:53.357 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:52 smithi016 conmon[32727]: 2022-03-21T18:51:52.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:54.362 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:53 smithi016 conmon[32727]: 2022-03-21T18:51:53.943+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:54 smithi016 ceph-mon[22010]: pgmap v2993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:54 smithi079 ceph-mon[23158]: pgmap v2993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:55.363 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:54 smithi016 conmon[32727]: 2022-03-21T18:51:54.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:56.369 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:55 smithi016 conmon[32727]: 2022-03-21T18:51:55.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:56 smithi016 ceph-mon[22010]: pgmap v2994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:56 smithi079 ceph-mon[23158]: pgmap v2994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:51:57.370 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:56 smithi016 conmon[32727]: 2022-03-21T18:51:56.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:58.375 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:57 smithi016 conmon[32727]: 2022-03-21T18:51:57.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: pgmap v2995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:51:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:51:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:51:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:51:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: pgmap v2995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:51:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:51:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:51:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:51:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:51:59.377 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:51:59 smithi016 conmon[32727]: 2022-03-21T18:51:59.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:51:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:51:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:51:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:51:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:51:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:00.382 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:00 smithi016 conmon[32727]: 2022-03-21T18:52:00.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:00 smithi016 ceph-mon[22010]: pgmap v2996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:00 smithi079 ceph-mon[23158]: pgmap v2996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:01.383 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:01 smithi016 conmon[32727]: 2022-03-21T18:52:01.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:02.388 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:02 smithi016 conmon[32727]: 2022-03-21T18:52:02.072+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:02 smithi016 ceph-mon[22010]: pgmap v2997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:02 smithi079 ceph-mon[23158]: pgmap v2997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:03.389 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:03 smithi016 conmon[32727]: 2022-03-21T18:52:03.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:04.395 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:04 smithi016 conmon[32727]: 2022-03-21T18:52:04.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:04 smithi016 ceph-mon[22010]: pgmap v2998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:04 smithi079 ceph-mon[23158]: pgmap v2998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:05.396 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:05 smithi016 conmon[32727]: 2022-03-21T18:52:05.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:06.402 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:06 smithi016 conmon[32727]: 2022-03-21T18:52:06.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:06 smithi016 ceph-mon[22010]: pgmap v2999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:06 smithi079 ceph-mon[23158]: pgmap v2999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:07.403 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:07 smithi016 conmon[32727]: 2022-03-21T18:52:07.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:08.405 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:08 smithi016 conmon[32727]: 2022-03-21T18:52:08.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:08 smithi016 ceph-mon[22010]: pgmap v3000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:08 smithi079 ceph-mon[23158]: pgmap v3000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:09.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:09 smithi016 conmon[32727]: 2022-03-21T18:52:09.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:10 smithi016 conmon[32727]: 2022-03-21T18:52:10.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:10 smithi079 ceph-mon[23158]: pgmap v3001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:10 smithi016 ceph-mon[22010]: pgmap v3001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:11 smithi016 conmon[32727]: 2022-03-21T18:52:11.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:12.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:12 smithi016 conmon[32727]: 2022-03-21T18:52:12.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:12 smithi079 ceph-mon[23158]: pgmap v3002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:12 smithi016 ceph-mon[22010]: pgmap v3002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:13 smithi016 conmon[32727]: 2022-03-21T18:52:13.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:14 smithi016 conmon[32727]: 2022-03-21T18:52:14.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:14 smithi079 ceph-mon[23158]: pgmap v3003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:14 smithi016 ceph-mon[22010]: pgmap v3003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:15 smithi016 conmon[32727]: 2022-03-21T18:52:15.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:16 smithi016 ceph-mon[22010]: pgmap v3004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:16 smithi016 conmon[32727]: 2022-03-21T18:52:16.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:16 smithi079 ceph-mon[23158]: pgmap v3004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:17 smithi016 conmon[32727]: 2022-03-21T18:52:17.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:18 smithi016 ceph-mon[22010]: pgmap v3005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:18 smithi016 conmon[32727]: 2022-03-21T18:52:18.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:18 smithi079 ceph-mon[23158]: pgmap v3005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:19 smithi016 conmon[32727]: 2022-03-21T18:52:19.236+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:20.453 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:20 smithi016 conmon[32727]: 2022-03-21T18:52:20.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:20 smithi079 ceph-mon[23158]: pgmap v3006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:20 smithi016 ceph-mon[22010]: pgmap v3006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:21.454 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:21 smithi016 conmon[32727]: 2022-03-21T18:52:21.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:22.459 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:22 smithi016 conmon[32727]: 2022-03-21T18:52:22.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:22 smithi079 ceph-mon[23158]: pgmap v3007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:22 smithi016 ceph-mon[22010]: pgmap v3007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:23.461 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:23 smithi016 conmon[32727]: 2022-03-21T18:52:23.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:24 smithi016 ceph-mon[22010]: pgmap v3008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:24.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:24 smithi016 conmon[32727]: 2022-03-21T18:52:24.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:24 smithi079 ceph-mon[23158]: pgmap v3008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:25 smithi016 conmon[32727]: 2022-03-21T18:52:25.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:26 smithi016 ceph-mon[22010]: pgmap v3009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:26 smithi016 conmon[32727]: 2022-03-21T18:52:26.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:26 smithi079 ceph-mon[23158]: pgmap v3009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:27 smithi016 conmon[32727]: 2022-03-21T18:52:27.286+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:28 smithi016 ceph-mon[22010]: pgmap v3010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:28 smithi016 conmon[32727]: 2022-03-21T18:52:28.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:28 smithi079 ceph-mon[23158]: pgmap v3010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[32727]: 2022-03-21T18:52:29.292+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:52:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:52:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:52:29.677 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:29.677 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:29.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:52:29.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:52:29.678 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:52:30.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: 2022-03-21T18:52:29.708+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:52:30.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:30.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: yield 2022-03-21T18:52:30.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:30.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:52:30.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:52:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:30.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:52:30.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:30.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:52:30.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:30.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:52:30.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:30.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:30.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:30 smithi016 conmon[32727]: 2022-03-21T18:52:30.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:30 smithi016 ceph-mon[22010]: pgmap v3011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:52:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:52:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:52:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:52:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:52:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:52:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:30 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:52:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:30 smithi079 ceph-mon[23158]: pgmap v3011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:52:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:52:30.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:30.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:30.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:30.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:52:30.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:30.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:52:30.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:30.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:52:30.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:30.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:30.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:30.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:52:30.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:30.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:30.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:30.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:30 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:52:31.486 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:31 smithi016 conmon[32727]: 2022-03-21T18:52:31.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:32.491 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[32727]: 2022-03-21T18:52:32.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:32.783 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:32.783 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:32 smithi016 ceph-mon[22010]: pgmap v3012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:32 smithi079 ceph-mon[23158]: pgmap v3012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:33.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 2022-03-21T18:52:32.780+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:52:33.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: yield 2022-03-21T18:52:33.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:52:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:52:33.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:52:33.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:52:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 2022-03-21T18:52:32.781+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:52:33.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:52:33.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: yield 2022-03-21T18:52:33.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:52:33.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:52:33.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:52:33.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:52:33.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 2022-03-21T18:52:33.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:52:33.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: 2022-03-21T18:52:33.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:52:33.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:52:33.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:52:33.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:52:33.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:52:33.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:52:33.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:52:33.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:52:33.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:52:33.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:52:33.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:52:33.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:52:33.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:52:33.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:52:33.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:52:33.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:52:33.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:52:33.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:52:33.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:52:33.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:52:33.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:52:33.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:52:33.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise 2022-03-21T18:52:33.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:52:33.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:52:33.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:52:33.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:52:33.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:52:32 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:33 smithi016 conmon[32727]: 2022-03-21T18:52:33.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:33 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:52:33.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:52:33.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:52:33.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:52:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:52:33.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:33 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:52:33.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:52:33.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:52:33.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:52:33.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:52:33.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:52:33.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:52:33.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:52:33.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:52:33.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:52:33.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:52:33.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:52:33.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:52:33.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:52:33.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:52:33.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:52:33.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:52:33.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:52:33.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:52:33.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:52:33.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:52:33.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:52:33.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:52:33.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:52:33.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:52:33.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:52:33.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:52:33.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:52:33.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:52:33.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:52:33.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:52:33.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:52:33.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:52:33.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:52:33.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:52:33.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:33 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:52:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:52:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:52:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:52:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:52:33.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:33 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:52:33.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:52:33.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:52:33.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:52:33.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:52:33.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:52:33.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:52:33.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:52:33.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:52:33.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:52:33.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:52:33.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:52:33.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:52:33.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:52:33.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:52:33.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:52:33.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:52:33.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:33.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:52:33.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:52:33.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:52:33.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:52:33.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:52:33.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:52:33.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:52:33.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:52:33.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:52:33.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:52:33.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:52:33.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:52:33.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:52:33.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:52:33.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:52:33.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:52:33.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:52:33.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:52:33.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:52:33.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:52:33.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:52:33.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:52:33.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:52:33.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:52:33.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:52:33.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:52:33.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:52:33.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:52:33.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:52:33.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:52:33.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:52:34.498 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:34 smithi016 conmon[32727]: 2022-03-21T18:52:34.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:34 smithi016 ceph-mon[22010]: pgmap v3013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:34 smithi079 ceph-mon[23158]: pgmap v3013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:35 smithi016 conmon[32727]: 2022-03-21T18:52:35.271+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:36.504 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:36 smithi016 conmon[32727]: 2022-03-21T18:52:36.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:36 smithi016 ceph-mon[22010]: pgmap v3014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:36 smithi079 ceph-mon[23158]: pgmap v3014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:37 smithi016 conmon[32727]: 2022-03-21T18:52:37.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:38 smithi016 ceph-mon[22010]: pgmap v3015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:38 smithi016 conmon[32727]: 2022-03-21T18:52:38.301+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:38 smithi079 ceph-mon[23158]: pgmap v3015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:39 smithi016 conmon[32727]: 2022-03-21T18:52:39.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:40 smithi016 ceph-mon[22010]: pgmap v3016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:40 smithi016 conmon[32727]: 2022-03-21T18:52:40.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:40 smithi079 ceph-mon[23158]: pgmap v3016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:41 smithi016 conmon[32727]: 2022-03-21T18:52:41.301+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:42 smithi016 ceph-mon[22010]: pgmap v3017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:42 smithi016 conmon[32727]: 2022-03-21T18:52:42.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:42 smithi079 ceph-mon[23158]: pgmap v3017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:43 smithi016 conmon[32727]: 2022-03-21T18:52:43.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:44.528 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:44 smithi016 conmon[32727]: 2022-03-21T18:52:44.259+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:44 smithi016 ceph-mon[22010]: pgmap v3018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:44 smithi079 ceph-mon[23158]: pgmap v3018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:45.530 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:45 smithi016 conmon[32727]: 2022-03-21T18:52:45.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:46.535 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:46 smithi016 conmon[32727]: 2022-03-21T18:52:46.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:46 smithi016 ceph-mon[22010]: pgmap v3019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:46 smithi079 ceph-mon[23158]: pgmap v3019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:47.536 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:47 smithi016 conmon[32727]: 2022-03-21T18:52:47.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:48.541 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:48 smithi016 conmon[32727]: 2022-03-21T18:52:48.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:48 smithi016 ceph-mon[22010]: pgmap v3020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:48 smithi079 ceph-mon[23158]: pgmap v3020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:49.543 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:49 smithi016 conmon[32727]: 2022-03-21T18:52:49.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:50.548 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:50 smithi016 conmon[32727]: 2022-03-21T18:52:50.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:50 smithi016 ceph-mon[22010]: pgmap v3021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:50 smithi079 ceph-mon[23158]: pgmap v3021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:51.549 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:51 smithi016 conmon[32727]: 2022-03-21T18:52:51.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:52 smithi016 conmon[32727]: 2022-03-21T18:52:52.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:52 smithi016 ceph-mon[22010]: pgmap v3022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:52 smithi079 ceph-mon[23158]: pgmap v3022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:53 smithi016 conmon[32727]: 2022-03-21T18:52:53.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:54 smithi016 conmon[32727]: 2022-03-21T18:52:54.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:54 smithi016 ceph-mon[22010]: pgmap v3023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:54 smithi079 ceph-mon[23158]: pgmap v3023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:55 smithi016 conmon[32727]: 2022-03-21T18:52:55.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:56.567 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:56 smithi016 conmon[32727]: 2022-03-21T18:52:56.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:56 smithi016 ceph-mon[22010]: pgmap v3024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:56 smithi079 ceph-mon[23158]: pgmap v3024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:52:57.568 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:57 smithi016 conmon[32727]: 2022-03-21T18:52:57.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:58.573 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:58 smithi016 conmon[32727]: 2022-03-21T18:52:58.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: pgmap v3025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:52:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:52:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:52:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:52:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: pgmap v3025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:52:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:52:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:52:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:52:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:52:59.574 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:52:59 smithi016 conmon[32727]: 2022-03-21T18:52:59.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:52:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:52:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:52:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:52:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:52:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:00.579 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:00 smithi016 conmon[32727]: 2022-03-21T18:53:00.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:00 smithi016 ceph-mon[22010]: pgmap v3026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:00 smithi079 ceph-mon[23158]: pgmap v3026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:01.581 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:01 smithi016 conmon[32727]: 2022-03-21T18:53:01.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:02 smithi016 conmon[32727]: 2022-03-21T18:53:02.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:02 smithi016 ceph-mon[22010]: pgmap v3027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:02 smithi079 ceph-mon[23158]: pgmap v3027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:03 smithi016 conmon[32727]: 2022-03-21T18:53:03.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:04.591 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:04 smithi016 conmon[32727]: 2022-03-21T18:53:04.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:04 smithi016 ceph-mon[22010]: pgmap v3028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:04 smithi079 ceph-mon[23158]: pgmap v3028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:05 smithi016 conmon[32727]: 2022-03-21T18:53:05.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:06 smithi016 conmon[32727]: 2022-03-21T18:53:06.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:06 smithi016 ceph-mon[22010]: pgmap v3029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:06 smithi079 ceph-mon[23158]: pgmap v3029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:07 smithi016 conmon[32727]: 2022-03-21T18:53:07.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:08 smithi016 conmon[32727]: 2022-03-21T18:53:08.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:08 smithi016 ceph-mon[22010]: pgmap v3030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:08 smithi079 ceph-mon[23158]: pgmap v3030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:09 smithi016 conmon[32727]: 2022-03-21T18:53:09.148+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:10 smithi016 conmon[32727]: 2022-03-21T18:53:10.148+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:10 smithi016 ceph-mon[22010]: pgmap v3031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:10 smithi079 ceph-mon[23158]: pgmap v3031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:11.611 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:11 smithi016 conmon[32727]: 2022-03-21T18:53:11.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:12.616 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:12 smithi016 conmon[32727]: 2022-03-21T18:53:12.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:12 smithi016 ceph-mon[22010]: pgmap v3032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:12 smithi079 ceph-mon[23158]: pgmap v3032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:13.618 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:13 smithi016 conmon[32727]: 2022-03-21T18:53:13.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:14.624 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:14 smithi016 conmon[32727]: 2022-03-21T18:53:14.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:14 smithi016 ceph-mon[22010]: pgmap v3033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:14 smithi079 ceph-mon[23158]: pgmap v3033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:15.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:15 smithi016 conmon[32727]: 2022-03-21T18:53:15.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:16.631 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:16 smithi016 conmon[32727]: 2022-03-21T18:53:16.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:16 smithi016 ceph-mon[22010]: pgmap v3034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:16 smithi079 ceph-mon[23158]: pgmap v3034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:17.632 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:17 smithi016 conmon[32727]: 2022-03-21T18:53:17.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:18.637 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:18 smithi016 conmon[32727]: 2022-03-21T18:53:18.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:18 smithi016 ceph-mon[22010]: pgmap v3035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:18 smithi079 ceph-mon[23158]: pgmap v3035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:19.638 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:19 smithi016 conmon[32727]: 2022-03-21T18:53:19.219+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:20.644 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:20 smithi016 conmon[32727]: 2022-03-21T18:53:20.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:20 smithi016 ceph-mon[22010]: pgmap v3036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:20 smithi079 ceph-mon[23158]: pgmap v3036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:21.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:21 smithi016 conmon[32727]: 2022-03-21T18:53:21.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:22.650 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:22 smithi016 conmon[32727]: 2022-03-21T18:53:22.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:22 smithi016 ceph-mon[22010]: pgmap v3037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:22 smithi079 ceph-mon[23158]: pgmap v3037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:23.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:23 smithi016 conmon[32727]: 2022-03-21T18:53:23.294+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:24.656 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:24 smithi016 conmon[32727]: 2022-03-21T18:53:24.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:24 smithi016 ceph-mon[22010]: pgmap v3038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:24 smithi079 ceph-mon[23158]: pgmap v3038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:25.657 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:25 smithi016 conmon[32727]: 2022-03-21T18:53:25.274+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:26.663 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:26 smithi016 conmon[32727]: 2022-03-21T18:53:26.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:26 smithi016 ceph-mon[22010]: pgmap v3039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:26 smithi079 ceph-mon[23158]: pgmap v3039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:27 smithi016 conmon[32727]: 2022-03-21T18:53:27.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:28 smithi016 conmon[32727]: 2022-03-21T18:53:28.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:28 smithi079 ceph-mon[23158]: pgmap v3040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:28 smithi016 ceph-mon[22010]: pgmap v3040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:29 smithi016 conmon[32727]: 2022-03-21T18:53:29.340+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:30 smithi016 conmon[32727]: 2022-03-21T18:53:30.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:30 smithi079 ceph-mon[23158]: pgmap v3041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:30 smithi016 ceph-mon[22010]: pgmap v3041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:31 smithi016 conmon[32727]: 2022-03-21T18:53:31.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:32 smithi016 conmon[32727]: 2022-03-21T18:53:32.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:32 smithi016 ceph-mon[22010]: pgmap v3042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:33.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:33.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:32 smithi079 ceph-mon[23158]: pgmap v3042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:33.638 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:33 smithi016 conmon[32727]: 2022-03-21T18:53:33.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:53:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:53:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:53:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:53:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:33 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:53:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:53:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:53:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:53:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:53:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:33 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T18:53:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:34 smithi016 conmon[32727]: 2022-03-21T18:53:34.342+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:34 smithi016 ceph-mon[22010]: pgmap v3043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:34 smithi079 ceph-mon[23158]: pgmap v3043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[32727]: 2022-03-21T18:53:35.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:36.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: 2022-03-21T18:53:35.884+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:36.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:53:36.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:36.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: yield 2022-03-21T18:53:36.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:36.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:36.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:36.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:53:36.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:36.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:53:36.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:36.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:53:36.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:36.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:53:36.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:36.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:53:36.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:36.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:36.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:35 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:36.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:36 smithi016 conmon[32727]: 2022-03-21T18:53:36.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:36 smithi016 ceph-mon[22010]: pgmap v3044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:53:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:53:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:53:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:53:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:53:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:53:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:37.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:37.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:37.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:53:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:36 smithi079 ceph-mon[23158]: pgmap v3044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:53:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:53:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:37.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:37.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:37.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:53:37.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:37.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:53:37.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:37.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:53:37.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:37.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:53:37.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:37.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:37.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:37.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:53:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:37 smithi016 conmon[32727]: 2022-03-21T18:53:37.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[32727]: 2022-03-21T18:53:38.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:38 smithi016 ceph-mon[22010]: pgmap v3045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 2022-03-21T18:53:38.957+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:53:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: yield 2022-03-21T18:53:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:53:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:53:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:53:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:53:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:39.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:53:39.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:39.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:39.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:39.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 2022-03-21T18:53:38.957+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:53:39.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:53:39.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:39.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: yield 2022-03-21T18:53:39.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:39.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:53:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:39.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:53:39.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:39.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:53:39.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:39.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:53:39.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:39.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:53:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:39.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 2022-03-21T18:53:39.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:53:39.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: 2022-03-21T18:53:39.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:53:39.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:53:39.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:53:39.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:53:39.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:53:39.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:53:39.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:53:39.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:53:39.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:53:39.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:53:39.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:53:39.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:53:39.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:53:39.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:53:39.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:53:39.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:53:39.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:53:39.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:53:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:53:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:53:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:53:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:53:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise 2022-03-21T18:53:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:53:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:53:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:53:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:53:39.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:53:38 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:38 smithi079 ceph-mon[23158]: pgmap v3045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:39 smithi016 conmon[32727]: 2022-03-21T18:53:39.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:53:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:53:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:53:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:53:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:53:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:39 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:53:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:53:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:53:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:53:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:53:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:40.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:53:40.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:40.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:40.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:53:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:53:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:53:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:53:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:53:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:53:40.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:53:40.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:53:40.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:53:40.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:53:40.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:53:40.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:53:40.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:53:40.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:53:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:53:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:53:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:53:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:53:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:53:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:53:40.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:53:40.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:53:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:53:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:53:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:53:40.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:53:40.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:53:40.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:53:40.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:53:40.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.189 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:53:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:40.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:40.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:53:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:53:40.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:53:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:40.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:53:40.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:40.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:40.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:39 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:53:40.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:53:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:53:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:53:40.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:53:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:53:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:53:40.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:53:40.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:53:40.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:53:40.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:53:40.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:53:40.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:53:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:53:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:53:40.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:53:40.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:53:40.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:53:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:53:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:53:40.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:53:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:53:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:53:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:53:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:53:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:53:40.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:53:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:53:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:53:40.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:53:40.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:53:40.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:53:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:53:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:53:40.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:53:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:53:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:53:40.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:53:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:53:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:53:40.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:53:40.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:53:40.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:53:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:53:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:53:40.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:53:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:53:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:53:40.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:40.748 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:40 smithi016 conmon[32727]: 2022-03-21T18:53:40.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:40 smithi016 ceph-mon[22010]: pgmap v3046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:40 smithi079 ceph-mon[23158]: pgmap v3046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:41.753 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:41 smithi016 conmon[32727]: 2022-03-21T18:53:41.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:42 smithi016 conmon[32727]: 2022-03-21T18:53:42.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:42 smithi016 ceph-mon[22010]: pgmap v3047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:42 smithi079 ceph-mon[23158]: pgmap v3047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:43 smithi016 conmon[32727]: 2022-03-21T18:53:43.369+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:44 smithi016 conmon[32727]: 2022-03-21T18:53:44.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:44 smithi016 ceph-mon[22010]: pgmap v3048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:44 smithi079 ceph-mon[23158]: pgmap v3048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:45 smithi016 conmon[32727]: 2022-03-21T18:53:45.402+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:46.769 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:46 smithi016 conmon[32727]: 2022-03-21T18:53:46.446+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:46 smithi016 ceph-mon[22010]: pgmap v3049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:46 smithi079 ceph-mon[23158]: pgmap v3049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:47.773 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:47 smithi016 conmon[32727]: 2022-03-21T18:53:47.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:48.774 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:48 smithi016 conmon[32727]: 2022-03-21T18:53:48.426+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:48 smithi016 ceph-mon[22010]: pgmap v3050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:48 smithi079 ceph-mon[23158]: pgmap v3050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:49.776 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:49 smithi016 conmon[32727]: 2022-03-21T18:53:49.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:50.777 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:50 smithi016 conmon[32727]: 2022-03-21T18:53:50.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:50 smithi016 ceph-mon[22010]: pgmap v3051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:50 smithi079 ceph-mon[23158]: pgmap v3051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:51.783 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:51 smithi016 conmon[32727]: 2022-03-21T18:53:51.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:52.784 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:52 smithi016 conmon[32727]: 2022-03-21T18:53:52.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:52 smithi016 ceph-mon[22010]: pgmap v3052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:52 smithi079 ceph-mon[23158]: pgmap v3052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:53.789 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:53 smithi016 conmon[32727]: 2022-03-21T18:53:53.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:54.791 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:54 smithi016 conmon[32727]: 2022-03-21T18:53:54.448+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:54 smithi016 ceph-mon[22010]: pgmap v3053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:54 smithi079 ceph-mon[23158]: pgmap v3053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:53:55.795 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:55 smithi016 conmon[32727]: 2022-03-21T18:53:55.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:56.797 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:56 smithi016 conmon[32727]: 2022-03-21T18:53:56.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:56 smithi016 ceph-mon[22010]: pgmap v3054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:56 smithi079 ceph-mon[23158]: pgmap v3054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:53:57.802 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:57 smithi016 conmon[32727]: 2022-03-21T18:53:57.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:58.804 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:58 smithi016 conmon[32727]: 2022-03-21T18:53:58.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:53:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: pgmap v3055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:53:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:53:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:53:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:53:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:53:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: pgmap v3055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:53:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:53:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:53:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:53:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:53:59.805 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:53:59 smithi016 conmon[32727]: 2022-03-21T18:53:59.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:53:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:53:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:00.807 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:00 smithi016 conmon[32727]: 2022-03-21T18:54:00.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:00 smithi016 ceph-mon[22010]: pgmap v3056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:00 smithi079 ceph-mon[23158]: pgmap v3056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:01.812 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:01 smithi016 conmon[32727]: 2022-03-21T18:54:01.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:02.813 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:02 smithi016 conmon[32727]: 2022-03-21T18:54:02.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:02 smithi016 ceph-mon[22010]: pgmap v3057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:02 smithi079 ceph-mon[23158]: pgmap v3057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:03.818 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:03 smithi016 conmon[32727]: 2022-03-21T18:54:03.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:04.820 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:04 smithi016 conmon[32727]: 2022-03-21T18:54:04.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:04 smithi016 ceph-mon[22010]: pgmap v3058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:04 smithi079 ceph-mon[23158]: pgmap v3058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:05.825 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:05 smithi016 conmon[32727]: 2022-03-21T18:54:05.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:06.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:06.826 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:06 smithi016 conmon[32727]: 2022-03-21T18:54:06.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:06 smithi016 ceph-mon[22010]: pgmap v3059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:06 smithi079 ceph-mon[23158]: pgmap v3059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:07.831 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:07 smithi016 conmon[32727]: 2022-03-21T18:54:07.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:08 smithi016 ceph-mon[22010]: pgmap v3060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:08 smithi016 conmon[32727]: 2022-03-21T18:54:08.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:08 smithi079 ceph-mon[23158]: pgmap v3060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:09 smithi016 conmon[32727]: 2022-03-21T18:54:09.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:10.835 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:10 smithi016 conmon[32727]: 2022-03-21T18:54:10.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:10 smithi016 ceph-mon[22010]: pgmap v3061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:10 smithi079 ceph-mon[23158]: pgmap v3061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:11.841 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:11 smithi016 conmon[32727]: 2022-03-21T18:54:11.581+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:12 smithi016 ceph-mon[22010]: pgmap v3062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:12 smithi016 conmon[32727]: 2022-03-21T18:54:12.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:12 smithi079 ceph-mon[23158]: pgmap v3062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:13.847 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:13 smithi016 conmon[32727]: 2022-03-21T18:54:13.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:14.848 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:14 smithi016 conmon[32727]: 2022-03-21T18:54:14.529+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:14 smithi016 ceph-mon[22010]: pgmap v3063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:14 smithi079 ceph-mon[23158]: pgmap v3063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:15.853 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:15 smithi016 conmon[32727]: 2022-03-21T18:54:15.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:16.855 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:16 smithi016 conmon[32727]: 2022-03-21T18:54:16.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:16 smithi016 ceph-mon[22010]: pgmap v3064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:16 smithi079 ceph-mon[23158]: pgmap v3064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:17.861 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:17 smithi016 conmon[32727]: 2022-03-21T18:54:17.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:18.861 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:18 smithi016 conmon[32727]: 2022-03-21T18:54:18.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:18 smithi016 ceph-mon[22010]: pgmap v3065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:18 smithi079 ceph-mon[23158]: pgmap v3065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:19.862 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:19 smithi016 conmon[32727]: 2022-03-21T18:54:19.560+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:20.863 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:20 smithi016 conmon[32727]: 2022-03-21T18:54:20.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:20 smithi016 ceph-mon[22010]: pgmap v3066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:20 smithi079 ceph-mon[23158]: pgmap v3066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:21.869 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:21 smithi016 conmon[32727]: 2022-03-21T18:54:21.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:22 smithi016 ceph-mon[22010]: pgmap v3067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:22 smithi016 conmon[32727]: 2022-03-21T18:54:22.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:22 smithi079 ceph-mon[23158]: pgmap v3067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:23 smithi016 conmon[32727]: 2022-03-21T18:54:23.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:24.876 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:24 smithi016 conmon[32727]: 2022-03-21T18:54:24.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:24 smithi016 ceph-mon[22010]: pgmap v3068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:24 smithi079 ceph-mon[23158]: pgmap v3068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:25.882 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:25 smithi016 conmon[32727]: 2022-03-21T18:54:25.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:26.883 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:26 smithi016 conmon[32727]: 2022-03-21T18:54:26.615+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:26 smithi016 ceph-mon[22010]: pgmap v3069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:26 smithi079 ceph-mon[23158]: pgmap v3069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:27.888 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:27 smithi016 conmon[32727]: 2022-03-21T18:54:27.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:28.890 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:28 smithi016 conmon[32727]: 2022-03-21T18:54:28.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:28 smithi016 ceph-mon[22010]: pgmap v3070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:28 smithi079 ceph-mon[23158]: pgmap v3070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:29 smithi016 conmon[32727]: 2022-03-21T18:54:29.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:30 smithi016 ceph-mon[22010]: pgmap v3071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:30 smithi016 conmon[32727]: 2022-03-21T18:54:30.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:30 smithi079 ceph-mon[23158]: pgmap v3071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:31 smithi016 conmon[32727]: 2022-03-21T18:54:31.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:32 smithi016 ceph-mon[22010]: pgmap v3072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:33.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:32 smithi016 conmon[32727]: 2022-03-21T18:54:32.682+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:32 smithi079 ceph-mon[23158]: pgmap v3072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:33 smithi016 conmon[32727]: 2022-03-21T18:54:33.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:34 smithi016 ceph-mon[22010]: pgmap v3073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:34 smithi016 conmon[32727]: 2022-03-21T18:54:34.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:34 smithi079 ceph-mon[23158]: pgmap v3073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:35 smithi016 conmon[32727]: 2022-03-21T18:54:35.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:36 smithi016 ceph-mon[22010]: pgmap v3074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:37.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:36 smithi016 conmon[32727]: 2022-03-21T18:54:36.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:36 smithi079 ceph-mon[23158]: pgmap v3074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:37 smithi016 conmon[32727]: 2022-03-21T18:54:37.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:39.034 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:38 smithi016 conmon[32727]: 2022-03-21T18:54:38.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:39.034 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:38 smithi016 ceph-mon[22010]: pgmap v3075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:39.035 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:38 smithi079 ceph-mon[23158]: pgmap v3075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:39 smithi016 conmon[32727]: 2022-03-21T18:54:39.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:54:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:54:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:54:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:54:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:54:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:54:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:40 smithi016 ceph-mon[22010]: pgmap v3076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:40 smithi016 conmon[32727]: 2022-03-21T18:54:40.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:40 smithi079 ceph-mon[23158]: pgmap v3076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:42.063 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:42.064 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:41 smithi016 conmon[32727]: 2022-03-21T18:54:41.760+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:42.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: 2022-03-21T18:54:42.060+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:42.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:54:42.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:42.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: yield 2022-03-21T18:54:42.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:54:42.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:42.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:54:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:54:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:42.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:54:42.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:42.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:54:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:42 smithi016 ceph-mon[22010]: pgmap v3077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:42 smithi016 conmon[32727]: 2022-03-21T18:54:42.722+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:42 smithi079 ceph-mon[23158]: pgmap v3077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:43 smithi016 conmon[32727]: 2022-03-21T18:54:43.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:54:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:54:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:44.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:54:44.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:54:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:44.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:54:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:44.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:54:44.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:44.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:43 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:54:44.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:54:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:54:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:44.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:54:44.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:54:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:44.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:54:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:44.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:54:44.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:44.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:43 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:54:44.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:45.135 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:44 smithi016 conmon[32727]: 2022-03-21T18:54:44.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:45.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:44 smithi016 ceph-mon[22010]: pgmap v3078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:45.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:45.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:44 smithi079 ceph-mon[23158]: pgmap v3078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:45.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 2022-03-21T18:54:45.133+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:45.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:54:45.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:45.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: yield 2022-03-21T18:54:45.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:45.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:45.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:45.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:54:45.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:45.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:54:45.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:45.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:54:45.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:54:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:54:45.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:45.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:45.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:45.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 2022-03-21T18:54:45.134+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:54:45.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:54:45.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:45.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: yield 2022-03-21T18:54:45.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:54:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:54:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:45.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:54:45.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:45.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:54:45.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:45.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:54:45.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:45.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:45.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:45.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 2022-03-21T18:54:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:54:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: 2022-03-21T18:54:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:54:45.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:54:45.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:54:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:54:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:54:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:54:45.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:54:45.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:54:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:54:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:54:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:54:45.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:54:45.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:54:45.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:54:45.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:54:45.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:54:45.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:54:45.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:54:45.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:54:45.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:54:45.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:54:45.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:54:45.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise 2022-03-21T18:54:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:54:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:54:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:54:45.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:54:45.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:45.940 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:45 smithi016 conmon[32727]: 2022-03-21T18:54:45.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:46 smithi016 conmon[32727]: 2022-03-21T18:54:46.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:54:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:54:47.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:54:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:47.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:54:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:47.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:54:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:47.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:46 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:54:47.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:47.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:54:47.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:47.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:54:47.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:54:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:47.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:54:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:47.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:47.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:54:47.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:47.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:47.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:54:47.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:54:47.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:54:47.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:54:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:54:47.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:54:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:54:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:54:47.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:54:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:54:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:54:47.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:54:47.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:54:47.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:54:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:54:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:54:47.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:54:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:54:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:54:47.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:54:47.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:54:47.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:54:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:54:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:54:47.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:54:47.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:54:47.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:54:47.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:54:47.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:54:47.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:46 smithi016 ceph-mon[22010]: pgmap v3079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:47.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:54:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:54:47.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:54:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:47.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:54:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:47.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:54:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:47.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:46 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:54:47.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:54:47.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:54:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:54:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:54:47.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:54:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:54:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:54:47.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:54:47.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:54:47.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:54:47.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:54:47.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:54:47.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:54:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:54:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:54:47.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:54:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:54:47.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:54:47.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:54:47.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:54:47.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:54:47.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:54:47.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:54:47.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:54:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:54:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:54:47.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:54:47.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:54:47.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:54:47.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:54:47.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:54:47.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:54:47.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:54:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:54:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:54:47.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:54:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:54:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:54:47.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:54:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:54:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:54:47.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:54:47.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:54:47.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:54:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:54:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:54:47.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:54:47.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:46 smithi079 ceph-mon[23158]: pgmap v3079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:47.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:47 smithi016 conmon[32727]: 2022-03-21T18:54:47.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:48 smithi016 conmon[32727]: 2022-03-21T18:54:48.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:48 smithi079 ceph-mon[23158]: pgmap v3080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:48 smithi016 ceph-mon[22010]: pgmap v3080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:49 smithi016 conmon[32727]: 2022-03-21T18:54:49.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:50 smithi016 conmon[32727]: 2022-03-21T18:54:50.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:50 smithi079 ceph-mon[23158]: pgmap v3081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:50 smithi016 ceph-mon[22010]: pgmap v3081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:51 smithi016 conmon[32727]: 2022-03-21T18:54:51.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:52 smithi016 conmon[32727]: 2022-03-21T18:54:52.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:52 smithi079 ceph-mon[23158]: pgmap v3082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:52 smithi016 ceph-mon[22010]: pgmap v3082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:53 smithi016 conmon[32727]: 2022-03-21T18:54:53.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:54 smithi016 conmon[32727]: 2022-03-21T18:54:54.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:54 smithi079 ceph-mon[23158]: pgmap v3083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:54 smithi016 ceph-mon[22010]: pgmap v3083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:54:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:55 smithi016 conmon[32727]: 2022-03-21T18:54:55.581+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:56 smithi016 conmon[32727]: 2022-03-21T18:54:56.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:56 smithi079 ceph-mon[23158]: pgmap v3084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:56 smithi016 ceph-mon[22010]: pgmap v3084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:54:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:57 smithi016 conmon[32727]: 2022-03-21T18:54:57.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:58 smithi016 conmon[32727]: 2022-03-21T18:54:58.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:54:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: pgmap v3085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:54:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:54:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:54:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:54:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: pgmap v3085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:54:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:54:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:54:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:54:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:54:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:54:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:54:59 smithi016 conmon[32727]: 2022-03-21T18:54:59.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:54:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:54:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:00 smithi016 conmon[32727]: 2022-03-21T18:55:00.466+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:00 smithi016 ceph-mon[22010]: pgmap v3086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:00 smithi079 ceph-mon[23158]: pgmap v3086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:01 smithi016 conmon[32727]: 2022-03-21T18:55:01.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:02.402 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:02 smithi016 conmon[32727]: 2022-03-21T18:55:02.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:02 smithi016 ceph-mon[22010]: pgmap v3087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:02 smithi079 ceph-mon[23158]: pgmap v3087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:03 smithi016 conmon[32727]: 2022-03-21T18:55:03.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:04.409 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:04.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:04 smithi016 conmon[32727]: 2022-03-21T18:55:04.407+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:05.373 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:04 smithi016 ceph-mon[22010]: pgmap v3088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:05.373 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:05.374 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:04 smithi079 ceph-mon[23158]: pgmap v3088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:05.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:05 smithi016 conmon[32727]: 2022-03-21T18:55:05.372+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:06.383 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:06 smithi016 conmon[32727]: 2022-03-21T18:55:06.382+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:06 smithi016 ceph-mon[22010]: pgmap v3089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:06 smithi079 ceph-mon[23158]: pgmap v3089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:07 smithi016 conmon[32727]: 2022-03-21T18:55:07.417+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:08 smithi016 conmon[32727]: 2022-03-21T18:55:08.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:09 smithi016 ceph-mon[22010]: pgmap v3090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:09 smithi079 ceph-mon[23158]: pgmap v3090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:09 smithi016 conmon[32727]: 2022-03-21T18:55:09.424+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:10 smithi016 ceph-mon[22010]: pgmap v3091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:10 smithi079 ceph-mon[23158]: pgmap v3091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:10 smithi016 conmon[32727]: 2022-03-21T18:55:10.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:11 smithi016 conmon[32727]: 2022-03-21T18:55:11.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:12.468 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:12.468 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:12 smithi016 ceph-mon[22010]: pgmap v3092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:12 smithi079 ceph-mon[23158]: pgmap v3092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:12 smithi016 conmon[32727]: 2022-03-21T18:55:12.466+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:13.446 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:13 smithi016 conmon[32727]: 2022-03-21T18:55:13.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:14.439 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:14.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:14 smithi016 ceph-mon[22010]: pgmap v3093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:14.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:14 smithi079 ceph-mon[23158]: pgmap v3093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:14 smithi016 conmon[32727]: 2022-03-21T18:55:14.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:15.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:15 smithi016 conmon[32727]: 2022-03-21T18:55:15.446+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:16 smithi079 ceph-mon[23158]: pgmap v3094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:16 smithi016 ceph-mon[22010]: pgmap v3094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:16 smithi016 conmon[32727]: 2022-03-21T18:55:16.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:17.445 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:17 smithi016 conmon[32727]: 2022-03-21T18:55:17.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:18.470 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:18.471 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:18 smithi016 ceph-mon[22010]: pgmap v3095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:18 smithi079 ceph-mon[23158]: pgmap v3095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:18 smithi016 conmon[32727]: 2022-03-21T18:55:18.469+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:19.510 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:19.510 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:19 smithi016 conmon[32727]: 2022-03-21T18:55:19.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:20 smithi079 ceph-mon[23158]: pgmap v3096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:20.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:20.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:20 smithi016 ceph-mon[22010]: pgmap v3096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:20 smithi016 conmon[32727]: 2022-03-21T18:55:20.523+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:21.492 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:21 smithi016 conmon[32727]: 2022-03-21T18:55:21.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:22 smithi079 ceph-mon[23158]: pgmap v3097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:22.488 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:22.488 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:22 smithi016 ceph-mon[22010]: pgmap v3097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:22 smithi016 conmon[32727]: 2022-03-21T18:55:22.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:23.469 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:23 smithi016 conmon[32727]: 2022-03-21T18:55:23.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:24 smithi079 ceph-mon[23158]: pgmap v3098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:24.477 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:24.477 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:24 smithi016 ceph-mon[22010]: pgmap v3098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:24.477 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:24 smithi016 conmon[32727]: 2022-03-21T18:55:24.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:25.520 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:25 smithi016 conmon[32727]: 2022-03-21T18:55:25.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:26 smithi079 ceph-mon[23158]: pgmap v3099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:26.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:26.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:26 smithi016 ceph-mon[22010]: pgmap v3099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:26 smithi016 conmon[32727]: 2022-03-21T18:55:26.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:27.521 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:27 smithi016 conmon[32727]: 2022-03-21T18:55:27.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:28.478 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:28.478 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:28 smithi016 ceph-mon[22010]: pgmap v3100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:28 smithi079 ceph-mon[23158]: pgmap v3100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:28 smithi016 conmon[32727]: 2022-03-21T18:55:28.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:29 smithi016 conmon[32727]: 2022-03-21T18:55:29.450+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:30 smithi016 ceph-mon[22010]: pgmap v3101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:30 smithi016 conmon[32727]: 2022-03-21T18:55:30.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:30 smithi079 ceph-mon[23158]: pgmap v3101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:31 smithi016 conmon[32727]: 2022-03-21T18:55:31.405+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:32 smithi016 ceph-mon[22010]: pgmap v3102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:32 smithi016 conmon[32727]: 2022-03-21T18:55:32.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:32 smithi079 ceph-mon[23158]: pgmap v3102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:33 smithi016 conmon[32727]: 2022-03-21T18:55:33.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:34 smithi016 ceph-mon[22010]: pgmap v3103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:34 smithi016 conmon[32727]: 2022-03-21T18:55:34.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:34 smithi079 ceph-mon[23158]: pgmap v3103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:35 smithi016 conmon[32727]: 2022-03-21T18:55:35.349+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:36 smithi016 ceph-mon[22010]: pgmap v3104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:36 smithi016 conmon[32727]: 2022-03-21T18:55:36.323+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:36 smithi079 ceph-mon[23158]: pgmap v3104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:37 smithi016 conmon[32727]: 2022-03-21T18:55:37.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:38 smithi016 conmon[32727]: 2022-03-21T18:55:38.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:38 smithi016 ceph-mon[22010]: pgmap v3105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:38 smithi079 ceph-mon[23158]: pgmap v3105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:39 smithi016 conmon[32727]: 2022-03-21T18:55:39.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:40 smithi016 ceph-mon[22010]: pgmap v3106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:40 smithi016 conmon[32727]: 2022-03-21T18:55:40.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:40 smithi079 ceph-mon[23158]: pgmap v3106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:41 smithi016 conmon[32727]: 2022-03-21T18:55:41.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:42 smithi016 ceph-mon[22010]: pgmap v3107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:42 smithi016 conmon[32727]: 2022-03-21T18:55:42.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:42 smithi079 ceph-mon[23158]: pgmap v3107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:43 smithi016 conmon[32727]: 2022-03-21T18:55:43.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:44.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:44.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:44 smithi016 ceph-mon[22010]: pgmap v3108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:44.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:44.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:44 smithi016 conmon[32727]: 2022-03-21T18:55:44.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:44 smithi079 ceph-mon[23158]: pgmap v3108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:44.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:55:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:55:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:55:45.597 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[32727]: 2022-03-21T18:55:45.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:45.597 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:45.598 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:55:45.598 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:55:45.598 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:55:45.914 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: 2022-03-21T18:55:45.741+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:55:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: yield 2022-03-21T18:55:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:55:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:55:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:55:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:55:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:55:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:45.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:46 smithi016 conmon[32727]: 2022-03-21T18:55:46.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:46 smithi016 ceph-mon[22010]: pgmap v3109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:55:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:55:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:55:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:46.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:55:46.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:46.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:55:46.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:46.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:46.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:46.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:55:46.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:46.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:46.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:46.682 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:55:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:46 smithi079 ceph-mon[23158]: pgmap v3109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:55:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:55:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:55:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:55:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:55:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:55:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:46.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:46.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:46.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:55:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:47 smithi016 conmon[32727]: 2022-03-21T18:55:47.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:48 smithi016 ceph-mon[22010]: pgmap v3110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:48.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[32727]: 2022-03-21T18:55:48.295+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:48 smithi079 ceph-mon[23158]: pgmap v3110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:49.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 2022-03-21T18:55:48.813+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:55:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: yield 2022-03-21T18:55:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:55:49.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:55:49.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:55:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:55:49.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 2022-03-21T18:55:48.814+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:55:49.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:55:49.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: yield 2022-03-21T18:55:49.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:55:49.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:55:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:55:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:55:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 2022-03-21T18:55:49.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:55:49.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: 2022-03-21T18:55:49.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:55:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:55:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:55:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:55:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:55:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:55:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:55:49.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:55:49.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:55:49.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:55:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:55:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:55:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:55:49.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:55:49.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:55:49.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:55:49.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:55:49.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:55:49.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:55:49.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:55:49.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:55:49.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:55:49.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise 2022-03-21T18:55:49.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:55:49.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:55:49.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:55:49.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:55:49.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:55:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:49 smithi016 conmon[32727]: 2022-03-21T18:55:49.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:55:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:55:49.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:55:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:55:49.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:55:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:49 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:55:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:55:49.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:55:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:55:49.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:55:49.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:55:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:55:49.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:55:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:55:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:55:49.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:55:49.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:55:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:55:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:55:49.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:55:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:55:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:55:49.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:55:49.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:55:49.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:55:49.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:55:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:55:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:55:49.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:55:49.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:55:49.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:55:49.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:55:49.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:55:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:55:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:55:49.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:55:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:55:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:55:49.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:55:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:55:49.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:55:49.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:55:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:55:49.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:49 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:55:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:55:49.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:55:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:55:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:55:49.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:55:49.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:55:49.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:55:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:55:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:55:49.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:55:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:55:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:55:49.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:55:49.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:55:49.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:55:49.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:55:49.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:55:49.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:55:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:55:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:55:49.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:55:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:55:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:55:49.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:55:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:55:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:55:49.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:55:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:55:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:55:49.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:55:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:55:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:55:49.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:55:49.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:55:49.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:55:49.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:55:49.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:55:49.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:55:49.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:55:49.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:55:49.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:55:49.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:55:49.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:55:49.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:55:49.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:55:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:55:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:55:49.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:50 smithi016 ceph-mon[22010]: pgmap v3111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:50 smithi016 conmon[32727]: 2022-03-21T18:55:50.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:50 smithi079 ceph-mon[23158]: pgmap v3111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:51 smithi016 conmon[32727]: 2022-03-21T18:55:51.292+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:52 smithi016 ceph-mon[22010]: pgmap v3112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:52 smithi016 conmon[32727]: 2022-03-21T18:55:52.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:52 smithi079 ceph-mon[23158]: pgmap v3112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:53 smithi016 conmon[32727]: 2022-03-21T18:55:53.347+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:54 smithi016 ceph-mon[22010]: pgmap v3113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:54 smithi016 conmon[32727]: 2022-03-21T18:55:54.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:54 smithi079 ceph-mon[23158]: pgmap v3113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:55.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:55 smithi016 conmon[32727]: 2022-03-21T18:55:55.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:56 smithi016 conmon[32727]: 2022-03-21T18:55:56.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:56 smithi016 ceph-mon[22010]: pgmap v3114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:56 smithi079 ceph-mon[23158]: pgmap v3114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:55:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:57 smithi016 conmon[32727]: 2022-03-21T18:55:57.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:58.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:58 smithi016 ceph-mon[22010]: pgmap v3115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:58.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:58 smithi016 conmon[32727]: 2022-03-21T18:55:58.356+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:58 smithi079 ceph-mon[23158]: pgmap v3115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:55:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:55:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:55:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:55:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:55:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:55:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:55:59.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:55:59 smithi016 conmon[32727]: 2022-03-21T18:55:59.318+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:55:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:55:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:55:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:55:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:55:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:55:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:55:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:00 smithi016 ceph-mon[22010]: pgmap v3116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:00 smithi016 conmon[32727]: 2022-03-21T18:56:00.309+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:00.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:00 smithi079 ceph-mon[23158]: pgmap v3116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:01 smithi016 conmon[32727]: 2022-03-21T18:56:01.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:02 smithi016 conmon[32727]: 2022-03-21T18:56:02.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:02 smithi016 ceph-mon[22010]: pgmap v3117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:02 smithi079 ceph-mon[23158]: pgmap v3117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:03 smithi016 conmon[32727]: 2022-03-21T18:56:03.228+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:04 smithi016 ceph-mon[22010]: pgmap v3118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:04 smithi016 conmon[32727]: 2022-03-21T18:56:04.267+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:04 smithi079 ceph-mon[23158]: pgmap v3118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:05 smithi016 conmon[32727]: 2022-03-21T18:56:05.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:06 smithi016 conmon[32727]: 2022-03-21T18:56:06.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:06 smithi016 ceph-mon[22010]: pgmap v3119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:06 smithi079 ceph-mon[23158]: pgmap v3119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:07 smithi016 conmon[32727]: 2022-03-21T18:56:07.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:08 smithi016 conmon[32727]: 2022-03-21T18:56:08.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:08 smithi016 ceph-mon[22010]: pgmap v3120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:08 smithi079 ceph-mon[23158]: pgmap v3120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:09 smithi016 conmon[32727]: 2022-03-21T18:56:09.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:10 smithi016 ceph-mon[22010]: pgmap v3121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:10 smithi016 conmon[32727]: 2022-03-21T18:56:10.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:10 smithi079 ceph-mon[23158]: pgmap v3121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:11 smithi016 conmon[32727]: 2022-03-21T18:56:11.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:12 smithi016 ceph-mon[22010]: pgmap v3122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:12 smithi016 conmon[32727]: 2022-03-21T18:56:12.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:12 smithi079 ceph-mon[23158]: pgmap v3122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:13 smithi016 conmon[32727]: 2022-03-21T18:56:13.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:14 smithi016 ceph-mon[22010]: pgmap v3123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:14 smithi016 conmon[32727]: 2022-03-21T18:56:14.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:14 smithi079 ceph-mon[23158]: pgmap v3123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:15 smithi016 conmon[32727]: 2022-03-21T18:56:15.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:16 smithi016 ceph-mon[22010]: pgmap v3124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:16 smithi016 conmon[32727]: 2022-03-21T18:56:16.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:16 smithi079 ceph-mon[23158]: pgmap v3124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:17 smithi016 conmon[32727]: 2022-03-21T18:56:17.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:18.379 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:18 smithi016 conmon[32727]: 2022-03-21T18:56:18.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:18 smithi016 ceph-mon[22010]: pgmap v3125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:18 smithi079 ceph-mon[23158]: pgmap v3125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:19.381 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:19 smithi016 conmon[32727]: 2022-03-21T18:56:19.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:20.386 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:20 smithi016 conmon[32727]: 2022-03-21T18:56:20.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:20 smithi016 ceph-mon[22010]: pgmap v3126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:20 smithi079 ceph-mon[23158]: pgmap v3126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:21.387 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:21 smithi016 conmon[32727]: 2022-03-21T18:56:21.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:22.392 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:22 smithi016 conmon[32727]: 2022-03-21T18:56:22.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:22 smithi016 ceph-mon[22010]: pgmap v3127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:22 smithi079 ceph-mon[23158]: pgmap v3127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:23.393 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:23 smithi016 conmon[32727]: 2022-03-21T18:56:23.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:24.397 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:24 smithi016 conmon[32727]: 2022-03-21T18:56:24.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:24 smithi016 ceph-mon[22010]: pgmap v3128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:24 smithi079 ceph-mon[23158]: pgmap v3128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:25.399 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:25 smithi016 conmon[32727]: 2022-03-21T18:56:25.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:26.404 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:26 smithi016 conmon[32727]: 2022-03-21T18:56:26.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:26 smithi016 ceph-mon[22010]: pgmap v3129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:26 smithi079 ceph-mon[23158]: pgmap v3129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:27.406 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:27 smithi016 conmon[32727]: 2022-03-21T18:56:27.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:28.410 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:28 smithi016 conmon[32727]: 2022-03-21T18:56:28.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:28 smithi016 ceph-mon[22010]: pgmap v3130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:28 smithi079 ceph-mon[23158]: pgmap v3130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:29.412 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:29 smithi016 conmon[32727]: 2022-03-21T18:56:29.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:30 smithi016 conmon[32727]: 2022-03-21T18:56:30.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:30 smithi079 ceph-mon[23158]: pgmap v3131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:30 smithi016 ceph-mon[22010]: pgmap v3131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:31.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:31 smithi016 conmon[32727]: 2022-03-21T18:56:31.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:32 smithi016 conmon[32727]: 2022-03-21T18:56:32.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:32 smithi079 ceph-mon[23158]: pgmap v3132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:32 smithi016 ceph-mon[22010]: pgmap v3132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:33.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:33 smithi016 conmon[32727]: 2022-03-21T18:56:33.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:34 smithi016 conmon[32727]: 2022-03-21T18:56:34.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:34 smithi079 ceph-mon[23158]: pgmap v3133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:34 smithi016 ceph-mon[22010]: pgmap v3133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:35 smithi016 conmon[32727]: 2022-03-21T18:56:35.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:36 smithi016 conmon[32727]: 2022-03-21T18:56:36.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:36 smithi079 ceph-mon[23158]: pgmap v3134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:36 smithi016 ceph-mon[22010]: pgmap v3134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:37.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:37 smithi016 conmon[32727]: 2022-03-21T18:56:37.062+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:38.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:38 smithi016 conmon[32727]: 2022-03-21T18:56:38.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:38 smithi079 ceph-mon[23158]: pgmap v3135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:38 smithi016 ceph-mon[22010]: pgmap v3135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:39.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:39 smithi016 conmon[32727]: 2022-03-21T18:56:39.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:40.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:40 smithi016 conmon[32727]: 2022-03-21T18:56:40.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:40 smithi079 ceph-mon[23158]: pgmap v3136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:40 smithi016 ceph-mon[22010]: pgmap v3136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:41.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:41 smithi016 conmon[32727]: 2022-03-21T18:56:41.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:42 smithi016 conmon[32727]: 2022-03-21T18:56:42.104+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:42 smithi079 ceph-mon[23158]: pgmap v3137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:42 smithi016 ceph-mon[22010]: pgmap v3137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:43 smithi016 conmon[32727]: 2022-03-21T18:56:43.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:44 smithi016 conmon[32727]: 2022-03-21T18:56:44.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:44 smithi079 ceph-mon[23158]: pgmap v3138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:44 smithi016 ceph-mon[22010]: pgmap v3138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:45.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:45 smithi016 conmon[32727]: 2022-03-21T18:56:45.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:46.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:46 smithi016 conmon[32727]: 2022-03-21T18:56:46.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:46 smithi079 ceph-mon[23158]: pgmap v3139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:46 smithi016 ceph-mon[22010]: pgmap v3139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:47.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:47 smithi016 conmon[32727]: 2022-03-21T18:56:47.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:48.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:48 smithi016 conmon[32727]: 2022-03-21T18:56:48.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:48.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:48.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:48 smithi016 ceph-mon[22010]: pgmap v3140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:48.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:48.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:48 smithi079 ceph-mon[23158]: pgmap v3140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:49 smithi016 conmon[32727]: 2022-03-21T18:56:49.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:56:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:56:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:56:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:56:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:56:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:56:49.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:50 smithi016 conmon[32727]: 2022-03-21T18:56:50.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:50 smithi016 ceph-mon[22010]: pgmap v3141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:50 smithi079 ceph-mon[23158]: pgmap v3141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:51.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[32727]: 2022-03-21T18:56:51.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:52 smithi016 conmon[32727]: 2022-03-21T18:56:52.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:52.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: 2022-03-21T18:56:51.917+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:56:52.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: yield 2022-03-21T18:56:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:52.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:52.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:52.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:56:52.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:52.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:56:52.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:52.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:56:52.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:56:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:52.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:56:52.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:52.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:52.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:52 smithi016 ceph-mon[22010]: pgmap v3142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:52 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:56:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:52.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:56:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:52.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:56:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:52.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:56:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:56:52.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:52.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:52.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:52.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:56:52.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:52.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:52.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:52 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:56:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:52 smithi079 ceph-mon[23158]: pgmap v3142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:52 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:56:52.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:56:52.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:52.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:52.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:56:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:52.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:56:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:52.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:56:52.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:52.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:52.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:52.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:56:52.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:52.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:52.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:52.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:52 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:56:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:53 smithi016 conmon[32727]: 2022-03-21T18:56:53.053+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[32727]: 2022-03-21T18:56:54.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:54 smithi016 ceph-mon[22010]: pgmap v3143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:54 smithi079 ceph-mon[23158]: pgmap v3143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:55 smithi016 conmon[32727]: 2022-03-21T18:56:55.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:55.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 2022-03-21T18:56:54.989+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:56:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: yield 2022-03-21T18:56:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:56:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:56:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:56:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:56:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 2022-03-21T18:56:54.990+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:56:55.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:56:55.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: yield 2022-03-21T18:56:55.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:56:55.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:56:55.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:56:55.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:56:55.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 2022-03-21T18:56:55.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:56:55.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: 2022-03-21T18:56:55.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:56:55.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:56:55.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:56:55.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:56:55.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:56:55.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:56:55.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:56:55.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:56:55.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:56:55.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:56:55.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:56:55.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:56:55.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:56:55.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:56:55.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:56:55.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:56:55.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:56:55.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:56:55.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:56:55.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:56:55.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:56:55.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:56:55.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise 2022-03-21T18:56:55.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:56:55.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:56:55.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:56:55.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:56:55.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:56:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:56:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:56:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:56:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:56:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:56:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:55 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:56:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:56:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:56:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:56:55.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:56:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:56:55.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:56:55.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:56:55.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:56:55.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:56:55.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:56:55.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:56:55.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:56:55.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:56:55.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:56:55.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:56:55.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:56:55.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:56:55.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:56:55.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:56:55.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:56:55.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:56:55.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:56:55.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:56:55.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:56:55.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:56:55.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:56:55.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:56:55.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:56:55.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:56:55.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:56:55.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:56:55.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:56:55.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:56:55.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:56:55.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:56:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:56:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:56:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:56:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:56:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:55 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:56:55.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:56:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:56:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:56:55.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:56:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:56:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:56:55.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:56:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:56:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:56:55.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:56:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:56:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:56:55.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:56:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:56:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:56:55.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:56:55.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:55.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:56:55.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:56:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:56:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:56:55.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:56:55.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:56:55.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:56:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:56:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:56:55.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:56:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:56:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:56:55.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:56:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:56:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:56:55.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:56:55.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:56:55.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:56:55.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:56:55.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:56:55.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:56:55.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:56:55.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:56:55.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:56:55.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:56:56.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:56:56.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:56:56.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:56:56.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:56:56.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:56:56.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:56:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:56 smithi016 conmon[32727]: 2022-03-21T18:56:56.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:56 smithi016 ceph-mon[22010]: pgmap v3144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:56 smithi079 ceph-mon[23158]: pgmap v3144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:56:57.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:57 smithi016 conmon[32727]: 2022-03-21T18:56:57.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:58 smithi016 conmon[32727]: 2022-03-21T18:56:58.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: pgmap v3145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:56:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:56:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:56:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:56:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: pgmap v3145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:56:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:56:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:56:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:56:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:56:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:56:59 smithi016 conmon[32727]: 2022-03-21T18:56:59.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:56:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:56:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:56:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:56:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:56:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:00.514 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:00 smithi016 conmon[32727]: 2022-03-21T18:57:00.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:00 smithi016 ceph-mon[22010]: pgmap v3146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:00 smithi079 ceph-mon[23158]: pgmap v3146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:01.514 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:01 smithi016 conmon[32727]: 2022-03-21T18:57:01.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:02.519 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:02 smithi016 conmon[32727]: 2022-03-21T18:57:02.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:02 smithi016 ceph-mon[22010]: pgmap v3147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:02.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:02 smithi079 ceph-mon[23158]: pgmap v3147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:03.521 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:03 smithi016 conmon[32727]: 2022-03-21T18:57:03.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:04.526 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:04 smithi016 conmon[32727]: 2022-03-21T18:57:04.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:04 smithi016 ceph-mon[22010]: pgmap v3148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:04 smithi079 ceph-mon[23158]: pgmap v3148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:05.528 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:05 smithi016 conmon[32727]: 2022-03-21T18:57:05.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:06 smithi016 conmon[32727]: 2022-03-21T18:57:06.297+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:06 smithi016 ceph-mon[22010]: pgmap v3149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:06 smithi079 ceph-mon[23158]: pgmap v3149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:07 smithi016 conmon[32727]: 2022-03-21T18:57:07.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:08 smithi016 conmon[32727]: 2022-03-21T18:57:08.345+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:08 smithi016 ceph-mon[22010]: pgmap v3150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:08 smithi079 ceph-mon[23158]: pgmap v3150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:09 smithi016 conmon[32727]: 2022-03-21T18:57:09.301+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:10.545 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:10 smithi016 conmon[32727]: 2022-03-21T18:57:10.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:10 smithi016 ceph-mon[22010]: pgmap v3151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:10 smithi079 ceph-mon[23158]: pgmap v3151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:11.547 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:11 smithi016 conmon[32727]: 2022-03-21T18:57:11.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:12.553 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:12 smithi016 conmon[32727]: 2022-03-21T18:57:12.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:12 smithi016 ceph-mon[22010]: pgmap v3152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:12 smithi079 ceph-mon[23158]: pgmap v3152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:13.553 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:13 smithi016 conmon[32727]: 2022-03-21T18:57:13.279+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:14.560 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:14 smithi016 conmon[32727]: 2022-03-21T18:57:14.288+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:14 smithi016 ceph-mon[22010]: pgmap v3153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:14 smithi079 ceph-mon[23158]: pgmap v3153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:15 smithi016 conmon[32727]: 2022-03-21T18:57:15.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:16 smithi016 ceph-mon[22010]: pgmap v3154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:16 smithi016 conmon[32727]: 2022-03-21T18:57:16.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:16 smithi079 ceph-mon[23158]: pgmap v3154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:17 smithi016 conmon[32727]: 2022-03-21T18:57:17.362+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:18 smithi016 ceph-mon[22010]: pgmap v3155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:18 smithi016 conmon[32727]: 2022-03-21T18:57:18.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:18 smithi079 ceph-mon[23158]: pgmap v3155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:19.574 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:19 smithi016 conmon[32727]: 2022-03-21T18:57:19.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:20 smithi016 ceph-mon[22010]: pgmap v3156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:20 smithi016 conmon[32727]: 2022-03-21T18:57:20.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:20 smithi079 ceph-mon[23158]: pgmap v3156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:21 smithi016 conmon[32727]: 2022-03-21T18:57:21.372+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:22.586 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:22 smithi016 conmon[32727]: 2022-03-21T18:57:22.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:22.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:22.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:22 smithi016 ceph-mon[22010]: pgmap v3157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:22 smithi079 ceph-mon[23158]: pgmap v3157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:23 smithi016 conmon[32727]: 2022-03-21T18:57:23.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:24.592 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:24 smithi016 conmon[32727]: 2022-03-21T18:57:24.320+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:24 smithi016 ceph-mon[22010]: pgmap v3158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:24 smithi079 ceph-mon[23158]: pgmap v3158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:25.593 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:25 smithi016 conmon[32727]: 2022-03-21T18:57:25.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:26.598 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:26 smithi016 conmon[32727]: 2022-03-21T18:57:26.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:26 smithi016 ceph-mon[22010]: pgmap v3159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:26 smithi079 ceph-mon[23158]: pgmap v3159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:27 smithi016 conmon[32727]: 2022-03-21T18:57:27.359+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:28.604 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:28 smithi016 conmon[32727]: 2022-03-21T18:57:28.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:28 smithi016 ceph-mon[22010]: pgmap v3160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:28 smithi079 ceph-mon[23158]: pgmap v3160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:29.606 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:29 smithi016 conmon[32727]: 2022-03-21T18:57:29.306+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:30.611 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:30 smithi016 conmon[32727]: 2022-03-21T18:57:30.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:30 smithi016 ceph-mon[22010]: pgmap v3161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:30 smithi079 ceph-mon[23158]: pgmap v3161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:31.612 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:31 smithi016 conmon[32727]: 2022-03-21T18:57:31.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:32.617 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:32 smithi016 conmon[32727]: 2022-03-21T18:57:32.315+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:32 smithi016 ceph-mon[22010]: pgmap v3162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:32 smithi079 ceph-mon[23158]: pgmap v3162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:33.619 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:33 smithi016 conmon[32727]: 2022-03-21T18:57:33.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:34.623 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:34 smithi016 conmon[32727]: 2022-03-21T18:57:34.297+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:34 smithi016 ceph-mon[22010]: pgmap v3163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:34 smithi079 ceph-mon[23158]: pgmap v3163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:35.624 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:35 smithi016 conmon[32727]: 2022-03-21T18:57:35.289+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:36.630 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:36 smithi016 conmon[32727]: 2022-03-21T18:57:36.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:36 smithi016 ceph-mon[22010]: pgmap v3164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:36 smithi079 ceph-mon[23158]: pgmap v3164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:37.631 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:37 smithi016 conmon[32727]: 2022-03-21T18:57:37.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:38.636 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:38 smithi016 conmon[32727]: 2022-03-21T18:57:38.229+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:38 smithi016 ceph-mon[22010]: pgmap v3165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:38 smithi079 ceph-mon[23158]: pgmap v3165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:39.637 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:39 smithi016 conmon[32727]: 2022-03-21T18:57:39.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:40.642 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:40 smithi016 conmon[32727]: 2022-03-21T18:57:40.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:40 smithi016 ceph-mon[22010]: pgmap v3166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:40 smithi079 ceph-mon[23158]: pgmap v3166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:41.643 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:41 smithi016 conmon[32727]: 2022-03-21T18:57:41.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:42.649 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:42 smithi016 conmon[32727]: 2022-03-21T18:57:42.284+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:42 smithi016 ceph-mon[22010]: pgmap v3167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:42 smithi079 ceph-mon[23158]: pgmap v3167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:43.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:43 smithi016 conmon[32727]: 2022-03-21T18:57:43.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:44.655 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:44 smithi016 conmon[32727]: 2022-03-21T18:57:44.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:44 smithi016 ceph-mon[22010]: pgmap v3168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:44 smithi079 ceph-mon[23158]: pgmap v3168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:45.656 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:45 smithi016 conmon[32727]: 2022-03-21T18:57:45.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:46.662 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:46 smithi016 conmon[32727]: 2022-03-21T18:57:46.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:46 smithi016 ceph-mon[22010]: pgmap v3169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:46 smithi079 ceph-mon[23158]: pgmap v3169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:47.663 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:47 smithi016 conmon[32727]: 2022-03-21T18:57:47.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:48 smithi016 conmon[32727]: 2022-03-21T18:57:48.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:48 smithi079 ceph-mon[23158]: pgmap v3170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:48 smithi016 ceph-mon[22010]: pgmap v3170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:49 smithi016 conmon[32727]: 2022-03-21T18:57:49.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:50.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:50.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:50 smithi016 conmon[32727]: 2022-03-21T18:57:50.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:50 smithi079 ceph-mon[23158]: pgmap v3171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:50 smithi016 ceph-mon[22010]: pgmap v3171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:51 smithi016 conmon[32727]: 2022-03-21T18:57:51.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:52.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:52 smithi016 conmon[32727]: 2022-03-21T18:57:52.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:52 smithi079 ceph-mon[23158]: pgmap v3172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:53.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:53.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:52 smithi016 ceph-mon[22010]: pgmap v3172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:53 smithi016 conmon[32727]: 2022-03-21T18:57:53.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:54.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:54 smithi016 conmon[32727]: 2022-03-21T18:57:54.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:54 smithi079 ceph-mon[23158]: pgmap v3173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:55.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:55.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:54 smithi016 ceph-mon[22010]: pgmap v3173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:55.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:57:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[32727]: 2022-03-21T18:57:55.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:57:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:57:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:57:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:57:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:57:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:57:55.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: 2022-03-21T18:57:55.469+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:55.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:57:55.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:55.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: yield 2022-03-21T18:57:55.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:55.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:55.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:55.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:57:55.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:55.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:57:55.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:55.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:57:55.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:55.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:57:55.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:55.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:57:55.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:55.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:55.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:56 smithi016 conmon[32727]: 2022-03-21T18:57:56.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:57:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:57:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:57:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:57:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:57:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:57:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:57:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:56 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:57:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:56 smithi079 ceph-mon[23158]: pgmap v3174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:57.022 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:57.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:57.023 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:57:57.023 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:57.024 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:57:57.024 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:57.024 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:57.025 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:57.025 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:57:57.025 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:57.026 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:57:57.026 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:57.026 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:57:57.027 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:57.027 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:57:57.027 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:57.028 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:57:57.028 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:57.028 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:57.029 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:57.029 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:56 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:57:57.030 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:56 smithi016 ceph-mon[22010]: pgmap v3174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:57:57.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:57 smithi016 conmon[32727]: 2022-03-21T18:57:57.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:58.064 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[32727]: 2022-03-21T18:57:58.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: pgmap v3175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:57:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:57:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:57:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:57:58.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 2022-03-21T18:57:58.541+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:58.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:57:58.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:58.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: yield 2022-03-21T18:57:58.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:58.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:58.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:58.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:57:58.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:58.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:57:58.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:58.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:57:58.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:58.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:57:58.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:58.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:57:58.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:58.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:58.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:58.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 2022-03-21T18:57:58.542+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:57:58.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:57:58.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:58.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: yield 2022-03-21T18:57:58.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:58.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:58.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:58.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:57:58.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:58.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:57:58.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:58.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:57:58.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:58.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:57:58.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:58.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:57:58.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:58.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:58.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:58.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 2022-03-21T18:57:58.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:57:58.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: 2022-03-21T18:57:58.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:57:58.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:57:58.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:57:58.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:57:58.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:57:58.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:57:58.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:57:58.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:57:58.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:57:58.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:57:58.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:57:58.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:57:58.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:57:58.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:57:58.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:57:58.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:57:58.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:57:58.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:57:58.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:57:58.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:57:58.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:57:58.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:57:58.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise 2022-03-21T18:57:58.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:57:58.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:57:58.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:57:58.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:57:58.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:57:58 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: pgmap v3175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:57:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:57:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:57:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:57:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:57:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:57:59 smithi016 conmon[32727]: 2022-03-21T18:57:59.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:57:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:57:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:59 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:57:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:57:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:57:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:57:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:57:59.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:59.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:57:59.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:59.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:57:59.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:59.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:59.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:59.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:59 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:57:59.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:57:59.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:57:59.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:57:59.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:57:59.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:57:59.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:57:59.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:57:59.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:57:59.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:57:59.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:57:59.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:57:59.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:57:59.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:57:59.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:57:59.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:57:59.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:57:59.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:57:59.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:59.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:57:59.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:57:59.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:57:59.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:57:59.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:57:59.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:57:59.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:57:59.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:57:59.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:57:59.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:57:59.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:57:59.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:57:59.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:57:59.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:57:59.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:57:59.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:57:59.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:57:59.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:57:59.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:57:59.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:57:59.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:57:59.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:57:59.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:57:59.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:57:59.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:57:59.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:57:59.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:57:59.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:57:59.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:57:59.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:57:59.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:57:59.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:57:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:00.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:00.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:59 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:58:00.034 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:58:00.034 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:58:00.034 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:58:00.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:58:00.035 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:58:00.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:58:00.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:58:00.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:58:00.036 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:58:00.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:58:00.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:58:00.037 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:58:00.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:58:00.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:58:00.038 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:58:00.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:58:00.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:58:00.039 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:58:00.040 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:59 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:58:00.040 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:58:00.040 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:58:00.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:58:00.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:58:00.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:58:00.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:58:00.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:58:00.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:58:00.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:58:00.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:58:00.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:58:00.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:58:00.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:58:00.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:58:00.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:58:00.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:58:00.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:58:00.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:58:00.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:58:00.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:58:00.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:58:00.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:58:00.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:58:00.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:58:00.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:58:00.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:58:00.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:58:00.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:58:00.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:58:00.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:58:00.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:58:00.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:58:00.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:58:00.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:58:00.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:58:00.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:58:00.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:58:00.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:58:00.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:58:00.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:58:00.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:58:00.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:58:00.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:58:00.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:58:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:58:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:58:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:58:00.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:58:00.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:58:00.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:57:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:00 smithi016 conmon[32727]: 2022-03-21T18:58:00.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:00 smithi079 ceph-mon[23158]: pgmap v3176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:00.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:00.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:00 smithi016 ceph-mon[22010]: pgmap v3176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:00 smithi016 conmon[32727]: 2022-03-21T18:58:00.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:02.036 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:02 smithi016 conmon[32727]: 2022-03-21T18:58:02.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:02 smithi079 ceph-mon[23158]: pgmap v3177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:03.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:03.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:02 smithi016 ceph-mon[22010]: pgmap v3177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:03 smithi016 conmon[32727]: 2022-03-21T18:58:03.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:04.065 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:04 smithi016 conmon[32727]: 2022-03-21T18:58:04.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:04 smithi079 ceph-mon[23158]: pgmap v3178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:05.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:05.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:04 smithi016 ceph-mon[22010]: pgmap v3178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:05.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:05 smithi016 conmon[32727]: 2022-03-21T18:58:05.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:06.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:06 smithi016 conmon[32727]: 2022-03-21T18:58:06.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:06.980 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:06.981 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:06 smithi016 ceph-mon[22010]: pgmap v3179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:06 smithi079 ceph-mon[23158]: pgmap v3179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:07.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:06 smithi016 conmon[32727]: 2022-03-21T18:58:06.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:07.999 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:07 smithi016 conmon[32727]: 2022-03-21T18:58:07.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:09.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:09.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:08 smithi016 ceph-mon[22010]: pgmap v3180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:08 smithi079 ceph-mon[23158]: pgmap v3180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:09 smithi016 conmon[32727]: 2022-03-21T18:58:09.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:10 smithi016 conmon[32727]: 2022-03-21T18:58:10.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:11.022 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:11.022 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:10 smithi016 ceph-mon[22010]: pgmap v3181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:10 smithi079 ceph-mon[23158]: pgmap v3181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:11 smithi016 conmon[32727]: 2022-03-21T18:58:11.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:12.064 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:12 smithi016 conmon[32727]: 2022-03-21T18:58:12.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:13.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:13.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:12 smithi016 ceph-mon[22010]: pgmap v3182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:12 smithi079 ceph-mon[23158]: pgmap v3182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:13 smithi016 conmon[32727]: 2022-03-21T18:58:13.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:14.035 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:14 smithi016 conmon[32727]: 2022-03-21T18:58:14.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:15.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:15.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:14 smithi016 ceph-mon[22010]: pgmap v3183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:15.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:14 smithi079 ceph-mon[23158]: pgmap v3183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:15 smithi016 conmon[32727]: 2022-03-21T18:58:15.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:16.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:16 smithi016 conmon[32727]: 2022-03-21T18:58:16.068+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:17.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:17.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:16 smithi016 ceph-mon[22010]: pgmap v3184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:16 smithi079 ceph-mon[23158]: pgmap v3184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:17 smithi016 conmon[32727]: 2022-03-21T18:58:17.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:18.035 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:18 smithi016 conmon[32727]: 2022-03-21T18:58:18.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:18 smithi016 ceph-mon[22010]: pgmap v3185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:18 smithi016 conmon[32727]: 2022-03-21T18:58:18.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:18 smithi079 ceph-mon[23158]: pgmap v3185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:20.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:20.034 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:20 smithi016 conmon[32727]: 2022-03-21T18:58:20.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:21.066 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:21.066 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:20 smithi016 ceph-mon[22010]: pgmap v3186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:20 smithi079 ceph-mon[23158]: pgmap v3186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:21 smithi016 conmon[32727]: 2022-03-21T18:58:21.065+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:22.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:22 smithi016 conmon[32727]: 2022-03-21T18:58:22.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:22 smithi016 ceph-mon[22010]: pgmap v3187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:22 smithi016 conmon[32727]: 2022-03-21T18:58:22.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:22 smithi079 ceph-mon[23158]: pgmap v3187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:24 smithi016 conmon[32727]: 2022-03-21T18:58:24.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:24 smithi016 ceph-mon[22010]: pgmap v3188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:25 smithi016 conmon[32727]: 2022-03-21T18:58:25.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:24 smithi079 ceph-mon[23158]: pgmap v3188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:26.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:26 smithi016 conmon[32727]: 2022-03-21T18:58:26.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:27.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:27.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:26 smithi016 ceph-mon[22010]: pgmap v3189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:26 smithi079 ceph-mon[23158]: pgmap v3189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:27.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:27 smithi016 conmon[32727]: 2022-03-21T18:58:27.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:28.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:28 smithi016 conmon[32727]: 2022-03-21T18:58:28.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:29.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:29.077 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:28 smithi016 ceph-mon[22010]: pgmap v3190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:28 smithi079 ceph-mon[23158]: pgmap v3190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:29 smithi016 conmon[32727]: 2022-03-21T18:58:29.074+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:30 smithi016 conmon[32727]: 2022-03-21T18:58:30.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:31.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:31.092 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:30 smithi016 ceph-mon[22010]: pgmap v3191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:30 smithi079 ceph-mon[23158]: pgmap v3191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:31.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:31 smithi016 conmon[32727]: 2022-03-21T18:58:31.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:32.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:32 smithi016 conmon[32727]: 2022-03-21T18:58:32.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:33.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:33.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:32 smithi016 ceph-mon[22010]: pgmap v3192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:32 smithi079 ceph-mon[23158]: pgmap v3192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:33.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:33 smithi016 conmon[32727]: 2022-03-21T18:58:33.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:34.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:34 smithi016 conmon[32727]: 2022-03-21T18:58:34.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:35.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:34 smithi016 ceph-mon[22010]: pgmap v3193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:35.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:34 smithi079 ceph-mon[23158]: pgmap v3193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:35 smithi016 conmon[32727]: 2022-03-21T18:58:35.112+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:36.078 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:36 smithi016 conmon[32727]: 2022-03-21T18:58:36.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:36 smithi016 ceph-mon[22010]: pgmap v3194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:37 smithi016 conmon[32727]: 2022-03-21T18:58:37.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:36 smithi079 ceph-mon[23158]: pgmap v3194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:38 smithi016 conmon[32727]: 2022-03-21T18:58:38.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:39 smithi016 conmon[32727]: 2022-03-21T18:58:39.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:38 smithi016 ceph-mon[22010]: pgmap v3195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:38 smithi079 ceph-mon[23158]: pgmap v3195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:40 smithi016 conmon[32727]: 2022-03-21T18:58:40.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:40 smithi016 conmon[32727]: 2022-03-21T18:58:40.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:40 smithi016 ceph-mon[22010]: pgmap v3196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:40 smithi079 ceph-mon[23158]: pgmap v3196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:41 smithi016 conmon[32727]: 2022-03-21T18:58:41.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:43.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:42 smithi016 conmon[32727]: 2022-03-21T18:58:42.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:42 smithi016 ceph-mon[22010]: pgmap v3197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:42 smithi079 ceph-mon[23158]: pgmap v3197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:43 smithi016 conmon[32727]: 2022-03-21T18:58:43.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:44 smithi016 ceph-mon[22010]: pgmap v3198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:44 smithi016 conmon[32727]: 2022-03-21T18:58:44.906+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:44 smithi079 ceph-mon[23158]: pgmap v3198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T18:58:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:45 smithi016 conmon[32727]: 2022-03-21T18:58:45.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:46 smithi016 ceph-mon[22010]: pgmap v3199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:46 smithi016 conmon[32727]: 2022-03-21T18:58:46.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:46 smithi079 ceph-mon[23158]: pgmap v3199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:47 smithi016 conmon[32727]: 2022-03-21T18:58:47.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:48 smithi016 ceph-mon[22010]: pgmap v3200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:48 smithi016 conmon[32727]: 2022-03-21T18:58:48.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:48 smithi079 ceph-mon[23158]: pgmap v3200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:49 smithi016 conmon[32727]: 2022-03-21T18:58:49.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:50 smithi016 ceph-mon[22010]: pgmap v3201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:50 smithi016 conmon[32727]: 2022-03-21T18:58:50.875+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:50 smithi079 ceph-mon[23158]: pgmap v3201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:51 smithi016 conmon[32727]: 2022-03-21T18:58:51.891+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:52 smithi016 ceph-mon[22010]: pgmap v3202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:52 smithi016 conmon[32727]: 2022-03-21T18:58:52.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:52 smithi079 ceph-mon[23158]: pgmap v3202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:53 smithi016 conmon[32727]: 2022-03-21T18:58:53.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:54 smithi016 ceph-mon[22010]: pgmap v3203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5776 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:54 smithi016 conmon[32727]: 2022-03-21T18:58:54.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:54 smithi079 ceph-mon[23158]: pgmap v3203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5776 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:58:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:55 smithi016 conmon[32727]: 2022-03-21T18:58:55.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:57.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:56 smithi016 ceph-mon[22010]: pgmap v3204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:57.489 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:56 smithi016 conmon[32727]: 2022-03-21T18:58:56.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:57.490 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:57.491 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:56 smithi079 ceph-mon[23158]: pgmap v3204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:58:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:57 smithi016 conmon[32727]: 2022-03-21T18:58:57.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:58 smithi016 conmon[32727]: 2022-03-21T18:58:58.910+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:58:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: pgmap v3205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:58:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:58:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:58:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:58:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:58:59.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:58:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: pgmap v3205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:58:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:58:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:58:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:58:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T18:58:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T18:58:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T18:59:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:58:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:58:59 smithi016 conmon[32727]: 2022-03-21T18:58:59.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:58:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:00 smithi016 ceph-mon[22010]: pgmap v3206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:00 smithi016 conmon[32727]: 2022-03-21T18:59:00.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:00 smithi079 ceph-mon[23158]: pgmap v3206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[32727]: 2022-03-21T18:59:01.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:01.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: 2022-03-21T18:59:01.645+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:59:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: yield 2022-03-21T18:59:01.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:01.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:01.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:01.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:59:01.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:59:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:59:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:59:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:01.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:59:01.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:01.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:01.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:02 smithi016 conmon[32727]: 2022-03-21T18:59:02.843+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:59:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:59:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:59:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:59:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:59:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:59:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:02 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:59:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:02 smithi016 ceph-mon[22010]: pgmap v3207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:59:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:59:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:59:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:59:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:59:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:59:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:03.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:02 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T18:59:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:02 smithi079 ceph-mon[23158]: pgmap v3207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:03.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:03 smithi016 conmon[32727]: 2022-03-21T18:59:03.830+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:04 smithi016 ceph-mon[22010]: pgmap v3208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[32727]: 2022-03-21T18:59:04.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 2022-03-21T18:59:04.717+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:05.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:59:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: yield 2022-03-21T18:59:05.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:05.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:59:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:05.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:59:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:59:05.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:59:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:05.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:59:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:05.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 2022-03-21T18:59:04.718+0000 7f92706db700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:59:05.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:59:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:05.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: yield 2022-03-21T18:59:05.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:05.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:05.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:05.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T18:59:05.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:05.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T18:59:05.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:05.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T18:59:05.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:05.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T18:59:05.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T18:59:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:05.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 2022-03-21T18:59:05.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T18:59:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: 2022-03-21T18:59:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T18:59:05.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:59:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T18:59:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:59:05.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T18:59:05.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:59:05.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:59:05.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:59:05.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:59:05.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:59:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T18:59:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:59:05.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:59:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:59:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T18:59:05.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:59:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T18:59:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:59:05.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:59:05.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:59:05.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T18:59:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:59:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise 2022-03-21T18:59:05.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:59:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T18:59:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:59:05.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T18:59:05.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 18:59:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:04 smithi079 ceph-mon[23158]: pgmap v3208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:05 smithi016 conmon[32727]: 2022-03-21T18:59:05.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:59:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:59:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:59:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:59:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:59:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:05 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:59:06.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:06.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T18:59:06.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:06.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:06.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T18:59:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T18:59:06.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:06.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T18:59:06.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:06.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:06.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:06.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T18:59:06.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:06.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:06.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:59:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:59:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T18:59:06.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:59:06.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T18:59:06.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:59:06.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T18:59:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:59:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:59:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:59:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:59:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:59:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:59:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:59:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:59:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:59:06.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T18:59:06.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:59:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T18:59:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:59:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:59:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:59:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:59:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:59:06.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T18:59:06.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:59:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:59:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:59:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T18:59:06.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:59:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:59:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:59:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:59:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:59:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:06.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:05 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T18:59:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T18:59:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T18:59:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T18:59:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T18:59:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T18:59:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T18:59:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T18:59:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T18:59:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T18:59:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T18:59:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T18:59:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T18:59:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T18:59:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T18:59:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T18:59:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T18:59:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:59:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T18:59:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T18:59:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T18:59:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T18:59:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T18:59:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T18:59:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T18:59:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T18:59:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T18:59:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T18:59:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T18:59:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T18:59:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T18:59:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T18:59:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T18:59:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T18:59:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T18:59:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T18:59:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T18:59:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T18:59:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T18:59:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T18:59:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T18:59:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T18:59:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T18:59:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T18:59:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T18:59:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T18:59:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T18:59:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T18:59:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:06 smithi016 ceph-mon[22010]: pgmap v3209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:06 smithi016 conmon[32727]: 2022-03-21T18:59:06.779+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:06 smithi079 ceph-mon[23158]: pgmap v3209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:07 smithi016 conmon[32727]: 2022-03-21T18:59:07.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:08 smithi016 ceph-mon[22010]: pgmap v3210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:08 smithi016 conmon[32727]: 2022-03-21T18:59:08.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:08 smithi079 ceph-mon[23158]: pgmap v3210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:09 smithi016 conmon[32727]: 2022-03-21T18:59:09.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:10 smithi016 conmon[32727]: 2022-03-21T18:59:10.827+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:10 smithi016 ceph-mon[22010]: pgmap v3211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:10 smithi079 ceph-mon[23158]: pgmap v3211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:11 smithi016 conmon[32727]: 2022-03-21T18:59:11.866+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:12 smithi016 ceph-mon[22010]: pgmap v3212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:12 smithi016 conmon[32727]: 2022-03-21T18:59:12.906+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:12 smithi079 ceph-mon[23158]: pgmap v3212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:13 smithi016 conmon[32727]: 2022-03-21T18:59:13.909+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:14 smithi079 ceph-mon[23158]: pgmap v3213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:14 smithi016 conmon[32727]: 2022-03-21T18:59:14.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:14 smithi016 ceph-mon[22010]: pgmap v3213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:15 smithi016 conmon[32727]: 2022-03-21T18:59:15.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:16 smithi079 ceph-mon[23158]: pgmap v3214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:16 smithi016 ceph-mon[22010]: pgmap v3214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:17.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:16 smithi016 conmon[32727]: 2022-03-21T18:59:16.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:18 smithi016 conmon[32727]: 2022-03-21T18:59:18.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:18 smithi079 ceph-mon[23158]: pgmap v3215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:18 smithi016 ceph-mon[22010]: pgmap v3215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:19 smithi016 conmon[32727]: 2022-03-21T18:59:19.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:19 smithi016 conmon[32727]: 2022-03-21T18:59:19.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:20 smithi079 ceph-mon[23158]: pgmap v3216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:20 smithi016 ceph-mon[22010]: pgmap v3216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:20 smithi016 conmon[32727]: 2022-03-21T18:59:20.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:22 smithi016 conmon[32727]: 2022-03-21T18:59:22.015+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:22 smithi079 ceph-mon[23158]: pgmap v3217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:22 smithi016 ceph-mon[22010]: pgmap v3217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:23 smithi016 conmon[32727]: 2022-03-21T18:59:23.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:24 smithi016 conmon[32727]: 2022-03-21T18:59:24.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:24 smithi079 ceph-mon[23158]: pgmap v3218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:24 smithi016 ceph-mon[22010]: pgmap v3218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:25 smithi016 conmon[32727]: 2022-03-21T18:59:24.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:26 smithi016 conmon[32727]: 2022-03-21T18:59:26.039+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:26 smithi079 ceph-mon[23158]: pgmap v3219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:26 smithi016 ceph-mon[22010]: pgmap v3219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:27 smithi016 conmon[32727]: 2022-03-21T18:59:27.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:28 smithi016 conmon[32727]: 2022-03-21T18:59:28.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:28 smithi079 ceph-mon[23158]: pgmap v3220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:28 smithi016 ceph-mon[22010]: pgmap v3220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:29 smithi016 conmon[32727]: 2022-03-21T18:59:29.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:30 smithi016 conmon[32727]: 2022-03-21T18:59:30.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:30 smithi016 ceph-mon[22010]: pgmap v3221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:31 smithi016 conmon[32727]: 2022-03-21T18:59:31.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:30 smithi079 ceph-mon[23158]: pgmap v3221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:32 smithi016 conmon[32727]: 2022-03-21T18:59:32.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:32 smithi016 ceph-mon[22010]: pgmap v3222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:32 smithi016 conmon[32727]: 2022-03-21T18:59:32.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:32 smithi079 ceph-mon[23158]: pgmap v3222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:33 smithi016 conmon[32727]: 2022-03-21T18:59:33.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:34 smithi016 ceph-mon[22010]: pgmap v3223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:34 smithi016 conmon[32727]: 2022-03-21T18:59:34.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:34 smithi079 ceph-mon[23158]: pgmap v3223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:36 smithi016 conmon[32727]: 2022-03-21T18:59:36.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:36 smithi016 ceph-mon[22010]: pgmap v3224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:36 smithi016 conmon[32727]: 2022-03-21T18:59:36.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:36 smithi079 ceph-mon[23158]: pgmap v3224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:37 smithi016 conmon[32727]: 2022-03-21T18:59:37.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:39 smithi016 ceph-mon[22010]: pgmap v3225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:38 smithi016 conmon[32727]: 2022-03-21T18:59:38.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:39 smithi079 ceph-mon[23158]: pgmap v3225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:39 smithi016 conmon[32727]: 2022-03-21T18:59:39.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:41 smithi016 ceph-mon[22010]: pgmap v3226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:40 smithi016 conmon[32727]: 2022-03-21T18:59:40.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:41 smithi079 ceph-mon[23158]: pgmap v3226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:41 smithi016 conmon[32727]: 2022-03-21T18:59:41.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:43 smithi016 ceph-mon[22010]: pgmap v3227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:42 smithi016 conmon[32727]: 2022-03-21T18:59:42.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:43 smithi079 ceph-mon[23158]: pgmap v3227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:43 smithi016 conmon[32727]: 2022-03-21T18:59:43.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:45 smithi016 ceph-mon[22010]: pgmap v3228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:44 smithi016 conmon[32727]: 2022-03-21T18:59:44.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:45 smithi079 ceph-mon[23158]: pgmap v3228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:45 smithi016 conmon[32727]: 2022-03-21T18:59:45.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:47 smithi016 ceph-mon[22010]: pgmap v3229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:46 smithi016 conmon[32727]: 2022-03-21T18:59:46.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:47 smithi079 ceph-mon[23158]: pgmap v3229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:47 smithi016 conmon[32727]: 2022-03-21T18:59:47.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:49 smithi016 ceph-mon[22010]: pgmap v3230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:48 smithi016 conmon[32727]: 2022-03-21T18:59:48.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:49 smithi079 ceph-mon[23158]: pgmap v3230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:50 smithi016 conmon[32727]: 2022-03-21T18:59:50.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:51 smithi016 conmon[32727]: 2022-03-21T18:59:51.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:51 smithi016 ceph-mon[22010]: pgmap v3231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:51 smithi079 ceph-mon[23158]: pgmap v3231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:52 smithi016 conmon[32727]: 2022-03-21T18:59:52.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:53 smithi016 conmon[32727]: 2022-03-21T18:59:53.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:53 smithi016 ceph-mon[22010]: pgmap v3232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:53 smithi079 ceph-mon[23158]: pgmap v3232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:54 smithi016 conmon[32727]: 2022-03-21T18:59:54.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:55 smithi016 conmon[32727]: 2022-03-21T18:59:55.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:55 smithi016 ceph-mon[22010]: pgmap v3233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:55 smithi079 ceph-mon[23158]: pgmap v3233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T18:59:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:56 smithi016 conmon[32727]: 2022-03-21T18:59:56.004+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:57 smithi016 ceph-mon[22010]: pgmap v3234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:56 smithi016 conmon[32727]: 2022-03-21T18:59:56.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:57 smithi079 ceph-mon[23158]: pgmap v3234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T18:59:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:58 smithi016 ceph-mon[22010]: pgmap v3235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:57 smithi016 conmon[32727]: 2022-03-21T18:59:57.950+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:58 smithi079 ceph-mon[23158]: pgmap v3235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T18:59:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:58 smithi016 conmon[32727]: 2022-03-21T18:59:58.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T18:59:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:59:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:59:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:59:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 18:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:59:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T18:59:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:59:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T18:59:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T18:59:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 18:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:00:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 18:59:59 smithi016 conmon[32727]: 2022-03-21T18:59:59.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: pgmap v3236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops 2022-03-21T19:00:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:00:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:00:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:00:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T19:00:00.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T19:00:00.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops 2022-03-21T19:00:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: pgmap v3236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops 2022-03-21T19:00:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:00:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:00:00.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T19:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 4h, current state unknown, last acting [] 2022-03-21T19:00:00.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5840 sec, osd.4 has slow ops 2022-03-21T19:00:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:01 smithi016 conmon[32727]: 2022-03-21T19:00:01.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:02 smithi016 ceph-mon[22010]: pgmap v3237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:02 smithi016 conmon[32727]: 2022-03-21T19:00:02.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:02 smithi079 ceph-mon[23158]: pgmap v3237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:03 smithi016 conmon[32727]: 2022-03-21T19:00:03.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:04 smithi016 conmon[32727]: 2022-03-21T19:00:04.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:04 smithi016 ceph-mon[22010]: pgmap v3238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:04 smithi079 ceph-mon[23158]: pgmap v3238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:05 smithi016 conmon[32727]: 2022-03-21T19:00:05.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:00:05.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:00:05.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:00:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:00:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:00:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:00:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:06 smithi016 ceph-mon[22010]: pgmap v3239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:06.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:06 smithi016 conmon[32727]: 2022-03-21T19:00:06.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:06 smithi079 ceph-mon[23158]: pgmap v3239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[32727]: 2022-03-21T19:00:07.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:08.103 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: 2022-03-21T19:00:07.821+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:00:08.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:08.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: yield 2022-03-21T19:00:08.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:08.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:08.106 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:08.106 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:00:08.107 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:08.107 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:00:08.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:08.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:00:08.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:08.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:00:08.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:08.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:00:08.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:08.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:08.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:08 smithi016 conmon[32727]: 2022-03-21T19:00:08.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:08 smithi016 ceph-mon[22010]: pgmap v3240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:00:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:00:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:00:08.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:08.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:00:08.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:08.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:00:08.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:08.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:08.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:08.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:00:08.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:08.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:08.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:08 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:00:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:08 smithi079 ceph-mon[23158]: pgmap v3240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:00:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:00:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:00:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:00:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:00:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:00:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:08 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:00:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:09 smithi016 conmon[32727]: 2022-03-21T19:00:09.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[32727]: 2022-03-21T19:00:10.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:10 smithi016 ceph-mon[22010]: pgmap v3241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:10 smithi079 ceph-mon[23158]: pgmap v3241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:11 smithi016 conmon[32727]: 2022-03-21T19:00:11.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:11.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 2022-03-21T19:00:10.893+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:00:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: yield 2022-03-21T19:00:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:00:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:00:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:00:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:00:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 2022-03-21T19:00:10.894+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:00:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:00:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: yield 2022-03-21T19:00:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:00:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:00:11.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:00:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:00:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 2022-03-21T19:00:11.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:00:11.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: 2022-03-21T19:00:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:00:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:00:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:00:11.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:00:11.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:00:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:00:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:00:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:00:11.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:00:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:00:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:00:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:00:11.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:00:11.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:00:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:00:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:00:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:00:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:00:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:00:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:00:11.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:00:11.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:00:11.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise 2022-03-21T19:00:11.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:00:11.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:00:11.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:00:11.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:00:11.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:00:10 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:00:11.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:00:11.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:00:11.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:00:11.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:00:11.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:11 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:00:11.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:00:11.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:00:11.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:00:11.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:00:11.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:00:11.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:00:11.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:00:11.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:00:11.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:00:11.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:00:11.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:00:11.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:00:11.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:00:11.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:00:11.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:00:11.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:00:11.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:00:11.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:00:11.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:00:11.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:00:11.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:00:11.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:00:11.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:00:11.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:00:11.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:00:11.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:00:11.696 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:00:11.696 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:00:11.697 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:00:11.697 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:00:11.698 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:00:11.698 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:00:11.698 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:00:11.699 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:00:11.699 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:00:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:00:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:00:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:00:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:00:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:11 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:00:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:00:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:00:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:00:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:00:11.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:00:11.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:00:11.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:00:11.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:00:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:00:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:00:11.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:00:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:00:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:00:11.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:00:11.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:00:11.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:00:11.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:11.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:00:11.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:00:11.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:00:11.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:00:11.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:00:11.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:00:11.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:00:11.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:00:11.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:00:11.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:00:11.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:00:11.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:00:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:00:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:00:11.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:00:11.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:00:11.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:00:11.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:00:11.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:00:11.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:00:11.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:00:11.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:00:11.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:00:11.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:00:11.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:00:11.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:00:11.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:00:11.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:00:11.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:00:11.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:00:11.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:00:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:12 smithi016 ceph-mon[22010]: pgmap v3242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:12 smithi016 conmon[32727]: 2022-03-21T19:00:12.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:12 smithi079 ceph-mon[23158]: pgmap v3242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:13 smithi016 conmon[32727]: 2022-03-21T19:00:13.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:14 smithi016 ceph-mon[22010]: pgmap v3243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:14 smithi016 conmon[32727]: 2022-03-21T19:00:14.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:14 smithi079 ceph-mon[23158]: pgmap v3243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:15.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:15 smithi016 conmon[32727]: 2022-03-21T19:00:15.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:16 smithi016 ceph-mon[22010]: pgmap v3244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:16 smithi016 conmon[32727]: 2022-03-21T19:00:16.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:16 smithi079 ceph-mon[23158]: pgmap v3244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:17 smithi016 conmon[32727]: 2022-03-21T19:00:17.051+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:18.289 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:18 smithi016 conmon[32727]: 2022-03-21T19:00:18.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:18 smithi016 ceph-mon[22010]: pgmap v3245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:18 smithi079 ceph-mon[23158]: pgmap v3245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:19.291 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:19 smithi016 conmon[32727]: 2022-03-21T19:00:19.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:20.295 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:19 smithi016 conmon[32727]: 2022-03-21T19:00:19.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:20 smithi016 ceph-mon[22010]: pgmap v3246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:20 smithi079 ceph-mon[23158]: pgmap v3246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:21.297 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:20 smithi016 conmon[32727]: 2022-03-21T19:00:20.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:21 smithi016 conmon[32727]: 2022-03-21T19:00:21.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:22 smithi016 ceph-mon[22010]: pgmap v3247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:22 smithi079 ceph-mon[23158]: pgmap v3247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:23.303 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:22 smithi016 conmon[32727]: 2022-03-21T19:00:22.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:24.307 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:23 smithi016 conmon[32727]: 2022-03-21T19:00:23.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:24 smithi016 ceph-mon[22010]: pgmap v3248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:24 smithi079 ceph-mon[23158]: pgmap v3248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:24 smithi016 conmon[32727]: 2022-03-21T19:00:24.903+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:26.315 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:25 smithi016 conmon[32727]: 2022-03-21T19:00:25.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:26 smithi016 ceph-mon[22010]: pgmap v3249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:26 smithi079 ceph-mon[23158]: pgmap v3249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:27.316 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:26 smithi016 conmon[32727]: 2022-03-21T19:00:26.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:28.321 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:27 smithi016 conmon[32727]: 2022-03-21T19:00:27.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:28 smithi016 ceph-mon[22010]: pgmap v3250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:28 smithi079 ceph-mon[23158]: pgmap v3250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:29.323 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:28 smithi016 conmon[32727]: 2022-03-21T19:00:28.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:30.328 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:29 smithi016 conmon[32727]: 2022-03-21T19:00:29.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:30 smithi016 ceph-mon[22010]: pgmap v3251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:30 smithi079 ceph-mon[23158]: pgmap v3251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:31.329 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:30 smithi016 conmon[32727]: 2022-03-21T19:00:30.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:32.333 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:31 smithi016 conmon[32727]: 2022-03-21T19:00:31.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:32 smithi016 ceph-mon[22010]: pgmap v3252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:32 smithi079 ceph-mon[23158]: pgmap v3252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:33.335 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:33 smithi016 conmon[32727]: 2022-03-21T19:00:33.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:34.342 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:33 smithi016 conmon[32727]: 2022-03-21T19:00:33.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:34 smithi016 ceph-mon[22010]: pgmap v3253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:34 smithi079 ceph-mon[23158]: pgmap v3253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:35.343 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:34 smithi016 conmon[32727]: 2022-03-21T19:00:34.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:36.348 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:35 smithi016 conmon[32727]: 2022-03-21T19:00:35.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:36 smithi016 ceph-mon[22010]: pgmap v3254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:36 smithi079 ceph-mon[23158]: pgmap v3254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:37.350 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:37 smithi016 conmon[32727]: 2022-03-21T19:00:37.033+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:38.355 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:38 smithi016 conmon[32727]: 2022-03-21T19:00:38.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:38 smithi016 ceph-mon[22010]: pgmap v3255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:38 smithi079 ceph-mon[23158]: pgmap v3255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:39.356 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:38 smithi016 conmon[32727]: 2022-03-21T19:00:38.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:40.361 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:39 smithi016 conmon[32727]: 2022-03-21T19:00:39.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:40 smithi016 ceph-mon[22010]: pgmap v3256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:40 smithi079 ceph-mon[23158]: pgmap v3256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:41.363 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:40 smithi016 conmon[32727]: 2022-03-21T19:00:40.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:42.368 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:41 smithi016 conmon[32727]: 2022-03-21T19:00:41.932+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:42.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:42.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:42 smithi016 ceph-mon[22010]: pgmap v3257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:42 smithi079 ceph-mon[23158]: pgmap v3257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:43.369 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:42 smithi016 conmon[32727]: 2022-03-21T19:00:42.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:44.373 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:43 smithi016 conmon[32727]: 2022-03-21T19:00:43.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:44 smithi016 ceph-mon[22010]: pgmap v3258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:44 smithi079 ceph-mon[23158]: pgmap v3258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:45.375 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:44 smithi016 conmon[32727]: 2022-03-21T19:00:44.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:45 smithi016 conmon[32727]: 2022-03-21T19:00:45.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:46 smithi016 ceph-mon[22010]: pgmap v3259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:46 smithi079 ceph-mon[23158]: pgmap v3259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:46 smithi016 conmon[32727]: 2022-03-21T19:00:46.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:47 smithi016 conmon[32727]: 2022-03-21T19:00:47.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:48 smithi016 ceph-mon[22010]: pgmap v3260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:48 smithi079 ceph-mon[23158]: pgmap v3260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:48 smithi016 conmon[32727]: 2022-03-21T19:00:48.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:50.393 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:49 smithi016 conmon[32727]: 2022-03-21T19:00:49.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:50 smithi016 ceph-mon[22010]: pgmap v3261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:50 smithi079 ceph-mon[23158]: pgmap v3261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:51.394 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:50 smithi016 conmon[32727]: 2022-03-21T19:00:50.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:52.399 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:51 smithi016 conmon[32727]: 2022-03-21T19:00:51.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:52 smithi016 ceph-mon[22010]: pgmap v3262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:52 smithi079 ceph-mon[23158]: pgmap v3262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:53.400 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:52 smithi016 conmon[32727]: 2022-03-21T19:00:52.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:54.405 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:53 smithi016 conmon[32727]: 2022-03-21T19:00:53.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:54 smithi016 ceph-mon[22010]: pgmap v3263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5896 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:54 smithi079 ceph-mon[23158]: pgmap v3263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:00:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5896 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:55.407 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:54 smithi016 conmon[32727]: 2022-03-21T19:00:54.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:56.412 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:55 smithi016 conmon[32727]: 2022-03-21T19:00:55.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:56 smithi016 ceph-mon[22010]: pgmap v3264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:56 smithi079 ceph-mon[23158]: pgmap v3264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2 op/s 2022-03-21T19:00:57.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:56 smithi016 conmon[32727]: 2022-03-21T19:00:56.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:58 smithi016 conmon[32727]: 2022-03-21T19:00:58.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: pgmap v3265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:00:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:00:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:00:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: pgmap v3265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:00:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:00:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:00:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:00:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:00:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:00:59 smithi016 conmon[32727]: 2022-03-21T19:00:59.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:00:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:00:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:00:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:00:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:00:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:00 smithi016 conmon[32727]: 2022-03-21T19:01:00.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:00 smithi079 ceph-mon[23158]: pgmap v3266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:00 smithi016 ceph-mon[22010]: pgmap v3266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:00 smithi016 conmon[32727]: 2022-03-21T19:01:00.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:01 smithi016 conmon[32727]: 2022-03-21T19:01:01.950+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:02 smithi079 ceph-mon[23158]: pgmap v3267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:02.907 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:02.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:02 smithi016 ceph-mon[22010]: pgmap v3267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:02 smithi016 conmon[32727]: 2022-03-21T19:01:02.906+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:03 smithi016 conmon[32727]: 2022-03-21T19:01:03.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:04 smithi079 ceph-mon[23158]: pgmap v3268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:04 smithi016 ceph-mon[22010]: pgmap v3268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:04 smithi016 conmon[32727]: 2022-03-21T19:01:04.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:05 smithi016 conmon[32727]: 2022-03-21T19:01:05.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:06 smithi079 ceph-mon[23158]: pgmap v3269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:06.877 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:06.877 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:06 smithi016 ceph-mon[22010]: pgmap v3269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:07.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:06 smithi016 conmon[32727]: 2022-03-21T19:01:06.875+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:07.850 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:07.850 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:07 smithi016 conmon[32727]: 2022-03-21T19:01:07.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:08 smithi079 ceph-mon[23158]: pgmap v3270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:08.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:08 smithi016 ceph-mon[22010]: pgmap v3270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:08.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:08 smithi016 conmon[32727]: 2022-03-21T19:01:08.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:09.848 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:09.849 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:09 smithi016 conmon[32727]: 2022-03-21T19:01:09.847+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:10 smithi079 ceph-mon[23158]: pgmap v3271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:10.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:10 smithi016 ceph-mon[22010]: pgmap v3271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:10.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:10 smithi016 conmon[32727]: 2022-03-21T19:01:10.809+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:01:11.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:01:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:01:11.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:11.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:01:11.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:01:11.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:01:11.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: 2022-03-21T19:01:11.501+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:11.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:01:11.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:11.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: yield 2022-03-21T19:01:11.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:11.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:11.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:11.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:01:11.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:11.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:01:11.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:11.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:01:11.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:11.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:01:11.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:11.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:01:11.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:11.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:11.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:11 smithi016 conmon[32727]: 2022-03-21T19:01:11.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:01:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:01:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:12 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:01:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:12.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:12 smithi079 ceph-mon[23158]: pgmap v3272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:12.768 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:12.768 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:01:12.769 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:12.769 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:01:12.769 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:12.770 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:12.770 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:12.771 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:01:12.771 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:12.771 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:01:12.772 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:12.772 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:01:12.772 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:12.773 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:12.773 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:12.773 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:01:12.774 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:12.774 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:12.774 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:12.775 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:12 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:01:12.775 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:12.775 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:12 smithi016 ceph-mon[22010]: pgmap v3272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:13.789 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:13.789 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:12 smithi016 conmon[32727]: 2022-03-21T19:01:12.766+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:13 smithi016 conmon[32727]: 2022-03-21T19:01:13.787+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:14.802 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:01:14.803 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:01:14.803 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: pgmap v3273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:01:14.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:01:14.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:01:14.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:01:14.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5916 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:14.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 2022-03-21T19:01:14.573+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:14.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:01:14.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:14.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: yield 2022-03-21T19:01:14.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:14.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:14.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:14.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:01:14.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:14.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:01:14.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:14.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:01:14.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:14.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:01:14.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:14.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:01:14.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:14.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:14.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:14.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 2022-03-21T19:01:14.574+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:01:14.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:01:14.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:14.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: yield 2022-03-21T19:01:14.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:14.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:14.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:14.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:01:14.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:14.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:01:14.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:14.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:01:14.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:14.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:01:14.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:14.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:01:14.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:14.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:14.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:14.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 2022-03-21T19:01:14.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:01:14.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: 2022-03-21T19:01:14.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:01:14.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:01:14.829 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:01:14.829 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:01:14.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:01:14.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:01:14.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:01:14.831 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:01:14.831 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:01:14.831 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:01:14.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:01:14.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:01:14.832 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:01:14.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:01:14.833 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:01:14.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:01:14.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:01:14.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:01:14.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:01:14.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:01:14.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:01:14.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:01:14.836 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise 2022-03-21T19:01:14.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:01:14.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:01:14.837 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:01:14.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:01:14.838 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:01:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:01:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: pgmap v3273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:01:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:01:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:01:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:01:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:01:14.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5916 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:14 smithi016 conmon[32727]: 2022-03-21T19:01:14.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:15.844 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:15.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:01:15.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:15.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:15.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:15.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:01:15.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:15.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:01:15.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:15.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:01:15.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:15.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:15.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:15.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:01:15.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:15.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:15.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:15 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:01:15.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:15.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:01:15.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:15.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:15.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:15.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:01:15.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:15.854 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:01:15.854 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:15.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:01:15.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:15.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:15.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:15.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:01:15.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:15.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:15.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:01:15.858 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:01:15.858 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:01:15.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:01:15.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:01:15.860 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:01:15.860 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:01:15.861 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:01:15.861 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:01:15.861 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:01:15.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:01:15.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:01:15.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:01:15.863 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:01:15.863 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:01:15.864 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:01:15.864 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:01:15.864 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:01:15.865 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:01:15.865 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:01:15.865 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:01:15.866 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:01:15.866 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:01:15.866 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:01:15.867 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:01:15.867 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:01:15.867 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:01:15.868 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:01:15.868 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:01:15.868 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:01:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:01:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:01:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:01:15.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:15.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:01:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:15 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:01:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:01:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:01:15.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:01:15.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:01:15.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:01:15.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:01:15.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:01:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:01:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:01:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:01:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:01:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:01:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:01:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:01:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:01:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:01:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:01:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:01:15.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:01:15.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:01:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:01:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:01:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:01:15.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:01:15.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:01:15.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:01:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:01:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:01:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:01:15.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:01:15.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:01:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:01:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:01:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:01:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:01:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:01:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:01:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:01:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:01:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:01:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:01:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:01:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:01:15.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:01:15.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:01:15.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:01:15.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:01:15.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:15 smithi016 conmon[32727]: 2022-03-21T19:01:15.841+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:16 smithi016 ceph-mon[22010]: pgmap v3274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:16 smithi016 conmon[32727]: 2022-03-21T19:01:16.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:16 smithi079 ceph-mon[23158]: pgmap v3274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:17 smithi016 conmon[32727]: 2022-03-21T19:01:17.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:18.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:18 smithi016 ceph-mon[22010]: pgmap v3275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:18.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:18 smithi079 ceph-mon[23158]: pgmap v3275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:18 smithi016 conmon[32727]: 2022-03-21T19:01:18.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:19.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:19.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:19 smithi016 conmon[32727]: 2022-03-21T19:01:19.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:20.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:20 smithi016 ceph-mon[22010]: pgmap v3276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:20.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:20 smithi079 ceph-mon[23158]: pgmap v3276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:20 smithi016 conmon[32727]: 2022-03-21T19:01:20.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:21.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:21 smithi016 conmon[32727]: 2022-03-21T19:01:21.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:22.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:22 smithi016 ceph-mon[22010]: pgmap v3277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:22 smithi079 ceph-mon[23158]: pgmap v3277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:23.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:22 smithi016 conmon[32727]: 2022-03-21T19:01:22.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:23.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:23.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:23 smithi016 conmon[32727]: 2022-03-21T19:01:23.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:24.882 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:24 smithi016 ceph-mon[22010]: pgmap v3278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:24.883 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:24.883 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5926 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:24 smithi079 ceph-mon[23158]: pgmap v3278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5926 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:24 smithi016 conmon[32727]: 2022-03-21T19:01:24.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:25.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:25 smithi016 conmon[32727]: 2022-03-21T19:01:25.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:26.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:26 smithi016 ceph-mon[22010]: pgmap v3279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:26 smithi079 ceph-mon[23158]: pgmap v3279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:26 smithi016 conmon[32727]: 2022-03-21T19:01:26.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:27.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:27.890 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:28.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:27 smithi016 conmon[32727]: 2022-03-21T19:01:27.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:28 smithi016 conmon[32727]: 2022-03-21T19:01:28.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:28 smithi016 ceph-mon[22010]: pgmap v3280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:28 smithi079 ceph-mon[23158]: pgmap v3280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:29 smithi016 conmon[32727]: 2022-03-21T19:01:29.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:30 smithi016 conmon[32727]: 2022-03-21T19:01:30.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:30 smithi016 ceph-mon[22010]: pgmap v3281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:30 smithi079 ceph-mon[23158]: pgmap v3281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:31 smithi016 conmon[32727]: 2022-03-21T19:01:31.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:32 smithi016 ceph-mon[22010]: pgmap v3282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:32 smithi016 conmon[32727]: 2022-03-21T19:01:32.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:32 smithi079 ceph-mon[23158]: pgmap v3282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:33 smithi016 conmon[32727]: 2022-03-21T19:01:33.846+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:34 smithi016 ceph-mon[22010]: pgmap v3283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5936 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:34 smithi016 conmon[32727]: 2022-03-21T19:01:34.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:34 smithi079 ceph-mon[23158]: pgmap v3283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5936 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:35 smithi016 conmon[32727]: 2022-03-21T19:01:35.830+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:36 smithi016 ceph-mon[22010]: pgmap v3284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:36 smithi016 conmon[32727]: 2022-03-21T19:01:36.785+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:36 smithi079 ceph-mon[23158]: pgmap v3284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:37 smithi016 conmon[32727]: 2022-03-21T19:01:37.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:38 smithi016 ceph-mon[22010]: pgmap v3285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:38 smithi016 conmon[32727]: 2022-03-21T19:01:38.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:38 smithi079 ceph-mon[23158]: pgmap v3285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5941 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:39 smithi016 conmon[32727]: 2022-03-21T19:01:39.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5941 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:40 smithi079 ceph-mon[23158]: pgmap v3286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:40 smithi016 ceph-mon[22010]: pgmap v3286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:40 smithi016 conmon[32727]: 2022-03-21T19:01:40.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:41 smithi016 conmon[32727]: 2022-03-21T19:01:41.806+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:42 smithi079 ceph-mon[23158]: pgmap v3287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:42 smithi016 ceph-mon[22010]: pgmap v3287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:42 smithi016 conmon[32727]: 2022-03-21T19:01:42.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:43 smithi016 conmon[32727]: 2022-03-21T19:01:43.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:44 smithi079 ceph-mon[23158]: pgmap v3288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:44 smithi016 conmon[32727]: 2022-03-21T19:01:44.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:44 smithi016 ceph-mon[22010]: pgmap v3288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:45 smithi016 conmon[32727]: 2022-03-21T19:01:45.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:46 smithi079 ceph-mon[23158]: pgmap v3289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:46 smithi016 conmon[32727]: 2022-03-21T19:01:46.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:46 smithi016 ceph-mon[22010]: pgmap v3289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:47 smithi016 conmon[32727]: 2022-03-21T19:01:47.722+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:48 smithi079 ceph-mon[23158]: pgmap v3290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:48 smithi016 conmon[32727]: 2022-03-21T19:01:48.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:48 smithi016 ceph-mon[22010]: pgmap v3290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:49 smithi016 conmon[32727]: 2022-03-21T19:01:49.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:50 smithi079 ceph-mon[23158]: pgmap v3291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:50 smithi016 conmon[32727]: 2022-03-21T19:01:50.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:50 smithi016 ceph-mon[22010]: pgmap v3291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:51 smithi016 conmon[32727]: 2022-03-21T19:01:51.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:52 smithi016 conmon[32727]: 2022-03-21T19:01:52.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:52 smithi016 ceph-mon[22010]: pgmap v3292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:52 smithi079 ceph-mon[23158]: pgmap v3292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:53 smithi016 conmon[32727]: 2022-03-21T19:01:53.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:54 smithi016 ceph-mon[22010]: pgmap v3293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:54.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:54 smithi016 conmon[32727]: 2022-03-21T19:01:54.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:54 smithi079 ceph-mon[23158]: pgmap v3293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:01:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:55 smithi016 conmon[32727]: 2022-03-21T19:01:55.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:56 smithi016 ceph-mon[22010]: pgmap v3294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:56 smithi016 conmon[32727]: 2022-03-21T19:01:56.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:56 smithi079 ceph-mon[23158]: pgmap v3294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:57 smithi016 conmon[32727]: 2022-03-21T19:01:57.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:58 smithi016 conmon[32727]: 2022-03-21T19:01:58.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:01:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: pgmap v3295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:01:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:01:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: pgmap v3295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:01:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:01:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:01:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:01:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:01:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:01:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:01:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:01:59.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:01:59 smithi016 conmon[32727]: 2022-03-21T19:01:59.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:01:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:00 smithi016 ceph-mon[22010]: pgmap v3296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:00 smithi016 conmon[32727]: 2022-03-21T19:02:00.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:00 smithi079 ceph-mon[23158]: pgmap v3296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:01 smithi016 conmon[32727]: 2022-03-21T19:02:01.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:02 smithi016 ceph-mon[22010]: pgmap v3297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:02 smithi016 conmon[32727]: 2022-03-21T19:02:02.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:02 smithi079 ceph-mon[23158]: pgmap v3297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:03 smithi016 conmon[32727]: 2022-03-21T19:02:03.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:04 smithi016 ceph-mon[22010]: pgmap v3298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:04 smithi016 conmon[32727]: 2022-03-21T19:02:04.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:04 smithi079 ceph-mon[23158]: pgmap v3298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:05 smithi016 conmon[32727]: 2022-03-21T19:02:05.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:06 smithi016 ceph-mon[22010]: pgmap v3299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:06 smithi016 conmon[32727]: 2022-03-21T19:02:06.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:06 smithi079 ceph-mon[23158]: pgmap v3299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:07 smithi016 conmon[32727]: 2022-03-21T19:02:07.614+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:08 smithi016 ceph-mon[22010]: pgmap v3300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:08 smithi016 conmon[32727]: 2022-03-21T19:02:08.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:08 smithi079 ceph-mon[23158]: pgmap v3300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:09 smithi016 conmon[32727]: 2022-03-21T19:02:09.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:10 smithi016 ceph-mon[22010]: pgmap v3301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:10 smithi016 conmon[32727]: 2022-03-21T19:02:10.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:10 smithi079 ceph-mon[23158]: pgmap v3301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:11 smithi016 conmon[32727]: 2022-03-21T19:02:11.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:12 smithi016 ceph-mon[22010]: pgmap v3302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:12 smithi016 conmon[32727]: 2022-03-21T19:02:12.561+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:12 smithi079 ceph-mon[23158]: pgmap v3302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:13 smithi016 conmon[32727]: 2022-03-21T19:02:13.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:14 smithi016 conmon[32727]: 2022-03-21T19:02:14.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: pgmap v3303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:02:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:02:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:02:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: pgmap v3303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:02:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:02:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:02:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:15 smithi016 conmon[32727]: 2022-03-21T19:02:15.622+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:16 smithi016 ceph-mon[22010]: pgmap v3304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:16 smithi016 conmon[32727]: 2022-03-21T19:02:16.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:16 smithi079 ceph-mon[23158]: pgmap v3304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[32727]: 2022-03-21T19:02:17.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:17.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: 2022-03-21T19:02:17.677+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:17.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:02:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: yield 2022-03-21T19:02:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:02:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:02:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:02:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:02:17.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:17.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:02:17.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:17.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:17.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:18 smithi016 conmon[32727]: 2022-03-21T19:02:18.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:02:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:02:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:02:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:02:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:02:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:02:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:02:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:18 smithi016 ceph-mon[22010]: pgmap v3305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:02:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:02:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:02:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:02:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:02:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:02:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:02:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:18 smithi079 ceph-mon[23158]: pgmap v3305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:19 smithi016 conmon[32727]: 2022-03-21T19:02:19.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:20 smithi016 ceph-mon[22010]: pgmap v3306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[32727]: 2022-03-21T19:02:20.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 2022-03-21T19:02:20.749+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:02:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: yield 2022-03-21T19:02:20.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:20.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:20.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:02:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:02:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:02:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:02:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:02:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:20.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 2022-03-21T19:02:20.750+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:02:20.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:02:20.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:20.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: yield 2022-03-21T19:02:20.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:02:20.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:20.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:02:20.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:02:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:02:20.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:20.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:02:20.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:20.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:20.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:20.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 2022-03-21T19:02:20.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:02:20.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: 2022-03-21T19:02:20.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:02:20.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:02:20.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:02:20.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:02:20.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:02:20.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:02:20.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:02:20.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:02:20.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:02:20.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:02:20.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:02:20.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:02:20.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:02:20.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:02:20.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:02:20.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:02:20.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:02:20.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:02:20.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:02:20.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:02:20.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:02:20.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:02:20.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise 2022-03-21T19:02:20.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:02:20.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:02:20.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:02:20.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:02:20.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:02:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:20 smithi079 ceph-mon[23158]: pgmap v3306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:21 smithi016 conmon[32727]: 2022-03-21T19:02:21.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:02:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:02:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:02:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:02:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:02:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:02:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:02:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:02:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:02:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:02:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:02:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:02:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:02:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:02:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:02:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:02:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:02:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:02:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:02:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:02:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:02:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:02:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:02:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:02:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:02:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:02:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:02:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:02:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:02:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:02:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:02:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:02:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:02:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:02:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:02:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:02:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:02:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:02:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:02:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:02:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:02:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:02:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:02:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:02:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:02:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:02:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:02:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:02:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:02:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:02:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:02:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:02:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:02:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:02:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:02:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:02:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:02:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:02:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:02:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:02:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:02:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:02:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:02:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:02:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:02:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:02:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:02:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:02:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:02:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:02:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:02:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:02:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:02:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:02:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:02:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:02:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:02:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:02:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:02:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:02:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:02:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:02:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:02:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:02:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:02:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:02:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:02:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:02:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:02:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:02:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:02:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:02:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:02:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:02:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:02:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:02:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:02:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:02:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:22 smithi016 ceph-mon[22010]: pgmap v3307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:22 smithi016 conmon[32727]: 2022-03-21T19:02:22.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:22 smithi079 ceph-mon[23158]: pgmap v3307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:23 smithi016 conmon[32727]: 2022-03-21T19:02:23.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:24 smithi016 conmon[32727]: 2022-03-21T19:02:24.689+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:24 smithi016 ceph-mon[22010]: pgmap v3308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:24 smithi079 ceph-mon[23158]: pgmap v3308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:25 smithi016 conmon[32727]: 2022-03-21T19:02:25.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:26 smithi016 ceph-mon[22010]: pgmap v3309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:26 smithi016 conmon[32727]: 2022-03-21T19:02:26.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:26 smithi079 ceph-mon[23158]: pgmap v3309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:27 smithi016 conmon[32727]: 2022-03-21T19:02:27.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:28 smithi016 ceph-mon[22010]: pgmap v3310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:28 smithi016 conmon[32727]: 2022-03-21T19:02:28.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:28 smithi079 ceph-mon[23158]: pgmap v3310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:29 smithi016 conmon[32727]: 2022-03-21T19:02:29.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:30 smithi016 ceph-mon[22010]: pgmap v3311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:30 smithi016 conmon[32727]: 2022-03-21T19:02:30.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:30 smithi079 ceph-mon[23158]: pgmap v3311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:31 smithi016 conmon[32727]: 2022-03-21T19:02:31.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:32 smithi016 conmon[32727]: 2022-03-21T19:02:32.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:32 smithi016 ceph-mon[22010]: pgmap v3312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:32 smithi079 ceph-mon[23158]: pgmap v3312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:33 smithi016 conmon[32727]: 2022-03-21T19:02:33.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:34 smithi016 conmon[32727]: 2022-03-21T19:02:34.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:34 smithi016 ceph-mon[22010]: pgmap v3313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 5995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:34 smithi079 ceph-mon[23158]: pgmap v3313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 5995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:35 smithi016 conmon[32727]: 2022-03-21T19:02:35.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:36 smithi016 conmon[32727]: 2022-03-21T19:02:36.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:36 smithi016 ceph-mon[22010]: pgmap v3314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:36 smithi079 ceph-mon[23158]: pgmap v3314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:37 smithi016 conmon[32727]: 2022-03-21T19:02:37.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:38 smithi016 conmon[32727]: 2022-03-21T19:02:38.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:38 smithi016 ceph-mon[22010]: pgmap v3315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:38 smithi079 ceph-mon[23158]: pgmap v3315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:39.906 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:39 smithi016 conmon[32727]: 2022-03-21T19:02:39.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:40.907 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:40 smithi016 conmon[32727]: 2022-03-21T19:02:40.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:40 smithi016 ceph-mon[22010]: pgmap v3316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:40 smithi079 ceph-mon[23158]: pgmap v3316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:41 smithi016 conmon[32727]: 2022-03-21T19:02:41.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:42 smithi016 ceph-mon[22010]: pgmap v3317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:42 smithi016 conmon[32727]: 2022-03-21T19:02:42.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:42 smithi079 ceph-mon[23158]: pgmap v3317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:43 smithi016 conmon[32727]: 2022-03-21T19:02:43.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:44 smithi016 ceph-mon[22010]: pgmap v3318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:44 smithi016 conmon[32727]: 2022-03-21T19:02:44.733+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:44 smithi079 ceph-mon[23158]: pgmap v3318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:46.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:45 smithi016 conmon[32727]: 2022-03-21T19:02:45.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:46 smithi016 ceph-mon[22010]: pgmap v3319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:46 smithi016 conmon[32727]: 2022-03-21T19:02:46.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:46 smithi079 ceph-mon[23158]: pgmap v3319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:47 smithi016 conmon[32727]: 2022-03-21T19:02:47.696+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:48 smithi016 ceph-mon[22010]: pgmap v3320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:49.198 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:49.198 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:48 smithi016 conmon[32727]: 2022-03-21T19:02:48.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:48 smithi079 ceph-mon[23158]: pgmap v3320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:49 smithi016 conmon[32727]: 2022-03-21T19:02:49.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:50 smithi016 ceph-mon[22010]: pgmap v3321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:50 smithi016 conmon[32727]: 2022-03-21T19:02:50.790+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:50 smithi079 ceph-mon[23158]: pgmap v3321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:51 smithi016 conmon[32727]: 2022-03-21T19:02:51.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:52 smithi016 conmon[32727]: 2022-03-21T19:02:52.876+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:52 smithi016 ceph-mon[22010]: pgmap v3322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:52 smithi079 ceph-mon[23158]: pgmap v3322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:53 smithi016 conmon[32727]: 2022-03-21T19:02:53.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:54 smithi016 ceph-mon[22010]: pgmap v3323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:55.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:54 smithi016 conmon[32727]: 2022-03-21T19:02:54.884+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:54 smithi079 ceph-mon[23158]: pgmap v3323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:02:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:02:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:55 smithi016 conmon[32727]: 2022-03-21T19:02:55.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:56 smithi016 conmon[32727]: 2022-03-21T19:02:56.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:56 smithi016 ceph-mon[22010]: pgmap v3324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:56 smithi079 ceph-mon[23158]: pgmap v3324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:57 smithi016 conmon[32727]: 2022-03-21T19:02:57.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:58 smithi016 conmon[32727]: 2022-03-21T19:02:58.877+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:02:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: pgmap v3325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:02:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:02:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:02:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:02:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:02:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: pgmap v3325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:02:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:02:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:02:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6021 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:02:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:02:59 smithi016 conmon[32727]: 2022-03-21T19:02:59.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6021 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:02:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:00 smithi016 conmon[32727]: 2022-03-21T19:03:00.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:00 smithi016 ceph-mon[22010]: pgmap v3326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:00 smithi079 ceph-mon[23158]: pgmap v3326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:01 smithi016 conmon[32727]: 2022-03-21T19:03:01.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:02 smithi016 conmon[32727]: 2022-03-21T19:03:02.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:02 smithi016 ceph-mon[22010]: pgmap v3327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:02 smithi079 ceph-mon[23158]: pgmap v3327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:03 smithi016 conmon[32727]: 2022-03-21T19:03:03.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:04 smithi016 conmon[32727]: 2022-03-21T19:03:04.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:04 smithi016 ceph-mon[22010]: pgmap v3328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6026 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:04 smithi079 ceph-mon[23158]: pgmap v3328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6026 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:05 smithi016 conmon[32727]: 2022-03-21T19:03:05.824+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:06 smithi016 ceph-mon[22010]: pgmap v3329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:06 smithi016 conmon[32727]: 2022-03-21T19:03:06.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:06 smithi079 ceph-mon[23158]: pgmap v3329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:07 smithi016 conmon[32727]: 2022-03-21T19:03:07.797+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:08 smithi016 ceph-mon[22010]: pgmap v3330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:08 smithi016 conmon[32727]: 2022-03-21T19:03:08.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:08 smithi079 ceph-mon[23158]: pgmap v3330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:09 smithi016 conmon[32727]: 2022-03-21T19:03:09.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6031 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6031 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:10 smithi016 ceph-mon[22010]: pgmap v3331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:10 smithi016 conmon[32727]: 2022-03-21T19:03:10.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:10 smithi079 ceph-mon[23158]: pgmap v3331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:11 smithi016 conmon[32727]: 2022-03-21T19:03:11.872+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:12 smithi016 ceph-mon[22010]: pgmap v3332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:12 smithi016 conmon[32727]: 2022-03-21T19:03:12.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:12 smithi079 ceph-mon[23158]: pgmap v3332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:13 smithi016 conmon[32727]: 2022-03-21T19:03:13.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:14 smithi016 conmon[32727]: 2022-03-21T19:03:14.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:15 smithi016 ceph-mon[22010]: pgmap v3333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:15 smithi079 ceph-mon[23158]: pgmap v3333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:15 smithi016 conmon[32727]: 2022-03-21T19:03:15.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:17 smithi016 ceph-mon[22010]: pgmap v3334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:16 smithi016 conmon[32727]: 2022-03-21T19:03:16.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:17 smithi079 ceph-mon[23158]: pgmap v3334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:17 smithi016 conmon[32727]: 2022-03-21T19:03:17.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:19 smithi016 ceph-mon[22010]: pgmap v3335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:18 smithi016 conmon[32727]: 2022-03-21T19:03:18.784+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:19 smithi079 ceph-mon[23158]: pgmap v3335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6041 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:19 smithi016 conmon[32727]: 2022-03-21T19:03:19.788+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6041 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:20 smithi016 conmon[32727]: 2022-03-21T19:03:20.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:21 smithi016 ceph-mon[22010]: pgmap v3336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:03:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:03:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:03:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:21.211 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:21 smithi079 ceph-mon[23158]: pgmap v3336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:21.211 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:03:21.211 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:03:21.212 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:03:21.212 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:21.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: 2022-03-21T19:03:21.229+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:21.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:03:21.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:21.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: yield 2022-03-21T19:03:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:21.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:03:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:21.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:03:21.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:21.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:03:21.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:03:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:21.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:03:21.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:21.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:21.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:21 smithi016 conmon[32727]: 2022-03-21T19:03:21.832+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:22 smithi016 conmon[32727]: 2022-03-21T19:03:22.808+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:23 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:03:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:03:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:23.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:03:23.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:23.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:03:23.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:23.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:03:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:03:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:23.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:23.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:23 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:03:23.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:23 smithi016 ceph-mon[22010]: pgmap v3337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:23.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:23 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:03:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:03:23.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:23.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:03:23.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:23.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:03:23.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:23.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:03:23.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:23.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:23.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:23.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:03:23.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:23.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:23.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:23.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:23 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:03:23.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:23 smithi079 ceph-mon[23158]: pgmap v3337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:23.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:23 smithi016 conmon[32727]: 2022-03-21T19:03:23.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:24.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 2022-03-21T19:03:24.301+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:03:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: yield 2022-03-21T19:03:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:03:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:03:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:03:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:03:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:03:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 2022-03-21T19:03:24.302+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:03:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:03:24.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:24.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: yield 2022-03-21T19:03:24.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:24.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:24.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:24.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:03:24.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:24.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:03:24.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:24.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:03:24.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:24.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:03:24.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:24.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:03:24.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:24.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:24.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:24.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 2022-03-21T19:03:24.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:03:24.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: 2022-03-21T19:03:24.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:03:24.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:03:24.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:03:24.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:03:24.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:03:24.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:03:24.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:03:24.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:03:24.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:03:24.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:03:24.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:03:24.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:03:24.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:03:24.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:03:24.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:03:24.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:03:24.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:03:24.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:03:24.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:03:24.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:03:24.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:03:24.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:03:24.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise 2022-03-21T19:03:24.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:03:24.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:03:24.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:03:24.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:03:24.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:25 smithi016 ceph-mon[22010]: pgmap v3338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:24 smithi016 conmon[32727]: 2022-03-21T19:03:24.824+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:25 smithi079 ceph-mon[23158]: pgmap v3338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:25 smithi016 conmon[32727]: 2022-03-21T19:03:25.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:26 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:03:26.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:26.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:26.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:26.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:03:26.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:26.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:03:26.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:26.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:03:26.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:26.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:26.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:26.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:03:26.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:26.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:26.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:26 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:03:26.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:26.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:03:26.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:26.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:26.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:26.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:03:26.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:26.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:03:26.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:26.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:03:26.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:26.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:26.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:26.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:03:26.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:26.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:26.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:03:26.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:03:26.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:03:26.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:03:26.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:03:26.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:03:26.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:03:26.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:03:26.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:03:26.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:03:26.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:03:26.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:03:26.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:03:26.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:03:26.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:03:26.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:03:26.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:03:26.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:03:26.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:03:26.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:03:26.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:03:26.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:03:26.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:03:26.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:03:26.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:03:26.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:03:26.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:03:26.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:03:26.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:03:26.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:26 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:03:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:03:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:03:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:03:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:03:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:26 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:03:26.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:03:26.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:03:26.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:03:26.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:03:26.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:03:26.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:03:26.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:03:26.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:03:26.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:03:26.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:03:26.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:03:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:03:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:03:26.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:03:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:03:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:03:26.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:03:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:03:26.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:03:26.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:03:26.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:03:26.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:03:26.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:03:26.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:03:26.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:03:26.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:03:26.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:03:26.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:03:26.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:03:26.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:03:26.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:03:26.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:03:26.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:03:26.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:03:26.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:03:26.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:03:26.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:03:26.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:03:26.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:03:26.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:03:26.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:03:26.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:03:26.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:03:26.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:03:26.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:03:26.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:03:26.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:03:26.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:26 smithi016 conmon[32727]: 2022-03-21T19:03:26.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:27 smithi016 ceph-mon[22010]: pgmap v3339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:27 smithi079 ceph-mon[23158]: pgmap v3339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:27 smithi016 conmon[32727]: 2022-03-21T19:03:27.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:28 smithi016 conmon[32727]: 2022-03-21T19:03:28.838+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:29 smithi016 ceph-mon[22010]: pgmap v3340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:29 smithi079 ceph-mon[23158]: pgmap v3340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:29 smithi016 conmon[32727]: 2022-03-21T19:03:29.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:30 smithi016 conmon[32727]: 2022-03-21T19:03:30.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:31 smithi016 ceph-mon[22010]: pgmap v3341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:31 smithi079 ceph-mon[23158]: pgmap v3341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:31 smithi016 conmon[32727]: 2022-03-21T19:03:31.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:32 smithi016 conmon[32727]: 2022-03-21T19:03:32.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:33 smithi016 ceph-mon[22010]: pgmap v3342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:33.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:33 smithi079 ceph-mon[23158]: pgmap v3342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:34.060 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:33 smithi016 conmon[32727]: 2022-03-21T19:03:33.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:35.061 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:34 smithi016 conmon[32727]: 2022-03-21T19:03:34.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:35 smithi016 ceph-mon[22010]: pgmap v3343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:35 smithi079 ceph-mon[23158]: pgmap v3343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:36.066 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:35 smithi016 conmon[32727]: 2022-03-21T19:03:35.756+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:37.068 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:36 smithi016 conmon[32727]: 2022-03-21T19:03:36.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:37 smithi016 ceph-mon[22010]: pgmap v3344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:37 smithi079 ceph-mon[23158]: pgmap v3344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:38.074 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:37 smithi016 conmon[32727]: 2022-03-21T19:03:37.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:39 smithi016 ceph-mon[22010]: pgmap v3345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:38 smithi016 conmon[32727]: 2022-03-21T19:03:38.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:39 smithi079 ceph-mon[23158]: pgmap v3345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:40.076 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:39 smithi016 conmon[32727]: 2022-03-21T19:03:39.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:41.077 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:40 smithi016 conmon[32727]: 2022-03-21T19:03:40.821+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:41 smithi016 ceph-mon[22010]: pgmap v3346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:41 smithi079 ceph-mon[23158]: pgmap v3346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:41 smithi016 conmon[32727]: 2022-03-21T19:03:41.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:43 smithi016 ceph-mon[22010]: pgmap v3347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:42 smithi016 conmon[32727]: 2022-03-21T19:03:42.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:43 smithi079 ceph-mon[23158]: pgmap v3347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:43 smithi016 conmon[32727]: 2022-03-21T19:03:43.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:45 smithi016 ceph-mon[22010]: pgmap v3348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:44 smithi016 conmon[32727]: 2022-03-21T19:03:44.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:45 smithi079 ceph-mon[23158]: pgmap v3348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:45 smithi016 conmon[32727]: 2022-03-21T19:03:45.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:47 smithi016 ceph-mon[22010]: pgmap v3349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:46 smithi016 conmon[32727]: 2022-03-21T19:03:46.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:47 smithi079 ceph-mon[23158]: pgmap v3349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:48.101 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:47 smithi016 conmon[32727]: 2022-03-21T19:03:47.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:48 smithi016 ceph-mon[22010]: pgmap v3350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:48 smithi079 ceph-mon[23158]: pgmap v3350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:49.103 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:48 smithi016 conmon[32727]: 2022-03-21T19:03:48.803+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:50.104 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:49 smithi016 conmon[32727]: 2022-03-21T19:03:49.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6071 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:50 smithi016 ceph-mon[22010]: pgmap v3351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6071 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:50 smithi079 ceph-mon[23158]: pgmap v3351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:50 smithi016 conmon[32727]: 2022-03-21T19:03:50.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:51 smithi016 conmon[32727]: 2022-03-21T19:03:51.712+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:52 smithi016 ceph-mon[22010]: pgmap v3352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:52 smithi079 ceph-mon[23158]: pgmap v3352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:52 smithi016 conmon[32727]: 2022-03-21T19:03:52.724+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:53 smithi016 conmon[32727]: 2022-03-21T19:03:53.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:54 smithi016 ceph-mon[22010]: pgmap v3353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:54 smithi079 ceph-mon[23158]: pgmap v3353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:03:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:54 smithi016 conmon[32727]: 2022-03-21T19:03:54.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:55.815 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:55 smithi016 conmon[32727]: 2022-03-21T19:03:55.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:56 smithi016 ceph-mon[22010]: pgmap v3354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:56 smithi079 ceph-mon[23158]: pgmap v3354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:56 smithi016 conmon[32727]: 2022-03-21T19:03:56.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:57 smithi016 conmon[32727]: 2022-03-21T19:03:57.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:58 smithi016 ceph-mon[22010]: pgmap v3355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:58 smithi079 ceph-mon[23158]: pgmap v3355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:03:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:58 smithi016 conmon[32727]: 2022-03-21T19:03:58.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:03:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:03:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:03:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:03:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:03:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:03:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:03:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:03:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:03:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:03:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:03:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:03:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:03:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:03:59 smithi016 conmon[32727]: 2022-03-21T19:03:59.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:00 smithi016 ceph-mon[22010]: pgmap v3356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:00 smithi079 ceph-mon[23158]: pgmap v3356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:00 smithi016 conmon[32727]: 2022-03-21T19:04:00.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:01 smithi016 conmon[32727]: 2022-03-21T19:04:01.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:02 smithi016 ceph-mon[22010]: pgmap v3357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:02 smithi079 ceph-mon[23158]: pgmap v3357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:02 smithi016 conmon[32727]: 2022-03-21T19:04:02.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:03 smithi016 conmon[32727]: 2022-03-21T19:04:03.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:04 smithi016 ceph-mon[22010]: pgmap v3358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:04 smithi079 ceph-mon[23158]: pgmap v3358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:04 smithi016 conmon[32727]: 2022-03-21T19:04:04.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:05 smithi016 conmon[32727]: 2022-03-21T19:04:05.670+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:06 smithi016 ceph-mon[22010]: pgmap v3359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:06 smithi079 ceph-mon[23158]: pgmap v3359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:06 smithi016 conmon[32727]: 2022-03-21T19:04:06.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:07 smithi016 conmon[32727]: 2022-03-21T19:04:07.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:08 smithi016 ceph-mon[22010]: pgmap v3360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:08 smithi079 ceph-mon[23158]: pgmap v3360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:08 smithi016 conmon[32727]: 2022-03-21T19:04:08.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:09 smithi016 conmon[32727]: 2022-03-21T19:04:09.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:10 smithi016 ceph-mon[22010]: pgmap v3361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:10 smithi079 ceph-mon[23158]: pgmap v3361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:10 smithi016 conmon[32727]: 2022-03-21T19:04:10.697+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:11 smithi016 conmon[32727]: 2022-03-21T19:04:11.669+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:12.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:12.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:12 smithi016 ceph-mon[22010]: pgmap v3362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:12 smithi079 ceph-mon[23158]: pgmap v3362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:12 smithi016 conmon[32727]: 2022-03-21T19:04:12.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:13.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:13 smithi016 conmon[32727]: 2022-03-21T19:04:13.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:14 smithi016 ceph-mon[22010]: pgmap v3363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:14 smithi079 ceph-mon[23158]: pgmap v3363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:14 smithi016 conmon[32727]: 2022-03-21T19:04:14.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:15 smithi016 conmon[32727]: 2022-03-21T19:04:15.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:16 smithi016 ceph-mon[22010]: pgmap v3364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:16 smithi079 ceph-mon[23158]: pgmap v3364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:16 smithi016 conmon[32727]: 2022-03-21T19:04:16.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:17 smithi016 conmon[32727]: 2022-03-21T19:04:17.735+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:18 smithi016 ceph-mon[22010]: pgmap v3365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:18 smithi079 ceph-mon[23158]: pgmap v3365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:18 smithi016 conmon[32727]: 2022-03-21T19:04:18.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:19 smithi016 conmon[32727]: 2022-03-21T19:04:19.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:20 smithi016 ceph-mon[22010]: pgmap v3366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:20 smithi079 ceph-mon[23158]: pgmap v3366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:20 smithi016 conmon[32727]: 2022-03-21T19:04:20.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:21 smithi016 conmon[32727]: 2022-03-21T19:04:21.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:22 smithi016 ceph-mon[22010]: pgmap v3367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:22 smithi079 ceph-mon[23158]: pgmap v3367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:22 smithi016 conmon[32727]: 2022-03-21T19:04:22.725+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:23 smithi016 conmon[32727]: 2022-03-21T19:04:23.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: pgmap v3368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:04:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:04:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:04:24.706 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:24.706 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: pgmap v3368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:24.707 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:24.707 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:04:24.707 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:04:24.708 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:04:25.153 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:24 smithi016 conmon[32727]: 2022-03-21T19:04:24.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:25 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:04:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:25 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:04:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:25 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:04:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:25 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:04:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:25 smithi016 conmon[32727]: 2022-03-21T19:04:25.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:26 smithi079 ceph-mon[23158]: pgmap v3369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:26 smithi016 ceph-mon[22010]: pgmap v3369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:26 smithi016 conmon[32727]: 2022-03-21T19:04:26.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:27.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: 2022-03-21T19:04:27.405+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:27.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:04:27.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:27.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: yield 2022-03-21T19:04:27.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:27.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:27.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:27.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:04:27.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:27.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:04:27.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:27.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:04:27.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:27.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:04:27.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:27.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:04:27.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:27.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:27.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:27 smithi016 conmon[32727]: 2022-03-21T19:04:27.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:28 smithi016 conmon[32727]: 2022-03-21T19:04:28.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:04:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:04:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:29.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:04:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:29.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:04:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:04:29.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:29.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:29.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:29.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:04:29.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:29.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:29.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:29.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:28 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:04:29.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:29.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:28 smithi016 ceph-mon[22010]: pgmap v3370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:04:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:04:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:29.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:29.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:04:29.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:29.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:04:29.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:29.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:04:29.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:29.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:29.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:29.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:04:29.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:29.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:29.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:29.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:28 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:04:29.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:29.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:28 smithi079 ceph-mon[23158]: pgmap v3370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:29 smithi016 conmon[32727]: 2022-03-21T19:04:29.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:30.734 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 2022-03-21T19:04:30.476+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:30.734 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:04:30.735 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:30.735 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: yield 2022-03-21T19:04:30.735 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:30.736 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:30.736 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:30.736 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:04:30.737 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:30.737 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:04:30.737 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:30.738 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:04:30.738 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:30.739 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:04:30.739 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:30.739 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:04:30.740 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:30.740 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:30.740 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:30.741 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 2022-03-21T19:04:30.477+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:04:30.741 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:04:30.741 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:30.742 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: yield 2022-03-21T19:04:30.742 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:30.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:30.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:30.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:04:30.744 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:30.744 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:04:30.744 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:30.745 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:04:30.745 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:30.745 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:04:30.746 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:30.746 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:04:30.746 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:30.747 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:30.747 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:30.748 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 2022-03-21T19:04:30.748 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:04:30.748 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: 2022-03-21T19:04:30.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:04:30.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:04:30.749 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:04:30.750 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:04:30.750 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:04:30.750 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:04:30.751 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:04:30.751 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:04:30.751 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:04:30.752 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:04:30.752 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:04:30.753 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:04:30.753 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:04:30.753 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:04:30.754 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:04:30.754 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:04:30.754 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:04:30.755 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:04:30.755 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:04:30.755 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:04:30.756 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:04:30.756 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:04:30.756 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise 2022-03-21T19:04:30.757 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:04:30.757 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:04:30.757 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:04:30.758 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:04:30.758 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:30 smithi016 ceph-mon[22010]: pgmap v3371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:30 smithi016 conmon[32727]: 2022-03-21T19:04:30.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:30 smithi079 ceph-mon[23158]: pgmap v3371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:31 smithi016 conmon[32727]: 2022-03-21T19:04:31.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:04:32.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:32.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:32.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:32.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:04:32.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:32.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:04:32.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:32.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:04:32.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:32.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:32.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:32.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:04:32.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:32.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:32.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:04:32.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:32.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:04:32.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:32.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:32.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:32.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:04:32.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:32.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:04:32.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:32.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:04:32.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:32.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:32.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:32.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:04:32.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:32.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:32.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:04:32.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:04:32.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:04:32.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:04:32.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:04:32.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:04:32.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:04:32.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:04:32.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:04:32.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:04:32.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:04:32.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:04:32.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:04:32.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:04:32.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:04:32.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:04:32.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:04:32.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:04:32.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:04:32.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:04:32.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:04:32.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:04:32.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:04:32.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:04:32.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:04:32.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:04:32.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:04:32.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:04:32.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:04:32.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.193 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:04:32.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:32.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:32.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:32.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:04:32.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:32.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:04:32.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:32.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:04:32.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:32.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:32.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:32.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:04:32.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:32.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:32.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:04:32.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:04:32.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:04:32.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:04:32.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:04:32.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:04:32.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:04:32.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:04:32.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:04:32.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:04:32.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:04:32.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:04:32.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:04:32.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:04:32.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:04:32.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:04:32.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:04:32.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:04:32.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:04:32.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:04:32.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:04:32.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:04:32.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:04:32.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:04:32.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:04:32.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:04:32.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:04:32.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:04:32.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:04:32.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:04:32.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:04:32.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:04:32.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:04:32.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:04:32.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:04:32.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:04:32.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:04:32.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:04:32.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:04:32.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:04:32.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:04:32.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:04:32.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:04:32.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:04:32.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:04:32.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:04:32.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:04:32.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:04:32.251 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:32 smithi016 conmon[32727]: 2022-03-21T19:04:32.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:32 smithi016 ceph-mon[22010]: pgmap v3372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:32 smithi079 ceph-mon[23158]: pgmap v3372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:33 smithi016 conmon[32727]: 2022-03-21T19:04:33.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:34 smithi016 conmon[32727]: 2022-03-21T19:04:34.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:34 smithi016 ceph-mon[22010]: pgmap v3373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6116 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:34 smithi079 ceph-mon[23158]: pgmap v3373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6116 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:35 smithi016 conmon[32727]: 2022-03-21T19:04:35.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:36 smithi016 ceph-mon[22010]: pgmap v3374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:36 smithi016 conmon[32727]: 2022-03-21T19:04:36.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:36 smithi079 ceph-mon[23158]: pgmap v3374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:37 smithi016 conmon[32727]: 2022-03-21T19:04:37.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:38 smithi016 ceph-mon[22010]: pgmap v3375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:38 smithi016 conmon[32727]: 2022-03-21T19:04:38.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:38 smithi079 ceph-mon[23158]: pgmap v3375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:39 smithi016 conmon[32727]: 2022-03-21T19:04:39.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:40 smithi016 ceph-mon[22010]: pgmap v3376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:40 smithi016 conmon[32727]: 2022-03-21T19:04:40.923+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:40 smithi079 ceph-mon[23158]: pgmap v3376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:41 smithi016 conmon[32727]: 2022-03-21T19:04:41.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:42 smithi016 ceph-mon[22010]: pgmap v3377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:42 smithi016 conmon[32727]: 2022-03-21T19:04:42.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:42 smithi079 ceph-mon[23158]: pgmap v3377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:43 smithi016 conmon[32727]: 2022-03-21T19:04:43.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:44 smithi079 ceph-mon[23158]: pgmap v3378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6126 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:44 smithi016 ceph-mon[22010]: pgmap v3378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6126 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:44 smithi016 conmon[32727]: 2022-03-21T19:04:44.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:46 smithi016 conmon[32727]: 2022-03-21T19:04:46.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:46 smithi079 ceph-mon[23158]: pgmap v3379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:46 smithi016 ceph-mon[22010]: pgmap v3379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:47 smithi016 conmon[32727]: 2022-03-21T19:04:47.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:48 smithi016 conmon[32727]: 2022-03-21T19:04:48.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:48 smithi079 ceph-mon[23158]: pgmap v3380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:49 smithi016 conmon[32727]: 2022-03-21T19:04:49.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:48 smithi016 ceph-mon[22010]: pgmap v3380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:50 smithi016 conmon[32727]: 2022-03-21T19:04:50.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:50 smithi079 ceph-mon[23158]: pgmap v3381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:50 smithi016 ceph-mon[22010]: pgmap v3381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:51 smithi016 conmon[32727]: 2022-03-21T19:04:51.051+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:52 smithi016 conmon[32727]: 2022-03-21T19:04:52.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:52 smithi079 ceph-mon[23158]: pgmap v3382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:52 smithi016 ceph-mon[22010]: pgmap v3382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:53 smithi016 conmon[32727]: 2022-03-21T19:04:53.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:53 smithi016 conmon[32727]: 2022-03-21T19:04:53.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:54 smithi079 ceph-mon[23158]: pgmap v3383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:54 smithi016 ceph-mon[22010]: pgmap v3383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:04:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:04:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:55 smithi016 conmon[32727]: 2022-03-21T19:04:55.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:56 smithi016 conmon[32727]: 2022-03-21T19:04:56.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:56 smithi016 ceph-mon[22010]: pgmap v3384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:57 smithi016 conmon[32727]: 2022-03-21T19:04:57.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:56 smithi079 ceph-mon[23158]: pgmap v3384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:58 smithi016 conmon[32727]: 2022-03-21T19:04:58.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:04:59 smithi016 conmon[32727]: 2022-03-21T19:04:59.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:04:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: pgmap v3385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:04:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:04:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:04:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:04:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: pgmap v3385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:04:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:04:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:04:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:04:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:04:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:05:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:04:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:00 smithi016 conmon[32727]: 2022-03-21T19:05:00.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:04:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:00 smithi016 ceph-mon[22010]: pgmap v3386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:01.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:01 smithi016 conmon[32727]: 2022-03-21T19:05:01.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:00 smithi079 ceph-mon[23158]: pgmap v3386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:02.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:02 smithi016 conmon[32727]: 2022-03-21T19:05:02.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:02 smithi016 ceph-mon[22010]: pgmap v3387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:03 smithi016 conmon[32727]: 2022-03-21T19:05:03.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:02 smithi079 ceph-mon[23158]: pgmap v3387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:04 smithi016 conmon[32727]: 2022-03-21T19:05:04.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:04 smithi016 ceph-mon[22010]: pgmap v3388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:05 smithi016 conmon[32727]: 2022-03-21T19:05:05.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:04 smithi079 ceph-mon[23158]: pgmap v3388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:06 smithi016 conmon[32727]: 2022-03-21T19:05:06.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:07 smithi016 ceph-mon[22010]: pgmap v3389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:07 smithi016 conmon[32727]: 2022-03-21T19:05:07.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:07 smithi079 ceph-mon[23158]: pgmap v3389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:08 smithi016 conmon[32727]: 2022-03-21T19:05:08.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:09 smithi016 ceph-mon[22010]: pgmap v3390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:09 smithi016 conmon[32727]: 2022-03-21T19:05:09.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:09 smithi079 ceph-mon[23158]: pgmap v3390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:10 smithi016 conmon[32727]: 2022-03-21T19:05:10.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:11.283 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:11 smithi016 ceph-mon[22010]: pgmap v3391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:11.283 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:11 smithi079 ceph-mon[23158]: pgmap v3391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:11 smithi016 conmon[32727]: 2022-03-21T19:05:11.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:12.291 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:12 smithi016 conmon[32727]: 2022-03-21T19:05:12.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:13.299 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:13 smithi016 ceph-mon[22010]: pgmap v3392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:13.300 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:13 smithi079 ceph-mon[23158]: pgmap v3392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:13 smithi016 conmon[32727]: 2022-03-21T19:05:13.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:14 smithi016 conmon[32727]: 2022-03-21T19:05:14.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:15.284 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:15 smithi016 ceph-mon[22010]: pgmap v3393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:15.284 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:15.285 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:15 smithi079 ceph-mon[23158]: pgmap v3393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:15 smithi016 conmon[32727]: 2022-03-21T19:05:15.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:16.318 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:16 smithi016 conmon[32727]: 2022-03-21T19:05:16.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:17.301 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:17 smithi016 ceph-mon[22010]: pgmap v3394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:17.302 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:17 smithi079 ceph-mon[23158]: pgmap v3394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:17.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:17 smithi016 conmon[32727]: 2022-03-21T19:05:17.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:18.333 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:18.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:18 smithi016 conmon[32727]: 2022-03-21T19:05:18.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:19.331 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:19 smithi016 ceph-mon[22010]: pgmap v3395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:19.332 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:19 smithi079 ceph-mon[23158]: pgmap v3395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:19 smithi016 conmon[32727]: 2022-03-21T19:05:19.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:20 smithi016 conmon[32727]: 2022-03-21T19:05:20.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:21.330 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:21 smithi016 ceph-mon[22010]: pgmap v3396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:21.330 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:21 smithi079 ceph-mon[23158]: pgmap v3396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:21 smithi016 conmon[32727]: 2022-03-21T19:05:21.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:22.334 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:22 smithi016 conmon[32727]: 2022-03-21T19:05:22.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:23.328 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:23 smithi016 ceph-mon[22010]: pgmap v3397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:23.328 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:23 smithi079 ceph-mon[23158]: pgmap v3397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:23 smithi016 conmon[32727]: 2022-03-21T19:05:23.326+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:24.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:24 smithi016 conmon[32727]: 2022-03-21T19:05:24.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:25.343 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:25 smithi016 ceph-mon[22010]: pgmap v3398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:25.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:25.344 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:25 smithi079 ceph-mon[23158]: pgmap v3398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:25 smithi016 conmon[32727]: 2022-03-21T19:05:25.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:26 smithi016 conmon[32727]: 2022-03-21T19:05:26.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:27.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:27 smithi016 ceph-mon[22010]: pgmap v3399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:27.340 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:27 smithi079 ceph-mon[23158]: pgmap v3399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:27 smithi016 conmon[32727]: 2022-03-21T19:05:27.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:28.355 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:28 smithi016 conmon[32727]: 2022-03-21T19:05:28.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:29.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:29 smithi016 ceph-mon[22010]: pgmap v3400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:29.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:29 smithi079 ceph-mon[23158]: pgmap v3400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:29 smithi016 conmon[32727]: 2022-03-21T19:05:29.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:30.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:30.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:30 smithi016 conmon[32727]: 2022-03-21T19:05:30.400+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:31 smithi079 ceph-mon[23158]: pgmap v3401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:05:31.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:05:31.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:05:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:31 smithi016 ceph-mon[22010]: pgmap v3401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:05:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:05:31.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:05:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:31 smithi016 conmon[32727]: 2022-03-21T19:05:31.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:32 smithi016 conmon[32727]: 2022-03-21T19:05:32.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:33 smithi016 ceph-mon[22010]: pgmap v3402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:33 smithi079 ceph-mon[23158]: pgmap v3402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[32727]: 2022-03-21T19:05:33.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:33.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: 2022-03-21T19:05:33.580+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:33.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:05:33.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:33.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: yield 2022-03-21T19:05:33.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:33.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:33.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:33.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:05:33.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:33.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:05:33.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:33.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:05:33.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:05:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:33.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:05:33.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:33.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:33.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:34 smithi016 conmon[32727]: 2022-03-21T19:05:34.436+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:35 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:05:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:05:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:35.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:35.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:35.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:05:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:05:35.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:05:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:35.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:05:35.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:35.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:35 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:05:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:35 smithi016 ceph-mon[22010]: pgmap v3403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:35.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:35 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:05:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:05:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:35.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:05:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:05:35.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:05:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:35.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:05:35.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:35.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:35 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:05:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:35 smithi079 ceph-mon[23158]: pgmap v3403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:35.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:35 smithi016 conmon[32727]: 2022-03-21T19:05:35.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[32727]: 2022-03-21T19:05:36.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:36.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 2022-03-21T19:05:36.652+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:36.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:05:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: yield 2022-03-21T19:05:36.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:36.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:05:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:36.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:05:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:05:36.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:05:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:36.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:05:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:36.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 2022-03-21T19:05:36.653+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:05:36.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:05:36.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:36.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: yield 2022-03-21T19:05:36.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:36.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:36.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:36.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:05:36.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:36.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:05:36.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:36.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:05:36.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:36.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:05:36.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:36.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:05:36.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:36.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:36.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:36.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 2022-03-21T19:05:36.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:05:36.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: 2022-03-21T19:05:36.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:05:36.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:05:36.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:05:36.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:05:36.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:05:36.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:05:36.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:05:36.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:05:36.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:05:36.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:05:36.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:05:36.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:05:36.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:05:36.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:05:36.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:05:36.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:05:36.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:05:36.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:05:36.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:05:36.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:05:36.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:05:36.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:05:36.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise 2022-03-21T19:05:36.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:05:36.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:05:36.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:05:36.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:05:36.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:05:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:37 smithi016 ceph-mon[22010]: pgmap v3404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:37 smithi079 ceph-mon[23158]: pgmap v3404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:37 smithi016 conmon[32727]: 2022-03-21T19:05:37.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:05:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:38.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:05:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:38.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:05:38.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:38.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:05:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:05:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:38.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:38 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:05:38.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:38.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:05:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:38.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:05:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:38.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:05:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:05:38.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:38.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:38.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:05:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:38.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:38.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:05:38.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:05:38.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:05:38.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:05:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:05:38.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:05:38.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:05:38.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:05:38.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:05:38.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:05:38.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:05:38.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:05:38.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:05:38.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:05:38.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:05:38.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:05:38.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:05:38.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:05:38.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:05:38.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:05:38.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:05:38.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:05:38.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:05:38.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:05:38.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:05:38.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:05:38.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:05:38.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:05:38.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:05:38.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.440 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:05:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:38.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:38.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:05:38.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:38.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:05:38.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:38.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:05:38.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:38.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:38.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:38.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:05:38.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:38.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:38.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:38 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:05:38.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:05:38.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:05:38.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:05:38.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:05:38.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:05:38.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:05:38.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:05:38.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:05:38.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:05:38.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:05:38.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:05:38.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:05:38.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:05:38.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:05:38.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:05:38.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:05:38.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:05:38.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:05:38.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:05:38.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:05:38.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:05:38.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:05:38.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:05:38.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:05:38.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:05:38.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:05:38.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:05:38.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:05:38.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:05:38.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:05:38.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:05:38.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:05:38.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:05:38.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:05:38.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:05:38.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:05:38.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:05:38.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:05:38.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:05:38.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:05:38.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:05:38.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:05:38.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:05:38.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:05:38.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:05:38.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:05:38.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:05:38.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:38 smithi016 conmon[32727]: 2022-03-21T19:05:38.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:39.372 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:39 smithi016 ceph-mon[22010]: pgmap v3405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:39.373 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:39 smithi079 ceph-mon[23158]: pgmap v3405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:39 smithi016 conmon[32727]: 2022-03-21T19:05:39.370+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:40 smithi016 conmon[32727]: 2022-03-21T19:05:40.329+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:41 smithi016 ceph-mon[22010]: pgmap v3406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:41 smithi016 conmon[32727]: 2022-03-21T19:05:41.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:41 smithi079 ceph-mon[23158]: pgmap v3406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:42 smithi016 conmon[32727]: 2022-03-21T19:05:42.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:43 smithi016 ceph-mon[22010]: pgmap v3407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:43 smithi016 conmon[32727]: 2022-03-21T19:05:43.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:43 smithi079 ceph-mon[23158]: pgmap v3407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:44 smithi016 conmon[32727]: 2022-03-21T19:05:44.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:45 smithi016 ceph-mon[22010]: pgmap v3408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:45 smithi016 conmon[32727]: 2022-03-21T19:05:45.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:45 smithi079 ceph-mon[23158]: pgmap v3408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:46 smithi016 conmon[32727]: 2022-03-21T19:05:46.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:46 smithi016 ceph-mon[22010]: pgmap v3409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:46 smithi079 ceph-mon[23158]: pgmap v3409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:47 smithi016 conmon[32727]: 2022-03-21T19:05:47.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:48 smithi016 conmon[32727]: 2022-03-21T19:05:48.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:48 smithi016 ceph-mon[22010]: pgmap v3410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:48 smithi079 ceph-mon[23158]: pgmap v3410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:49 smithi016 conmon[32727]: 2022-03-21T19:05:49.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:50 smithi016 ceph-mon[22010]: pgmap v3411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:50 smithi016 conmon[32727]: 2022-03-21T19:05:50.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:50 smithi079 ceph-mon[23158]: pgmap v3411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:51.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:51 smithi016 conmon[32727]: 2022-03-21T19:05:51.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:52.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:52 smithi016 ceph-mon[22010]: pgmap v3412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:52.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:52 smithi016 conmon[32727]: 2022-03-21T19:05:52.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:52 smithi079 ceph-mon[23158]: pgmap v3412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:53 smithi016 conmon[32727]: 2022-03-21T19:05:53.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:54 smithi016 conmon[32727]: 2022-03-21T19:05:54.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:54 smithi016 ceph-mon[22010]: pgmap v3413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:54 smithi079 ceph-mon[23158]: pgmap v3413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:05:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:55 smithi016 conmon[32727]: 2022-03-21T19:05:55.162+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:56 smithi016 ceph-mon[22010]: pgmap v3414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:56 smithi016 conmon[32727]: 2022-03-21T19:05:56.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:56 smithi079 ceph-mon[23158]: pgmap v3414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:57 smithi016 conmon[32727]: 2022-03-21T19:05:57.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:58 smithi016 ceph-mon[22010]: pgmap v3415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:58 smithi016 conmon[32727]: 2022-03-21T19:05:58.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:58 smithi079 ceph-mon[23158]: pgmap v3415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:05:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:05:59 smithi016 conmon[32727]: 2022-03-21T19:05:59.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:05:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:05:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:05:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:05:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:05:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:05:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:05:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:05:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:05:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:05:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:05:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:05:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:05:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:00 smithi016 conmon[32727]: 2022-03-21T19:06:00.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:00 smithi016 ceph-mon[22010]: pgmap v3416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:00 smithi079 ceph-mon[23158]: pgmap v3416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:01 smithi016 conmon[32727]: 2022-03-21T19:06:01.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:02 smithi016 conmon[32727]: 2022-03-21T19:06:02.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:02 smithi016 ceph-mon[22010]: pgmap v3417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:02 smithi079 ceph-mon[23158]: pgmap v3417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:03.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:03 smithi016 conmon[32727]: 2022-03-21T19:06:03.122+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:04 smithi016 conmon[32727]: 2022-03-21T19:06:04.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:04 smithi016 ceph-mon[22010]: pgmap v3418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:04 smithi079 ceph-mon[23158]: pgmap v3418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:05 smithi016 conmon[32727]: 2022-03-21T19:06:05.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:06 smithi016 ceph-mon[22010]: pgmap v3419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:06 smithi016 conmon[32727]: 2022-03-21T19:06:06.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:06 smithi079 ceph-mon[23158]: pgmap v3419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:07 smithi016 conmon[32727]: 2022-03-21T19:06:07.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:08 smithi016 ceph-mon[22010]: pgmap v3420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:08 smithi016 conmon[32727]: 2022-03-21T19:06:08.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:08 smithi079 ceph-mon[23158]: pgmap v3420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:09 smithi016 conmon[32727]: 2022-03-21T19:06:09.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:10 smithi016 ceph-mon[22010]: pgmap v3421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:10 smithi016 conmon[32727]: 2022-03-21T19:06:10.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:10 smithi079 ceph-mon[23158]: pgmap v3421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:11.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:11 smithi016 conmon[32727]: 2022-03-21T19:06:11.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:12 smithi016 conmon[32727]: 2022-03-21T19:06:12.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:12 smithi016 ceph-mon[22010]: pgmap v3422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:12 smithi079 ceph-mon[23158]: pgmap v3422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:13 smithi016 conmon[32727]: 2022-03-21T19:06:13.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:14.393 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:14 smithi016 conmon[32727]: 2022-03-21T19:06:14.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:14 smithi016 ceph-mon[22010]: pgmap v3423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:14 smithi079 ceph-mon[23158]: pgmap v3423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:15.394 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:15 smithi016 conmon[32727]: 2022-03-21T19:06:15.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:16 smithi016 ceph-mon[22010]: pgmap v3424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:16 smithi016 conmon[32727]: 2022-03-21T19:06:16.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:16 smithi079 ceph-mon[23158]: pgmap v3424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:17.401 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:17 smithi016 conmon[32727]: 2022-03-21T19:06:17.132+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:18.406 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:18 smithi016 conmon[32727]: 2022-03-21T19:06:18.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:18 smithi016 ceph-mon[22010]: pgmap v3425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:18 smithi079 ceph-mon[23158]: pgmap v3425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:19.408 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:19 smithi016 conmon[32727]: 2022-03-21T19:06:19.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:20.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:20 smithi016 conmon[32727]: 2022-03-21T19:06:20.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:20 smithi016 ceph-mon[22010]: pgmap v3426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:20 smithi079 ceph-mon[23158]: pgmap v3426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:21 smithi016 conmon[32727]: 2022-03-21T19:06:21.060+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:22 smithi016 conmon[32727]: 2022-03-21T19:06:22.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:22 smithi079 ceph-mon[23158]: pgmap v3427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:22 smithi016 ceph-mon[22010]: pgmap v3427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:23 smithi016 conmon[32727]: 2022-03-21T19:06:23.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:24 smithi016 conmon[32727]: 2022-03-21T19:06:24.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:24 smithi079 ceph-mon[23158]: pgmap v3428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:24 smithi016 ceph-mon[22010]: pgmap v3428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:25.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:25 smithi016 conmon[32727]: 2022-03-21T19:06:25.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:26 smithi016 conmon[32727]: 2022-03-21T19:06:26.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:26 smithi079 ceph-mon[23158]: pgmap v3429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:26 smithi016 ceph-mon[22010]: pgmap v3429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:27 smithi016 conmon[32727]: 2022-03-21T19:06:27.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:28 smithi016 conmon[32727]: 2022-03-21T19:06:28.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:28 smithi079 ceph-mon[23158]: pgmap v3430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:28 smithi016 ceph-mon[22010]: pgmap v3430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:29 smithi016 conmon[32727]: 2022-03-21T19:06:29.124+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:30 smithi016 conmon[32727]: 2022-03-21T19:06:30.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:30 smithi079 ceph-mon[23158]: pgmap v3431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:30 smithi016 ceph-mon[22010]: pgmap v3431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:31.449 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:31 smithi016 conmon[32727]: 2022-03-21T19:06:31.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:32.454 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:32 smithi016 conmon[32727]: 2022-03-21T19:06:32.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:32 smithi079 ceph-mon[23158]: pgmap v3432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:32 smithi016 ceph-mon[22010]: pgmap v3432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:33 smithi016 conmon[32727]: 2022-03-21T19:06:33.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:34.460 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:34 smithi016 conmon[32727]: 2022-03-21T19:06:34.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:34 smithi079 ceph-mon[23158]: pgmap v3433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:34 smithi016 ceph-mon[22010]: pgmap v3433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:35.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:35 smithi016 conmon[32727]: 2022-03-21T19:06:35.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:36 smithi016 conmon[32727]: 2022-03-21T19:06:36.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:36 smithi079 ceph-mon[23158]: pgmap v3434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:36.729 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:36.729 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:36 smithi016 ceph-mon[22010]: pgmap v3434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:37.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[32727]: 2022-03-21T19:06:37.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:37.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: 2022-03-21T19:06:37.260+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:37.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:06:37.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:37.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: yield 2022-03-21T19:06:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:37.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:06:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:37.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:06:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:06:37.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:37.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:06:37.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:06:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:37.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:37.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:06:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:06:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:06:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:06:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:06:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:06:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:38 smithi016 conmon[32727]: 2022-03-21T19:06:38.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:06:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:06:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:38.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:38.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:06:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:06:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:06:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:38.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:06:38.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:38.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:38.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:38.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:38 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:06:38.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:38 smithi079 ceph-mon[23158]: pgmap v3435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:06:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:06:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:06:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:06:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:06:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:06:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:38.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:38.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:38 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:06:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:38 smithi016 ceph-mon[22010]: pgmap v3435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:39 smithi016 conmon[32727]: 2022-03-21T19:06:39.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[32727]: 2022-03-21T19:06:40.103+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:40.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 2022-03-21T19:06:40.332+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:40.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:06:40.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:40.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: yield 2022-03-21T19:06:40.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:40.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:40.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:40.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:06:40.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:40.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:06:40.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:06:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:40.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:06:40.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:40.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:06:40.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:40.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:40.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 2022-03-21T19:06:40.333+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:06:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:06:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:40.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: yield 2022-03-21T19:06:40.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:40.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:40.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:40.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:06:40.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:06:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:06:40.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:40.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:06:40.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:40.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:06:40.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:40.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:40.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:40.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 2022-03-21T19:06:40.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:06:40.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: 2022-03-21T19:06:40.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:06:40.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:06:40.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:06:40.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:06:40.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:06:40.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:06:40.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:06:40.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:06:40.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:06:40.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:06:40.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:06:40.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:06:40.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:06:40.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:06:40.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:06:40.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:06:40.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:06:40.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:06:40.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:06:40.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:06:40.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:06:40.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:06:40.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise 2022-03-21T19:06:40.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:06:40.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:06:40.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:06:40.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:06:40.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:06:40 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:40 smithi016 ceph-mon[22010]: pgmap v3436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:40 smithi079 ceph-mon[23158]: pgmap v3436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:41.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:41 smithi016 conmon[32727]: 2022-03-21T19:06:41.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:41 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:41.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:06:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:41.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:06:41.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:41.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:06:41.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:41.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:06:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:41.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:41.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:06:41.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:41.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:41.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:41 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:06:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:41.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:06:41.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:41.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:41.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:06:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:41.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:06:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:06:41.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:41.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:41.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:41.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:06:41.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:06:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:06:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:06:41.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:06:41.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:06:41.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:06:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:06:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:06:41.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:06:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:06:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:06:41.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:06:41.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:06:41.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:06:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:06:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:06:41.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:06:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:06:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:06:41.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:06:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:06:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:06:41.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:06:41.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:06:41.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:06:41.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:06:41.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:06:41.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:06:41.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:06:41.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:41 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:06:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:06:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:06:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:06:41.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:41.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:06:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:41 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:06:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:06:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:06:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:06:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:06:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:06:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:06:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:06:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:06:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:06:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:06:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:06:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:06:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:06:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:06:41.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:06:41.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:06:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:06:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:06:41.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:06:41.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:06:41.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:06:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:06:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:06:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:06:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:06:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:06:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:06:41.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:06:41.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:06:41.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:06:41.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:06:41.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:06:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:06:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:06:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:06:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:06:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:06:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:06:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:06:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:06:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:06:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:06:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:06:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:06:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:06:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:06:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:06:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:42 smithi016 conmon[32727]: 2022-03-21T19:06:42.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:42 smithi016 ceph-mon[22010]: pgmap v3437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:42 smithi079 ceph-mon[23158]: pgmap v3437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:43 smithi016 conmon[32727]: 2022-03-21T19:06:43.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:44 smithi016 conmon[32727]: 2022-03-21T19:06:44.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:44 smithi016 ceph-mon[22010]: pgmap v3438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:44 smithi079 ceph-mon[23158]: pgmap v3438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:45 smithi016 conmon[32727]: 2022-03-21T19:06:45.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:46.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:46 smithi016 conmon[32727]: 2022-03-21T19:06:46.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:46 smithi016 ceph-mon[22010]: pgmap v3439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:46 smithi079 ceph-mon[23158]: pgmap v3439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:47.503 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:47 smithi016 conmon[32727]: 2022-03-21T19:06:47.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:48.508 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:48 smithi016 conmon[32727]: 2022-03-21T19:06:48.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:48 smithi016 ceph-mon[22010]: pgmap v3440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:48 smithi079 ceph-mon[23158]: pgmap v3440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:49 smithi016 conmon[32727]: 2022-03-21T19:06:49.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:49.934 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:50.513 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:50 smithi016 conmon[32727]: 2022-03-21T19:06:50.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:50 smithi016 ceph-mon[22010]: pgmap v3441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:50 smithi079 ceph-mon[23158]: pgmap v3441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:51 smithi016 conmon[32727]: 2022-03-21T19:06:51.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:52 smithi016 conmon[32727]: 2022-03-21T19:06:52.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:52 smithi016 ceph-mon[22010]: pgmap v3442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:52 smithi079 ceph-mon[23158]: pgmap v3442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:53 smithi016 conmon[32727]: 2022-03-21T19:06:53.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:54.527 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:54 smithi016 conmon[32727]: 2022-03-21T19:06:54.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:54 smithi016 ceph-mon[22010]: pgmap v3443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:54 smithi079 ceph-mon[23158]: pgmap v3443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:06:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:55.529 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:55 smithi016 conmon[32727]: 2022-03-21T19:06:55.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:56.533 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:56 smithi016 conmon[32727]: 2022-03-21T19:06:56.246+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:56 smithi016 ceph-mon[22010]: pgmap v3444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:56 smithi079 ceph-mon[23158]: pgmap v3444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:57.535 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:57 smithi016 conmon[32727]: 2022-03-21T19:06:57.233+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:58.540 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:58 smithi016 conmon[32727]: 2022-03-21T19:06:58.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: pgmap v3445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:06:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:06:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:06:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:06:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: pgmap v3445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:06:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:06:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:06:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:06:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:06:59.541 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:06:59 smithi016 conmon[32727]: 2022-03-21T19:06:59.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:06:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:06:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:06:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:06:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:06:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:00.546 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:00 smithi016 conmon[32727]: 2022-03-21T19:07:00.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:00 smithi016 ceph-mon[22010]: pgmap v3446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:00 smithi079 ceph-mon[23158]: pgmap v3446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:01.547 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:01 smithi016 conmon[32727]: 2022-03-21T19:07:01.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:02.552 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:02 smithi016 conmon[32727]: 2022-03-21T19:07:02.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:02 smithi016 ceph-mon[22010]: pgmap v3447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:02 smithi079 ceph-mon[23158]: pgmap v3447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:03.553 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:03 smithi016 conmon[32727]: 2022-03-21T19:07:03.240+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:04.558 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:04 smithi016 conmon[32727]: 2022-03-21T19:07:04.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:04 smithi016 ceph-mon[22010]: pgmap v3448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:04 smithi079 ceph-mon[23158]: pgmap v3448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:05.559 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:05 smithi016 conmon[32727]: 2022-03-21T19:07:05.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:06.564 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:06 smithi016 conmon[32727]: 2022-03-21T19:07:06.284+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:06 smithi016 ceph-mon[22010]: pgmap v3449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:06 smithi079 ceph-mon[23158]: pgmap v3449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:07.566 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:07 smithi016 conmon[32727]: 2022-03-21T19:07:07.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:08.571 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:08 smithi016 conmon[32727]: 2022-03-21T19:07:08.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:08 smithi016 ceph-mon[22010]: pgmap v3450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:08 smithi079 ceph-mon[23158]: pgmap v3450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:09.572 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:09 smithi016 conmon[32727]: 2022-03-21T19:07:09.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:10.579 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:10 smithi016 conmon[32727]: 2022-03-21T19:07:10.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:10 smithi016 ceph-mon[22010]: pgmap v3451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:10 smithi079 ceph-mon[23158]: pgmap v3451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:11.580 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:11 smithi016 conmon[32727]: 2022-03-21T19:07:11.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:12 smithi016 conmon[32727]: 2022-03-21T19:07:12.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:12 smithi016 ceph-mon[22010]: pgmap v3452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:12 smithi079 ceph-mon[23158]: pgmap v3452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:13 smithi016 conmon[32727]: 2022-03-21T19:07:13.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:14 smithi016 conmon[32727]: 2022-03-21T19:07:14.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:14 smithi016 ceph-mon[22010]: pgmap v3453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:14 smithi079 ceph-mon[23158]: pgmap v3453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:15 smithi016 conmon[32727]: 2022-03-21T19:07:15.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:16 smithi016 conmon[32727]: 2022-03-21T19:07:16.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:16 smithi016 ceph-mon[22010]: pgmap v3454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:16 smithi079 ceph-mon[23158]: pgmap v3454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:17 smithi016 conmon[32727]: 2022-03-21T19:07:17.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:18.602 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:18 smithi016 conmon[32727]: 2022-03-21T19:07:18.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:18 smithi016 ceph-mon[22010]: pgmap v3455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:18 smithi079 ceph-mon[23158]: pgmap v3455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:19.604 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:19 smithi016 conmon[32727]: 2022-03-21T19:07:19.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:20.608 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:20 smithi016 conmon[32727]: 2022-03-21T19:07:20.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:20 smithi016 ceph-mon[22010]: pgmap v3456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:20 smithi079 ceph-mon[23158]: pgmap v3456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:21.610 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:21 smithi016 conmon[32727]: 2022-03-21T19:07:21.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:22.614 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:22 smithi016 conmon[32727]: 2022-03-21T19:07:22.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:22 smithi016 ceph-mon[22010]: pgmap v3457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:22 smithi079 ceph-mon[23158]: pgmap v3457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:23.616 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:23 smithi016 conmon[32727]: 2022-03-21T19:07:23.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:24.620 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:24 smithi016 conmon[32727]: 2022-03-21T19:07:24.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:24 smithi016 ceph-mon[22010]: pgmap v3458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:24 smithi079 ceph-mon[23158]: pgmap v3458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:25.622 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:25 smithi016 conmon[32727]: 2022-03-21T19:07:25.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:26.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:26 smithi016 conmon[32727]: 2022-03-21T19:07:26.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:26 smithi016 ceph-mon[22010]: pgmap v3459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:26 smithi079 ceph-mon[23158]: pgmap v3459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:27.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:27 smithi016 conmon[32727]: 2022-03-21T19:07:27.222+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:28.633 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:28 smithi016 conmon[32727]: 2022-03-21T19:07:28.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:28 smithi016 ceph-mon[22010]: pgmap v3460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:28 smithi079 ceph-mon[23158]: pgmap v3460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:29.635 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:29 smithi016 conmon[32727]: 2022-03-21T19:07:29.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:30.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:30 smithi016 conmon[32727]: 2022-03-21T19:07:30.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:30 smithi016 ceph-mon[22010]: pgmap v3461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:30 smithi079 ceph-mon[23158]: pgmap v3461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:31.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:31 smithi016 conmon[32727]: 2022-03-21T19:07:31.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:32.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:32 smithi016 conmon[32727]: 2022-03-21T19:07:32.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:32 smithi016 ceph-mon[22010]: pgmap v3462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:32 smithi079 ceph-mon[23158]: pgmap v3462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:33.647 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:33 smithi016 conmon[32727]: 2022-03-21T19:07:33.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:34 smithi016 conmon[32727]: 2022-03-21T19:07:34.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:34 smithi016 ceph-mon[22010]: pgmap v3463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:34 smithi079 ceph-mon[23158]: pgmap v3463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:35.654 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:35 smithi016 conmon[32727]: 2022-03-21T19:07:35.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:36 smithi016 conmon[32727]: 2022-03-21T19:07:36.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:36 smithi016 ceph-mon[22010]: pgmap v3464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:36 smithi079 ceph-mon[23158]: pgmap v3464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:37 smithi016 conmon[32727]: 2022-03-21T19:07:37.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:38 smithi016 conmon[32727]: 2022-03-21T19:07:38.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:38 smithi016 ceph-mon[22010]: pgmap v3465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:38 smithi079 ceph-mon[23158]: pgmap v3465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:39 smithi016 conmon[32727]: 2022-03-21T19:07:39.118+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:40.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:40.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:40.411 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:40 smithi016 conmon[32727]: 2022-03-21T19:07:40.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:41.054 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:40 smithi079 ceph-mon[23158]: pgmap v3466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:41.054 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:07:41.054 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:07:41.055 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:07:41.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:41.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:40 smithi016 ceph-mon[22010]: pgmap v3466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:41.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:07:41.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:07:41.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:07:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:41 smithi016 conmon[32727]: 2022-03-21T19:07:41.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:42.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:42 smithi016 conmon[32727]: 2022-03-21T19:07:42.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:42 smithi079 ceph-mon[23158]: pgmap v3467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:43.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:43.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:42 smithi016 ceph-mon[22010]: pgmap v3467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[32727]: 2022-03-21T19:07:43.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:43.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: 2022-03-21T19:07:43.436+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:43.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:07:43.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:43.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: yield 2022-03-21T19:07:43.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:43.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:43.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:43.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:07:43.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:43.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:07:43.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:43.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:07:43.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:43.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:07:43.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:43.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:07:43.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:43.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:43.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:43 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:44 smithi016 conmon[32727]: 2022-03-21T19:07:44.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:44 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:07:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:07:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:07:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:07:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:07:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:07:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:44.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:44 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:07:44.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:44 smithi079 ceph-mon[23158]: pgmap v3468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:44.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:45.125 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:45.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:44 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:45.126 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:07:45.127 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:45.127 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:07:45.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:45.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:45.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:45.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:07:45.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:45.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:07:45.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:45.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:07:45.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:45.131 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:45.131 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:45.131 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:07:45.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:45.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:45.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:45.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:44 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:07:45.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:44 smithi016 ceph-mon[22010]: pgmap v3468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:45.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:45 smithi016 conmon[32727]: 2022-03-21T19:07:45.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:46.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[32727]: 2022-03-21T19:07:46.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:46 smithi016 ceph-mon[22010]: pgmap v3469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:46.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 2022-03-21T19:07:46.508+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:46.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:07:46.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:46.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: yield 2022-03-21T19:07:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:46.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:07:46.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:46.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:07:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:07:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:07:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:07:46.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:46.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:46.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:46.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 2022-03-21T19:07:46.509+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:07:46.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:07:46.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:46.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: yield 2022-03-21T19:07:46.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:46.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:46.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:46.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:07:46.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:46.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:07:46.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:46.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:07:46.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:46.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:07:46.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:46.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:07:46.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:46.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:46.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:46.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 2022-03-21T19:07:46.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:07:46.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: 2022-03-21T19:07:46.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:07:46.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:07:46.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:07:46.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:07:46.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:07:46.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:07:46.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:07:46.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:07:46.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:07:46.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:07:46.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:07:46.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:07:46.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:07:46.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:07:46.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:07:46.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:07:46.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:07:46.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:07:46.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:07:46.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:07:46.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:07:46.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:07:46.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise 2022-03-21T19:07:46.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:07:46.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:07:46.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:07:46.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:07:46.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:07:46 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:46 smithi079 ceph-mon[23158]: pgmap v3469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:47 smithi016 conmon[32727]: 2022-03-21T19:07:47.167+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:07:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:47.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:07:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:47.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:07:47.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:47.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:07:47.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:47.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:07:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:47.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:07:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:47 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:07:47.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:07:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:07:47.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:47.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:07:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:07:47.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:47.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:07:47.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:47.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:07:47.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:47.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:07:47.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:07:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:07:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:07:47.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:07:47.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:07:47.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:07:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:07:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:07:47.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:07:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:07:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:07:47.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:07:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:07:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:07:47.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:07:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:07:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:07:47.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:07:47.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:07:47.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:07:47.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:07:47.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:07:47.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:07:47.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:07:47.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:07:47.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:07:47.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:07:47.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:07:47.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:07:47.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:48.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:48.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:48.147 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:07:48.147 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:48.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:07:48.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:48.148 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:48.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:48.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:07:48.149 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:48.150 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:07:48.150 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:48.151 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:07:48.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:48.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:48.152 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:48.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:07:48.153 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:48.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:48.154 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:48.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:47 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:07:48.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:07:48.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:07:48.155 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:07:48.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:07:48.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:07:48.156 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:07:48.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:07:48.157 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:07:48.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:07:48.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:07:48.158 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:07:48.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:07:48.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:07:48.159 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:07:48.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:07:48.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:07:48.160 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:07:48.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:48.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:07:48.161 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:07:48.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:07:48.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:07:48.162 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:07:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:07:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:07:48.163 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:07:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:07:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:07:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:07:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:07:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:07:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:07:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:07:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:07:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:07:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:07:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:07:48.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:07:48.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:07:48.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:07:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:07:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:07:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:07:48.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:07:48.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:07:48.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:07:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:07:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:07:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:07:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:48 smithi016 conmon[32727]: 2022-03-21T19:07:48.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:48 smithi079 ceph-mon[23158]: pgmap v3470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:49.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:49.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:48 smithi016 ceph-mon[22010]: pgmap v3470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:49 smithi016 conmon[32727]: 2022-03-21T19:07:49.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:50.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:50.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:50 smithi016 conmon[32727]: 2022-03-21T19:07:50.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:50 smithi079 ceph-mon[23158]: pgmap v3471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:51.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:51.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:50 smithi016 ceph-mon[22010]: pgmap v3471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:51 smithi016 conmon[32727]: 2022-03-21T19:07:51.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:52.078 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:52 smithi016 conmon[32727]: 2022-03-21T19:07:52.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:52 smithi079 ceph-mon[23158]: pgmap v3472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:53.126 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:53.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:52 smithi016 ceph-mon[22010]: pgmap v3472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:53 smithi016 conmon[32727]: 2022-03-21T19:07:53.124+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:54.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:54 smithi016 conmon[32727]: 2022-03-21T19:07:54.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:54 smithi079 ceph-mon[23158]: pgmap v3473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:55.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:55.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:54 smithi016 ceph-mon[22010]: pgmap v3473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:07:55.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:07:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:55 smithi016 conmon[32727]: 2022-03-21T19:07:55.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:56.125 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:56 smithi016 conmon[32727]: 2022-03-21T19:07:56.123+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:56 smithi079 ceph-mon[23158]: pgmap v3474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:57.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:57.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:56 smithi016 ceph-mon[22010]: pgmap v3474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:57 smithi016 conmon[32727]: 2022-03-21T19:07:57.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:58.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:58 smithi016 conmon[32727]: 2022-03-21T19:07:58.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:07:59.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: pgmap v3475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:07:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:07:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:07:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:07:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:07:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: pgmap v3475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:07:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:07:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:07:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:07:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:07:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:07:59 smithi016 conmon[32727]: 2022-03-21T19:07:59.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:00.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:00.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:07:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:07:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:00 smithi016 conmon[32727]: 2022-03-21T19:08:00.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:01.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:01.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:00 smithi016 ceph-mon[22010]: pgmap v3476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:00 smithi079 ceph-mon[23158]: pgmap v3476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:01 smithi016 conmon[32727]: 2022-03-21T19:08:01.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:02 smithi016 conmon[32727]: 2022-03-21T19:08:02.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:02 smithi016 ceph-mon[22010]: pgmap v3477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:02 smithi079 ceph-mon[23158]: pgmap v3477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:03 smithi016 conmon[32727]: 2022-03-21T19:08:03.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:04.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:04 smithi016 conmon[32727]: 2022-03-21T19:08:04.138+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:05.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:05.113 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:04 smithi016 ceph-mon[22010]: pgmap v3478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:05.113 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:04 smithi079 ceph-mon[23158]: pgmap v3478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:05 smithi016 conmon[32727]: 2022-03-21T19:08:05.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:06.142 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:06 smithi016 conmon[32727]: 2022-03-21T19:08:06.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:07.111 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:07.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:06 smithi016 ceph-mon[22010]: pgmap v3479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:06 smithi079 ceph-mon[23158]: pgmap v3479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:07 smithi016 conmon[32727]: 2022-03-21T19:08:07.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:08.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:08 smithi016 conmon[32727]: 2022-03-21T19:08:08.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:09.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:09.118 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:08 smithi016 ceph-mon[22010]: pgmap v3480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:08 smithi079 ceph-mon[23158]: pgmap v3480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:09 smithi016 conmon[32727]: 2022-03-21T19:08:09.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:10.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:10.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:10 smithi016 conmon[32727]: 2022-03-21T19:08:10.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:11.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:11.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:10 smithi016 ceph-mon[22010]: pgmap v3481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:10 smithi079 ceph-mon[23158]: pgmap v3481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:11 smithi016 conmon[32727]: 2022-03-21T19:08:11.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:12.104 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:12 smithi016 conmon[32727]: 2022-03-21T19:08:12.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:13.079 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:13.080 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:12 smithi016 ceph-mon[22010]: pgmap v3482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:12 smithi079 ceph-mon[23158]: pgmap v3482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:13 smithi016 conmon[32727]: 2022-03-21T19:08:13.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:14.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:14 smithi016 conmon[32727]: 2022-03-21T19:08:14.084+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:15.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:15.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:14 smithi016 ceph-mon[22010]: pgmap v3483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:15.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:14 smithi079 ceph-mon[23158]: pgmap v3483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:15 smithi016 conmon[32727]: 2022-03-21T19:08:15.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:16.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:16 smithi016 conmon[32727]: 2022-03-21T19:08:16.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:17.111 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:17.112 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:16 smithi016 ceph-mon[22010]: pgmap v3484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:16 smithi079 ceph-mon[23158]: pgmap v3484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:17 smithi016 conmon[32727]: 2022-03-21T19:08:17.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:18.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:18 smithi016 conmon[32727]: 2022-03-21T19:08:18.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:19.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:19.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:18 smithi016 ceph-mon[22010]: pgmap v3485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:18 smithi079 ceph-mon[23158]: pgmap v3485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:19 smithi016 conmon[32727]: 2022-03-21T19:08:19.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:20 smithi016 conmon[32727]: 2022-03-21T19:08:20.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:20 smithi016 ceph-mon[22010]: pgmap v3486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:20 smithi079 ceph-mon[23158]: pgmap v3486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:21 smithi016 conmon[32727]: 2022-03-21T19:08:21.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:22.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:22 smithi016 conmon[32727]: 2022-03-21T19:08:22.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:22 smithi016 ceph-mon[22010]: pgmap v3487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:22 smithi079 ceph-mon[23158]: pgmap v3487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:23 smithi016 conmon[32727]: 2022-03-21T19:08:23.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:24.135 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:24 smithi016 conmon[32727]: 2022-03-21T19:08:24.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:25.114 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:25.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:24 smithi016 ceph-mon[22010]: pgmap v3488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:25.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:24 smithi079 ceph-mon[23158]: pgmap v3488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:25.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:25 smithi016 conmon[32727]: 2022-03-21T19:08:25.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:26.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:26 smithi016 conmon[32727]: 2022-03-21T19:08:26.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:27.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:27.209 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:26 smithi016 ceph-mon[22010]: pgmap v3489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:26 smithi079 ceph-mon[23158]: pgmap v3489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:27 smithi016 conmon[32727]: 2022-03-21T19:08:27.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:28.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:28 smithi016 conmon[32727]: 2022-03-21T19:08:28.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:28 smithi016 ceph-mon[22010]: pgmap v3490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:28 smithi079 ceph-mon[23158]: pgmap v3490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:29 smithi016 conmon[32727]: 2022-03-21T19:08:29.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:30.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:30.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:30 smithi016 conmon[32727]: 2022-03-21T19:08:30.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:31.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:31.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:30 smithi016 ceph-mon[22010]: pgmap v3491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:31.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:30 smithi079 ceph-mon[23158]: pgmap v3491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:31 smithi016 conmon[32727]: 2022-03-21T19:08:31.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:32.089 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:32 smithi016 conmon[32727]: 2022-03-21T19:08:32.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:32 smithi016 ceph-mon[22010]: pgmap v3492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:33 smithi016 conmon[32727]: 2022-03-21T19:08:33.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:32 smithi079 ceph-mon[23158]: pgmap v3492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:34 smithi016 conmon[32727]: 2022-03-21T19:08:34.038+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:34 smithi016 ceph-mon[22010]: pgmap v3493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:35 smithi016 conmon[32727]: 2022-03-21T19:08:35.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:34 smithi079 ceph-mon[23158]: pgmap v3493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:36 smithi016 conmon[32727]: 2022-03-21T19:08:36.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:36 smithi016 ceph-mon[22010]: pgmap v3494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:37.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:37 smithi016 conmon[32727]: 2022-03-21T19:08:37.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:36 smithi079 ceph-mon[23158]: pgmap v3494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:38 smithi016 conmon[32727]: 2022-03-21T19:08:38.028+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:39 smithi016 conmon[32727]: 2022-03-21T19:08:39.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:38 smithi016 ceph-mon[22010]: pgmap v3495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:38 smithi079 ceph-mon[23158]: pgmap v3495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:40 smithi016 conmon[32727]: 2022-03-21T19:08:40.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:40 smithi016 ceph-mon[22010]: pgmap v3496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:40 smithi016 conmon[32727]: 2022-03-21T19:08:40.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:40 smithi079 ceph-mon[23158]: pgmap v3496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:41 smithi016 conmon[32727]: 2022-03-21T19:08:41.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:42 smithi016 ceph-mon[22010]: pgmap v3497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:42 smithi016 conmon[32727]: 2022-03-21T19:08:42.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:42 smithi079 ceph-mon[23158]: pgmap v3497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:44 smithi016 conmon[32727]: 2022-03-21T19:08:44.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:44 smithi016 ceph-mon[22010]: pgmap v3498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:44 smithi016 conmon[32727]: 2022-03-21T19:08:44.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:44 smithi079 ceph-mon[23158]: pgmap v3498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:45 smithi016 conmon[32727]: 2022-03-21T19:08:45.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[32727]: 2022-03-21T19:08:46.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: pgmap v3499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:08:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:08:47.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:08:47.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: 2022-03-21T19:08:46.988+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:47.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:08:47.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:47.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: yield 2022-03-21T19:08:47.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:47.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:47.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:08:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:08:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:08:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:47.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:08:47.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:47.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:08:47.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:47.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:47.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: pgmap v3499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:08:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:08:47.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:08:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:47 smithi016 conmon[32727]: 2022-03-21T19:08:47.932+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:08:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:48.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:08:48.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:48.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:08:48.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:48.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:08:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:08:48.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:48.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:48.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:48.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:08:48.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:48.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:48.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:48.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:47 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:08:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:48.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:48.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:08:48.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:48.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:08:48.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:48.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:48.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:48.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:08:48.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:48.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:08:48.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:48.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:08:48.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:48.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:48.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:48.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:08:48.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:48.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:48.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:48.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:47 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:08:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:48 smithi016 conmon[32727]: 2022-03-21T19:08:48.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:49.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:48 smithi016 ceph-mon[22010]: pgmap v3500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:48 smithi079 ceph-mon[23158]: pgmap v3500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:50.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:49 smithi016 conmon[32727]: 2022-03-21T19:08:49.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:50.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 2022-03-21T19:08:50.060+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:50.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:08:50.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:50.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: yield 2022-03-21T19:08:50.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:50.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:50.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:50.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:08:50.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:50.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:08:50.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:50.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:08:50.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:50.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:08:50.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:50.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:08:50.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:50.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:50.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:50.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 2022-03-21T19:08:50.061+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:08:50.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:08:50.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:50.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: yield 2022-03-21T19:08:50.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:50.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:50.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:50.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:08:50.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:50.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:08:50.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:50.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:08:50.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:50.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:08:50.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:50.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:08:50.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:50.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:50.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:50.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 2022-03-21T19:08:50.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:08:50.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: 2022-03-21T19:08:50.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:08:50.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:08:50.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:08:50.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:08:50.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:08:50.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:08:50.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:08:50.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:08:50.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:08:50.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:08:50.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:08:50.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:08:50.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:08:50.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:08:50.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:08:50.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:08:50.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:08:50.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:08:50.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:08:50.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:08:50.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:08:50.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:08:50.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise 2022-03-21T19:08:50.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:08:50.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:08:50.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:08:50.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:08:50.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:50 smithi016 conmon[32727]: 2022-03-21T19:08:50.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:50 smithi016 ceph-mon[22010]: pgmap v3501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:50 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:51.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:08:51.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:51.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:51.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:51.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:08:51.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:51.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:08:51.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:51.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:08:51.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:51.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:51.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:51.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:08:51.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:51.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:51.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:50 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:08:51.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:51.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:08:51.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:51.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:51.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:51.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:08:51.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:08:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:51.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:08:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:51.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:08:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:51.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:08:51.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:08:51.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:08:51.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:08:51.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:08:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:08:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:08:51.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:08:51.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:08:51.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:08:51.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:08:51.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:08:51.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:08:51.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:08:51.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:08:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:08:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:08:51.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:08:51.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:08:51.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:08:51.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:08:51.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:08:51.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:08:51.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:08:51.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:08:51.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:08:51.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:08:51.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:08:51.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:08:51.195 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:50 smithi079 ceph-mon[23158]: pgmap v3501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:50 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:08:51.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:51.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:51.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:51.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:08:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:08:51.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:08:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:51.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:08:51.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:51.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:51.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:50 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:08:51.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:08:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:08:51.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:08:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:08:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:08:51.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:08:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:08:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:08:51.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:08:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:08:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:08:51.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:08:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:08:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:08:51.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:08:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:08:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:51.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:08:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:08:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:08:51.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:08:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:08:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:08:51.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:08:51.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:08:51.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:08:51.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:08:51.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:08:51.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:08:51.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:08:51.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:08:51.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:08:51.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:08:51.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:08:51.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:08:51.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:08:51.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:08:51.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:08:51.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:08:51.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:08:51.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:08:51.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:08:51.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:08:51.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:08:51.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:08:51.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:08:51.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:08:51.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:08:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:51 smithi016 conmon[32727]: 2022-03-21T19:08:51.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:52 smithi016 ceph-mon[22010]: pgmap v3502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:52 smithi016 conmon[32727]: 2022-03-21T19:08:52.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:52 smithi079 ceph-mon[23158]: pgmap v3502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:53 smithi016 conmon[32727]: 2022-03-21T19:08:53.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:54 smithi016 conmon[32727]: 2022-03-21T19:08:54.843+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:54 smithi016 ceph-mon[22010]: pgmap v3503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6376 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:54 smithi079 ceph-mon[23158]: pgmap v3503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:08:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6376 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:08:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:55 smithi016 conmon[32727]: 2022-03-21T19:08:55.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:56 smithi016 ceph-mon[22010]: pgmap v3504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:56 smithi016 conmon[32727]: 2022-03-21T19:08:56.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:56 smithi079 ceph-mon[23158]: pgmap v3504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:57 smithi016 conmon[32727]: 2022-03-21T19:08:57.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:08:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: pgmap v3505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:08:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:08:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:08:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:08:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:08:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: pgmap v3505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:08:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:08:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:08:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:08:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:08:59.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:58 smithi016 conmon[32727]: 2022-03-21T19:08:58.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:08:59 smithi016 conmon[32727]: 2022-03-21T19:08:59.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:08:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6381 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:08:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6381 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:00 smithi079 ceph-mon[23158]: pgmap v3506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:00 smithi016 conmon[32727]: 2022-03-21T19:09:00.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:00 smithi016 ceph-mon[22010]: pgmap v3506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:01 smithi016 conmon[32727]: 2022-03-21T19:09:01.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:02 smithi079 ceph-mon[23158]: pgmap v3507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:02 smithi016 conmon[32727]: 2022-03-21T19:09:02.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:02 smithi016 ceph-mon[22010]: pgmap v3507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:03 smithi016 conmon[32727]: 2022-03-21T19:09:03.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:04 smithi016 ceph-mon[22010]: pgmap v3508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:04 smithi016 conmon[32727]: 2022-03-21T19:09:04.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:04 smithi079 ceph-mon[23158]: pgmap v3508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:05 smithi016 conmon[32727]: 2022-03-21T19:09:05.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:06 smithi079 ceph-mon[23158]: pgmap v3509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:06 smithi016 ceph-mon[22010]: pgmap v3509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:07.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:06 smithi016 conmon[32727]: 2022-03-21T19:09:06.923+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:07 smithi016 conmon[32727]: 2022-03-21T19:09:07.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:08 smithi079 ceph-mon[23158]: pgmap v3510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:08 smithi016 ceph-mon[22010]: pgmap v3510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:08 smithi016 conmon[32727]: 2022-03-21T19:09:08.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6391 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:09 smithi016 conmon[32727]: 2022-03-21T19:09:09.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6391 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:10 smithi079 ceph-mon[23158]: pgmap v3511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:10 smithi016 conmon[32727]: 2022-03-21T19:09:10.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:10 smithi016 ceph-mon[22010]: pgmap v3511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:11 smithi016 conmon[32727]: 2022-03-21T19:09:11.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:12 smithi016 conmon[32727]: 2022-03-21T19:09:12.909+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:12 smithi016 ceph-mon[22010]: pgmap v3512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:12 smithi079 ceph-mon[23158]: pgmap v3512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:09:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:13 smithi016 conmon[32727]: 2022-03-21T19:09:13.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:14 smithi016 conmon[32727]: 2022-03-21T19:09:14.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:14 smithi016 ceph-mon[22010]: pgmap v3513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:14 smithi079 ceph-mon[23158]: pgmap v3513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:15 smithi016 conmon[32727]: 2022-03-21T19:09:15.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:16 smithi016 ceph-mon[22010]: pgmap v3514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:16 smithi016 conmon[32727]: 2022-03-21T19:09:16.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:16 smithi079 ceph-mon[23158]: pgmap v3514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:17 smithi016 conmon[32727]: 2022-03-21T19:09:17.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:18 smithi016 conmon[32727]: 2022-03-21T19:09:18.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:19.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:18 smithi016 ceph-mon[22010]: pgmap v3515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:18 smithi079 ceph-mon[23158]: pgmap v3515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:20.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:19 smithi016 conmon[32727]: 2022-03-21T19:09:19.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:20 smithi016 ceph-mon[22010]: pgmap v3516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:21 smithi016 conmon[32727]: 2022-03-21T19:09:21.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:20 smithi079 ceph-mon[23158]: pgmap v3516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:22 smithi016 conmon[32727]: 2022-03-21T19:09:22.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:23 smithi016 conmon[32727]: 2022-03-21T19:09:23.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:23.725 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:23.726 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:22 smithi016 ceph-mon[22010]: pgmap v3517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:23.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:23.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:22 smithi079 ceph-mon[23158]: pgmap v3517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:24 smithi016 conmon[32727]: 2022-03-21T19:09:24.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:25 smithi016 conmon[32727]: 2022-03-21T19:09:25.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:24 smithi016 ceph-mon[22010]: pgmap v3518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:24 smithi079 ceph-mon[23158]: pgmap v3518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:26 smithi016 conmon[32727]: 2022-03-21T19:09:26.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:27 smithi016 conmon[32727]: 2022-03-21T19:09:27.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:26 smithi016 ceph-mon[22010]: pgmap v3519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:26 smithi079 ceph-mon[23158]: pgmap v3519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:28 smithi016 conmon[32727]: 2022-03-21T19:09:28.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:29 smithi016 conmon[32727]: 2022-03-21T19:09:29.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:29 smithi016 ceph-mon[22010]: pgmap v3520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:29 smithi079 ceph-mon[23158]: pgmap v3520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:30 smithi016 ceph-mon[22010]: pgmap v3521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:30 smithi016 conmon[32727]: 2022-03-21T19:09:30.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:30 smithi079 ceph-mon[23158]: pgmap v3521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:31.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:31 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:31.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:31 smithi016 conmon[32727]: 2022-03-21T19:09:31.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:31 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:32 smithi016 ceph-mon[22010]: pgmap v3522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:32 smithi016 conmon[32727]: 2022-03-21T19:09:32.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:32 smithi079 ceph-mon[23158]: pgmap v3522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:33.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:33 smithi016 conmon[32727]: 2022-03-21T19:09:33.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:34.387 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:34 smithi016 conmon[32727]: 2022-03-21T19:09:34.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:34 smithi016 ceph-mon[22010]: pgmap v3523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:34 smithi079 ceph-mon[23158]: pgmap v3523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:35 smithi016 conmon[32727]: 2022-03-21T19:09:35.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:35.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:36 smithi016 ceph-mon[22010]: pgmap v3524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:36 smithi016 conmon[32727]: 2022-03-21T19:09:36.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:36 smithi079 ceph-mon[23158]: pgmap v3524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:37 smithi016 conmon[32727]: 2022-03-21T19:09:37.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:38 smithi016 conmon[32727]: 2022-03-21T19:09:38.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:38 smithi016 ceph-mon[22010]: pgmap v3525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:38.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:38 smithi079 ceph-mon[23158]: pgmap v3525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:39 smithi016 conmon[32727]: 2022-03-21T19:09:39.246+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:39.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:40 smithi016 ceph-mon[22010]: pgmap v3526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:40 smithi016 conmon[32727]: 2022-03-21T19:09:40.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:40 smithi079 ceph-mon[23158]: pgmap v3526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:41.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:41 smithi016 conmon[32727]: 2022-03-21T19:09:41.179+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:42.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:42 smithi016 conmon[32727]: 2022-03-21T19:09:42.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:42 smithi016 ceph-mon[22010]: pgmap v3527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:42 smithi079 ceph-mon[23158]: pgmap v3527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:43 smithi016 conmon[32727]: 2022-03-21T19:09:43.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:44 smithi016 ceph-mon[22010]: pgmap v3528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:44 smithi016 conmon[32727]: 2022-03-21T19:09:44.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:44 smithi079 ceph-mon[23158]: pgmap v3528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:45 smithi016 conmon[32727]: 2022-03-21T19:09:45.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:46 smithi016 ceph-mon[22010]: pgmap v3529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:46 smithi016 conmon[32727]: 2022-03-21T19:09:46.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:46 smithi079 ceph-mon[23158]: pgmap v3529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:47.429 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:47 smithi016 conmon[32727]: 2022-03-21T19:09:47.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:48.434 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:48 smithi016 conmon[32727]: 2022-03-21T19:09:48.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:48 smithi079 ceph-mon[23158]: pgmap v3530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:48 smithi016 ceph-mon[22010]: pgmap v3530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:49.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:49 smithi016 conmon[32727]: 2022-03-21T19:09:49.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:50 smithi016 conmon[32727]: 2022-03-21T19:09:50.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:50 smithi079 ceph-mon[23158]: pgmap v3531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:09:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:09:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:09:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:50 smithi016 ceph-mon[22010]: pgmap v3531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:09:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:09:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:09:51.440 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:51 smithi016 conmon[32727]: 2022-03-21T19:09:51.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:52 smithi016 conmon[32727]: 2022-03-21T19:09:52.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:52.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:52 smithi016 ceph-mon[22010]: pgmap v3532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:52 smithi079 ceph-mon[23158]: pgmap v3532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:53.447 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[32727]: 2022-03-21T19:09:53.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:53.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: 2022-03-21T19:09:53.164+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:53.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:09:53.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:53.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: yield 2022-03-21T19:09:53.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:53.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:53.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:53.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:09:53.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:53.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:09:53.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:53.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:09:53.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:53.453 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:09:53.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:53.454 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:09:53.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:53.455 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:53.456 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:54 smithi016 conmon[32727]: 2022-03-21T19:09:54.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:09:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:09:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:09:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:09:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:09:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:09:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:09:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:54 smithi016 ceph-mon[22010]: pgmap v3533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:09:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:09:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:09:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:09:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:54.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:09:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:54.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:09:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:09:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:54 smithi079 ceph-mon[23158]: pgmap v3533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:09:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:55 smithi016 conmon[32727]: 2022-03-21T19:09:55.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:56 smithi016 ceph-mon[22010]: pgmap v3534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[32727]: 2022-03-21T19:09:56.286+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:56.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 2022-03-21T19:09:56.236+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:56.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:09:56.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:56.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: yield 2022-03-21T19:09:56.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:56.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:56.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:56.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:09:56.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:56.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:09:56.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:56.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:09:56.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:56.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:09:56.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:56.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:09:56.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:56.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:56.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:56.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 2022-03-21T19:09:56.237+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:09:56.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:09:56.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:56.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: yield 2022-03-21T19:09:56.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:56.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:56.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:56.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:09:56.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:56.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:09:56.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:56.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:09:56.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:56.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:09:56.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:56.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:09:56.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:56.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:56.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:56.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 2022-03-21T19:09:56.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:09:56.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: 2022-03-21T19:09:56.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:09:56.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:09:56.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:09:56.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:09:56.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:09:56.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:09:56.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:09:56.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:09:56.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:09:56.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:09:56.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:09:56.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:09:56.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:09:56.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:09:56.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:09:56.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:09:56.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:09:56.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:09:56.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:09:56.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:09:56.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:09:56.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:09:56.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise 2022-03-21T19:09:56.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:09:56.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:09:56.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:09:56.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:09:56.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:09:56 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:56 smithi079 ceph-mon[23158]: pgmap v3534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:57 smithi016 conmon[32727]: 2022-03-21T19:09:57.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:09:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:09:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:09:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:09:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:09:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:09:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:09:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:09:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:09:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:09:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:09:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:09:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:09:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:09:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:09:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:09:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:09:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:09:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:09:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:09:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:09:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:09:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:09:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:09:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:09:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:09:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:09:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:09:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:09:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:09:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:09:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:09:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:09:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:09:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:09:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:09:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:09:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:09:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:09:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:09:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:09:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:09:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:09:57.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:57.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:09:57.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:57.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:57.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:09:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:57.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:09:57.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:09:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:09:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:09:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:09:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:09:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:09:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:09:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:09:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:09:57.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:09:57.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:09:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:09:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:09:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:09:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:09:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:09:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:09:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:09:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:09:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:09:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:09:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:09:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:09:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:09:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:09:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:09:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:09:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:09:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:09:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:09:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:09:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:09:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:09:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:09:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:09:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:09:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:09:57.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:09:57.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:09:57.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:09:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:09:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:09:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:09:57.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:09:57.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:09:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:09:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:09:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:58 smithi016 conmon[32727]: 2022-03-21T19:09:58.323+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: pgmap v3535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:09:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:09:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:09:58.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:09:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: pgmap v3535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:09:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:09:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:09:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:09:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:09:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:09:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:09:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:09:59 smithi016 conmon[32727]: 2022-03-21T19:09:59.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:09:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:09:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:09:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:00 smithi016 conmon[32727]: 2022-03-21T19:10:00.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: pgmap v3536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops 2022-03-21T19:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:10:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:10:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:10:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops 2022-03-21T19:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: pgmap v3536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops 2022-03-21T19:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:10:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:10:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6440 sec, osd.4 has slow ops 2022-03-21T19:10:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:01 smithi016 conmon[32727]: 2022-03-21T19:10:01.405+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:02 smithi016 ceph-mon[22010]: pgmap v3537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:02 smithi016 conmon[32727]: 2022-03-21T19:10:02.436+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:02 smithi079 ceph-mon[23158]: pgmap v3537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:03 smithi016 conmon[32727]: 2022-03-21T19:10:03.485+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:04 smithi016 conmon[32727]: 2022-03-21T19:10:04.456+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:04 smithi016 ceph-mon[22010]: pgmap v3538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:04 smithi079 ceph-mon[23158]: pgmap v3538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:05 smithi016 conmon[32727]: 2022-03-21T19:10:05.418+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:06 smithi016 ceph-mon[22010]: pgmap v3539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:06 smithi016 conmon[32727]: 2022-03-21T19:10:06.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:06 smithi079 ceph-mon[23158]: pgmap v3539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:07 smithi016 conmon[32727]: 2022-03-21T19:10:07.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:08 smithi016 ceph-mon[22010]: pgmap v3540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:08 smithi016 conmon[32727]: 2022-03-21T19:10:08.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:08 smithi079 ceph-mon[23158]: pgmap v3540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:09 smithi016 conmon[32727]: 2022-03-21T19:10:09.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:10 smithi016 ceph-mon[22010]: pgmap v3541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:10 smithi016 conmon[32727]: 2022-03-21T19:10:10.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:10 smithi079 ceph-mon[23158]: pgmap v3541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:11 smithi016 conmon[32727]: 2022-03-21T19:10:11.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:12 smithi016 ceph-mon[22010]: pgmap v3542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:12 smithi016 conmon[32727]: 2022-03-21T19:10:12.389+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:12 smithi079 ceph-mon[23158]: pgmap v3542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:13 smithi016 conmon[32727]: 2022-03-21T19:10:13.417+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:14 smithi016 conmon[32727]: 2022-03-21T19:10:14.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:14 smithi016 ceph-mon[22010]: pgmap v3543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:14 smithi079 ceph-mon[23158]: pgmap v3543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:15 smithi016 conmon[32727]: 2022-03-21T19:10:15.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:16 smithi016 ceph-mon[22010]: pgmap v3544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:16 smithi016 conmon[32727]: 2022-03-21T19:10:16.465+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:16 smithi079 ceph-mon[23158]: pgmap v3544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:17 smithi016 conmon[32727]: 2022-03-21T19:10:17.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:18 smithi016 ceph-mon[22010]: pgmap v3545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:18 smithi016 conmon[32727]: 2022-03-21T19:10:18.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:18 smithi079 ceph-mon[23158]: pgmap v3545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:19 smithi016 conmon[32727]: 2022-03-21T19:10:19.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:20 smithi016 ceph-mon[22010]: pgmap v3546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:20 smithi016 conmon[32727]: 2022-03-21T19:10:20.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:20 smithi079 ceph-mon[23158]: pgmap v3546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:21 smithi016 conmon[32727]: 2022-03-21T19:10:21.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:22 smithi016 ceph-mon[22010]: pgmap v3547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:22 smithi016 conmon[32727]: 2022-03-21T19:10:22.334+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:22 smithi079 ceph-mon[23158]: pgmap v3547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:23.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:23 smithi016 conmon[32727]: 2022-03-21T19:10:23.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:24.550 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:24 smithi016 conmon[32727]: 2022-03-21T19:10:24.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:24 smithi016 ceph-mon[22010]: pgmap v3548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:24.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:24 smithi079 ceph-mon[23158]: pgmap v3548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:25.552 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:25 smithi016 conmon[32727]: 2022-03-21T19:10:25.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:26.557 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:26 smithi016 conmon[32727]: 2022-03-21T19:10:26.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:26 smithi016 ceph-mon[22010]: pgmap v3549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:26 smithi079 ceph-mon[23158]: pgmap v3549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:27.558 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:27 smithi016 conmon[32727]: 2022-03-21T19:10:27.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:28.563 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:28 smithi016 conmon[32727]: 2022-03-21T19:10:28.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:28 smithi016 ceph-mon[22010]: pgmap v3550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:28 smithi079 ceph-mon[23158]: pgmap v3550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:29.565 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:29 smithi016 conmon[32727]: 2022-03-21T19:10:29.239+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:30.570 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:30 smithi016 conmon[32727]: 2022-03-21T19:10:30.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:30 smithi016 ceph-mon[22010]: pgmap v3551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:30 smithi079 ceph-mon[23158]: pgmap v3551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:31.571 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:31 smithi016 conmon[32727]: 2022-03-21T19:10:31.285+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:32.576 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:32 smithi016 conmon[32727]: 2022-03-21T19:10:32.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:32 smithi016 ceph-mon[22010]: pgmap v3552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:32 smithi079 ceph-mon[23158]: pgmap v3552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:33.578 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:33 smithi016 conmon[32727]: 2022-03-21T19:10:33.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:34.582 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:34 smithi016 conmon[32727]: 2022-03-21T19:10:34.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:34 smithi016 ceph-mon[22010]: pgmap v3553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:34 smithi079 ceph-mon[23158]: pgmap v3553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:35.583 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:35 smithi016 conmon[32727]: 2022-03-21T19:10:35.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:36.588 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:36 smithi016 conmon[32727]: 2022-03-21T19:10:36.182+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:36 smithi016 ceph-mon[22010]: pgmap v3554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:36 smithi079 ceph-mon[23158]: pgmap v3554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:37 smithi016 conmon[32727]: 2022-03-21T19:10:37.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:38 smithi016 conmon[32727]: 2022-03-21T19:10:38.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:38 smithi016 ceph-mon[22010]: pgmap v3555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:38 smithi079 ceph-mon[23158]: pgmap v3555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:39 smithi016 conmon[32727]: 2022-03-21T19:10:39.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:40 smithi016 conmon[32727]: 2022-03-21T19:10:40.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:40 smithi016 ceph-mon[22010]: pgmap v3556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:40 smithi079 ceph-mon[23158]: pgmap v3556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:41 smithi016 conmon[32727]: 2022-03-21T19:10:41.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:42 smithi016 conmon[32727]: 2022-03-21T19:10:42.148+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:42 smithi016 ceph-mon[22010]: pgmap v3557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:42 smithi079 ceph-mon[23158]: pgmap v3557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:43.608 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:43 smithi016 conmon[32727]: 2022-03-21T19:10:43.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:44.612 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:44 smithi016 conmon[32727]: 2022-03-21T19:10:44.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:44 smithi016 ceph-mon[22010]: pgmap v3558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:44.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:44 smithi079 ceph-mon[23158]: pgmap v3558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:45 smithi016 conmon[32727]: 2022-03-21T19:10:45.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:46 smithi016 conmon[32727]: 2022-03-21T19:10:46.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:46 smithi016 ceph-mon[22010]: pgmap v3559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:46 smithi079 ceph-mon[23158]: pgmap v3559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:47 smithi016 conmon[32727]: 2022-03-21T19:10:47.149+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:48.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:48 smithi016 conmon[32727]: 2022-03-21T19:10:48.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:48 smithi016 ceph-mon[22010]: pgmap v3560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:48 smithi079 ceph-mon[23158]: pgmap v3560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:49 smithi016 conmon[32727]: 2022-03-21T19:10:49.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:50 smithi016 conmon[32727]: 2022-03-21T19:10:50.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:50 smithi016 ceph-mon[22010]: pgmap v3561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:50 smithi079 ceph-mon[23158]: pgmap v3561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:51 smithi016 conmon[32727]: 2022-03-21T19:10:51.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:52 smithi016 conmon[32727]: 2022-03-21T19:10:52.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:52 smithi016 ceph-mon[22010]: pgmap v3562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:52 smithi079 ceph-mon[23158]: pgmap v3562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:53 smithi016 conmon[32727]: 2022-03-21T19:10:53.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:54 smithi016 conmon[32727]: 2022-03-21T19:10:54.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:54 smithi016 ceph-mon[22010]: pgmap v3563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:54 smithi079 ceph-mon[23158]: pgmap v3563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:10:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:55 smithi016 conmon[32727]: 2022-03-21T19:10:55.007+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:56.289 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:55 smithi016 conmon[32727]: 2022-03-21T19:10:55.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:56 smithi016 ceph-mon[22010]: pgmap v3564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:10:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:10:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:10:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:56 smithi079 ceph-mon[23158]: pgmap v3564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:10:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:10:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:10:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:56 smithi016 conmon[32727]: 2022-03-21T19:10:56.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:57.819 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:57 smithi016 conmon[32727]: 2022-03-21T19:10:57.905+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:58 smithi016 conmon[32727]: 2022-03-21T19:10:58.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:10:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: pgmap v3565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:10:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:10:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:10:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:10:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: pgmap v3565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:10:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:10:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:10:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:10:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:10:59.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: 2022-03-21T19:10:59.340+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:10:59.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:10:59.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:10:59.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: yield 2022-03-21T19:10:59.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:10:59.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:10:59.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:10:59.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:10:59.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:10:59.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:10:59.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:10:59.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:10:59.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:10:59.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:10:59.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:10:59.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:10:59.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:10:59.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:10:59.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:10:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:10:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:10:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:10:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:10:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:10:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:10:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:10:59 smithi016 conmon[32727]: 2022-03-21T19:10:59.925+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:00 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:11:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:11:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:00.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:00.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:00.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:11:00.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:00.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:11:00.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:00.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:11:00.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:00.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:00.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:00.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:11:00.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:00.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:00.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:00.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:00 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:11:00.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:00 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T19:11:00.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:00.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:00 smithi016 ceph-mon[22010]: pgmap v3566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:00 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:11:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:11:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:00.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:00.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:00.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:11:00.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:00.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:11:00.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:00.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:11:00.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:00.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:00.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:00.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:11:00.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:00.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:00.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:00.985 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:00 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:11:00.985 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:00 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T19:11:00.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:00.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:00 smithi079 ceph-mon[23158]: pgmap v3566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:00 smithi016 conmon[32727]: 2022-03-21T19:11:00.917+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:01 smithi016 conmon[32727]: 2022-03-21T19:11:01.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 2022-03-21T19:11:02.413+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:11:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: yield 2022-03-21T19:11:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:11:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:11:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:11:02.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:02.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:11:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:11:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:02.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:02.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:02.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 2022-03-21T19:11:02.414+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:11:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:11:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: yield 2022-03-21T19:11:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:11:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:11:02.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:02.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:11:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:11:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:02.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:11:02.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:02.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 2022-03-21T19:11:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:11:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: 2022-03-21T19:11:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:11:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:11:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:11:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:11:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:11:02.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:11:02.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:11:02.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:11:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:11:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:11:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:11:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:11:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:11:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:11:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:11:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:11:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:11:02.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:11:02.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:11:02.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:11:02.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:11:02.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:11:02.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise 2022-03-21T19:11:02.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:11:02.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:11:02.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:11:02.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:11:02.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:02 smithi079 ceph-mon[23158]: pgmap v3567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:02 smithi016 conmon[32727]: 2022-03-21T19:11:02.892+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:02 smithi016 ceph-mon[22010]: pgmap v3567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:11:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:11:03.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:03.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:03.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:03.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:11:03.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:03.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:11:03.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:03.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:11:03.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:03.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:03.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:03.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:11:03.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:03.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:03.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:03.984 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:03 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:11:03.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:11:03.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:03.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:11:03.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:03.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:03.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:03.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:11:03.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:03.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:11:03.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:03.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:11:03.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:03.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:03.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:03.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:11:03.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:03.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:03.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:03.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:11:03.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:11:03.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:11:03.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:11:03.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:11:03.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:11:03.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:11:03.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:11:03.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:11:03.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:11:03.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:11:03.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:11:03.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:11:04.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:11:04.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:11:04.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:11:04.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:11:04.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:11:04.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:11:04.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:11:04.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:11:04.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:11:04.007 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:11:04.008 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:11:04.008 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:11:04.009 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:11:04.009 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:11:04.010 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:11:04.011 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:11:04.011 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:11:04.012 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:04.013 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:03 smithi016 conmon[32727]: 2022-03-21T19:11:03.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:11:04.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:04.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:11:04.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:04.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:04.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:04.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:11:04.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:04.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:11:04.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:04.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:11:04.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:04.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:04.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:04.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:11:04.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:04.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:04.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:04.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:03 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:11:04.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:11:04.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:11:04.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:11:04.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:11:04.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:11:04.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:11:04.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:11:04.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:11:04.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:11:04.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:11:04.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:11:04.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:11:04.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:11:04.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:11:04.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:11:04.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:11:04.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:11:04.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:04.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:11:04.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:11:04.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:11:04.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:11:04.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:11:04.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:11:04.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:11:04.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:11:04.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:11:04.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:11:04.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:11:04.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:11:04.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:11:04.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:11:04.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:11:04.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:11:04.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:11:04.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:11:04.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:11:04.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:11:04.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:11:04.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:11:04.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:11:04.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:11:04.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:11:04.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:11:04.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:11:04.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:11:04.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:11:04.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:11:04.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:11:04.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:04 smithi079 ceph-mon[23158]: pgmap v3568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6506 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:04 smithi016 conmon[32727]: 2022-03-21T19:11:04.904+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:04 smithi016 ceph-mon[22010]: pgmap v3568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6506 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:05.938 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:05 smithi016 conmon[32727]: 2022-03-21T19:11:05.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:06 smithi079 ceph-mon[23158]: pgmap v3569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:06 smithi016 ceph-mon[22010]: pgmap v3569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:06 smithi016 conmon[32727]: 2022-03-21T19:11:06.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:07 smithi016 conmon[32727]: 2022-03-21T19:11:07.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:08 smithi079 ceph-mon[23158]: pgmap v3570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:08 smithi016 ceph-mon[22010]: pgmap v3570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:08 smithi016 conmon[32727]: 2022-03-21T19:11:08.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6511 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6511 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:09 smithi016 conmon[32727]: 2022-03-21T19:11:09.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:10 smithi079 ceph-mon[23158]: pgmap v3571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:10 smithi016 ceph-mon[22010]: pgmap v3571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:10 smithi016 conmon[32727]: 2022-03-21T19:11:10.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:11 smithi016 conmon[32727]: 2022-03-21T19:11:11.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:12 smithi079 ceph-mon[23158]: pgmap v3572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:12 smithi016 conmon[32727]: 2022-03-21T19:11:12.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:12 smithi016 ceph-mon[22010]: pgmap v3572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:13 smithi016 conmon[32727]: 2022-03-21T19:11:13.875+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:14 smithi079 ceph-mon[23158]: pgmap v3573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6516 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:15.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:14 smithi016 conmon[32727]: 2022-03-21T19:11:14.847+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:15.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:15.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:14 smithi016 ceph-mon[22010]: pgmap v3573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:15.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6516 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:15 smithi016 conmon[32727]: 2022-03-21T19:11:15.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:16 smithi079 ceph-mon[23158]: pgmap v3574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:16 smithi016 ceph-mon[22010]: pgmap v3574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:16 smithi016 conmon[32727]: 2022-03-21T19:11:16.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:17 smithi016 conmon[32727]: 2022-03-21T19:11:17.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:18 smithi079 ceph-mon[23158]: pgmap v3575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:18 smithi016 ceph-mon[22010]: pgmap v3575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:18 smithi016 conmon[32727]: 2022-03-21T19:11:18.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6521 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:19 smithi016 conmon[32727]: 2022-03-21T19:11:19.824+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6521 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:21.379 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:20 smithi016 ceph-mon[22010]: pgmap v3576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:21.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:20 smithi016 conmon[32727]: 2022-03-21T19:11:20.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:21.380 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:21.381 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:20 smithi079 ceph-mon[23158]: pgmap v3576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:21 smithi016 conmon[32727]: 2022-03-21T19:11:21.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:22 smithi016 ceph-mon[22010]: pgmap v3577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:22 smithi016 conmon[32727]: 2022-03-21T19:11:22.925+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:22 smithi079 ceph-mon[23158]: pgmap v3577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:23 smithi016 conmon[32727]: 2022-03-21T19:11:23.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:24 smithi016 ceph-mon[22010]: pgmap v3578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:24 smithi016 conmon[32727]: 2022-03-21T19:11:24.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:24 smithi079 ceph-mon[23158]: pgmap v3578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:26.001 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:26 smithi016 conmon[32727]: 2022-03-21T19:11:25.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:27.021 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:27.022 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:26 smithi016 ceph-mon[22010]: pgmap v3579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:26 smithi079 ceph-mon[23158]: pgmap v3579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:27 smithi016 conmon[32727]: 2022-03-21T19:11:27.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:28.016 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:28 smithi016 conmon[32727]: 2022-03-21T19:11:28.014+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:29.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:28 smithi016 ceph-mon[22010]: pgmap v3580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:29.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:28 smithi079 ceph-mon[23158]: pgmap v3580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:29 smithi016 conmon[32727]: 2022-03-21T19:11:29.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:30.026 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:30 smithi016 conmon[32727]: 2022-03-21T19:11:30.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:31.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:31.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:30 smithi016 ceph-mon[22010]: pgmap v3581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:30 smithi079 ceph-mon[23158]: pgmap v3581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:31 smithi016 conmon[32727]: 2022-03-21T19:11:31.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:32.034 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:32 smithi016 conmon[32727]: 2022-03-21T19:11:32.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:32 smithi016 ceph-mon[22010]: pgmap v3582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:33.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:33 smithi016 conmon[32727]: 2022-03-21T19:11:33.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:32 smithi079 ceph-mon[23158]: pgmap v3582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:34 smithi016 conmon[32727]: 2022-03-21T19:11:34.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:35.049 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:35.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:34 smithi016 ceph-mon[22010]: pgmap v3583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:35.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:34 smithi079 ceph-mon[23158]: pgmap v3583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:35 smithi016 conmon[32727]: 2022-03-21T19:11:35.048+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:36 smithi016 conmon[32727]: 2022-03-21T19:11:36.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:37 smithi016 ceph-mon[22010]: pgmap v3584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:36 smithi016 conmon[32727]: 2022-03-21T19:11:36.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:37 smithi079 ceph-mon[23158]: pgmap v3584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:37 smithi016 conmon[32727]: 2022-03-21T19:11:37.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:39 smithi016 ceph-mon[22010]: pgmap v3585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:38 smithi016 conmon[32727]: 2022-03-21T19:11:38.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:39 smithi079 ceph-mon[23158]: pgmap v3585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:40.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:39 smithi016 conmon[32727]: 2022-03-21T19:11:39.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:41 smithi016 ceph-mon[22010]: pgmap v3586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:40 smithi016 conmon[32727]: 2022-03-21T19:11:40.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:41 smithi079 ceph-mon[23158]: pgmap v3586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:41 smithi016 conmon[32727]: 2022-03-21T19:11:41.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:43 smithi016 ceph-mon[22010]: pgmap v3587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:43.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:42 smithi016 conmon[32727]: 2022-03-21T19:11:42.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:43 smithi079 ceph-mon[23158]: pgmap v3587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:43 smithi016 conmon[32727]: 2022-03-21T19:11:43.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:44 smithi016 conmon[32727]: 2022-03-21T19:11:44.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:45 smithi016 ceph-mon[22010]: pgmap v3588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:45 smithi079 ceph-mon[23158]: pgmap v3588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:45 smithi016 conmon[32727]: 2022-03-21T19:11:45.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:47 smithi016 ceph-mon[22010]: pgmap v3589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:47.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:46 smithi016 conmon[32727]: 2022-03-21T19:11:46.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:47 smithi079 ceph-mon[23158]: pgmap v3589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:47 smithi016 conmon[32727]: 2022-03-21T19:11:47.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:49 smithi016 ceph-mon[22010]: pgmap v3590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:48 smithi016 conmon[32727]: 2022-03-21T19:11:48.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:49 smithi079 ceph-mon[23158]: pgmap v3590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:50.410 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:49 smithi016 conmon[32727]: 2022-03-21T19:11:49.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:50 smithi016 ceph-mon[22010]: pgmap v3591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:50 smithi079 ceph-mon[23158]: pgmap v3591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:51.412 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:50 smithi016 conmon[32727]: 2022-03-21T19:11:50.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:51 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6556 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:51 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6556 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:51 smithi016 conmon[32727]: 2022-03-21T19:11:51.919+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:52 smithi079 ceph-mon[23158]: pgmap v3592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:52.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:52 smithi016 ceph-mon[22010]: pgmap v3592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:52 smithi016 conmon[32727]: 2022-03-21T19:11:52.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:53 smithi016 conmon[32727]: 2022-03-21T19:11:53.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:54 smithi079 ceph-mon[23158]: pgmap v3593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:54 smithi016 ceph-mon[22010]: pgmap v3593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:11:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:54 smithi016 conmon[32727]: 2022-03-21T19:11:54.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:55 smithi016 conmon[32727]: 2022-03-21T19:11:55.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:56 smithi079 ceph-mon[23158]: pgmap v3594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:56 smithi016 ceph-mon[22010]: pgmap v3594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:56 smithi016 conmon[32727]: 2022-03-21T19:11:56.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:57 smithi016 conmon[32727]: 2022-03-21T19:11:57.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: pgmap v3595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:11:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:11:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:11:58.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:11:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: pgmap v3595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:11:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:11:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:11:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:11:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:11:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:58 smithi016 conmon[32727]: 2022-03-21T19:11:58.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:11:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:11:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:11:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:11:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:11:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:11:59 smithi016 conmon[32727]: 2022-03-21T19:11:59.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:00 smithi079 ceph-mon[23158]: pgmap v3596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:00 smithi016 ceph-mon[22010]: pgmap v3596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:01 smithi016 conmon[32727]: 2022-03-21T19:12:01.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:02 smithi016 conmon[32727]: 2022-03-21T19:12:02.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:02 smithi079 ceph-mon[23158]: pgmap v3597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:12:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:12:02.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:12:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:02 smithi016 ceph-mon[22010]: pgmap v3597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:12:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:12:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:12:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[32727]: 2022-03-21T19:12:03.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: 2022-03-21T19:12:03.021+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:12:03.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:03.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: yield 2022-03-21T19:12:03.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:03.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:12:03.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:12:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:03.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:12:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:12:03.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:03.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:12:03.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:03.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:03.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:12:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:12:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:12:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:12:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:12:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:12:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:03 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:12:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:12:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:12:03.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:03.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:03.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:03.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:12:03.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:03.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:12:03.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:03.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:12:03.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:03.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:03.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:03.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:12:03.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:03.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:03.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:03.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:03 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:12:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:04 smithi016 conmon[32727]: 2022-03-21T19:12:04.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:04 smithi079 ceph-mon[23158]: pgmap v3598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:04 smithi016 ceph-mon[22010]: pgmap v3598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:05 smithi016 conmon[32727]: 2022-03-21T19:12:05.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:06.219 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:06.219 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:06.220 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:06.220 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.220 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.221 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.221 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:05 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[32727]: 2022-03-21T19:12:06.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:05 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:06.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 2022-03-21T19:12:06.093+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:12:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: yield 2022-03-21T19:12:06.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:06.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:06.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:06.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:12:06.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:12:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:12:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:12:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:06.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:12:06.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:06.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:06.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:06.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 2022-03-21T19:12:06.094+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:12:06.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:12:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: yield 2022-03-21T19:12:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:12:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:12:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:12:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:12:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:12:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 2022-03-21T19:12:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:12:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: 2022-03-21T19:12:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:12:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:12:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:12:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:12:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:12:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:12:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:12:06.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:12:06.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:12:06.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:12:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:12:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:12:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:12:06.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:12:06.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:12:06.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:12:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:12:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:12:06.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:12:06.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:12:06.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:12:06.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:12:06.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise 2022-03-21T19:12:06.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:12:06.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:12:06.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:12:06.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:12:06.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:12:06 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:12:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:12:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: pgmap v3599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:07.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:07.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:07.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:12:07.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:07.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:07.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:12:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:07.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:12:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:12:07.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:07.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:07.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:12:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:07.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:06 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:12:07.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:12:07.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:07.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:12:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:12:07.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:07.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:12:07.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:07.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:12:07.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:12:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:12:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:12:07.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:12:07.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:12:07.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:12:07.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:12:07.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:12:07.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:12:07.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:12:07.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:12:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:12:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:12:07.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:12:07.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:12:07.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:12:07.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:12:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:12:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:12:07.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:12:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:12:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:12:07.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:12:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:12:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:12:07.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:12:07.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:12:07.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:12:07.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:12:07.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:07 smithi016 conmon[32727]: 2022-03-21T19:12:07.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:07.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:12:07.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:12:07.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: pgmap v3599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:07.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:07.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:12:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:07.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:12:07.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:12:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:07.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:12:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:07.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:07.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:12:07.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:07.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:06 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:12:07.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:12:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:12:07.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:12:07.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:12:07.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:12:07.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:12:07.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:12:07.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:12:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:12:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:12:07.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:12:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:12:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:12:07.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:12:07.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:12:07.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:12:07.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:07.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:12:07.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:12:07.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:12:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:12:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:12:07.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:12:07.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:12:07.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:12:07.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:12:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:12:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:12:07.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:12:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:12:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:12:07.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:12:07.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:12:07.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:12:07.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:12:07.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:12:07.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:12:07.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:12:07.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:12:07.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:12:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:12:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:12:07.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:12:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:12:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:12:07.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:12:07.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:12:07.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:12:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:08 smithi016 conmon[32727]: 2022-03-21T19:12:08.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:08 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:12:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:08.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:08 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:12:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:09 smithi016 ceph-mon[22010]: pgmap v3600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:09 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T19:12:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:09 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:12:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:09 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:12:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:09.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:09 smithi016 conmon[32727]: 2022-03-21T19:12:09.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:09 smithi079 ceph-mon[23158]: pgmap v3600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:09 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T19:12:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:09 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:12:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:09 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:12:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:10 smithi016 conmon[32727]: 2022-03-21T19:12:10.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:11 smithi016 ceph-mon[22010]: pgmap v3601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:11 smithi016 conmon[32727]: 2022-03-21T19:12:11.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:11 smithi079 ceph-mon[23158]: pgmap v3601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:12 smithi016 conmon[32727]: 2022-03-21T19:12:12.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:13 smithi016 ceph-mon[22010]: pgmap v3602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:13.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:13 smithi016 conmon[32727]: 2022-03-21T19:12:13.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:13 smithi079 ceph-mon[23158]: pgmap v3602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:14.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:14 smithi016 conmon[32727]: 2022-03-21T19:12:14.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:15 smithi016 conmon[32727]: 2022-03-21T19:12:15.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:15 smithi016 ceph-mon[22010]: pgmap v3603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:15 smithi079 ceph-mon[23158]: pgmap v3603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:16 smithi016 conmon[32727]: 2022-03-21T19:12:16.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:17 smithi016 conmon[32727]: 2022-03-21T19:12:17.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:17 smithi016 ceph-mon[22010]: pgmap v3604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:17 smithi079 ceph-mon[23158]: pgmap v3604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:18 smithi016 conmon[32727]: 2022-03-21T19:12:18.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:18.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:19 smithi016 ceph-mon[22010]: pgmap v3605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:19.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:19 smithi016 conmon[32727]: 2022-03-21T19:12:19.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:19 smithi079 ceph-mon[23158]: pgmap v3605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:20 smithi016 ceph-mon[22010]: pgmap v3606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:20 smithi016 conmon[32727]: 2022-03-21T19:12:20.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:20 smithi079 ceph-mon[23158]: pgmap v3606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:21 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:21.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:21 smithi016 conmon[32727]: 2022-03-21T19:12:21.222+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:21 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:22 smithi016 ceph-mon[22010]: pgmap v3607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:22 smithi016 conmon[32727]: 2022-03-21T19:12:22.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:22 smithi079 ceph-mon[23158]: pgmap v3607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:23 smithi016 conmon[32727]: 2022-03-21T19:12:23.270+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:24 smithi016 conmon[32727]: 2022-03-21T19:12:24.318+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:24 smithi016 ceph-mon[22010]: pgmap v3608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:24 smithi079 ceph-mon[23158]: pgmap v3608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:25 smithi016 conmon[32727]: 2022-03-21T19:12:25.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:26 smithi016 ceph-mon[22010]: pgmap v3609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:26.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:26 smithi016 conmon[32727]: 2022-03-21T19:12:26.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:26 smithi079 ceph-mon[23158]: pgmap v3609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:27.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:27 smithi016 conmon[32727]: 2022-03-21T19:12:27.321+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:28 smithi016 ceph-mon[22010]: pgmap v3610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:28 smithi016 conmon[32727]: 2022-03-21T19:12:28.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:28 smithi079 ceph-mon[23158]: pgmap v3610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:29.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:29 smithi016 conmon[32727]: 2022-03-21T19:12:29.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:30 smithi016 ceph-mon[22010]: pgmap v3611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:30 smithi016 conmon[32727]: 2022-03-21T19:12:30.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:30 smithi079 ceph-mon[23158]: pgmap v3611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:31 smithi016 conmon[32727]: 2022-03-21T19:12:31.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:32 smithi016 ceph-mon[22010]: pgmap v3612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:32.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:32 smithi016 conmon[32727]: 2022-03-21T19:12:32.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:32 smithi079 ceph-mon[23158]: pgmap v3612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:33 smithi016 conmon[32727]: 2022-03-21T19:12:33.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:34 smithi016 ceph-mon[22010]: pgmap v3613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:34 smithi016 conmon[32727]: 2022-03-21T19:12:34.271+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:34 smithi079 ceph-mon[23158]: pgmap v3613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:35 smithi016 conmon[32727]: 2022-03-21T19:12:35.321+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:36 smithi016 ceph-mon[22010]: pgmap v3614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:36 smithi016 conmon[32727]: 2022-03-21T19:12:36.304+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:36 smithi079 ceph-mon[23158]: pgmap v3614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:37.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:37 smithi016 conmon[32727]: 2022-03-21T19:12:37.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:38 smithi016 ceph-mon[22010]: pgmap v3615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:38 smithi016 conmon[32727]: 2022-03-21T19:12:38.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:38 smithi079 ceph-mon[23158]: pgmap v3615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:39 smithi016 conmon[32727]: 2022-03-21T19:12:39.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:40 smithi016 ceph-mon[22010]: pgmap v3616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:40 smithi016 conmon[32727]: 2022-03-21T19:12:40.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:40 smithi079 ceph-mon[23158]: pgmap v3616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:41 smithi016 conmon[32727]: 2022-03-21T19:12:41.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:42 smithi016 conmon[32727]: 2022-03-21T19:12:42.277+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:42 smithi016 ceph-mon[22010]: pgmap v3617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:42 smithi079 ceph-mon[23158]: pgmap v3617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:43 smithi016 conmon[32727]: 2022-03-21T19:12:43.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:44 smithi016 ceph-mon[22010]: pgmap v3618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:44.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:44 smithi016 conmon[32727]: 2022-03-21T19:12:44.288+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:44 smithi079 ceph-mon[23158]: pgmap v3618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:45 smithi016 conmon[32727]: 2022-03-21T19:12:45.318+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:46 smithi016 ceph-mon[22010]: pgmap v3619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:46 smithi016 conmon[32727]: 2022-03-21T19:12:46.334+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:46 smithi079 ceph-mon[23158]: pgmap v3619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:47 smithi016 conmon[32727]: 2022-03-21T19:12:47.308+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:47.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:48 smithi016 conmon[32727]: 2022-03-21T19:12:48.280+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:48 smithi016 ceph-mon[22010]: pgmap v3620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:48 smithi079 ceph-mon[23158]: pgmap v3620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:49.506 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:49 smithi016 conmon[32727]: 2022-03-21T19:12:49.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:50.512 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:50 smithi016 conmon[32727]: 2022-03-21T19:12:50.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:50 smithi016 ceph-mon[22010]: pgmap v3621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:50 smithi079 ceph-mon[23158]: pgmap v3621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:51.513 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:51 smithi016 conmon[32727]: 2022-03-21T19:12:51.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:52 smithi016 conmon[32727]: 2022-03-21T19:12:52.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:52 smithi016 ceph-mon[22010]: pgmap v3622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:52 smithi079 ceph-mon[23158]: pgmap v3622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:53 smithi016 conmon[32727]: 2022-03-21T19:12:53.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:54 smithi016 conmon[32727]: 2022-03-21T19:12:54.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:54 smithi016 ceph-mon[22010]: pgmap v3623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:54 smithi079 ceph-mon[23158]: pgmap v3623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:12:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:55 smithi016 conmon[32727]: 2022-03-21T19:12:55.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:56 smithi016 conmon[32727]: 2022-03-21T19:12:56.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:56 smithi016 ceph-mon[22010]: pgmap v3624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:56 smithi079 ceph-mon[23158]: pgmap v3624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:57 smithi016 conmon[32727]: 2022-03-21T19:12:57.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:58 smithi016 conmon[32727]: 2022-03-21T19:12:58.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: pgmap v3625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:12:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:12:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:12:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:12:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: pgmap v3625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:12:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:12:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:12:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:12:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:12:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:12:59 smithi016 conmon[32727]: 2022-03-21T19:12:59.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:12:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:12:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:12:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:12:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:12:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:00 smithi016 conmon[32727]: 2022-03-21T19:13:00.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:00 smithi016 ceph-mon[22010]: pgmap v3626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:00 smithi079 ceph-mon[23158]: pgmap v3626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:01 smithi016 conmon[32727]: 2022-03-21T19:13:01.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:02 smithi016 conmon[32727]: 2022-03-21T19:13:02.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:02 smithi016 ceph-mon[22010]: pgmap v3627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:02 smithi079 ceph-mon[23158]: pgmap v3627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:03 smithi016 conmon[32727]: 2022-03-21T19:13:03.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:04 smithi016 conmon[32727]: 2022-03-21T19:13:04.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:04 smithi016 ceph-mon[22010]: pgmap v3628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:04 smithi079 ceph-mon[23158]: pgmap v3628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:05 smithi016 conmon[32727]: 2022-03-21T19:13:05.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:06 smithi016 conmon[32727]: 2022-03-21T19:13:06.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:06 smithi016 ceph-mon[22010]: pgmap v3629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:06 smithi079 ceph-mon[23158]: pgmap v3629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:07.563 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:07 smithi016 conmon[32727]: 2022-03-21T19:13:07.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:08 smithi016 conmon[32727]: 2022-03-21T19:13:08.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:08 smithi016 ceph-mon[22010]: pgmap v3630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:13:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:13:08.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:08 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:08 smithi079 ceph-mon[23158]: pgmap v3630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:13:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:13:08.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:08 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:13:09.568 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:09 smithi016 conmon[32727]: 2022-03-21T19:13:09.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:10.575 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:10 smithi016 conmon[32727]: 2022-03-21T19:13:10.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:10 smithi016 ceph-mon[22010]: pgmap v3631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:10 smithi079 ceph-mon[23158]: pgmap v3631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:11.576 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[32727]: 2022-03-21T19:13:11.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:11.577 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: 2022-03-21T19:13:11.181+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:13:11.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:11.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: yield 2022-03-21T19:13:11.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:11.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:11.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:11.580 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:13:11.580 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:11.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:13:11.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:11.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:13:11.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:11.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:13:11.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:11.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:13:11.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:11.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:11.584 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:13:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:13:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:13:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:13:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:13:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:13:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:11 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:13:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:13:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:13:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:13:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:13:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:13:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:11 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:13:12.581 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:12 smithi016 conmon[32727]: 2022-03-21T19:13:12.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:12 smithi016 ceph-mon[22010]: pgmap v3632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:12 smithi079 ceph-mon[23158]: pgmap v3632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:13.582 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:13 smithi016 conmon[32727]: 2022-03-21T19:13:13.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:14.588 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[32727]: 2022-03-21T19:13:14.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:14.590 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 2022-03-21T19:13:14.253+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:14.590 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:13:14.590 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:14.591 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: yield 2022-03-21T19:13:14.591 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:14.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:14.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:14.592 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:13:14.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:14.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:13:14.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:14.594 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:13:14.594 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:14.594 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:13:14.595 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:14.595 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:13:14.595 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:14.596 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:14.596 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:14.597 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 2022-03-21T19:13:14.254+0000 7f9271ede700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:13:14.597 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:13:14.597 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:14.598 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: yield 2022-03-21T19:13:14.598 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:14.598 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:14.599 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:14.599 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:13:14.599 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:14.600 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:13:14.600 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:14.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:13:14.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:14.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:13:14.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:14.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:13:14.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:14.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:14.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:14.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 2022-03-21T19:13:14.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:13:14.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: 2022-03-21T19:13:14.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:13:14.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:13:14.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:13:14.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:13:14.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:13:14.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:13:14.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:13:14.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:13:14.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:13:14.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:13:14.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:13:14.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:13:14.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:13:14.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:13:14.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:13:14.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:13:14.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:13:14.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:13:14.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:13:14.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:13:14.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:13:14.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:13:14.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise 2022-03-21T19:13:14.613 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:13:14.613 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:13:14.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:13:14.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:13:14.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:13:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:14 smithi016 ceph-mon[22010]: pgmap v3633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:14 smithi079 ceph-mon[23158]: pgmap v3633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:15.590 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:15 smithi016 conmon[32727]: 2022-03-21T19:13:15.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:13:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:13:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:13:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:13:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:13:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:15 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:13:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:13:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:13:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:13:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:13:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:13:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:13:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:13:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:13:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:13:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:13:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:13:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:13:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:13:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:13:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:13:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:13:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:13:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:13:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:13:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:13:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:13:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:13:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:13:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:13:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:13:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:13:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:13:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:13:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:13:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:13:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:13:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:13:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:13:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:13:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:13:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:13:15.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:13:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:15.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:13:15.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:15.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:13:15.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:15 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:13:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:13:15.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:13:15.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:13:15.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:13:15.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:13:15.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:13:15.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:13:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:13:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:13:15.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:13:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:13:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:13:15.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:13:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:13:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:13:15.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:13:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:13:15.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:13:15.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:13:15.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:13:15.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:13:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:13:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:13:15.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:13:15.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:13:15.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:13:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:13:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:13:15.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:13:15.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:13:15.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:13:15.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:13:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:13:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:13:15.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:13:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:13:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:13:15.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:13:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:13:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:13:15.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:13:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:13:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:13:15.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:13:15.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:13:15.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:13:15.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:13:15.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:16.595 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:16 smithi016 conmon[32727]: 2022-03-21T19:13:16.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:16 smithi016 ceph-mon[22010]: pgmap v3634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:16 smithi079 ceph-mon[23158]: pgmap v3634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:17 smithi016 conmon[32727]: 2022-03-21T19:13:17.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:18.601 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:18 smithi016 conmon[32727]: 2022-03-21T19:13:18.329+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:19.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:18 smithi016 ceph-mon[22010]: pgmap v3635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:19.050 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:19.051 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:18 smithi079 ceph-mon[23158]: pgmap v3635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:19.602 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:19 smithi016 conmon[32727]: 2022-03-21T19:13:19.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:20.608 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:20 smithi016 conmon[32727]: 2022-03-21T19:13:20.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:20 smithi016 ceph-mon[22010]: pgmap v3636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:20 smithi079 ceph-mon[23158]: pgmap v3636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:21.609 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:21 smithi016 conmon[32727]: 2022-03-21T19:13:21.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:22.614 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:22 smithi016 conmon[32727]: 2022-03-21T19:13:22.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:22 smithi016 ceph-mon[22010]: pgmap v3637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:22 smithi079 ceph-mon[23158]: pgmap v3637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:23.615 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:23 smithi016 conmon[32727]: 2022-03-21T19:13:23.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:24.622 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:24 smithi016 conmon[32727]: 2022-03-21T19:13:24.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:24 smithi016 ceph-mon[22010]: pgmap v3638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:24 smithi079 ceph-mon[23158]: pgmap v3638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:25.623 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:25 smithi016 conmon[32727]: 2022-03-21T19:13:25.294+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:26.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:26 smithi016 conmon[32727]: 2022-03-21T19:13:26.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:26 smithi016 ceph-mon[22010]: pgmap v3639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:26 smithi079 ceph-mon[23158]: pgmap v3639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:27.629 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:27 smithi016 conmon[32727]: 2022-03-21T19:13:27.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:28.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:28.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:28 smithi016 ceph-mon[22010]: pgmap v3640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:28.636 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:28 smithi016 conmon[32727]: 2022-03-21T19:13:28.382+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:28 smithi079 ceph-mon[23158]: pgmap v3640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:29 smithi016 conmon[32727]: 2022-03-21T19:13:29.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:30 smithi016 conmon[32727]: 2022-03-21T19:13:30.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:30 smithi016 ceph-mon[22010]: pgmap v3641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:30 smithi079 ceph-mon[23158]: pgmap v3641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:31 smithi016 conmon[32727]: 2022-03-21T19:13:31.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:32 smithi016 conmon[32727]: 2022-03-21T19:13:32.426+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:32 smithi016 ceph-mon[22010]: pgmap v3642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:32 smithi079 ceph-mon[23158]: pgmap v3642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:33 smithi016 conmon[32727]: 2022-03-21T19:13:33.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:34 smithi016 ceph-mon[22010]: pgmap v3643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:34 smithi016 conmon[32727]: 2022-03-21T19:13:34.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:34 smithi079 ceph-mon[23158]: pgmap v3643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:35 smithi016 conmon[32727]: 2022-03-21T19:13:35.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:36 smithi016 ceph-mon[22010]: pgmap v3644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:36 smithi016 conmon[32727]: 2022-03-21T19:13:36.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:36 smithi079 ceph-mon[23158]: pgmap v3644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:37 smithi016 conmon[32727]: 2022-03-21T19:13:37.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:38 smithi016 ceph-mon[22010]: pgmap v3645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:38.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:38 smithi016 conmon[32727]: 2022-03-21T19:13:38.497+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:38 smithi079 ceph-mon[23158]: pgmap v3645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:39 smithi016 conmon[32727]: 2022-03-21T19:13:39.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:40 smithi016 ceph-mon[22010]: pgmap v3646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:40 smithi016 conmon[32727]: 2022-03-21T19:13:40.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:40 smithi079 ceph-mon[23158]: pgmap v3646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:41 smithi016 conmon[32727]: 2022-03-21T19:13:41.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:42 smithi016 ceph-mon[22010]: pgmap v3647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:42 smithi016 conmon[32727]: 2022-03-21T19:13:42.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:42 smithi079 ceph-mon[23158]: pgmap v3647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:43 smithi016 conmon[32727]: 2022-03-21T19:13:43.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:44 smithi016 conmon[32727]: 2022-03-21T19:13:44.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:44 smithi016 ceph-mon[22010]: pgmap v3648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:44 smithi079 ceph-mon[23158]: pgmap v3648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:45 smithi016 conmon[32727]: 2022-03-21T19:13:45.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:46 smithi016 ceph-mon[22010]: pgmap v3649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:46 smithi016 conmon[32727]: 2022-03-21T19:13:46.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:46 smithi079 ceph-mon[23158]: pgmap v3649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:47 smithi016 conmon[32727]: 2022-03-21T19:13:47.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:48 smithi016 ceph-mon[22010]: pgmap v3650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:48 smithi016 conmon[32727]: 2022-03-21T19:13:48.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:48 smithi079 ceph-mon[23158]: pgmap v3650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:49 smithi016 conmon[32727]: 2022-03-21T19:13:49.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:50 smithi016 ceph-mon[22010]: pgmap v3651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:50 smithi016 conmon[32727]: 2022-03-21T19:13:50.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:50 smithi079 ceph-mon[23158]: pgmap v3651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:51.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:51 smithi016 conmon[32727]: 2022-03-21T19:13:51.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:52 smithi016 ceph-mon[22010]: pgmap v3652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:52.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:52 smithi016 conmon[32727]: 2022-03-21T19:13:52.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:52 smithi079 ceph-mon[23158]: pgmap v3652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:53 smithi016 conmon[32727]: 2022-03-21T19:13:53.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:54 smithi079 ceph-mon[23158]: pgmap v3653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:54 smithi016 conmon[32727]: 2022-03-21T19:13:54.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:54 smithi016 ceph-mon[22010]: pgmap v3653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:13:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:13:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:55.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:55 smithi016 conmon[32727]: 2022-03-21T19:13:55.659+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:56 smithi079 ceph-mon[23158]: pgmap v3654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:56 smithi016 ceph-mon[22010]: pgmap v3654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:56 smithi016 conmon[32727]: 2022-03-21T19:13:56.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:57 smithi016 conmon[32727]: 2022-03-21T19:13:57.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:58 smithi016 conmon[32727]: 2022-03-21T19:13:58.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: pgmap v3655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:13:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:13:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:13:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:13:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:13:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: pgmap v3655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:13:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:13:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:14:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:13:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:13:59 smithi016 conmon[32727]: 2022-03-21T19:13:59.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:13:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:00 smithi016 ceph-mon[22010]: pgmap v3656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:00 smithi016 conmon[32727]: 2022-03-21T19:14:00.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:00 smithi079 ceph-mon[23158]: pgmap v3656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:01 smithi016 conmon[32727]: 2022-03-21T19:14:01.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:02 smithi016 ceph-mon[22010]: pgmap v3657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:02 smithi016 conmon[32727]: 2022-03-21T19:14:02.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:02 smithi079 ceph-mon[23158]: pgmap v3657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:03 smithi016 conmon[32727]: 2022-03-21T19:14:03.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:04 smithi016 ceph-mon[22010]: pgmap v3658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:04 smithi016 conmon[32727]: 2022-03-21T19:14:04.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:04 smithi079 ceph-mon[23158]: pgmap v3658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:05 smithi016 conmon[32727]: 2022-03-21T19:14:05.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:06 smithi016 ceph-mon[22010]: pgmap v3659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:06 smithi016 conmon[32727]: 2022-03-21T19:14:06.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:06 smithi079 ceph-mon[23158]: pgmap v3659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:07 smithi016 conmon[32727]: 2022-03-21T19:14:07.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:08 smithi016 ceph-mon[22010]: pgmap v3660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:08 smithi016 conmon[32727]: 2022-03-21T19:14:08.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:08 smithi079 ceph-mon[23158]: pgmap v3660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:09 smithi016 conmon[32727]: 2022-03-21T19:14:09.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:10.325 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:10 smithi016 ceph-mon[22010]: pgmap v3661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:10 smithi016 conmon[32727]: 2022-03-21T19:14:10.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:10 smithi079 ceph-mon[23158]: pgmap v3661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:11.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:11 smithi016 conmon[32727]: 2022-03-21T19:14:11.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:12 smithi016 ceph-mon[22010]: pgmap v3662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:12 smithi016 conmon[32727]: 2022-03-21T19:14:12.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:12 smithi079 ceph-mon[23158]: pgmap v3662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:13 smithi016 conmon[32727]: 2022-03-21T19:14:13.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:14 smithi016 conmon[32727]: 2022-03-21T19:14:14.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: pgmap v3663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:14:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:14:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:14:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: pgmap v3663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:14:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:14:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:14:15.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:15 smithi016 conmon[32727]: 2022-03-21T19:14:15.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:16 smithi016 ceph-mon[22010]: pgmap v3664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:16.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:16 smithi016 conmon[32727]: 2022-03-21T19:14:16.659+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:16 smithi079 ceph-mon[23158]: pgmap v3664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:17.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: 2022-03-21T19:14:17.357+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:17.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:14:17.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:17.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: yield 2022-03-21T19:14:17.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:17.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:17.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:17.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:14:17.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:17.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:14:17.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:17.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:14:17.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:17.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:14:17.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:17.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:14:17.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:17.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:17.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:17 smithi016 conmon[32727]: 2022-03-21T19:14:17.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:18 smithi016 conmon[32727]: 2022-03-21T19:14:18.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:14:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:14:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:14:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:14:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:14:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:14:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:18.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:18.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:18.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:14:18.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:18.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:18 smithi016 ceph-mon[22010]: pgmap v3665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:14:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:14:19.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:19.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:14:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:14:19.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:14:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:19.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:14:19.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:19.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:14:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:19.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:18 smithi079 ceph-mon[23158]: pgmap v3665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:19 smithi016 conmon[32727]: 2022-03-21T19:14:19.614+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:20.803 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[32727]: 2022-03-21T19:14:20.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:20.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 2022-03-21T19:14:20.429+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:20.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:14:20.805 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:20.805 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: yield 2022-03-21T19:14:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:20.806 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:14:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:20.807 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:14:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:14:20.808 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:20.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:14:20.809 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:14:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:20.810 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:20.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:20.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 2022-03-21T19:14:20.430+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:14:20.811 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:14:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: yield 2022-03-21T19:14:20.812 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:20.813 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:14:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:14:20.814 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:14:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:20.815 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:14:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:14:20.816 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:20.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:20.817 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:20.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 2022-03-21T19:14:20.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:14:20.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: 2022-03-21T19:14:20.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:14:20.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:14:20.819 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:14:20.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:14:20.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:14:20.820 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:14:20.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:14:20.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:14:20.821 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:14:20.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:14:20.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:14:20.822 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:14:20.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:14:20.823 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:14:20.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:14:20.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:14:20.824 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:14:20.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:14:20.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:14:20.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:14:20.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:14:20.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:14:20.826 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise 2022-03-21T19:14:20.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:14:20.827 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:14:20.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:14:20.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:14:20.828 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:14:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:20 smithi016 ceph-mon[22010]: pgmap v3666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:20 smithi079 ceph-mon[23158]: pgmap v3666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:21 smithi016 conmon[32727]: 2022-03-21T19:14:21.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:14:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:14:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:14:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:14:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:14:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:14:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:14:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:14:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:14:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:14:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:14:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:14:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:14:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:14:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:14:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:14:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:14:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:14:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:14:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:14:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:14:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:14:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:14:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:14:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:14:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:14:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:14:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:14:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:14:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:14:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:14:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:14:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:14:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:14:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:14:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:14:21.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:14:21.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:14:21.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:14:21.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:14:21.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:14:21.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:14:21.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:14:21.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:21.946 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:14:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:14:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:14:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:14:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:14:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:14:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:14:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:14:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:14:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:14:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:14:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:14:22.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:14:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:14:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:14:22.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:14:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:14:22.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:14:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:14:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:14:22.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:14:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:14:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:14:22.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:14:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:14:22.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:14:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:14:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:14:22.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:14:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:14:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:14:22.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:14:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:14:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:14:22.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:14:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:14:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:14:22.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:14:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:14:22.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:14:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:14:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:14:22.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:14:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:14:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:14:22.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:14:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:14:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:14:22.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:14:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:14:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:14:22.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:14:22.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:14:22.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:14:22.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:14:22.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:22 smithi016 conmon[32727]: 2022-03-21T19:14:22.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:22 smithi016 ceph-mon[22010]: pgmap v3667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:22 smithi079 ceph-mon[23158]: pgmap v3667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:23 smithi016 conmon[32727]: 2022-03-21T19:14:23.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:24 smithi016 conmon[32727]: 2022-03-21T19:14:24.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:24 smithi016 ceph-mon[22010]: pgmap v3668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:24 smithi079 ceph-mon[23158]: pgmap v3668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:25 smithi016 conmon[32727]: 2022-03-21T19:14:25.706+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:26 smithi016 conmon[32727]: 2022-03-21T19:14:26.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:26 smithi016 ceph-mon[22010]: pgmap v3669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:26 smithi079 ceph-mon[23158]: pgmap v3669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:27 smithi016 conmon[32727]: 2022-03-21T19:14:27.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:28 smithi016 conmon[32727]: 2022-03-21T19:14:28.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:28 smithi016 ceph-mon[22010]: pgmap v3670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:28 smithi079 ceph-mon[23158]: pgmap v3670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:29 smithi016 conmon[32727]: 2022-03-21T19:14:29.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:30 smithi016 conmon[32727]: 2022-03-21T19:14:30.664+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:31.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:30 smithi016 ceph-mon[22010]: pgmap v3671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:30 smithi079 ceph-mon[23158]: pgmap v3671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:31 smithi016 conmon[32727]: 2022-03-21T19:14:31.712+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:32 smithi016 ceph-mon[22010]: pgmap v3672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:32 smithi016 conmon[32727]: 2022-03-21T19:14:32.727+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:32 smithi079 ceph-mon[23158]: pgmap v3672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:33 smithi016 conmon[32727]: 2022-03-21T19:14:33.726+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:34 smithi016 ceph-mon[22010]: pgmap v3673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:34 smithi016 conmon[32727]: 2022-03-21T19:14:34.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:34 smithi079 ceph-mon[23158]: pgmap v3673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:35 smithi016 conmon[32727]: 2022-03-21T19:14:35.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:36 smithi016 ceph-mon[22010]: pgmap v3674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:36 smithi016 conmon[32727]: 2022-03-21T19:14:36.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:36 smithi079 ceph-mon[23158]: pgmap v3674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:37 smithi016 conmon[32727]: 2022-03-21T19:14:37.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:38 smithi016 ceph-mon[22010]: pgmap v3675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:38 smithi016 conmon[32727]: 2022-03-21T19:14:38.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:38 smithi079 ceph-mon[23158]: pgmap v3675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:39 smithi016 conmon[32727]: 2022-03-21T19:14:39.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:40 smithi016 conmon[32727]: 2022-03-21T19:14:40.828+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:40 smithi016 ceph-mon[22010]: pgmap v3676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:40 smithi079 ceph-mon[23158]: pgmap v3676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:41 smithi016 conmon[32727]: 2022-03-21T19:14:41.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:42 smithi016 conmon[32727]: 2022-03-21T19:14:42.756+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:43.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:42 smithi016 ceph-mon[22010]: pgmap v3677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:42 smithi079 ceph-mon[23158]: pgmap v3677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:43 smithi016 conmon[32727]: 2022-03-21T19:14:43.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:44 smithi016 ceph-mon[22010]: pgmap v3678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:44 smithi016 conmon[32727]: 2022-03-21T19:14:44.739+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:44 smithi079 ceph-mon[23158]: pgmap v3678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:45 smithi016 conmon[32727]: 2022-03-21T19:14:45.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:46 smithi016 ceph-mon[22010]: pgmap v3679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:46 smithi016 conmon[32727]: 2022-03-21T19:14:46.748+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:46 smithi079 ceph-mon[23158]: pgmap v3679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:47 smithi016 conmon[32727]: 2022-03-21T19:14:47.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:48 smithi016 conmon[32727]: 2022-03-21T19:14:48.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:48 smithi016 ceph-mon[22010]: pgmap v3680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:48 smithi079 ceph-mon[23158]: pgmap v3680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:49 smithi016 conmon[32727]: 2022-03-21T19:14:49.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:50 smithi016 ceph-mon[22010]: pgmap v3681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:50 smithi016 conmon[32727]: 2022-03-21T19:14:50.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:50 smithi079 ceph-mon[23158]: pgmap v3681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:51 smithi016 conmon[32727]: 2022-03-21T19:14:51.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:52 smithi016 ceph-mon[22010]: pgmap v3682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:52 smithi016 conmon[32727]: 2022-03-21T19:14:52.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:52 smithi079 ceph-mon[23158]: pgmap v3682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:53 smithi016 conmon[32727]: 2022-03-21T19:14:53.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:54 smithi016 conmon[32727]: 2022-03-21T19:14:54.726+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:54 smithi016 ceph-mon[22010]: pgmap v3683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:54 smithi079 ceph-mon[23158]: pgmap v3683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:14:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:14:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:55 smithi016 conmon[32727]: 2022-03-21T19:14:55.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:56 smithi016 ceph-mon[22010]: pgmap v3684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:57.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:56 smithi016 conmon[32727]: 2022-03-21T19:14:56.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:56 smithi079 ceph-mon[23158]: pgmap v3684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:58.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:57 smithi016 conmon[32727]: 2022-03-21T19:14:57.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:58 smithi016 conmon[32727]: 2022-03-21T19:14:58.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: pgmap v3685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:14:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: pgmap v3685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:15:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:14:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:14:59 smithi016 conmon[32727]: 2022-03-21T19:14:59.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:14:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:00 smithi016 ceph-mon[22010]: pgmap v3686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:00 smithi016 conmon[32727]: 2022-03-21T19:15:00.748+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:00 smithi079 ceph-mon[23158]: pgmap v3686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:01 smithi016 conmon[32727]: 2022-03-21T19:15:01.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:02 smithi016 ceph-mon[22010]: pgmap v3687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:02 smithi016 conmon[32727]: 2022-03-21T19:15:02.766+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:02 smithi079 ceph-mon[23158]: pgmap v3687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:03 smithi016 conmon[32727]: 2022-03-21T19:15:03.777+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:04 smithi016 conmon[32727]: 2022-03-21T19:15:04.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:04 smithi016 ceph-mon[22010]: pgmap v3688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:04 smithi079 ceph-mon[23158]: pgmap v3688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:05 smithi016 conmon[32727]: 2022-03-21T19:15:05.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:06 smithi016 ceph-mon[22010]: pgmap v3689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:06 smithi016 conmon[32727]: 2022-03-21T19:15:06.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:06 smithi079 ceph-mon[23158]: pgmap v3689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:08.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:07 smithi016 conmon[32727]: 2022-03-21T19:15:07.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:08 smithi016 ceph-mon[22010]: pgmap v3690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:09.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:08 smithi016 conmon[32727]: 2022-03-21T19:15:08.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:08 smithi079 ceph-mon[23158]: pgmap v3690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:09 smithi016 conmon[32727]: 2022-03-21T19:15:09.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:10 smithi016 ceph-mon[22010]: pgmap v3691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:10 smithi016 conmon[32727]: 2022-03-21T19:15:10.827+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:10 smithi079 ceph-mon[23158]: pgmap v3691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:11 smithi016 conmon[32727]: 2022-03-21T19:15:11.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:12 smithi016 ceph-mon[22010]: pgmap v3692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:12 smithi016 conmon[32727]: 2022-03-21T19:15:12.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:12 smithi079 ceph-mon[23158]: pgmap v3692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:13 smithi016 conmon[32727]: 2022-03-21T19:15:13.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:14 smithi016 ceph-mon[22010]: pgmap v3693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:14 smithi016 conmon[32727]: 2022-03-21T19:15:14.849+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:14 smithi079 ceph-mon[23158]: pgmap v3693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:15.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:15 smithi016 conmon[32727]: 2022-03-21T19:15:15.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:16 smithi016 ceph-mon[22010]: pgmap v3694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:17.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:16 smithi016 conmon[32727]: 2022-03-21T19:15:16.808+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:16 smithi079 ceph-mon[23158]: pgmap v3694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:17 smithi016 conmon[32727]: 2022-03-21T19:15:17.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:18 smithi016 ceph-mon[22010]: pgmap v3695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:19.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:18 smithi016 conmon[32727]: 2022-03-21T19:15:18.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:19.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:18 smithi079 ceph-mon[23158]: pgmap v3695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6759 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:19 smithi016 conmon[32727]: 2022-03-21T19:15:19.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6759 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:20 smithi016 conmon[32727]: 2022-03-21T19:15:20.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: pgmap v3696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:15:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:15:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:15:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:20 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: pgmap v3696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:15:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:15:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:15:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:20 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:21 smithi016 conmon[32727]: 2022-03-21T19:15:21.747+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:21 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:21 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:21 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:21 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:15:23.023 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:22 smithi016 conmon[32727]: 2022-03-21T19:15:22.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:23.024 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:23.024 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:23 smithi016 ceph-mon[22010]: pgmap v3697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:23.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: 2022-03-21T19:15:23.021+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:15:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: yield 2022-03-21T19:15:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:15:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:15:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:15:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:15:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:15:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:23 smithi079 ceph-mon[23158]: pgmap v3697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:23 smithi016 conmon[32727]: 2022-03-21T19:15:23.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:15:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:15:24.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:24.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:24.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:24.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:15:24.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:24.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:15:24.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:24.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:15:24.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:24.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:24.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:24.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:15:24.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:24.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:24.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:24.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:24 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:15:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:15:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:15:24.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:15:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:15:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:15:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:15:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:24 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:15:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:25 smithi016 ceph-mon[22010]: pgmap v3698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:24 smithi016 conmon[32727]: 2022-03-21T19:15:24.804+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:25 smithi079 ceph-mon[23158]: pgmap v3698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:26.095 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:25 smithi016 conmon[32727]: 2022-03-21T19:15:25.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:26.096 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:26.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 2022-03-21T19:15:26.093+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:26.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:15:26.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:26.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: yield 2022-03-21T19:15:26.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:26.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:26.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:26.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:15:26.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:26.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:15:26.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:26.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:15:26.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:26.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:15:26.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:26.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:15:26.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:26.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:26.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:26.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 2022-03-21T19:15:26.094+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:15:26.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:15:26.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:26.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: yield 2022-03-21T19:15:26.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:26.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:26.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:26.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:15:26.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:26.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:15:26.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:26.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:15:26.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:26.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:15:26.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:26.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:15:26.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:26.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:26.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:26.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 2022-03-21T19:15:26.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:15:26.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: 2022-03-21T19:15:26.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:15:26.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:15:26.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:15:26.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:15:26.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:15:26.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:15:26.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:15:26.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:15:26.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:15:26.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:15:26.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:15:26.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:15:26.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:15:26.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:15:26.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:15:26.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:15:26.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:15:26.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:15:26.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:15:26.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:15:26.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:15:26.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:15:26.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise 2022-03-21T19:15:26.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:15:26.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:15:26.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:15:26.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:15:26.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:26 smithi016 conmon[32727]: 2022-03-21T19:15:26.797+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:27 smithi016 ceph-mon[22010]: pgmap v3699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:27 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:15:27.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:27.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:15:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:15:27.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:27.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:15:27.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:27.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:27.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:27.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:15:27.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:27.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:27.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:27 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:15:27.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:27.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:15:27.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:27.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:27.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:27.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:15:27.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:27.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:15:27.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:27.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:15:27.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:27.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:27.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:27.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:15:27.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:27.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:27.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:15:27.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:15:27.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:15:27.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:15:27.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:15:27.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:15:27.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:15:27.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:15:27.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:15:27.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:15:27.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:15:27.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:15:27.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:15:27.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:15:27.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:15:27.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:15:27.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:15:27.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:15:27.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:15:27.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:15:27.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:15:27.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:15:27.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:15:27.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:15:27.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:15:27.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:15:27.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:15:27.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:15:27.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:15:27.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:27.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:27 smithi079 ceph-mon[23158]: pgmap v3699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:27.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:27 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:27.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:15:27.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:27.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:27.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:27.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:27.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:15:27.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:27.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:27.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:27.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:15:27.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:27.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:27.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:27 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:15:27.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:15:27.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:15:27.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:15:27.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:15:27.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:15:27.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:15:27.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:15:27.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:15:27.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:15:27.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:15:27.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:15:27.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:15:27.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:15:27.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:15:27.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:15:27.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:15:27.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:27.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:15:27.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:15:27.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:15:27.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:15:27.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:15:27.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:15:27.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:15:27.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:15:27.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:15:27.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:15:27.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:15:27.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:15:27.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:15:27.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:15:27.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:15:27.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:15:27.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:15:27.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:15:27.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:15:27.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:15:27.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:15:27.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:15:27.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:15:27.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:15:27.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:15:27.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:15:27.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:15:27.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:15:27.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:15:27.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:15:27.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:15:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:27 smithi016 conmon[32727]: 2022-03-21T19:15:27.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:29 smithi016 ceph-mon[22010]: pgmap v3700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:29.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:28 smithi016 conmon[32727]: 2022-03-21T19:15:28.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:29 smithi079 ceph-mon[23158]: pgmap v3700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:29 smithi016 conmon[32727]: 2022-03-21T19:15:29.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:31 smithi016 ceph-mon[22010]: pgmap v3701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:31.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:30 smithi016 conmon[32727]: 2022-03-21T19:15:30.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:31 smithi079 ceph-mon[23158]: pgmap v3701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:31 smithi016 conmon[32727]: 2022-03-21T19:15:31.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:33 smithi016 ceph-mon[22010]: pgmap v3702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:32 smithi016 conmon[32727]: 2022-03-21T19:15:32.863+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:33 smithi079 ceph-mon[23158]: pgmap v3702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:33 smithi016 conmon[32727]: 2022-03-21T19:15:33.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:35 smithi016 ceph-mon[22010]: pgmap v3703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6774 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:34 smithi016 conmon[32727]: 2022-03-21T19:15:34.849+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:35 smithi079 ceph-mon[23158]: pgmap v3703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6774 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:35 smithi016 conmon[32727]: 2022-03-21T19:15:35.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:37.051 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:36 smithi016 conmon[32727]: 2022-03-21T19:15:36.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:37 smithi016 ceph-mon[22010]: pgmap v3704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:37 smithi079 ceph-mon[23158]: pgmap v3704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:37 smithi016 conmon[32727]: 2022-03-21T19:15:37.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:39 smithi016 ceph-mon[22010]: pgmap v3705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:38 smithi016 conmon[32727]: 2022-03-21T19:15:38.834+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:39 smithi079 ceph-mon[23158]: pgmap v3705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:39 smithi016 conmon[32727]: 2022-03-21T19:15:39.863+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:41 smithi016 ceph-mon[22010]: pgmap v3706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:40 smithi016 conmon[32727]: 2022-03-21T19:15:40.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:41 smithi079 ceph-mon[23158]: pgmap v3706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:42.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:41 smithi016 conmon[32727]: 2022-03-21T19:15:41.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:43 smithi016 ceph-mon[22010]: pgmap v3707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:42 smithi016 conmon[32727]: 2022-03-21T19:15:42.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:43 smithi079 ceph-mon[23158]: pgmap v3707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:44.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:43 smithi016 conmon[32727]: 2022-03-21T19:15:43.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:44 smithi016 conmon[32727]: 2022-03-21T19:15:44.830+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:45 smithi016 ceph-mon[22010]: pgmap v3708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6784 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:45 smithi079 ceph-mon[23158]: pgmap v3708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6784 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:46.079 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:45 smithi016 conmon[32727]: 2022-03-21T19:15:45.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:47 smithi016 ceph-mon[22010]: pgmap v3709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:46 smithi016 conmon[32727]: 2022-03-21T19:15:46.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:47 smithi079 ceph-mon[23158]: pgmap v3709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:47 smithi016 conmon[32727]: 2022-03-21T19:15:47.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:49 smithi016 ceph-mon[22010]: pgmap v3710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:48 smithi016 conmon[32727]: 2022-03-21T19:15:48.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:49 smithi079 ceph-mon[23158]: pgmap v3710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6789 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:49 smithi016 conmon[32727]: 2022-03-21T19:15:49.919+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6789 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:51 smithi016 ceph-mon[22010]: pgmap v3711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:50 smithi016 conmon[32727]: 2022-03-21T19:15:50.903+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:51 smithi079 ceph-mon[23158]: pgmap v3711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:51 smithi016 conmon[32727]: 2022-03-21T19:15:51.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:53 smithi016 ceph-mon[22010]: pgmap v3712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:52 smithi016 conmon[32727]: 2022-03-21T19:15:52.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:53 smithi079 ceph-mon[23158]: pgmap v3712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:53 smithi016 conmon[32727]: 2022-03-21T19:15:53.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:55 smithi016 ceph-mon[22010]: pgmap v3713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:55.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:55.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:54 smithi016 conmon[32727]: 2022-03-21T19:15:54.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:55 smithi079 ceph-mon[23158]: pgmap v3713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:15:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:15:55.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:56 smithi016 conmon[32727]: 2022-03-21T19:15:56.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:57 smithi016 ceph-mon[22010]: pgmap v3714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:57.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:56 smithi016 conmon[32727]: 2022-03-21T19:15:56.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:57 smithi079 ceph-mon[23158]: pgmap v3714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:57 smithi016 conmon[32727]: 2022-03-21T19:15:57.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:58 smithi016 conmon[32727]: 2022-03-21T19:15:58.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:15:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: pgmap v3715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:15:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:15:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:15:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:15:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: pgmap v3715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:15:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:15:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:15:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:15:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:15:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:15:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:15:59 smithi016 conmon[32727]: 2022-03-21T19:15:59.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:00 smithi079 ceph-mon[23158]: pgmap v3716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:00 smithi016 ceph-mon[22010]: pgmap v3716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:00 smithi016 conmon[32727]: 2022-03-21T19:16:00.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:01 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:01 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:01 smithi016 conmon[32727]: 2022-03-21T19:16:01.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:02 smithi016 ceph-mon[22010]: pgmap v3717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:02 smithi079 ceph-mon[23158]: pgmap v3717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:02 smithi016 conmon[32727]: 2022-03-21T19:16:02.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:03 smithi016 conmon[32727]: 2022-03-21T19:16:03.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:04 smithi016 ceph-mon[22010]: pgmap v3718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:04 smithi079 ceph-mon[23158]: pgmap v3718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:05 smithi016 conmon[32727]: 2022-03-21T19:16:05.010+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:05 smithi016 conmon[32727]: 2022-03-21T19:16:05.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:06 smithi016 ceph-mon[22010]: pgmap v3719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:06 smithi079 ceph-mon[23158]: pgmap v3719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:06 smithi016 conmon[32727]: 2022-03-21T19:16:06.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:07 smithi016 conmon[32727]: 2022-03-21T19:16:07.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:08 smithi016 ceph-mon[22010]: pgmap v3720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:08 smithi079 ceph-mon[23158]: pgmap v3720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:08 smithi016 conmon[32727]: 2022-03-21T19:16:08.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:09 smithi016 conmon[32727]: 2022-03-21T19:16:09.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:10 smithi016 ceph-mon[22010]: pgmap v3721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:10 smithi079 ceph-mon[23158]: pgmap v3721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:10 smithi016 conmon[32727]: 2022-03-21T19:16:10.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:11.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:11 smithi016 conmon[32727]: 2022-03-21T19:16:11.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:12.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:13.269 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:12 smithi016 ceph-mon[22010]: pgmap v3722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:13.270 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:13.270 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:12 smithi079 ceph-mon[23158]: pgmap v3722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:13.271 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:12 smithi016 conmon[32727]: 2022-03-21T19:16:12.906+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:13 smithi016 conmon[32727]: 2022-03-21T19:16:13.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:14 smithi016 ceph-mon[22010]: pgmap v3723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:14 smithi079 ceph-mon[23158]: pgmap v3723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:14 smithi016 conmon[32727]: 2022-03-21T19:16:14.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:15 smithi016 conmon[32727]: 2022-03-21T19:16:15.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:16 smithi016 ceph-mon[22010]: pgmap v3724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:16 smithi079 ceph-mon[23158]: pgmap v3724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:16 smithi016 conmon[32727]: 2022-03-21T19:16:16.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:17 smithi016 conmon[32727]: 2022-03-21T19:16:17.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:18 smithi016 ceph-mon[22010]: pgmap v3725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:18 smithi079 ceph-mon[23158]: pgmap v3725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:19 smithi016 conmon[32727]: 2022-03-21T19:16:19.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:20 smithi016 conmon[32727]: 2022-03-21T19:16:20.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:20 smithi016 ceph-mon[22010]: pgmap v3726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:20 smithi079 ceph-mon[23158]: pgmap v3726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:21 smithi016 conmon[32727]: 2022-03-21T19:16:21.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:22 smithi016 conmon[32727]: 2022-03-21T19:16:22.076+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:22 smithi016 ceph-mon[22010]: pgmap v3727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:22 smithi079 ceph-mon[23158]: pgmap v3727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:23 smithi016 conmon[32727]: 2022-03-21T19:16:23.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:24 smithi016 conmon[32727]: 2022-03-21T19:16:24.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:25 smithi016 ceph-mon[22010]: pgmap v3728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:25 smithi016 conmon[32727]: 2022-03-21T19:16:24.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:25 smithi079 ceph-mon[23158]: pgmap v3728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:25 smithi016 conmon[32727]: 2022-03-21T19:16:25.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:16:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:16:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:26 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:16:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:16:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:16:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:26 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:16:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:26 smithi016 conmon[32727]: 2022-03-21T19:16:26.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:27 smithi016 ceph-mon[22010]: pgmap v3729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:27 smithi079 ceph-mon[23158]: pgmap v3729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:28 smithi016 conmon[32727]: 2022-03-21T19:16:28.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:28.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:29 smithi016 ceph-mon[22010]: pgmap v3730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[32727]: 2022-03-21T19:16:29.006+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:29.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: 2022-03-21T19:16:29.197+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:29.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:16:29.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: yield 2022-03-21T19:16:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:29.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:29.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:29.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:16:29.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:29.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:16:29.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:16:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:29.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:16:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:16:29.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:29.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:29.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:29 smithi079 ceph-mon[23158]: pgmap v3730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:30 smithi016 conmon[32727]: 2022-03-21T19:16:30.039+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:16:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:30.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:16:30.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:30.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:30.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:30.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:16:30.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:30.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:16:30.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:30.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:16:30.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:30.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:30.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:30.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:16:30.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:30.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:30.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:30.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:30 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:16:30.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:16:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:16:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:30.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:30.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:16:30.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:30.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:16:30.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:30.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:16:30.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:30.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:30.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:30.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:16:30.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:30.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:30.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:30.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:30 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:16:30.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:31 smithi016 ceph-mon[22010]: pgmap v3731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:31 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:31.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:31 smithi016 conmon[32727]: 2022-03-21T19:16:31.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:31 smithi079 ceph-mon[23158]: pgmap v3731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:31 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[32727]: 2022-03-21T19:16:32.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:32.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 2022-03-21T19:16:32.269+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:32.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:16:32.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:32.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: yield 2022-03-21T19:16:32.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:32.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:32.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:32.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:16:32.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:32.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:16:32.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:32.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:16:32.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:32.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:16:32.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:32.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:16:32.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:32.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:32.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:32.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 2022-03-21T19:16:32.270+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:16:32.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:16:32.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:32.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: yield 2022-03-21T19:16:32.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:32.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:32.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:32.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:16:32.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:32.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:16:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:16:32.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:16:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:32.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:16:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:32.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:32.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 2022-03-21T19:16:32.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:16:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: 2022-03-21T19:16:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:16:32.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:16:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:16:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:16:32.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:16:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:16:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:16:32.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:16:32.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:16:32.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:16:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:16:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:16:32.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:16:32.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:16:32.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:16:32.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:16:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:16:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:16:32.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:16:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:16:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:16:32.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:16:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise 2022-03-21T19:16:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:16:32.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:16:32.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:16:32.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:16:32.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:33 smithi016 ceph-mon[22010]: pgmap v3732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:32 smithi016 conmon[32727]: 2022-03-21T19:16:32.996+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:33 smithi079 ceph-mon[23158]: pgmap v3732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:34 smithi016 conmon[32727]: 2022-03-21T19:16:34.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:34 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:16:34.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:34.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:16:34.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:34.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:16:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:16:34.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:34.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:16:34.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:34.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:34 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:16:34.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:34.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:16:34.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:34.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:34.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:34.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:16:34.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:34.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:16:34.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:34.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:16:34.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:34.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:34.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:34.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:16:34.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:34.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:34.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:16:34.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:16:34.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:16:34.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:16:34.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:16:34.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:16:34.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:16:34.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:16:34.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:16:34.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:16:34.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:16:34.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:16:34.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:16:34.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:16:34.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:16:34.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:16:34.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:16:34.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:16:34.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:16:34.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:16:34.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:16:34.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:16:34.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:16:34.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:16:34.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:16:34.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:16:34.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:16:34.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:16:34.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:16:34.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:34 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:34.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:16:34.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:34.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:34.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:34.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:16:34.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:34.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:16:34.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:34.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:16:34.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:34.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:34.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:34.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:16:34.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:34.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:34.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:34 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:16:34.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:16:34.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:16:34.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:16:34.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:16:34.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:16:34.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:16:34.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:16:34.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:16:34.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:16:34.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:16:34.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:16:34.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:16:34.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:16:34.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:16:34.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:16:34.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:16:34.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:16:34.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:16:34.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:16:34.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:16:34.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:16:34.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:16:34.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:16:34.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:16:34.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:16:34.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:16:34.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:16:34.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:16:34.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:16:34.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:16:34.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:16:34.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:16:34.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:16:34.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:16:34.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:16:34.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:16:34.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:16:34.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:16:34.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:16:34.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:16:34.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:16:34.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:16:34.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:16:34.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:16:34.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:16:34.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:16:34.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:16:34.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:35 smithi016 ceph-mon[22010]: pgmap v3733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:35 smithi016 conmon[32727]: 2022-03-21T19:16:35.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:35 smithi079 ceph-mon[23158]: pgmap v3733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:36 smithi016 conmon[32727]: 2022-03-21T19:16:36.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:36 smithi016 conmon[32727]: 2022-03-21T19:16:36.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:37 smithi016 ceph-mon[22010]: pgmap v3734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:37 smithi079 ceph-mon[23158]: pgmap v3734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:38 smithi016 conmon[32727]: 2022-03-21T19:16:38.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:39 smithi016 ceph-mon[22010]: pgmap v3735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:38 smithi016 conmon[32727]: 2022-03-21T19:16:38.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:39 smithi079 ceph-mon[23158]: pgmap v3735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:39 smithi016 conmon[32727]: 2022-03-21T19:16:39.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:41 smithi016 ceph-mon[22010]: pgmap v3736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:41.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:40 smithi016 conmon[32727]: 2022-03-21T19:16:40.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:41 smithi079 ceph-mon[23158]: pgmap v3736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:42.207 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:41 smithi016 conmon[32727]: 2022-03-21T19:16:41.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:43.209 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:42 smithi016 conmon[32727]: 2022-03-21T19:16:42.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:43 smithi079 ceph-mon[23158]: pgmap v3737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:43 smithi016 ceph-mon[22010]: pgmap v3737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:43.894 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:44.215 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:43 smithi016 conmon[32727]: 2022-03-21T19:16:43.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:44 smithi016 conmon[32727]: 2022-03-21T19:16:44.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:45 smithi079 ceph-mon[23158]: pgmap v3738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:45 smithi016 ceph-mon[22010]: pgmap v3738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:46.220 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:45 smithi016 conmon[32727]: 2022-03-21T19:16:45.930+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:47.223 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:46 smithi016 conmon[32727]: 2022-03-21T19:16:46.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:47 smithi079 ceph-mon[23158]: pgmap v3739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:47 smithi016 ceph-mon[22010]: pgmap v3739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:48.227 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:47 smithi016 conmon[32727]: 2022-03-21T19:16:47.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:48 smithi016 conmon[32727]: 2022-03-21T19:16:48.891+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:49 smithi016 ceph-mon[22010]: pgmap v3740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:49 smithi079 ceph-mon[23158]: pgmap v3740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:49 smithi016 conmon[32727]: 2022-03-21T19:16:49.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:50 smithi016 ceph-mon[22010]: pgmap v3741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:50 smithi079 ceph-mon[23158]: pgmap v3741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:50 smithi016 conmon[32727]: 2022-03-21T19:16:50.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:51 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:51 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:51 smithi016 conmon[32727]: 2022-03-21T19:16:51.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:52 smithi016 ceph-mon[22010]: pgmap v3742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:52 smithi079 ceph-mon[23158]: pgmap v3742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:52 smithi016 conmon[32727]: 2022-03-21T19:16:52.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:53 smithi016 conmon[32727]: 2022-03-21T19:16:53.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:54 smithi016 ceph-mon[22010]: pgmap v3743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:54 smithi079 ceph-mon[23158]: pgmap v3743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:16:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:55 smithi016 conmon[32727]: 2022-03-21T19:16:55.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:55 smithi016 conmon[32727]: 2022-03-21T19:16:55.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:56 smithi016 ceph-mon[22010]: pgmap v3744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:56 smithi079 ceph-mon[23158]: pgmap v3744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:56 smithi016 conmon[32727]: 2022-03-21T19:16:56.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:57 smithi016 conmon[32727]: 2022-03-21T19:16:57.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: pgmap v3745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:16:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:16:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:16:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:16:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: pgmap v3745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:16:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:16:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:16:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:16:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:16:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:58 smithi016 conmon[32727]: 2022-03-21T19:16:58.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:16:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:16:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:16:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:16:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:16:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:16:59 smithi016 conmon[32727]: 2022-03-21T19:16:59.951+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:00 smithi016 ceph-mon[22010]: pgmap v3746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:00 smithi079 ceph-mon[23158]: pgmap v3746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:00 smithi016 conmon[32727]: 2022-03-21T19:17:00.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:01.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:01 smithi016 conmon[32727]: 2022-03-21T19:17:01.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:02 smithi016 ceph-mon[22010]: pgmap v3747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:02 smithi079 ceph-mon[23158]: pgmap v3747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:02 smithi016 conmon[32727]: 2022-03-21T19:17:02.948+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:03 smithi016 conmon[32727]: 2022-03-21T19:17:03.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:04.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:04.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:04 smithi016 ceph-mon[22010]: pgmap v3748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:04.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:04 smithi079 ceph-mon[23158]: pgmap v3748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:04 smithi016 conmon[32727]: 2022-03-21T19:17:04.905+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:05 smithi016 conmon[32727]: 2022-03-21T19:17:05.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:06 smithi016 ceph-mon[22010]: pgmap v3749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:06 smithi079 ceph-mon[23158]: pgmap v3749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:06 smithi016 conmon[32727]: 2022-03-21T19:17:06.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:07 smithi016 conmon[32727]: 2022-03-21T19:17:07.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:08 smithi016 ceph-mon[22010]: pgmap v3750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:08 smithi079 ceph-mon[23158]: pgmap v3750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:08 smithi016 conmon[32727]: 2022-03-21T19:17:08.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:09 smithi016 conmon[32727]: 2022-03-21T19:17:09.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:10 smithi016 ceph-mon[22010]: pgmap v3751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:10 smithi079 ceph-mon[23158]: pgmap v3751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:10 smithi016 conmon[32727]: 2022-03-21T19:17:10.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:11 smithi016 conmon[32727]: 2022-03-21T19:17:11.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:12 smithi016 ceph-mon[22010]: pgmap v3752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:12 smithi079 ceph-mon[23158]: pgmap v3752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:13 smithi016 conmon[32727]: 2022-03-21T19:17:13.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:14 smithi016 conmon[32727]: 2022-03-21T19:17:14.011+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:14 smithi016 ceph-mon[22010]: pgmap v3753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:14 smithi079 ceph-mon[23158]: pgmap v3753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:15 smithi016 conmon[32727]: 2022-03-21T19:17:15.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:16 smithi016 conmon[32727]: 2022-03-21T19:17:16.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:16 smithi016 ceph-mon[22010]: pgmap v3754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:16 smithi079 ceph-mon[23158]: pgmap v3754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:17 smithi016 conmon[32727]: 2022-03-21T19:17:17.111+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:18 smithi016 conmon[32727]: 2022-03-21T19:17:18.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:19 smithi079 ceph-mon[23158]: pgmap v3755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:19 smithi016 ceph-mon[22010]: pgmap v3755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:19 smithi016 conmon[32727]: 2022-03-21T19:17:19.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:20 smithi016 conmon[32727]: 2022-03-21T19:17:20.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:21 smithi016 ceph-mon[22010]: pgmap v3756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:21.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:21 smithi016 conmon[32727]: 2022-03-21T19:17:21.124+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:21 smithi079 ceph-mon[23158]: pgmap v3756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:22 smithi016 conmon[32727]: 2022-03-21T19:17:22.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:23 smithi016 ceph-mon[22010]: pgmap v3757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:23.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:23 smithi016 conmon[32727]: 2022-03-21T19:17:23.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:23 smithi079 ceph-mon[23158]: pgmap v3757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:24 smithi016 conmon[32727]: 2022-03-21T19:17:24.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:25 smithi016 ceph-mon[22010]: pgmap v3758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:25 smithi016 conmon[32727]: 2022-03-21T19:17:25.024+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:25 smithi079 ceph-mon[23158]: pgmap v3758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:26 smithi016 conmon[32727]: 2022-03-21T19:17:26.051+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:27 smithi016 ceph-mon[22010]: pgmap v3759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:27.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:27 smithi016 conmon[32727]: 2022-03-21T19:17:27.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:27 smithi079 ceph-mon[23158]: pgmap v3759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:28 smithi016 conmon[32727]: 2022-03-21T19:17:28.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:29 smithi016 ceph-mon[22010]: pgmap v3760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:29 smithi016 conmon[32727]: 2022-03-21T19:17:29.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:29 smithi079 ceph-mon[23158]: pgmap v3760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:30 smithi016 conmon[32727]: 2022-03-21T19:17:30.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:30 smithi016 ceph-mon[22010]: pgmap v3761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:30 smithi079 ceph-mon[23158]: pgmap v3761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:31 smithi016 conmon[32727]: 2022-03-21T19:17:31.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:31 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:31 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:32.317 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:32 smithi016 conmon[32727]: 2022-03-21T19:17:32.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:32 smithi016 ceph-mon[22010]: pgmap v3762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:17:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:17:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:32 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:17:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:32 smithi079 ceph-mon[23158]: pgmap v3762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:17:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:17:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:32 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:17:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:33 smithi016 conmon[32727]: 2022-03-21T19:17:33.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:33 smithi016 conmon[32727]: 2022-03-21T19:17:33.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:35.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:34 smithi016 ceph-mon[22010]: pgmap v3763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:35.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:35.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:34 smithi079 ceph-mon[23158]: pgmap v3763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:35.375 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:34 smithi016 conmon[32727]: 2022-03-21T19:17:34.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:35.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: 2022-03-21T19:17:35.373+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:35.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:17:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: yield 2022-03-21T19:17:35.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:35.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:35.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:35.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:17:35.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:35.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:17:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:17:35.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:35.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:17:35.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:35.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:17:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:35.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:35 smithi016 conmon[32727]: 2022-03-21T19:17:35.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:17:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:17:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:17:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:17:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:17:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:17:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:36.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:36.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:17:36.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:36.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:36 smithi016 ceph-mon[22010]: pgmap v3764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:17:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:17:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:17:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:17:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:17:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:17:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:17:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:36 smithi079 ceph-mon[23158]: pgmap v3764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:36 smithi016 conmon[32727]: 2022-03-21T19:17:36.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:37 smithi016 conmon[32727]: 2022-03-21T19:17:37.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:38.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:38.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:38 smithi016 ceph-mon[22010]: pgmap v3765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:38.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 2022-03-21T19:17:38.445+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:38.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:17:38.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:38.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: yield 2022-03-21T19:17:38.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:38.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:38.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:38.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:17:38.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:38.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:17:38.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:38.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:17:38.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:38.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:17:38.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:38.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:17:38.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:38.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:38.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:38.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 2022-03-21T19:17:38.446+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:17:38.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:17:38.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:38.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: yield 2022-03-21T19:17:38.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:38.881 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:38.881 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:38.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:17:38.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:38.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:17:38.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:38.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:17:38.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:38.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:17:38.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:38.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:17:38.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:38.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:38.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:38.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 2022-03-21T19:17:38.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:17:38.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: 2022-03-21T19:17:38.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:17:38.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:17:38.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:17:38.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:17:38.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:17:38.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:17:38.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:17:38.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:17:38.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:17:38.890 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:17:38.891 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:17:38.891 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:17:38.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:17:38.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:17:38.892 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:17:38.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:17:38.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:17:38.893 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:17:38.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:17:38.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:17:38.894 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:17:38.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:17:38.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise 2022-03-21T19:17:38.895 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:17:38.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:17:38.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:17:38.896 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:17:38.897 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:38 smithi079 ceph-mon[23158]: pgmap v3765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:38 smithi016 conmon[32727]: 2022-03-21T19:17:38.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:39.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.840 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.840 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:39.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:17:39.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:39.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:39.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:39.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:17:39.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:39.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:17:39.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:39.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:17:39.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:39.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:39.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:39.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:17:39.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:39.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:39.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.846 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:39 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:17:39.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:39.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:17:39.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:39.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:39.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:39.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:17:39.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:39.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:17:39.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:39.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:17:39.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:39.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:39.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:39.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:17:39.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:39.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:39.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:17:39.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:17:39.854 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:17:39.854 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.854 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:17:39.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:17:39.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:17:39.855 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:17:39.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:17:39.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:17:39.856 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:17:39.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:17:39.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:17:39.857 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:17:39.858 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:17:39.858 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:17:39.858 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:17:39.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:17:39.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:17:39.859 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:17:39.860 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:17:39.860 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:17:39.860 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:17:39.861 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:17:39.861 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:17:39.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:17:39.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:17:39.862 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:17:39.863 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:17:39.863 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:17:39.863 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:39.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:17:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:17:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:17:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:17:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:17:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:39 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:17:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:17:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:17:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:17:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:17:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:17:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:17:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:17:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:17:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:17:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:17:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:17:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:17:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:17:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:17:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:17:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:17:39.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:17:39.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:17:39.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:17:39.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:17:39.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:17:39.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:17:39.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:17:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:17:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:17:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:17:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:17:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:17:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:17:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:17:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:17:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:17:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:17:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:17:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:17:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:17:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:17:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:17:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:17:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:17:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:17:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:17:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:17:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:17:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:17:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:17:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:17:39.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:39.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:39 smithi016 conmon[32727]: 2022-03-21T19:17:39.837+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:40.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:40.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:40 smithi016 ceph-mon[22010]: pgmap v3766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:40 smithi079 ceph-mon[23158]: pgmap v3766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:40 smithi016 conmon[32727]: 2022-03-21T19:17:40.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:41.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:41 smithi016 conmon[32727]: 2022-03-21T19:17:41.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:42.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:42.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:42 smithi016 ceph-mon[22010]: pgmap v3767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:42 smithi079 ceph-mon[23158]: pgmap v3767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:42 smithi016 conmon[32727]: 2022-03-21T19:17:42.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:43.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:43 smithi016 conmon[32727]: 2022-03-21T19:17:43.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:44.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:44.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:44 smithi016 ceph-mon[22010]: pgmap v3768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:44.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:44 smithi079 ceph-mon[23158]: pgmap v3768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:44 smithi016 conmon[32727]: 2022-03-21T19:17:44.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:45 smithi016 conmon[32727]: 2022-03-21T19:17:45.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:46 smithi016 ceph-mon[22010]: pgmap v3769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:46 smithi016 conmon[32727]: 2022-03-21T19:17:46.747+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:46 smithi079 ceph-mon[23158]: pgmap v3769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:47 smithi016 conmon[32727]: 2022-03-21T19:17:47.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:48 smithi016 ceph-mon[22010]: pgmap v3770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:48 smithi016 conmon[32727]: 2022-03-21T19:17:48.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:48 smithi079 ceph-mon[23158]: pgmap v3770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:49.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:49.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6909 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6909 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:49 smithi016 conmon[32727]: 2022-03-21T19:17:49.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:50.857 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:50.857 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:50 smithi016 ceph-mon[22010]: pgmap v3771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:50 smithi079 ceph-mon[23158]: pgmap v3771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:50 smithi016 conmon[32727]: 2022-03-21T19:17:50.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:51.860 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:51 smithi016 conmon[32727]: 2022-03-21T19:17:51.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:52.847 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:52.848 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:52 smithi016 ceph-mon[22010]: pgmap v3772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:52 smithi079 ceph-mon[23158]: pgmap v3772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:52 smithi016 conmon[32727]: 2022-03-21T19:17:52.844+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:53.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:53 smithi016 conmon[32727]: 2022-03-21T19:17:53.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:54.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:54.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:54 smithi016 ceph-mon[22010]: pgmap v3773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:54.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:54 smithi079 ceph-mon[23158]: pgmap v3773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:17:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:54 smithi016 conmon[32727]: 2022-03-21T19:17:54.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:55.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:55 smithi016 conmon[32727]: 2022-03-21T19:17:55.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:56.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:56.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:56 smithi016 ceph-mon[22010]: pgmap v3774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:56 smithi079 ceph-mon[23158]: pgmap v3774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:56 smithi016 conmon[32727]: 2022-03-21T19:17:56.832+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:57 smithi016 conmon[32727]: 2022-03-21T19:17:57.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:58.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:58.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: pgmap v3775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:58.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:17:58.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:17:58.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:17:58.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:17:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: pgmap v3775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:17:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:17:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:17:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:17:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:17:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:58 smithi016 conmon[32727]: 2022-03-21T19:17:58.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:17:59.882 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:59.882 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:17:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6919 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:17:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:17:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:17:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6919 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:00.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:17:59 smithi016 conmon[32727]: 2022-03-21T19:17:59.880+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:00.893 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:00.893 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:00 smithi016 ceph-mon[22010]: pgmap v3776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:00 smithi079 ceph-mon[23158]: pgmap v3776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:00 smithi016 conmon[32727]: 2022-03-21T19:18:00.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:01.891 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:01 smithi016 conmon[32727]: 2022-03-21T19:18:01.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:02.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:02.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:02 smithi016 ceph-mon[22010]: pgmap v3777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:02 smithi079 ceph-mon[23158]: pgmap v3777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:02 smithi016 conmon[32727]: 2022-03-21T19:18:02.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:03.902 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:03 smithi016 conmon[32727]: 2022-03-21T19:18:03.900+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:04.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:04.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:04 smithi016 ceph-mon[22010]: pgmap v3778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:04.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6924 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:04 smithi079 ceph-mon[23158]: pgmap v3778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6924 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:04 smithi016 conmon[32727]: 2022-03-21T19:18:04.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:05.858 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:05 smithi016 conmon[32727]: 2022-03-21T19:18:05.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:06.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:06.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:06 smithi016 ceph-mon[22010]: pgmap v3779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:06 smithi079 ceph-mon[23158]: pgmap v3779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:06 smithi016 conmon[32727]: 2022-03-21T19:18:06.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:07.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:07 smithi016 conmon[32727]: 2022-03-21T19:18:07.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:08.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:08.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:08 smithi016 ceph-mon[22010]: pgmap v3780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:08 smithi079 ceph-mon[23158]: pgmap v3780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:08 smithi016 conmon[32727]: 2022-03-21T19:18:08.882+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:09.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:09.890 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:09 smithi016 conmon[32727]: 2022-03-21T19:18:09.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:10 smithi016 ceph-mon[22010]: pgmap v3781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:10 smithi016 conmon[32727]: 2022-03-21T19:18:10.844+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:10 smithi079 ceph-mon[23158]: pgmap v3781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:11 smithi016 conmon[32727]: 2022-03-21T19:18:11.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:12 smithi016 ceph-mon[22010]: pgmap v3782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:12 smithi016 conmon[32727]: 2022-03-21T19:18:12.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:12 smithi079 ceph-mon[23158]: pgmap v3782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:13.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:13 smithi016 conmon[32727]: 2022-03-21T19:18:13.806+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:14 smithi016 ceph-mon[22010]: pgmap v3783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:14 smithi016 conmon[32727]: 2022-03-21T19:18:14.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:14 smithi079 ceph-mon[23158]: pgmap v3783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:15 smithi016 conmon[32727]: 2022-03-21T19:18:15.771+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:16 smithi016 ceph-mon[22010]: pgmap v3784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:16 smithi016 conmon[32727]: 2022-03-21T19:18:16.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:16 smithi079 ceph-mon[23158]: pgmap v3784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:17 smithi016 conmon[32727]: 2022-03-21T19:18:17.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:18 smithi016 ceph-mon[22010]: pgmap v3785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:18 smithi016 conmon[32727]: 2022-03-21T19:18:18.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:18 smithi079 ceph-mon[23158]: pgmap v3785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:19 smithi016 conmon[32727]: 2022-03-21T19:18:19.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:20 smithi016 ceph-mon[22010]: pgmap v3786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:20 smithi016 conmon[32727]: 2022-03-21T19:18:20.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:20 smithi079 ceph-mon[23158]: pgmap v3786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:21.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:21 smithi016 conmon[32727]: 2022-03-21T19:18:21.691+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:22 smithi016 ceph-mon[22010]: pgmap v3787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:22 smithi016 conmon[32727]: 2022-03-21T19:18:22.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:22 smithi079 ceph-mon[23158]: pgmap v3787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:23.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:23 smithi016 conmon[32727]: 2022-03-21T19:18:23.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:24 smithi016 ceph-mon[22010]: pgmap v3788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:24 smithi016 conmon[32727]: 2022-03-21T19:18:24.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:24 smithi079 ceph-mon[23158]: pgmap v3788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:25 smithi016 conmon[32727]: 2022-03-21T19:18:25.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:26 smithi079 ceph-mon[23158]: pgmap v3789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:26 smithi016 ceph-mon[22010]: pgmap v3789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:27.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:26 smithi016 conmon[32727]: 2022-03-21T19:18:26.666+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:27 smithi016 conmon[32727]: 2022-03-21T19:18:27.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:28 smithi016 ceph-mon[22010]: pgmap v3790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:28 smithi016 conmon[32727]: 2022-03-21T19:18:28.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:28 smithi079 ceph-mon[23158]: pgmap v3790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:29 smithi016 conmon[32727]: 2022-03-21T19:18:29.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:30 smithi016 ceph-mon[22010]: pgmap v3791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:30 smithi016 conmon[32727]: 2022-03-21T19:18:30.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:30 smithi079 ceph-mon[23158]: pgmap v3791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:31 smithi016 conmon[32727]: 2022-03-21T19:18:31.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:32 smithi016 ceph-mon[22010]: pgmap v3792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:32 smithi016 conmon[32727]: 2022-03-21T19:18:32.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:32 smithi079 ceph-mon[23158]: pgmap v3792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:33 smithi016 conmon[32727]: 2022-03-21T19:18:33.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:34 smithi016 conmon[32727]: 2022-03-21T19:18:34.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:34 smithi016 ceph-mon[22010]: pgmap v3793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:34 smithi079 ceph-mon[23158]: pgmap v3793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:35 smithi016 conmon[32727]: 2022-03-21T19:18:35.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:36 smithi016 ceph-mon[22010]: pgmap v3794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:36 smithi016 conmon[32727]: 2022-03-21T19:18:36.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:36 smithi079 ceph-mon[23158]: pgmap v3794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:37 smithi016 conmon[32727]: 2022-03-21T19:18:37.662+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:38.885 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:38.886 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:38 smithi079 ceph-mon[23158]: pgmap v3795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:38.886 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:18:38.887 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:18:38.887 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:38 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:18:38.913 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:38 smithi016 conmon[32727]: 2022-03-21T19:18:38.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:38.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:38 smithi016 ceph-mon[22010]: pgmap v3795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:18:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:18:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:38 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:18:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: 2022-03-21T19:18:39.053+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:18:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: yield 2022-03-21T19:18:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:18:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:18:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:18:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:18:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:18:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:18:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:18:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:18:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:18:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:18:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:18:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:39 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:18:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:39 smithi016 conmon[32727]: 2022-03-21T19:18:39.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:18:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:18:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:18:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:18:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:18:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:18:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:39 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:18:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:40 smithi079 ceph-mon[23158]: pgmap v3796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:40 smithi016 ceph-mon[22010]: pgmap v3796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:40 smithi016 conmon[32727]: 2022-03-21T19:18:40.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:42.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:42.128 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:41 smithi016 conmon[32727]: 2022-03-21T19:18:41.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:42.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 2022-03-21T19:18:42.125+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:18:42.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: yield 2022-03-21T19:18:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:42.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:42.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:42.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:18:42.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:18:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:42.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:18:42.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:42.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:18:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:18:42.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:42.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:42.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 2022-03-21T19:18:42.126+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:18:42.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:18:42.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:42.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: yield 2022-03-21T19:18:42.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:42.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:42.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:42.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:18:42.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:42.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:18:42.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:42.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:18:42.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:42.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:18:42.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:42.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:18:42.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:42.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:42.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 2022-03-21T19:18:42.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:18:42.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: 2022-03-21T19:18:42.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:18:42.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:18:42.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:18:42.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:18:42.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:18:42.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:18:42.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:18:42.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:18:42.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:18:42.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:18:42.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:18:42.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:18:42.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:18:42.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:18:42.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:18:42.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:18:42.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:18:42.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:18:42.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:18:42.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:18:42.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:18:42.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:18:42.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise 2022-03-21T19:18:42.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:18:42.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:18:42.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:18:42.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:18:42.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:42 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:18:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:42.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:18:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:42.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:42.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:18:42.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:42.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:18:42.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:42.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:18:42.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:42.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:42.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:42.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:18:42.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:42.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:42.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:42 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:18:42.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:18:42.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:42.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:18:42.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:42.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:42.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:42.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:18:42.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:42.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:18:42.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:42.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:18:42.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:42.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:42.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:42.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:18:42.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:42.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:42.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:18:42.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:18:42.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:18:42.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:18:42.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:18:42.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:18:42.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:18:42.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:18:42.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:18:42.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:18:42.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:18:42.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:18:42.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:18:42.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:18:42.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:18:42.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:18:42.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:18:42.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:18:42.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:18:42.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:18:42.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:18:42.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:18:42.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:18:42.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:18:42.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:18:42.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:18:42.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:18:42.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:18:42.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:18:42.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:18:42.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:42.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:42 smithi079 ceph-mon[23158]: pgmap v3797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:42 smithi016 conmon[32727]: 2022-03-21T19:18:42.691+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:42 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:43.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:18:43.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:43.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:18:43.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:43.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:43.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:43.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:18:43.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:43.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:18:43.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:43.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:18:43.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:43.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:43.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:43.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:18:43.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:43.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:43.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:43.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:42 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:18:43.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:18:43.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:18:43.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:18:43.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:18:43.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:18:43.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:18:43.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:18:43.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:18:43.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:18:43.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:18:43.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:18:43.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:18:43.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:18:43.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:18:43.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:18:43.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:18:43.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:18:43.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:43.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:18:43.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:18:43.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:18:43.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:18:43.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:18:43.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:18:43.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:18:43.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:18:43.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:18:43.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:18:43.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:18:43.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:18:43.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:18:43.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:18:43.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:18:43.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:18:43.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:18:43.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:18:43.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:18:43.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:18:43.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:18:43.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:18:43.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:18:43.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:18:43.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:18:43.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:18:43.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:18:43.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:18:43.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:18:43.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:18:43.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:18:43.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:42 smithi016 ceph-mon[22010]: pgmap v3797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:44.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:43 smithi016 conmon[32727]: 2022-03-21T19:18:43.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:44 smithi016 conmon[32727]: 2022-03-21T19:18:44.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:44 smithi016 ceph-mon[22010]: pgmap v3798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:44 smithi079 ceph-mon[23158]: pgmap v3798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:45 smithi016 conmon[32727]: 2022-03-21T19:18:45.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:46 smithi016 conmon[32727]: 2022-03-21T19:18:46.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:46 smithi016 ceph-mon[22010]: pgmap v3799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:46 smithi079 ceph-mon[23158]: pgmap v3799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:47 smithi016 conmon[32727]: 2022-03-21T19:18:47.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:48.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:48 smithi016 conmon[32727]: 2022-03-21T19:18:48.771+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:48 smithi016 ceph-mon[22010]: pgmap v3800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:49.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:49.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:48 smithi079 ceph-mon[23158]: pgmap v3800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:49 smithi016 conmon[32727]: 2022-03-21T19:18:49.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:50 smithi016 ceph-mon[22010]: pgmap v3801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:51.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:50 smithi016 conmon[32727]: 2022-03-21T19:18:50.774+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:50 smithi079 ceph-mon[23158]: pgmap v3801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:51 smithi016 conmon[32727]: 2022-03-21T19:18:51.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:52 smithi016 ceph-mon[22010]: pgmap v3802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:52 smithi016 conmon[32727]: 2022-03-21T19:18:52.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:52 smithi079 ceph-mon[23158]: pgmap v3802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:18:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:54.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:53 smithi016 conmon[32727]: 2022-03-21T19:18:53.777+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:54 smithi016 ceph-mon[22010]: pgmap v3803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:54 smithi016 conmon[32727]: 2022-03-21T19:18:54.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:54 smithi079 ceph-mon[23158]: pgmap v3803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:18:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:18:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:55 smithi016 conmon[32727]: 2022-03-21T19:18:55.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:57.799 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:56 smithi016 ceph-mon[22010]: pgmap v3804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:57.799 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:56 smithi016 conmon[32727]: 2022-03-21T19:18:56.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:57.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:57.800 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:56 smithi079 ceph-mon[23158]: pgmap v3804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:57 smithi016 conmon[32727]: 2022-03-21T19:18:57.763+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:58 smithi016 conmon[32727]: 2022-03-21T19:18:58.749+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:18:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: pgmap v3805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:18:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:18:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:18:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:18:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:18:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: pgmap v3805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:18:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:18:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:18:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:18:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:19:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:18:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:00.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:18:59 smithi016 conmon[32727]: 2022-03-21T19:18:59.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:18:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:00 smithi016 ceph-mon[22010]: pgmap v3806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:00 smithi016 conmon[32727]: 2022-03-21T19:19:00.662+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:00 smithi079 ceph-mon[23158]: pgmap v3806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:01 smithi016 conmon[32727]: 2022-03-21T19:19:01.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:02 smithi016 ceph-mon[22010]: pgmap v3807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:02 smithi016 conmon[32727]: 2022-03-21T19:19:02.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:02 smithi079 ceph-mon[23158]: pgmap v3807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:03.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:03 smithi016 conmon[32727]: 2022-03-21T19:19:03.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:04 smithi016 ceph-mon[22010]: pgmap v3808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:05.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:04 smithi016 conmon[32727]: 2022-03-21T19:19:04.677+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:04 smithi079 ceph-mon[23158]: pgmap v3808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:05 smithi016 conmon[32727]: 2022-03-21T19:19:05.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:06 smithi016 ceph-mon[22010]: pgmap v3809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:06 smithi016 conmon[32727]: 2022-03-21T19:19:06.659+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:06 smithi079 ceph-mon[23158]: pgmap v3809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:07 smithi016 conmon[32727]: 2022-03-21T19:19:07.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:08 smithi016 ceph-mon[22010]: pgmap v3810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:08 smithi016 conmon[32727]: 2022-03-21T19:19:08.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:08 smithi079 ceph-mon[23158]: pgmap v3810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:09 smithi016 conmon[32727]: 2022-03-21T19:19:09.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:10 smithi016 ceph-mon[22010]: pgmap v3811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:10 smithi016 conmon[32727]: 2022-03-21T19:19:10.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:11.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:10 smithi079 ceph-mon[23158]: pgmap v3811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:11 smithi016 conmon[32727]: 2022-03-21T19:19:11.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:12.822 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:12 smithi016 conmon[32727]: 2022-03-21T19:19:12.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:12 smithi016 ceph-mon[22010]: pgmap v3812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:12 smithi079 ceph-mon[23158]: pgmap v3812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:13.823 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:13 smithi016 conmon[32727]: 2022-03-21T19:19:13.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:14.827 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:14 smithi016 conmon[32727]: 2022-03-21T19:19:14.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:14 smithi016 ceph-mon[22010]: pgmap v3813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 6995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:14 smithi079 ceph-mon[23158]: pgmap v3813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 6995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:15 smithi016 conmon[32727]: 2022-03-21T19:19:15.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:16.834 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:16 smithi016 conmon[32727]: 2022-03-21T19:19:16.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:17.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:16 smithi016 ceph-mon[22010]: pgmap v3814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:16 smithi079 ceph-mon[23158]: pgmap v3814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:17 smithi016 conmon[32727]: 2022-03-21T19:19:17.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:18 smithi016 conmon[32727]: 2022-03-21T19:19:18.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:18 smithi016 ceph-mon[22010]: pgmap v3815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:18 smithi079 ceph-mon[23158]: pgmap v3815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:19 smithi016 conmon[32727]: 2022-03-21T19:19:19.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:20 smithi016 conmon[32727]: 2022-03-21T19:19:20.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:20 smithi016 ceph-mon[22010]: pgmap v3816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:20 smithi079 ceph-mon[23158]: pgmap v3816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:21 smithi016 conmon[32727]: 2022-03-21T19:19:21.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:22 smithi016 conmon[32727]: 2022-03-21T19:19:22.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:22 smithi016 ceph-mon[22010]: pgmap v3817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:22 smithi079 ceph-mon[23158]: pgmap v3817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:23 smithi016 conmon[32727]: 2022-03-21T19:19:23.706+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:24 smithi016 conmon[32727]: 2022-03-21T19:19:24.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:24 smithi016 ceph-mon[22010]: pgmap v3818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:24 smithi079 ceph-mon[23158]: pgmap v3818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:25 smithi016 conmon[32727]: 2022-03-21T19:19:25.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:26 smithi016 conmon[32727]: 2022-03-21T19:19:26.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:27.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:26 smithi016 ceph-mon[22010]: pgmap v3819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:26 smithi079 ceph-mon[23158]: pgmap v3819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:27 smithi016 conmon[32727]: 2022-03-21T19:19:27.800+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:28 smithi016 conmon[32727]: 2022-03-21T19:19:28.757+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:28 smithi016 ceph-mon[22010]: pgmap v3820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:28 smithi079 ceph-mon[23158]: pgmap v3820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:29 smithi016 conmon[32727]: 2022-03-21T19:19:29.766+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:30 smithi016 conmon[32727]: 2022-03-21T19:19:30.733+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:30 smithi016 ceph-mon[22010]: pgmap v3821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:31.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:30 smithi079 ceph-mon[23158]: pgmap v3821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:31 smithi016 conmon[32727]: 2022-03-21T19:19:31.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:32 smithi016 ceph-mon[22010]: pgmap v3822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:33.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:32 smithi016 conmon[32727]: 2022-03-21T19:19:32.749+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:32 smithi079 ceph-mon[23158]: pgmap v3822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:34.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:33 smithi016 conmon[32727]: 2022-03-21T19:19:33.748+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:34 smithi016 conmon[32727]: 2022-03-21T19:19:34.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:34 smithi016 ceph-mon[22010]: pgmap v3823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:34 smithi079 ceph-mon[23158]: pgmap v3823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:35 smithi016 conmon[32727]: 2022-03-21T19:19:35.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:36.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:36 smithi016 ceph-mon[22010]: pgmap v3824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:36 smithi016 conmon[32727]: 2022-03-21T19:19:36.733+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:36 smithi079 ceph-mon[23158]: pgmap v3824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:38.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:37 smithi016 conmon[32727]: 2022-03-21T19:19:37.684+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:38 smithi016 ceph-mon[22010]: pgmap v3825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:39.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:38 smithi016 conmon[32727]: 2022-03-21T19:19:38.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:38 smithi079 ceph-mon[23158]: pgmap v3825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:39 smithi016 conmon[32727]: 2022-03-21T19:19:39.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:40 smithi016 conmon[32727]: 2022-03-21T19:19:40.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:40 smithi016 ceph-mon[22010]: pgmap v3826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:40 smithi079 ceph-mon[23158]: pgmap v3826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:41 smithi016 conmon[32727]: 2022-03-21T19:19:41.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:42.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:42 smithi016 conmon[32727]: 2022-03-21T19:19:42.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:19:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:19:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:19:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:42 smithi079 ceph-mon[23158]: pgmap v3827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:19:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:19:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:19:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:42 smithi016 ceph-mon[22010]: pgmap v3827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:44.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:43 smithi016 conmon[32727]: 2022-03-21T19:19:43.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:44 smithi016 conmon[32727]: 2022-03-21T19:19:44.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:44 smithi016 ceph-mon[22010]: pgmap v3828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:44 smithi079 ceph-mon[23158]: pgmap v3828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:45.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: 2022-03-21T19:19:45.228+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:45.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:19:45.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:45.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: yield 2022-03-21T19:19:45.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:45.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:45.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:19:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:45.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:19:45.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:45.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:19:45.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:45.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:19:45.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:45.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:19:45.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:45.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:45.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:45 smithi016 conmon[32727]: 2022-03-21T19:19:45.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:45 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:46.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:19:46.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:46.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:19:46.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:46.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:46.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:46.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:19:46.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:46.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:19:46.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:46.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:19:46.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:46.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:46.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:46.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:19:46.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:46.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:46.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:46.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:45 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:19:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:45 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:19:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:46.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:19:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:46.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:19:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:46.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:19:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:19:46.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:46.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:19:46.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:46.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:46.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:46.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:45 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:19:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:46 smithi016 conmon[32727]: 2022-03-21T19:19:46.682+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:46 smithi016 ceph-mon[22010]: pgmap v3829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:46 smithi079 ceph-mon[23158]: pgmap v3829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:48.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:47 smithi016 conmon[32727]: 2022-03-21T19:19:47.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:48.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 2022-03-21T19:19:48.300+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:48.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:19:48.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:48.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: yield 2022-03-21T19:19:48.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:48.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:48.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:48.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:19:48.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:48.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:19:48.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:48.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:19:48.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:48.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:19:48.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:48.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:19:48.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:48.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:48.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:48.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 2022-03-21T19:19:48.301+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:19:48.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:19:48.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:48.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: yield 2022-03-21T19:19:48.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:48.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:48.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:48.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:19:48.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:48.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:19:48.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:48.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:19:48.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:48.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:19:48.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:48.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:19:48.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:48.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:48.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:48.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 2022-03-21T19:19:48.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:19:48.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: 2022-03-21T19:19:48.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:19:48.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:19:48.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:19:48.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:19:48.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:19:48.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:19:48.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:19:48.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:19:48.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:19:48.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:19:48.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:19:48.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:19:48.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:19:48.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:19:48.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:19:48.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:19:48.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:19:48.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:19:48.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:19:48.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:19:48.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:19:48.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:19:48.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise 2022-03-21T19:19:48.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:19:48.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:19:48.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:19:48.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:19:48.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:48.939 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:48 smithi016 conmon[32727]: 2022-03-21T19:19:48.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:48 smithi079 ceph-mon[23158]: pgmap v3830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:48 smithi016 ceph-mon[22010]: pgmap v3830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:49 smithi016 conmon[32727]: 2022-03-21T19:19:49.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:49 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:19:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:50.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:19:50.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:19:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:50.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:19:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:50.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:19:50.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:49 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:19:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:50.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:19:50.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:50.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:50.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:50.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:19:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:19:50.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:19:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:50.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:19:50.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:50.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:50.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:19:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:19:50.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:19:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:19:50.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:19:50.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:19:50.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:19:50.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:19:50.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:19:50.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:19:50.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:19:50.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:19:50.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:19:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:19:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:19:50.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:19:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:19:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:19:50.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:19:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:19:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:19:50.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:19:50.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:19:50.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:19:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:19:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:19:50.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:19:50.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:19:50.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:19:50.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.250 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:50.250 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:49 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:19:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:50.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:50.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:19:50.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:50.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:19:50.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:50.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:19:50.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:50.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:50.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:50.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:19:50.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:50.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:50.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.425 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:49 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:19:50.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:19:50.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:19:50.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:19:50.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:19:50.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:19:50.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:19:50.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:19:50.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:19:50.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:19:50.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:19:50.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:19:50.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:19:50.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:19:50.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:19:50.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:19:50.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:19:50.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:19:50.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:19:50.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:19:50.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:19:50.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:19:50.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:19:50.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:19:50.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:19:50.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:19:50.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:19:50.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:19:50.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:19:50.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:19:50.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:19:50.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:19:50.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:19:50.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:19:50.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:19:50.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:19:50.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:19:50.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:19:50.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:19:50.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:19:50.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:19:50.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:19:50.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:19:50.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:19:50.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:19:50.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:19:50.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:19:50.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:19:50.450 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:50.450 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:50 smithi016 conmon[32727]: 2022-03-21T19:19:50.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:50 smithi079 ceph-mon[23158]: pgmap v3831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:50 smithi016 ceph-mon[22010]: pgmap v3831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:51 smithi016 conmon[32727]: 2022-03-21T19:19:51.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:52 smithi016 conmon[32727]: 2022-03-21T19:19:52.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:52 smithi079 ceph-mon[23158]: pgmap v3832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:52 smithi016 ceph-mon[22010]: pgmap v3832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:53 smithi016 conmon[32727]: 2022-03-21T19:19:53.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:54 smithi016 conmon[32727]: 2022-03-21T19:19:54.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:54 smithi079 ceph-mon[23158]: pgmap v3833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:54 smithi016 ceph-mon[22010]: pgmap v3833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:19:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:19:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:55 smithi016 conmon[32727]: 2022-03-21T19:19:55.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:56 smithi016 conmon[32727]: 2022-03-21T19:19:56.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:56 smithi079 ceph-mon[23158]: pgmap v3834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:56 smithi016 ceph-mon[22010]: pgmap v3834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:57 smithi016 conmon[32727]: 2022-03-21T19:19:57.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:58.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:58 smithi016 conmon[32727]: 2022-03-21T19:19:58.561+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:19:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: pgmap v3835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:19:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:19:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:19:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: pgmap v3835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:19:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:19:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:19:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:19:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:19:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:19:59 smithi016 conmon[32727]: 2022-03-21T19:19:59.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:19:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:19:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:00 smithi016 conmon[32727]: 2022-03-21T19:20:00.466+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops 2022-03-21T19:20:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:20:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:20:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:20:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops 2022-03-21T19:20:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:00 smithi016 ceph-mon[22010]: pgmap v3836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops 2022-03-21T19:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:20:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:20:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:20:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7040 sec, osd.4 has slow ops 2022-03-21T19:20:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:00 smithi079 ceph-mon[23158]: pgmap v3836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:01 smithi016 conmon[32727]: 2022-03-21T19:20:01.499+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:02 smithi016 conmon[32727]: 2022-03-21T19:20:02.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:02 smithi016 ceph-mon[22010]: pgmap v3837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:02 smithi079 ceph-mon[23158]: pgmap v3837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:03 smithi016 conmon[32727]: 2022-03-21T19:20:03.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:04 smithi016 conmon[32727]: 2022-03-21T19:20:04.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:04 smithi016 ceph-mon[22010]: pgmap v3838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:05.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:04 smithi079 ceph-mon[23158]: pgmap v3838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:05 smithi016 conmon[32727]: 2022-03-21T19:20:05.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:06 smithi016 conmon[32727]: 2022-03-21T19:20:06.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:06 smithi016 ceph-mon[22010]: pgmap v3839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:06 smithi079 ceph-mon[23158]: pgmap v3839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:07 smithi016 conmon[32727]: 2022-03-21T19:20:07.561+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:08 smithi016 conmon[32727]: 2022-03-21T19:20:08.523+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:09 smithi016 ceph-mon[22010]: pgmap v3840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:09.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:09 smithi079 ceph-mon[23158]: pgmap v3840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:09 smithi016 conmon[32727]: 2022-03-21T19:20:09.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:10 smithi016 conmon[32727]: 2022-03-21T19:20:10.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:11 smithi016 ceph-mon[22010]: pgmap v3841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:11 smithi079 ceph-mon[23158]: pgmap v3841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:11 smithi016 conmon[32727]: 2022-03-21T19:20:11.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:12 smithi016 conmon[32727]: 2022-03-21T19:20:12.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:13 smithi016 ceph-mon[22010]: pgmap v3842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:13 smithi079 ceph-mon[23158]: pgmap v3842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:13.558 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:13 smithi016 conmon[32727]: 2022-03-21T19:20:13.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:14 smithi016 conmon[32727]: 2022-03-21T19:20:14.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:15 smithi016 ceph-mon[22010]: pgmap v3843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:15 smithi079 ceph-mon[23158]: pgmap v3843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:15 smithi016 conmon[32727]: 2022-03-21T19:20:15.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:16 smithi016 conmon[32727]: 2022-03-21T19:20:16.581+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:17 smithi016 ceph-mon[22010]: pgmap v3844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:17 smithi079 ceph-mon[23158]: pgmap v3844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:17 smithi016 conmon[32727]: 2022-03-21T19:20:17.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:18 smithi016 conmon[32727]: 2022-03-21T19:20:18.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:19 smithi016 ceph-mon[22010]: pgmap v3845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:19 smithi079 ceph-mon[23158]: pgmap v3845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:19 smithi016 conmon[32727]: 2022-03-21T19:20:19.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:20 smithi016 conmon[32727]: 2022-03-21T19:20:20.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:21 smithi016 ceph-mon[22010]: pgmap v3846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:21 smithi079 ceph-mon[23158]: pgmap v3846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:21 smithi016 conmon[32727]: 2022-03-21T19:20:21.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:22 smithi016 conmon[32727]: 2022-03-21T19:20:22.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:23 smithi016 ceph-mon[22010]: pgmap v3847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:23 smithi079 ceph-mon[23158]: pgmap v3847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:23 smithi016 conmon[32727]: 2022-03-21T19:20:23.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:24 smithi016 conmon[32727]: 2022-03-21T19:20:24.456+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:25 smithi016 ceph-mon[22010]: pgmap v3848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:25 smithi079 ceph-mon[23158]: pgmap v3848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:25.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:25 smithi016 conmon[32727]: 2022-03-21T19:20:25.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:26 smithi016 conmon[32727]: 2022-03-21T19:20:26.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:27 smithi016 ceph-mon[22010]: pgmap v3849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:27 smithi079 ceph-mon[23158]: pgmap v3849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:27 smithi016 conmon[32727]: 2022-03-21T19:20:27.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:28 smithi016 conmon[32727]: 2022-03-21T19:20:28.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:29 smithi016 ceph-mon[22010]: pgmap v3850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:29 smithi079 ceph-mon[23158]: pgmap v3850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:29 smithi016 conmon[32727]: 2022-03-21T19:20:29.584+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:30 smithi016 conmon[32727]: 2022-03-21T19:20:30.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:31 smithi016 ceph-mon[22010]: pgmap v3851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:31 smithi079 ceph-mon[23158]: pgmap v3851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:31 smithi016 conmon[32727]: 2022-03-21T19:20:31.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:32 smithi016 conmon[32727]: 2022-03-21T19:20:32.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:33 smithi016 ceph-mon[22010]: pgmap v3852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:33 smithi079 ceph-mon[23158]: pgmap v3852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:33 smithi016 conmon[32727]: 2022-03-21T19:20:33.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:34 smithi016 conmon[32727]: 2022-03-21T19:20:34.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:35 smithi016 ceph-mon[22010]: pgmap v3853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:35 smithi079 ceph-mon[23158]: pgmap v3853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:35 smithi016 conmon[32727]: 2022-03-21T19:20:35.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:36 smithi016 conmon[32727]: 2022-03-21T19:20:36.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:37 smithi016 ceph-mon[22010]: pgmap v3854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:37.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:37 smithi079 ceph-mon[23158]: pgmap v3854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:37 smithi016 conmon[32727]: 2022-03-21T19:20:37.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:38 smithi016 conmon[32727]: 2022-03-21T19:20:38.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:39 smithi016 ceph-mon[22010]: pgmap v3855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:39 smithi079 ceph-mon[23158]: pgmap v3855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:39 smithi016 conmon[32727]: 2022-03-21T19:20:39.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:40 smithi016 conmon[32727]: 2022-03-21T19:20:40.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:41 smithi016 ceph-mon[22010]: pgmap v3856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:41 smithi079 ceph-mon[23158]: pgmap v3856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:41 smithi016 conmon[32727]: 2022-03-21T19:20:41.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:42 smithi016 conmon[32727]: 2022-03-21T19:20:42.508+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:43 smithi016 ceph-mon[22010]: pgmap v3857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:43 smithi079 ceph-mon[23158]: pgmap v3857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:43 smithi016 conmon[32727]: 2022-03-21T19:20:43.524+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:44 smithi016 conmon[32727]: 2022-03-21T19:20:44.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:45 smithi016 ceph-mon[22010]: pgmap v3858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:45 smithi079 ceph-mon[23158]: pgmap v3858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:45 smithi016 conmon[32727]: 2022-03-21T19:20:45.569+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:46 smithi016 conmon[32727]: 2022-03-21T19:20:46.560+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:47 smithi016 ceph-mon[22010]: pgmap v3859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:47 smithi079 ceph-mon[23158]: pgmap v3859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:47 smithi016 conmon[32727]: 2022-03-21T19:20:47.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:48.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:48.382 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:48 smithi016 conmon[32727]: 2022-03-21T19:20:48.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:49 smithi016 ceph-mon[22010]: pgmap v3860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:20:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:20:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:20:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:49 smithi079 ceph-mon[23158]: pgmap v3860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:20:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:20:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:20:50.132 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:49 smithi016 conmon[32727]: 2022-03-21T19:20:49.694+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:51.137 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:50 smithi016 conmon[32727]: 2022-03-21T19:20:50.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:51.408 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:51.408 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:51 smithi016 ceph-mon[22010]: pgmap v3861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:51 smithi079 ceph-mon[23158]: pgmap v3861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:51.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: 2022-03-21T19:20:51.404+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:51.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:20:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: yield 2022-03-21T19:20:51.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:51.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:20:51.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:51.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:20:51.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:51.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:20:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:20:51.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:51.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:20:51.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:51.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:51.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:52.139 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:51 smithi016 conmon[32727]: 2022-03-21T19:20:51.689+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:52.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:53.143 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:52 smithi016 conmon[32727]: 2022-03-21T19:20:52.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:53 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:20:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:20:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:53.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:53.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:53.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:20:53.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:53.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:20:53.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:53.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:20:53.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:53.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:53.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:53.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:20:53.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:53.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:53.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:53.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:53 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:20:53.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:53 smithi016 ceph-mon[22010]: pgmap v3862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:53.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:53 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:20:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:20:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:53.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:53.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:53.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:20:53.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:53.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:20:53.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:53.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:20:53.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:53.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:53.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:53.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:20:53.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:53.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:53.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:53.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:53 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:20:53.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:53 smithi079 ceph-mon[23158]: pgmap v3862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:53.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:54.148 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:53 smithi016 conmon[32727]: 2022-03-21T19:20:53.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[32727]: 2022-03-21T19:20:54.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 2022-03-21T19:20:54.476+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:20:54.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: yield 2022-03-21T19:20:54.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:54.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:20:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:20:54.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:20:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:54.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:20:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:20:54.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:54.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 2022-03-21T19:20:54.477+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:20:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:20:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:54.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: yield 2022-03-21T19:20:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:54.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:20:54.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:20:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:54.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:20:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:20:54.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:20:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:54.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 2022-03-21T19:20:54.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:20:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: 2022-03-21T19:20:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:20:54.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:20:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:20:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:20:54.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:20:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:20:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:20:54.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:20:54.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:20:54.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:20:54.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:20:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:20:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:20:54.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:20:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:20:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:20:54.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:20:54.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:20:54.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:20:54.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:20:54.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:20:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:20:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise 2022-03-21T19:20:54.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:20:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:20:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:20:54.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:20:54.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:20:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:55 smithi016 ceph-mon[22010]: pgmap v3863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:55 smithi079 ceph-mon[23158]: pgmap v3863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:20:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:20:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:56.151 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:55 smithi016 conmon[32727]: 2022-03-21T19:20:55.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:20:56.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:56.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:56.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:56.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:20:56.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:56.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:20:56.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:56.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:20:56.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:56.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:56.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:56.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:20:56.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:56.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:56.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:56 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:20:56.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:56.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:20:56.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:56.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:56.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:56.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:20:56.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:56.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:20:56.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:56.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:20:56.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:56.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:56.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:56.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:20:56.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:56.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:56.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:20:56.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:20:56.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:20:56.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:20:56.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:20:56.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:20:56.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:20:56.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:20:56.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:20:56.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:20:56.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:20:56.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:20:56.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:20:56.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:20:56.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:20:56.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:20:56.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:20:56.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:20:56.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:20:56.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:20:56.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:20:56.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:20:56.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:20:56.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:20:56.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:20:56.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:20:56.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:20:56.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:20:56.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:20:56.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.441 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:20:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:56.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:56.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:20:56.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:56.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:20:56.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:56.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:20:56.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:56.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:56.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:56.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:20:56.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:56.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:56.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:56 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:20:56.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:20:56.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:20:56.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:20:56.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:20:56.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:20:56.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:20:56.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:20:56.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:20:56.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:20:56.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:20:56.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:20:56.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:20:56.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:20:56.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:20:56.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:20:56.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:20:56.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:20:56.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:20:56.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:20:56.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:20:56.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:20:56.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:20:56.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:20:56.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:20:56.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:20:56.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:20:56.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:20:56.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:20:56.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:20:56.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:20:56.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:20:56.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:20:56.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:20:56.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:20:56.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:20:56.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:20:56.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:20:56.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:20:56.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:20:56.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:20:56.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:20:56.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:20:56.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:20:56.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:20:56.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:20:56.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:20:56.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:20:56.500 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:57.156 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:56 smithi016 conmon[32727]: 2022-03-21T19:20:56.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:57 smithi016 ceph-mon[22010]: pgmap v3864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:57 smithi079 ceph-mon[23158]: pgmap v3864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:58.158 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:57 smithi016 conmon[32727]: 2022-03-21T19:20:57.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:59.163 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:58 smithi016 conmon[32727]: 2022-03-21T19:20:58.787+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:20:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: pgmap v3865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:20:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:20:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:20:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:20:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:20:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:20:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: pgmap v3865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:20:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:20:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:20:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:20:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:20:59.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:20:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:20:59 smithi016 conmon[32727]: 2022-03-21T19:20:59.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:01.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:00 smithi016 conmon[32727]: 2022-03-21T19:21:00.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:01 smithi079 ceph-mon[23158]: pgmap v3866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:01 smithi016 ceph-mon[22010]: pgmap v3866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:02.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:01 smithi016 conmon[32727]: 2022-03-21T19:21:01.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:02 smithi016 conmon[32727]: 2022-03-21T19:21:02.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:03 smithi079 ceph-mon[23158]: pgmap v3867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:03 smithi016 ceph-mon[22010]: pgmap v3867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:03 smithi016 conmon[32727]: 2022-03-21T19:21:03.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:04 smithi016 conmon[32727]: 2022-03-21T19:21:04.803+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:05 smithi079 ceph-mon[23158]: pgmap v3868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:05 smithi016 ceph-mon[22010]: pgmap v3868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:05.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:06.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:05 smithi016 conmon[32727]: 2022-03-21T19:21:05.774+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:06 smithi016 conmon[32727]: 2022-03-21T19:21:06.809+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:07 smithi079 ceph-mon[23158]: pgmap v3869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:07 smithi016 ceph-mon[22010]: pgmap v3869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:07 smithi016 conmon[32727]: 2022-03-21T19:21:07.846+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:08 smithi016 conmon[32727]: 2022-03-21T19:21:08.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:09 smithi079 ceph-mon[23158]: pgmap v3870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:09 smithi016 ceph-mon[22010]: pgmap v3870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:10.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:09 smithi016 conmon[32727]: 2022-03-21T19:21:09.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:10 smithi016 conmon[32727]: 2022-03-21T19:21:10.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:11 smithi079 ceph-mon[23158]: pgmap v3871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:11 smithi016 ceph-mon[22010]: pgmap v3871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:11.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:11 smithi016 conmon[32727]: 2022-03-21T19:21:11.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:12 smithi016 conmon[32727]: 2022-03-21T19:21:12.790+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:13 smithi079 ceph-mon[23158]: pgmap v3872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:13 smithi016 ceph-mon[22010]: pgmap v3872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:13 smithi016 conmon[32727]: 2022-03-21T19:21:13.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:14 smithi016 conmon[32727]: 2022-03-21T19:21:14.799+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:15 smithi079 ceph-mon[23158]: pgmap v3873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:15 smithi016 ceph-mon[22010]: pgmap v3873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:15 smithi016 conmon[32727]: 2022-03-21T19:21:15.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:16 smithi016 conmon[32727]: 2022-03-21T19:21:16.772+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:17 smithi079 ceph-mon[23158]: pgmap v3874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:17 smithi016 ceph-mon[22010]: pgmap v3874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:17 smithi016 conmon[32727]: 2022-03-21T19:21:17.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:18 smithi016 conmon[32727]: 2022-03-21T19:21:18.739+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:19 smithi016 ceph-mon[22010]: pgmap v3875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:19 smithi079 ceph-mon[23158]: pgmap v3875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:19 smithi016 conmon[32727]: 2022-03-21T19:21:19.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:20 smithi016 ceph-mon[22010]: pgmap v3876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:20 smithi016 conmon[32727]: 2022-03-21T19:21:20.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:20 smithi079 ceph-mon[23158]: pgmap v3876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:21 smithi016 conmon[32727]: 2022-03-21T19:21:21.694+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:21 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:21 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:22 smithi016 conmon[32727]: 2022-03-21T19:21:22.738+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:22 smithi016 ceph-mon[22010]: pgmap v3877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:22 smithi079 ceph-mon[23158]: pgmap v3877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:23 smithi016 conmon[32727]: 2022-03-21T19:21:23.726+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:23.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:24 smithi016 ceph-mon[22010]: pgmap v3878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:24 smithi016 conmon[32727]: 2022-03-21T19:21:24.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:24 smithi079 ceph-mon[23158]: pgmap v3878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:25 smithi016 conmon[32727]: 2022-03-21T19:21:25.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:26 smithi016 ceph-mon[22010]: pgmap v3879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:26.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:26 smithi016 conmon[32727]: 2022-03-21T19:21:26.680+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:26 smithi079 ceph-mon[23158]: pgmap v3879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:27.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:27 smithi016 conmon[32727]: 2022-03-21T19:21:27.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:28 smithi016 ceph-mon[22010]: pgmap v3880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:28.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:28 smithi016 conmon[32727]: 2022-03-21T19:21:28.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:28 smithi079 ceph-mon[23158]: pgmap v3880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:29.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:29 smithi016 conmon[32727]: 2022-03-21T19:21:29.662+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:30 smithi016 ceph-mon[22010]: pgmap v3881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:30.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:30 smithi016 conmon[32727]: 2022-03-21T19:21:30.695+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:30 smithi079 ceph-mon[23158]: pgmap v3881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:31.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:31 smithi016 conmon[32727]: 2022-03-21T19:21:31.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:32 smithi016 ceph-mon[22010]: pgmap v3882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:32.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:32 smithi016 conmon[32727]: 2022-03-21T19:21:32.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:32 smithi079 ceph-mon[23158]: pgmap v3882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:33 smithi016 conmon[32727]: 2022-03-21T19:21:33.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:34 smithi016 conmon[32727]: 2022-03-21T19:21:34.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:34 smithi016 ceph-mon[22010]: pgmap v3883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:34 smithi079 ceph-mon[23158]: pgmap v3883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:35 smithi016 conmon[32727]: 2022-03-21T19:21:35.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:36 smithi016 ceph-mon[22010]: pgmap v3884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:36 smithi016 conmon[32727]: 2022-03-21T19:21:36.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:36 smithi079 ceph-mon[23158]: pgmap v3884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:37 smithi016 conmon[32727]: 2022-03-21T19:21:37.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:38 smithi016 ceph-mon[22010]: pgmap v3885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:38.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:38 smithi016 conmon[32727]: 2022-03-21T19:21:38.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:38 smithi079 ceph-mon[23158]: pgmap v3885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:39 smithi016 conmon[32727]: 2022-03-21T19:21:39.680+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:40 smithi016 ceph-mon[22010]: pgmap v3886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:40 smithi016 conmon[32727]: 2022-03-21T19:21:40.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:40 smithi079 ceph-mon[23158]: pgmap v3886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:41.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:41 smithi016 conmon[32727]: 2022-03-21T19:21:41.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:42 smithi016 conmon[32727]: 2022-03-21T19:21:42.569+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:42.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:42.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:42 smithi016 ceph-mon[22010]: pgmap v3887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:42 smithi079 ceph-mon[23158]: pgmap v3887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:43 smithi016 conmon[32727]: 2022-03-21T19:21:43.553+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:44 smithi016 conmon[32727]: 2022-03-21T19:21:44.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:44 smithi016 ceph-mon[22010]: pgmap v3888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:44 smithi079 ceph-mon[23158]: pgmap v3888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:45 smithi016 conmon[32727]: 2022-03-21T19:21:45.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:46 smithi016 conmon[32727]: 2022-03-21T19:21:46.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:46.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:46 smithi016 ceph-mon[22010]: pgmap v3889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:46 smithi079 ceph-mon[23158]: pgmap v3889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:47 smithi016 conmon[32727]: 2022-03-21T19:21:47.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:48 smithi016 ceph-mon[22010]: pgmap v3890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:48 smithi016 conmon[32727]: 2022-03-21T19:21:48.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:48 smithi079 ceph-mon[23158]: pgmap v3890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:49 smithi016 conmon[32727]: 2022-03-21T19:21:49.460+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:51.065 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:50 smithi016 ceph-mon[22010]: pgmap v3891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:51.066 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:50 smithi016 conmon[32727]: 2022-03-21T19:21:50.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:51.066 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:51.067 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:50 smithi079 ceph-mon[23158]: pgmap v3891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:51.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:51 smithi016 conmon[32727]: 2022-03-21T19:21:51.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:52 smithi016 ceph-mon[22010]: pgmap v3892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:52.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:52 smithi016 conmon[32727]: 2022-03-21T19:21:52.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:52 smithi079 ceph-mon[23158]: pgmap v3892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:53.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:53 smithi016 conmon[32727]: 2022-03-21T19:21:53.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:54 smithi016 conmon[32727]: 2022-03-21T19:21:54.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: pgmap v3893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:21:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:21:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:21:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:54.931 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:54.931 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: pgmap v3893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:21:54.932 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:21:54.932 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:21:54.933 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:21:54.933 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:55.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: 2022-03-21T19:21:55.084+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:21:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:55.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: yield 2022-03-21T19:21:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:55.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:21:55.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:21:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:55.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:21:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:21:55.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:21:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:55.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:55.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:55 smithi016 conmon[32727]: 2022-03-21T19:21:55.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:21:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:21:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:55.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:21:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:55.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:21:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:21:55.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:55.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:21:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:55.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:55 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:21:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:21:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:21:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:55.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:21:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:55.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:21:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:21:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:21:55.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:55.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:55 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:21:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:56 smithi016 ceph-mon[22010]: pgmap v3894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:56.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:56 smithi016 conmon[32727]: 2022-03-21T19:21:56.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:56 smithi079 ceph-mon[23158]: pgmap v3894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:57.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:57 smithi016 conmon[32727]: 2022-03-21T19:21:57.523+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:58.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 2022-03-21T19:21:58.156+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:21:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: yield 2022-03-21T19:21:58.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:21:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:21:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:21:58.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:21:58.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 2022-03-21T19:21:58.157+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:21:58.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:21:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: yield 2022-03-21T19:21:58.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:21:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:21:58.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:21:58.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:21:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 2022-03-21T19:21:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:21:58.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: 2022-03-21T19:21:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:21:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:21:58.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:21:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:21:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:21:58.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:21:58.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:21:58.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:21:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:21:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:21:58.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:21:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:21:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:21:58.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:21:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:21:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:21:58.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:21:58.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:21:58.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:21:58.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:21:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:21:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:21:58.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise 2022-03-21T19:21:58.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:21:58.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:21:58.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:21:58.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:21:58.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:58 smithi016 conmon[32727]: 2022-03-21T19:21:58.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:21:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:21:58.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:21:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:21:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:21:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:21:58.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:21:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:21:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:21:58.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:21:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:21:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:21:58.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:21:58.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:21:58.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:21:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:21:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:21:58.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:21:58.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:21:58.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:21:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:21:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:21:58.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:21:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:21:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:21:58.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:21:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:21:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:21:58.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:21:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:21:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:21:58.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:21:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:21:58.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:21:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:21:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:21:58.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:21:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:21:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:21:58.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:21:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: pgmap v3895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:58.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:21:58.942 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:21:58.942 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:21:58.942 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:21:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:21:58.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:21:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:21:58.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:21:58.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:21:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:21:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:21:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:21:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:21:58.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:21:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:21:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:21:58.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:21:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:21:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:21:58.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:21:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:21:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:21:58.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:21:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:21:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:21:58.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:21:58.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:21:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:21:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:21:58.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:21:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:21:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:21:58.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:21:58.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:21:58.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:21:58.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:21:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:21:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:21:58.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:21:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:21:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:21:58.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:21:58.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:21:58.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:21:58.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:21:58.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:21:58.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:21:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:21:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:21:58.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:21:58.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:21:58.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:21:58.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:21:58.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:21:58.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:21:58.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:21:58.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:21:58.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: pgmap v3895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:21:58.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:21:58.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:21:58.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:21:59.000 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:21:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:21:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:21:59.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:21:59 smithi016 conmon[32727]: 2022-03-21T19:21:59.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:21:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:21:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:21:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:00 smithi016 ceph-mon[22010]: pgmap v3896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:00 smithi016 conmon[32727]: 2022-03-21T19:22:00.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:00 smithi079 ceph-mon[23158]: pgmap v3896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:01 smithi016 conmon[32727]: 2022-03-21T19:22:01.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:02 smithi016 ceph-mon[22010]: pgmap v3897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:02.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:02 smithi016 conmon[32727]: 2022-03-21T19:22:02.456+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:02 smithi079 ceph-mon[23158]: pgmap v3897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:03.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:03 smithi016 conmon[32727]: 2022-03-21T19:22:03.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:03.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:04 smithi016 ceph-mon[22010]: pgmap v3898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:04.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:04 smithi016 conmon[32727]: 2022-03-21T19:22:04.442+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:04 smithi079 ceph-mon[23158]: pgmap v3898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:05.691 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:05 smithi016 conmon[32727]: 2022-03-21T19:22:05.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:06.696 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:06 smithi016 conmon[32727]: 2022-03-21T19:22:06.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:06 smithi079 ceph-mon[23158]: pgmap v3899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:06 smithi016 ceph-mon[22010]: pgmap v3899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:07.698 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:07 smithi016 conmon[32727]: 2022-03-21T19:22:07.440+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:08.703 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:08 smithi016 conmon[32727]: 2022-03-21T19:22:08.416+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:08 smithi079 ceph-mon[23158]: pgmap v3900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:08 smithi016 ceph-mon[22010]: pgmap v3900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:09 smithi016 conmon[32727]: 2022-03-21T19:22:09.400+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:10.712 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:10 smithi016 conmon[32727]: 2022-03-21T19:22:10.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:10 smithi079 ceph-mon[23158]: pgmap v3901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:10 smithi016 ceph-mon[22010]: pgmap v3901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:11 smithi016 conmon[32727]: 2022-03-21T19:22:11.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:12 smithi016 conmon[32727]: 2022-03-21T19:22:12.406+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:12 smithi079 ceph-mon[23158]: pgmap v3902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:12 smithi016 ceph-mon[22010]: pgmap v3902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:13 smithi016 conmon[32727]: 2022-03-21T19:22:13.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:14.723 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:14 smithi016 conmon[32727]: 2022-03-21T19:22:14.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:14 smithi016 ceph-mon[22010]: pgmap v3903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:14 smithi079 ceph-mon[23158]: pgmap v3903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:15.725 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:15 smithi016 conmon[32727]: 2022-03-21T19:22:15.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:16 smithi016 ceph-mon[22010]: pgmap v3904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:16 smithi016 conmon[32727]: 2022-03-21T19:22:16.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:16 smithi079 ceph-mon[23158]: pgmap v3904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:17.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:17 smithi016 conmon[32727]: 2022-03-21T19:22:17.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:18.737 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:18 smithi016 conmon[32727]: 2022-03-21T19:22:18.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:18 smithi016 ceph-mon[22010]: pgmap v3905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:18 smithi079 ceph-mon[23158]: pgmap v3905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:19.738 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:19 smithi016 conmon[32727]: 2022-03-21T19:22:19.475+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:20 smithi016 ceph-mon[22010]: pgmap v3906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:20 smithi016 conmon[32727]: 2022-03-21T19:22:20.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:20 smithi079 ceph-mon[23158]: pgmap v3906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:21.745 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:21 smithi016 conmon[32727]: 2022-03-21T19:22:21.460+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:22.183 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:22.750 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:22 smithi016 conmon[32727]: 2022-03-21T19:22:22.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:22 smithi016 ceph-mon[22010]: pgmap v3907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:22 smithi079 ceph-mon[23158]: pgmap v3907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:23.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:23 smithi016 conmon[32727]: 2022-03-21T19:22:23.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:24 smithi016 conmon[32727]: 2022-03-21T19:22:24.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:24 smithi016 ceph-mon[22010]: pgmap v3908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:24.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:24 smithi079 ceph-mon[23158]: pgmap v3908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:25.758 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:25 smithi016 conmon[32727]: 2022-03-21T19:22:25.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:26.763 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:26 smithi016 conmon[32727]: 2022-03-21T19:22:26.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:26 smithi016 ceph-mon[22010]: pgmap v3909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:26 smithi079 ceph-mon[23158]: pgmap v3909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:27.764 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:27 smithi016 conmon[32727]: 2022-03-21T19:22:27.497+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:28.769 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:28 smithi016 conmon[32727]: 2022-03-21T19:22:28.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:28 smithi016 ceph-mon[22010]: pgmap v3910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:28 smithi079 ceph-mon[23158]: pgmap v3910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:29.771 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:29 smithi016 conmon[32727]: 2022-03-21T19:22:29.499+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:30.776 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:30 smithi016 conmon[32727]: 2022-03-21T19:22:30.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:30 smithi016 ceph-mon[22010]: pgmap v3911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:31.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:31.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:30 smithi079 ceph-mon[23158]: pgmap v3911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:31.778 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:31 smithi016 conmon[32727]: 2022-03-21T19:22:31.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:32.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:32 smithi016 ceph-mon[22010]: pgmap v3912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:32.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:32 smithi016 conmon[32727]: 2022-03-21T19:22:32.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:32 smithi079 ceph-mon[23158]: pgmap v3912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:33.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:33 smithi016 conmon[32727]: 2022-03-21T19:22:33.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:34.789 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:34 smithi016 conmon[32727]: 2022-03-21T19:22:34.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:34 smithi016 ceph-mon[22010]: pgmap v3913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:35.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:35.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:34 smithi079 ceph-mon[23158]: pgmap v3913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:35.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:35.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:35.920 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:35 smithi016 conmon[32727]: 2022-03-21T19:22:35.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:36.796 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:36 smithi016 conmon[32727]: 2022-03-21T19:22:36.524+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:36 smithi016 ceph-mon[22010]: pgmap v3914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:36 smithi079 ceph-mon[23158]: pgmap v3914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:37 smithi016 conmon[32727]: 2022-03-21T19:22:37.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:37.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:38.803 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:38 smithi016 conmon[32727]: 2022-03-21T19:22:38.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:38 smithi016 ceph-mon[22010]: pgmap v3915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:38 smithi079 ceph-mon[23158]: pgmap v3915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:39.804 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:39 smithi016 conmon[32727]: 2022-03-21T19:22:39.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:40.810 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:40 smithi016 conmon[32727]: 2022-03-21T19:22:40.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:40 smithi016 ceph-mon[22010]: pgmap v3916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:40 smithi079 ceph-mon[23158]: pgmap v3916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:41.811 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:41 smithi016 conmon[32727]: 2022-03-21T19:22:41.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:42.816 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:42 smithi016 conmon[32727]: 2022-03-21T19:22:42.497+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:42 smithi016 ceph-mon[22010]: pgmap v3917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:43.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:42 smithi079 ceph-mon[23158]: pgmap v3917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:43.818 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:43 smithi016 conmon[32727]: 2022-03-21T19:22:43.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:44.825 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:44 smithi016 conmon[32727]: 2022-03-21T19:22:44.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:44 smithi016 ceph-mon[22010]: pgmap v3918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:44 smithi079 ceph-mon[23158]: pgmap v3918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:45.827 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:45 smithi016 conmon[32727]: 2022-03-21T19:22:45.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:46.831 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:46 smithi016 conmon[32727]: 2022-03-21T19:22:46.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:46 smithi016 ceph-mon[22010]: pgmap v3919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:46 smithi079 ceph-mon[23158]: pgmap v3919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:47.833 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:47 smithi016 conmon[32727]: 2022-03-21T19:22:47.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:48 smithi016 conmon[32727]: 2022-03-21T19:22:48.409+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:48 smithi016 ceph-mon[22010]: pgmap v3920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:48 smithi079 ceph-mon[23158]: pgmap v3920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:49.839 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:49 smithi016 conmon[32727]: 2022-03-21T19:22:49.417+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:50 smithi016 conmon[32727]: 2022-03-21T19:22:50.372+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:50 smithi016 ceph-mon[22010]: pgmap v3921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:50 smithi079 ceph-mon[23158]: pgmap v3921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:51 smithi016 conmon[32727]: 2022-03-21T19:22:51.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:52 smithi016 conmon[32727]: 2022-03-21T19:22:52.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:52 smithi016 ceph-mon[22010]: pgmap v3922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:52 smithi079 ceph-mon[23158]: pgmap v3922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:53 smithi016 conmon[32727]: 2022-03-21T19:22:53.359+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:54 smithi016 conmon[32727]: 2022-03-21T19:22:54.390+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:54 smithi016 ceph-mon[22010]: pgmap v3923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:54 smithi079 ceph-mon[23158]: pgmap v3923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:22:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:22:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:55 smithi016 conmon[32727]: 2022-03-21T19:22:55.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:56 smithi016 conmon[32727]: 2022-03-21T19:22:56.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:56 smithi016 ceph-mon[22010]: pgmap v3924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:56 smithi079 ceph-mon[23158]: pgmap v3924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:57 smithi016 conmon[32727]: 2022-03-21T19:22:57.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:58.207 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:58 smithi016 conmon[32727]: 2022-03-21T19:22:58.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:22:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:22:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:22:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:22:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: pgmap v3925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:22:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:22:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:22:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:22:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:22:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:22:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:22:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:22:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: pgmap v3925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:22:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:22:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:22:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:22:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:22:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:22:59 smithi016 conmon[32727]: 2022-03-21T19:22:59.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:22:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:22:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:00 smithi016 conmon[32727]: 2022-03-21T19:23:00.294+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:00 smithi079 ceph-mon[23158]: pgmap v3926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:00 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:23:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:00 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:23:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:00 smithi016 ceph-mon[22010]: pgmap v3926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:00 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:23:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:00 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:23:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[32727]: 2022-03-21T19:23:01.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:01.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: 2022-03-21T19:23:01.260+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:01.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:23:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: yield 2022-03-21T19:23:01.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:01.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:23:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:23:01.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:23:01.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:23:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:01.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:23:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:01.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:23:02.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:23:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:23:02.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:23:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:23:02.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:23:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:02.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:02.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:02.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:23:02.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:02.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:23:02.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:02.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:23:02.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:02.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:02.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:02.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:23:02.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:02.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:02.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:02.180 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:23:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:23:02.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:23:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:23:02.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 198.1M 2022-03-21T19:23:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-03-21T19:23:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:02.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:23:02.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:02.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:23:02.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:02.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:23:02.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:23:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:02.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:23:02.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:02.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:23:02.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:02.241 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:23:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:02 smithi016 conmon[32727]: 2022-03-21T19:23:02.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:02 smithi016 ceph-mon[22010]: pgmap v3927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:02 smithi079 ceph-mon[23158]: pgmap v3927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:03 smithi016 conmon[32727]: 2022-03-21T19:23:03.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[32727]: 2022-03-21T19:23:04.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 2022-03-21T19:23:04.332+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:04.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:23:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: yield 2022-03-21T19:23:04.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:04.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:23:04.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:23:04.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:23:04.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:23:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:04.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:23:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:04.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 2022-03-21T19:23:04.333+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:23:04.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:23:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: yield 2022-03-21T19:23:04.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:04.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:23:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:23:04.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:23:04.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:23:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:04.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:23:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:04.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 2022-03-21T19:23:04.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:23:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: 2022-03-21T19:23:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:23:04.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:23:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:23:04.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:23:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:23:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:23:04.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:23:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:23:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:23:04.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:23:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:23:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:23:04.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:23:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:23:04.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:23:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:23:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:23:04.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:23:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:23:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:23:04.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:23:04.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:23:04.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise 2022-03-21T19:23:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:23:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:23:04.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:23:04.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:23:04.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:23:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:04 smithi016 ceph-mon[22010]: pgmap v3928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:04 smithi079 ceph-mon[23158]: pgmap v3928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:05 smithi016 conmon[32727]: 2022-03-21T19:23:05.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:23:06.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:06.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:23:06.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:23:06.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:23:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:06.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:23:06.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:06.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:06.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:05 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:23:06.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:23:06.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:06.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:06.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:23:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:06.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:23:06.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:06.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:23:06.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:06.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:23:06.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:23:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:23:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:23:06.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:23:06.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:23:06.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:23:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:23:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:23:06.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:23:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:23:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:23:06.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:23:06.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:23:06.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:23:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:23:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:23:06.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:23:06.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:23:06.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:23:06.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:23:06.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:23:06.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:23:06.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:23:06.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:23:06.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:23:06.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:23:06.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:23:06.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:23:06.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:23:06.194 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.194 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:23:06.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:06.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:23:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:23:06.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:23:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:06.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:23:06.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:06.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:05 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:23:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:23:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:23:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:23:06.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:23:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:23:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:23:06.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:23:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:23:06.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:23:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:23:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:23:06.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:23:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:23:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:23:06.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:23:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:23:06.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:23:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:23:06.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:23:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:23:06.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:23:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:23:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:23:06.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:23:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:23:06.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:23:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:23:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:23:06.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:23:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:23:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:23:06.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:23:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:23:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:23:06.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:23:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:23:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:23:06.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:23:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:23:06.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:23:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:23:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:23:06.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:23:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:23:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:23:06.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:23:06.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:23:06.250 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:06 smithi016 conmon[32727]: 2022-03-21T19:23:06.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:06 smithi016 ceph-mon[22010]: pgmap v3929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:07.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:07.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:06 smithi079 ceph-mon[23158]: pgmap v3929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:07 smithi016 conmon[32727]: 2022-03-21T19:23:07.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:08 smithi016 conmon[32727]: 2022-03-21T19:23:08.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:08 smithi016 ceph-mon[22010]: pgmap v3930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:08 smithi079 ceph-mon[23158]: pgmap v3930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:09 smithi016 conmon[32727]: 2022-03-21T19:23:09.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:10 smithi016 conmon[32727]: 2022-03-21T19:23:10.309+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:10 smithi016 ceph-mon[22010]: pgmap v3931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:10 smithi079 ceph-mon[23158]: pgmap v3931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:11 smithi016 conmon[32727]: 2022-03-21T19:23:11.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:12 smithi016 conmon[32727]: 2022-03-21T19:23:12.239+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:12 smithi016 ceph-mon[22010]: pgmap v3932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:12 smithi079 ceph-mon[23158]: pgmap v3932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:13 smithi016 conmon[32727]: 2022-03-21T19:23:13.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:14.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:14.262 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:14 smithi016 conmon[32727]: 2022-03-21T19:23:14.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:14 smithi079 ceph-mon[23158]: pgmap v3933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:15.240 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:15.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:14 smithi016 ceph-mon[22010]: pgmap v3933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:15.241 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:15 smithi016 conmon[32727]: 2022-03-21T19:23:15.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:16.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:16 smithi016 conmon[32727]: 2022-03-21T19:23:16.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:17.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:17.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:16 smithi016 ceph-mon[22010]: pgmap v3934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:16 smithi079 ceph-mon[23158]: pgmap v3934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:17 smithi016 conmon[32727]: 2022-03-21T19:23:17.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:18 smithi016 conmon[32727]: 2022-03-21T19:23:18.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:19.201 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:19.201 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:18 smithi016 ceph-mon[22010]: pgmap v3935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:18 smithi079 ceph-mon[23158]: pgmap v3935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:19 smithi016 conmon[32727]: 2022-03-21T19:23:19.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:20.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:20.236 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:20 smithi016 conmon[32727]: 2022-03-21T19:23:20.233+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:20 smithi079 ceph-mon[23158]: pgmap v3936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:21.267 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:21.268 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:20 smithi016 ceph-mon[22010]: pgmap v3936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:21 smithi016 conmon[32727]: 2022-03-21T19:23:21.265+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:22.249 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:22 smithi016 conmon[32727]: 2022-03-21T19:23:22.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:23.215 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:23.216 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:22 smithi016 ceph-mon[22010]: pgmap v3937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:22 smithi079 ceph-mon[23158]: pgmap v3937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:23 smithi016 conmon[32727]: 2022-03-21T19:23:23.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:24.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:24 smithi016 conmon[32727]: 2022-03-21T19:23:24.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:24 smithi079 ceph-mon[23158]: pgmap v3938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:25 smithi016 conmon[32727]: 2022-03-21T19:23:25.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:24 smithi016 ceph-mon[22010]: pgmap v3938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:25.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:26.227 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:26 smithi016 conmon[32727]: 2022-03-21T19:23:26.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:26 smithi079 ceph-mon[23158]: pgmap v3939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:27.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:27.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:26 smithi016 ceph-mon[22010]: pgmap v3939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:27 smithi016 conmon[32727]: 2022-03-21T19:23:27.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:28.280 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:28 smithi016 conmon[32727]: 2022-03-21T19:23:28.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:28 smithi079 ceph-mon[23158]: pgmap v3940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:29.279 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:29.279 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:28 smithi016 ceph-mon[22010]: pgmap v3940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:29 smithi016 conmon[32727]: 2022-03-21T19:23:29.276+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:30.233 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:30.233 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:30 smithi016 conmon[32727]: 2022-03-21T19:23:30.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:30 smithi016 ceph-mon[22010]: pgmap v3941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:31.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:31 smithi016 conmon[32727]: 2022-03-21T19:23:31.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:30 smithi079 ceph-mon[23158]: pgmap v3941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:32 smithi016 conmon[32727]: 2022-03-21T19:23:32.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:32 smithi016 ceph-mon[22010]: pgmap v3942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:33 smithi016 conmon[32727]: 2022-03-21T19:23:33.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:32 smithi079 ceph-mon[23158]: pgmap v3942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:34.256 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:34 smithi016 conmon[32727]: 2022-03-21T19:23:34.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:34 smithi016 ceph-mon[22010]: pgmap v3943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:35 smithi016 conmon[32727]: 2022-03-21T19:23:35.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:34 smithi079 ceph-mon[23158]: pgmap v3943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:36 smithi016 conmon[32727]: 2022-03-21T19:23:36.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:37.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:36 smithi016 ceph-mon[22010]: pgmap v3944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:37 smithi016 conmon[32727]: 2022-03-21T19:23:37.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:36 smithi079 ceph-mon[23158]: pgmap v3944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:38 smithi016 conmon[32727]: 2022-03-21T19:23:38.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:39 smithi016 conmon[32727]: 2022-03-21T19:23:39.122+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:39.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:38 smithi016 ceph-mon[22010]: pgmap v3945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:38 smithi079 ceph-mon[23158]: pgmap v3945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:40 smithi016 conmon[32727]: 2022-03-21T19:23:40.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:41 smithi016 conmon[32727]: 2022-03-21T19:23:41.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:41 smithi016 ceph-mon[22010]: pgmap v3946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:41 smithi079 ceph-mon[23158]: pgmap v3946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:42 smithi016 conmon[32727]: 2022-03-21T19:23:42.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:43.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:43 smithi016 ceph-mon[22010]: pgmap v3947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:43.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:43 smithi016 conmon[32727]: 2022-03-21T19:23:43.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:43 smithi079 ceph-mon[23158]: pgmap v3947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:44 smithi016 conmon[32727]: 2022-03-21T19:23:44.219+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:45 smithi016 ceph-mon[22010]: pgmap v3948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:45.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:45.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:45 smithi016 conmon[32727]: 2022-03-21T19:23:45.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:45 smithi079 ceph-mon[23158]: pgmap v3948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:46 smithi016 conmon[32727]: 2022-03-21T19:23:46.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:47 smithi016 ceph-mon[22010]: pgmap v3949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:47.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:47 smithi016 conmon[32727]: 2022-03-21T19:23:47.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:47 smithi079 ceph-mon[23158]: pgmap v3949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:48.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:48 smithi016 conmon[32727]: 2022-03-21T19:23:48.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:49 smithi016 ceph-mon[22010]: pgmap v3950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:49.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:49 smithi016 conmon[32727]: 2022-03-21T19:23:49.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:49 smithi079 ceph-mon[23158]: pgmap v3950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:50.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:50 smithi016 conmon[32727]: 2022-03-21T19:23:50.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:51.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:51 smithi016 ceph-mon[22010]: pgmap v3951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:51.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:51 smithi016 conmon[32727]: 2022-03-21T19:23:51.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:51 smithi079 ceph-mon[23158]: pgmap v3951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:52 smithi016 conmon[32727]: 2022-03-21T19:23:52.098+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:53 smithi016 ceph-mon[22010]: pgmap v3952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:53.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:53 smithi016 conmon[32727]: 2022-03-21T19:23:53.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:53 smithi079 ceph-mon[23158]: pgmap v3952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:54.501 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:54 smithi016 conmon[32727]: 2022-03-21T19:23:54.091+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:55 smithi016 ceph-mon[22010]: pgmap v3953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:55.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:55 smithi016 conmon[32727]: 2022-03-21T19:23:55.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:55 smithi079 ceph-mon[23158]: pgmap v3953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:23:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:23:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:56 smithi016 ceph-mon[22010]: pgmap v3954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:56.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:56 smithi016 conmon[32727]: 2022-03-21T19:23:56.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:56 smithi079 ceph-mon[23158]: pgmap v3954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:57 smithi016 conmon[32727]: 2022-03-21T19:23:57.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:58 smithi016 conmon[32727]: 2022-03-21T19:23:58.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:23:59 smithi016 conmon[32727]: 2022-03-21T19:23:59.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:23:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: pgmap v3955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:23:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:23:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:23:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:23:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:23:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:23:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: pgmap v3955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:23:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:23:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:23:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:23:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:23:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:23:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:00 smithi016 conmon[32727]: 2022-03-21T19:24:00.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:00 smithi016 ceph-mon[22010]: pgmap v3956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:00 smithi079 ceph-mon[23158]: pgmap v3956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:01.586 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:01 smithi016 conmon[32727]: 2022-03-21T19:24:01.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:01 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:01 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:02.591 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:02 smithi016 conmon[32727]: 2022-03-21T19:24:02.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:02 smithi016 ceph-mon[22010]: pgmap v3957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:02 smithi079 ceph-mon[23158]: pgmap v3957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:03.538 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:03 smithi016 conmon[32727]: 2022-03-21T19:24:03.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:03.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:04.597 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:04 smithi016 conmon[32727]: 2022-03-21T19:24:04.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:04 smithi079 ceph-mon[23158]: pgmap v3958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:24:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:24:04.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:24:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:04 smithi016 ceph-mon[22010]: pgmap v3958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:24:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:24:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:24:05.598 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:05 smithi016 conmon[32727]: 2022-03-21T19:24:05.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:06.604 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[32727]: 2022-03-21T19:24:06.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:06 smithi016 ceph-mon[22010]: pgmap v3959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:06.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: 2022-03-21T19:24:06.605+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:06.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:24:06.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: yield 2022-03-21T19:24:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:06.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:24:06.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:06.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:24:06.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:06.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:24:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:24:06.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:06.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:24:06.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:06.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:06.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:06 smithi079 ceph-mon[23158]: pgmap v3959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:07.607 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:07 smithi016 conmon[32727]: 2022-03-21T19:24:07.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:07 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:24:07.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:07.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:24:07.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:07.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:07.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:07.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:24:07.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:07.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:24:07.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:07.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:24:07.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:07.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:07.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:07.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:24:07.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:07.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:07.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:07.926 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:07 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:24:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:07 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:24:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:24:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:07.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:07.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:07.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:24:07.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:07.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:24:07.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:07.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:24:07.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:07.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:07.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:07.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:24:07.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:07.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:07.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:07.986 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:07 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:24:08.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:08 smithi016 conmon[32727]: 2022-03-21T19:24:08.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:08 smithi016 ceph-mon[22010]: pgmap v3960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:08 smithi079 ceph-mon[23158]: pgmap v3960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[32727]: 2022-03-21T19:24:09.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:09.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 2022-03-21T19:24:09.677+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:09.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:24:09.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:09.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: yield 2022-03-21T19:24:09.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:09.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:09.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:09.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:24:09.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:09.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:24:09.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:09.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:24:09.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:09.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:24:09.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:09.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:24:09.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:09.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:09.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:09.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 2022-03-21T19:24:09.678+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:24:09.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:24:09.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:09.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: yield 2022-03-21T19:24:09.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:09.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:09.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:09.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:24:09.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:09.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:24:09.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:09.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:24:09.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:09.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:24:09.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:09.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:24:09.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:09.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:09.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:09.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 2022-03-21T19:24:09.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:24:09.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: 2022-03-21T19:24:09.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:24:09.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:24:09.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:24:09.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:24:09.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:24:09.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:24:09.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:24:09.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:24:09.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:24:09.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:24:09.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:24:09.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:24:09.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:24:09.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:24:09.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:24:09.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:24:09.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:24:09.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:24:09.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:24:09.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:24:09.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:24:09.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:24:09.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise 2022-03-21T19:24:09.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:24:09.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:24:09.960 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:24:09.961 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:24:09.961 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:24:09 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:09.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:10 smithi016 conmon[32727]: 2022-03-21T19:24:10.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:24:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:24:10.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:24:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:10.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:24:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:10.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:10.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:24:10.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:10.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:10.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:10 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:24:10.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:10.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:24:10.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:10.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:10.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:10.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:24:10.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:10.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:24:10.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:10.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:24:10.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:10.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:10.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:10.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:24:10.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:10.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:10.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:24:10.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:24:10.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:24:10.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:24:10.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:24:10.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:24:10.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:24:10.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:24:10.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:24:10.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:24:10.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:24:10.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:24:10.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:24:10.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:24:10.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:24:10.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:24:10.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:24:10.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:24:10.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:24:10.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:24:10.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:24:10.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:24:10.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:24:10.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:24:10.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:24:10.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:24:10.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:24:10.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:24:10.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:24:10.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:10.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:10 smithi016 ceph-mon[22010]: pgmap v3961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:24:10.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:10.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:24:10.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:24:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:10.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:24:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:10.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:10.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:24:10.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:10.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:10.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:10 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:24:10.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:24:10.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:24:10.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:24:10.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:24:10.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:24:10.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:24:10.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:24:10.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:24:10.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:24:10.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:24:10.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:24:10.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:24:10.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:24:10.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:24:10.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:24:10.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:24:10.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:24:10.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:24:10.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:24:10.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:24:10.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:24:10.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:24:10.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:24:10.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:24:10.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:24:10.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:24:10.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:24:10.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:24:10.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:24:10.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:24:10.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:24:10.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:24:10.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:24:10.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:24:10.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:24:10.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:24:10.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:24:10.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:24:10.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:24:10.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:24:10.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:24:10.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:24:10.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:24:10.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:24:10.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:24:10.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:24:10.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:24:10.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:10.999 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:10 smithi079 ceph-mon[23158]: pgmap v3961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:11 smithi016 conmon[32727]: 2022-03-21T19:24:11.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:12 smithi016 conmon[32727]: 2022-03-21T19:24:12.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:12.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:12 smithi016 ceph-mon[22010]: pgmap v3962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:12 smithi079 ceph-mon[23158]: pgmap v3962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:13 smithi016 conmon[32727]: 2022-03-21T19:24:13.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:14 smithi016 conmon[32727]: 2022-03-21T19:24:14.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:14 smithi016 ceph-mon[22010]: pgmap v3963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:14 smithi079 ceph-mon[23158]: pgmap v3963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:15 smithi016 conmon[32727]: 2022-03-21T19:24:15.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:16 smithi016 conmon[32727]: 2022-03-21T19:24:16.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:16 smithi016 ceph-mon[22010]: pgmap v3964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:16 smithi079 ceph-mon[23158]: pgmap v3964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:17 smithi016 conmon[32727]: 2022-03-21T19:24:17.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:18 smithi016 conmon[32727]: 2022-03-21T19:24:18.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:18 smithi016 ceph-mon[22010]: pgmap v3965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:18 smithi079 ceph-mon[23158]: pgmap v3965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:19 smithi016 conmon[32727]: 2022-03-21T19:24:19.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:20 smithi016 conmon[32727]: 2022-03-21T19:24:20.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:20 smithi016 ceph-mon[22010]: pgmap v3966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:20 smithi079 ceph-mon[23158]: pgmap v3966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:21 smithi016 conmon[32727]: 2022-03-21T19:24:21.079+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:22 smithi016 conmon[32727]: 2022-03-21T19:24:22.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:22 smithi016 ceph-mon[22010]: pgmap v3967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:22.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:22 smithi079 ceph-mon[23158]: pgmap v3967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:23 smithi016 conmon[32727]: 2022-03-21T19:24:23.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:24 smithi016 conmon[32727]: 2022-03-21T19:24:24.105+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:25.068 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:24 smithi079 ceph-mon[23158]: pgmap v3968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:25.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:25.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:25.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:24 smithi016 ceph-mon[22010]: pgmap v3968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:25.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:25 smithi016 conmon[32727]: 2022-03-21T19:24:25.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:26.103 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:26 smithi016 conmon[32727]: 2022-03-21T19:24:26.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:26 smithi079 ceph-mon[23158]: pgmap v3969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:27.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:27.102 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:26 smithi016 ceph-mon[22010]: pgmap v3969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:27 smithi016 conmon[32727]: 2022-03-21T19:24:27.099+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:28.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:28 smithi016 conmon[32727]: 2022-03-21T19:24:28.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:28 smithi079 ceph-mon[23158]: pgmap v3970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:29.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:29.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:28 smithi016 ceph-mon[22010]: pgmap v3970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:29 smithi016 conmon[32727]: 2022-03-21T19:24:29.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:30.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:30.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:30 smithi016 conmon[32727]: 2022-03-21T19:24:30.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:30 smithi079 ceph-mon[23158]: pgmap v3971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:30 smithi016 ceph-mon[22010]: pgmap v3971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:31 smithi016 conmon[32727]: 2022-03-21T19:24:31.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:32 smithi016 conmon[32727]: 2022-03-21T19:24:32.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:32 smithi079 ceph-mon[23158]: pgmap v3972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:32 smithi016 ceph-mon[22010]: pgmap v3972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:33 smithi016 conmon[32727]: 2022-03-21T19:24:33.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:34 smithi016 conmon[32727]: 2022-03-21T19:24:34.181+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:34 smithi079 ceph-mon[23158]: pgmap v3973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:34 smithi016 ceph-mon[22010]: pgmap v3973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:35 smithi016 conmon[32727]: 2022-03-21T19:24:35.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:36 smithi016 conmon[32727]: 2022-03-21T19:24:36.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:36 smithi079 ceph-mon[23158]: pgmap v3974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:37.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:36 smithi016 ceph-mon[22010]: pgmap v3974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:37 smithi016 conmon[32727]: 2022-03-21T19:24:37.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:38 smithi016 conmon[32727]: 2022-03-21T19:24:38.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:38 smithi079 ceph-mon[23158]: pgmap v3975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:38 smithi016 ceph-mon[22010]: pgmap v3975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:39 smithi016 conmon[32727]: 2022-03-21T19:24:39.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:40 smithi016 conmon[32727]: 2022-03-21T19:24:40.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:40 smithi079 ceph-mon[23158]: pgmap v3976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:40 smithi016 ceph-mon[22010]: pgmap v3976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:41 smithi016 conmon[32727]: 2022-03-21T19:24:41.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:42 smithi016 conmon[32727]: 2022-03-21T19:24:42.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:42 smithi016 ceph-mon[22010]: pgmap v3977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:42 smithi079 ceph-mon[23158]: pgmap v3977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:43 smithi016 conmon[32727]: 2022-03-21T19:24:43.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:44 smithi016 conmon[32727]: 2022-03-21T19:24:44.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:44 smithi016 ceph-mon[22010]: pgmap v3978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:44 smithi079 ceph-mon[23158]: pgmap v3978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:45 smithi016 conmon[32727]: 2022-03-21T19:24:45.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:46 smithi016 conmon[32727]: 2022-03-21T19:24:46.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:46 smithi016 ceph-mon[22010]: pgmap v3979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:46 smithi079 ceph-mon[23158]: pgmap v3979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:47 smithi016 conmon[32727]: 2022-03-21T19:24:47.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:48.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:48.157 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:48 smithi016 conmon[32727]: 2022-03-21T19:24:48.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:48 smithi016 ceph-mon[22010]: pgmap v3980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:48 smithi079 ceph-mon[23158]: pgmap v3980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:49 smithi016 conmon[32727]: 2022-03-21T19:24:49.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:50.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:50 smithi016 conmon[32727]: 2022-03-21T19:24:50.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:51.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:51.143 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:51.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:50 smithi016 ceph-mon[22010]: pgmap v3981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:50 smithi079 ceph-mon[23158]: pgmap v3981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:51 smithi016 conmon[32727]: 2022-03-21T19:24:51.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:52.158 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:52 smithi016 conmon[32727]: 2022-03-21T19:24:52.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:53.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:53.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:52 smithi016 ceph-mon[22010]: pgmap v3982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:52 smithi079 ceph-mon[23158]: pgmap v3982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:53 smithi016 conmon[32727]: 2022-03-21T19:24:53.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:54.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:54 smithi016 conmon[32727]: 2022-03-21T19:24:54.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:54 smithi016 ceph-mon[22010]: pgmap v3983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:54 smithi079 ceph-mon[23158]: pgmap v3983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:24:55.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:24:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:55 smithi016 conmon[32727]: 2022-03-21T19:24:55.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:56.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:56.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:56 smithi016 conmon[32727]: 2022-03-21T19:24:56.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:57 smithi016 ceph-mon[22010]: pgmap v3984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:57 smithi016 conmon[32727]: 2022-03-21T19:24:57.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:57 smithi079 ceph-mon[23158]: pgmap v3984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:58 smithi016 conmon[32727]: 2022-03-21T19:24:58.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:24:59 smithi016 conmon[32727]: 2022-03-21T19:24:59.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:24:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: pgmap v3985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:24:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:24:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:24:59.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:24:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:24:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:24:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: pgmap v3985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:24:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:24:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:24:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:24:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:24:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:25:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:00 smithi016 conmon[32727]: 2022-03-21T19:25:00.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:01 smithi016 ceph-mon[22010]: pgmap v3986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:01 smithi016 conmon[32727]: 2022-03-21T19:25:01.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:01 smithi079 ceph-mon[23158]: pgmap v3986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:02 smithi016 conmon[32727]: 2022-03-21T19:25:02.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:03 smithi016 ceph-mon[22010]: pgmap v3987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:03.669 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:03 smithi016 conmon[32727]: 2022-03-21T19:25:03.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:03 smithi079 ceph-mon[23158]: pgmap v3987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:04 smithi016 conmon[32727]: 2022-03-21T19:25:04.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:05 smithi016 ceph-mon[22010]: pgmap v3988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:05 smithi016 conmon[32727]: 2022-03-21T19:25:05.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:05 smithi079 ceph-mon[23158]: pgmap v3988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:06 smithi016 conmon[32727]: 2022-03-21T19:25:06.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:07.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:07.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:07 smithi016 ceph-mon[22010]: pgmap v3989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:07.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:07 smithi016 conmon[32727]: 2022-03-21T19:25:07.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:07 smithi079 ceph-mon[23158]: pgmap v3989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:08 smithi016 ceph-mon[22010]: pgmap v3990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:08 smithi016 conmon[32727]: 2022-03-21T19:25:08.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:08 smithi079 ceph-mon[23158]: pgmap v3990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:09 smithi016 conmon[32727]: 2022-03-21T19:25:09.240+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:10 smithi016 conmon[32727]: 2022-03-21T19:25:10.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:10 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:25:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:10 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:25:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:10 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:25:10.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:10 smithi016 ceph-mon[22010]: pgmap v3991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:10 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:25:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:10 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:25:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:10 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:25:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:10 smithi079 ceph-mon[23158]: pgmap v3991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:11.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[32727]: 2022-03-21T19:25:11.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: 2022-03-21T19:25:11.117+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:25:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:11.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: yield 2022-03-21T19:25:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:11.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:25:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:11.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:25:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:25:11.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:25:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:11.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:25:11.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:11.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:11.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:11 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:25:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:25:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:25:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:25:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:11.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:25:11.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:11.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:11.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:11.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:25:11.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:11.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:11.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.929 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:11 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:25:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:11 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:25:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:11.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:25:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:11.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:25:11.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:25:11.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:25:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:11.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:25:11.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:11.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:11.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:11 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:25:12.608 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:12 smithi016 conmon[32727]: 2022-03-21T19:25:12.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:12.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:12 smithi016 ceph-mon[22010]: pgmap v3992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:12 smithi079 ceph-mon[23158]: pgmap v3992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:13.609 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:13 smithi016 conmon[32727]: 2022-03-21T19:25:13.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:14.613 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[32727]: 2022-03-21T19:25:14.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:14.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 2022-03-21T19:25:14.189+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.615 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:25:14.615 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.616 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: yield 2022-03-21T19:25:14.616 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.616 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.617 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.617 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:25:14.617 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:25:14.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:25:14.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:25:14.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 2022-03-21T19:25:14.190+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:25:14.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:25:14.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: yield 2022-03-21T19:25:14.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:25:14.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:25:14.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:25:14.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:25:14.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 2022-03-21T19:25:14.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:25:14.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: 2022-03-21T19:25:14.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:25:14.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:25:14.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:25:14.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:25:14.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:25:14.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:25:14.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:25:14.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:25:14.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:25:14.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:25:14.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:25:14.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:25:14.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:25:14.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:25:14.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:25:14.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:25:14.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:25:14.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:25:14.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:25:14.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:25:14.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:25:14.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:25:14.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise 2022-03-21T19:25:14.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:25:14.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:25:14.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:25:14.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:25:14.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:25:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:14 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:25:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:25:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:25:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:25:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:25:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:14 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:25:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:25:14.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:25:14.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:25:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:25:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:25:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:25:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:25:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:25:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:25:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:25:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:25:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:25:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:25:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:25:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:25:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:25:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:25:14.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:25:14.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:25:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:25:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:25:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:25:14.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:25:14.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:25:14.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:25:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:25:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:25:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:25:14.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:25:14.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:25:14.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:25:14.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:25:14.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:25:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:25:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:14 smithi016 ceph-mon[22010]: pgmap v3993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:14 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:25:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:25:14.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:25:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:25:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:25:14.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:14 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:25:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:25:14.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:25:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:25:14.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:25:14.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:25:14.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:25:14.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:25:14.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:25:14.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:25:14.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:25:14.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:25:14.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:25:14.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:25:14.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:25:14.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:25:14.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:25:14.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:25:14.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:25:14.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:25:14.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:25:14.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:25:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:25:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:25:14.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:25:14.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:25:14.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:25:14.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:25:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:25:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:25:14.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:25:14.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:25:14.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:25:14.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:25:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:25:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:25:14.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:25:14.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:25:14.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:25:14.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:25:14.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:25:14.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:25:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:25:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:25:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:25:14.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:25:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:25:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:25:14.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:14 smithi079 ceph-mon[23158]: pgmap v3993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:15.615 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:15 smithi016 conmon[32727]: 2022-03-21T19:25:15.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:16.620 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:16 smithi016 conmon[32727]: 2022-03-21T19:25:16.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:16 smithi016 ceph-mon[22010]: pgmap v3994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:16 smithi079 ceph-mon[23158]: pgmap v3994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:17.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:17 smithi016 conmon[32727]: 2022-03-21T19:25:17.204+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:18.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:18 smithi016 conmon[32727]: 2022-03-21T19:25:18.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:18 smithi016 ceph-mon[22010]: pgmap v3995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:18 smithi079 ceph-mon[23158]: pgmap v3995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:19.628 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:19 smithi016 conmon[32727]: 2022-03-21T19:25:19.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:20.633 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:20 smithi016 conmon[32727]: 2022-03-21T19:25:20.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:20 smithi016 ceph-mon[22010]: pgmap v3996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:20 smithi079 ceph-mon[23158]: pgmap v3996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:21.634 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:21 smithi016 conmon[32727]: 2022-03-21T19:25:21.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:22 smithi016 conmon[32727]: 2022-03-21T19:25:22.162+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:22.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:22 smithi016 ceph-mon[22010]: pgmap v3997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:22 smithi079 ceph-mon[23158]: pgmap v3997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:23 smithi016 conmon[32727]: 2022-03-21T19:25:23.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:24.645 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:24 smithi016 conmon[32727]: 2022-03-21T19:25:24.201+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:24 smithi016 ceph-mon[22010]: pgmap v3998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:24.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:24 smithi079 ceph-mon[23158]: pgmap v3998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:25.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:25 smithi016 conmon[32727]: 2022-03-21T19:25:25.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:26.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:26 smithi016 conmon[32727]: 2022-03-21T19:25:26.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:26 smithi016 ceph-mon[22010]: pgmap v3999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:26 smithi079 ceph-mon[23158]: pgmap v3999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:27.653 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:27 smithi016 conmon[32727]: 2022-03-21T19:25:27.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:28.657 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:28 smithi016 conmon[32727]: 2022-03-21T19:25:28.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:28 smithi016 ceph-mon[22010]: pgmap v4000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:28 smithi079 ceph-mon[23158]: pgmap v4000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:29.659 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:29 smithi016 conmon[32727]: 2022-03-21T19:25:29.242+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:30 smithi016 conmon[32727]: 2022-03-21T19:25:30.243+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:30 smithi016 ceph-mon[22010]: pgmap v4001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:30 smithi079 ceph-mon[23158]: pgmap v4001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:31 smithi016 conmon[32727]: 2022-03-21T19:25:31.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:32 smithi016 conmon[32727]: 2022-03-21T19:25:32.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:32 smithi079 ceph-mon[23158]: pgmap v4002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:32 smithi016 ceph-mon[22010]: pgmap v4002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:33 smithi016 conmon[32727]: 2022-03-21T19:25:33.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:34 smithi016 conmon[32727]: 2022-03-21T19:25:34.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:34 smithi079 ceph-mon[23158]: pgmap v4003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:34 smithi016 ceph-mon[22010]: pgmap v4003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:35 smithi016 conmon[32727]: 2022-03-21T19:25:35.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:36 smithi016 conmon[32727]: 2022-03-21T19:25:36.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:36 smithi079 ceph-mon[23158]: pgmap v4004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:37.144 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:37.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:36 smithi016 ceph-mon[22010]: pgmap v4004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:37 smithi016 conmon[32727]: 2022-03-21T19:25:37.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:38.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:38.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:38 smithi016 conmon[32727]: 2022-03-21T19:25:38.145+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:38 smithi079 ceph-mon[23158]: pgmap v4005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:39.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:39.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:38 smithi016 ceph-mon[22010]: pgmap v4005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:39 smithi016 conmon[32727]: 2022-03-21T19:25:39.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:40.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:40.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:40 smithi016 conmon[32727]: 2022-03-21T19:25:40.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:40 smithi079 ceph-mon[23158]: pgmap v4006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:41.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:41.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:40 smithi016 ceph-mon[22010]: pgmap v4006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:41 smithi016 conmon[32727]: 2022-03-21T19:25:41.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:42.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:42 smithi016 conmon[32727]: 2022-03-21T19:25:42.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:42 smithi079 ceph-mon[23158]: pgmap v4007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:43.106 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:43.107 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:42 smithi016 ceph-mon[22010]: pgmap v4007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:43 smithi016 conmon[32727]: 2022-03-21T19:25:43.104+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:44.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:44 smithi016 conmon[32727]: 2022-03-21T19:25:44.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:44 smithi079 ceph-mon[23158]: pgmap v4008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:45.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:45.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:44 smithi016 ceph-mon[22010]: pgmap v4008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:45.150 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:45 smithi016 conmon[32727]: 2022-03-21T19:25:45.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:46 smithi016 conmon[32727]: 2022-03-21T19:25:46.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:46.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:46 smithi079 ceph-mon[23158]: pgmap v4009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:46 smithi016 ceph-mon[22010]: pgmap v4009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:47 smithi016 conmon[32727]: 2022-03-21T19:25:47.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:48.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:48 smithi016 conmon[32727]: 2022-03-21T19:25:48.236+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:48 smithi079 ceph-mon[23158]: pgmap v4010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:48 smithi016 ceph-mon[22010]: pgmap v4010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:49 smithi016 conmon[32727]: 2022-03-21T19:25:49.276+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:50 smithi016 conmon[32727]: 2022-03-21T19:25:50.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:50 smithi016 ceph-mon[22010]: pgmap v4011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:50 smithi079 ceph-mon[23158]: pgmap v4011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:51 smithi016 conmon[32727]: 2022-03-21T19:25:51.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:52 smithi016 conmon[32727]: 2022-03-21T19:25:52.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:52 smithi016 ceph-mon[22010]: pgmap v4012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:52 smithi079 ceph-mon[23158]: pgmap v4012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:53 smithi016 conmon[32727]: 2022-03-21T19:25:53.178+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:54 smithi016 conmon[32727]: 2022-03-21T19:25:54.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:54 smithi016 ceph-mon[22010]: pgmap v4013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:54 smithi079 ceph-mon[23158]: pgmap v4013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:25:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:25:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:55 smithi016 conmon[32727]: 2022-03-21T19:25:55.228+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:56 smithi016 conmon[32727]: 2022-03-21T19:25:56.181+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:57.330 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:57.330 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:56 smithi016 ceph-mon[22010]: pgmap v4014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:57.331 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:57.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:57.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:56 smithi079 ceph-mon[23158]: pgmap v4014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:57 smithi016 conmon[32727]: 2022-03-21T19:25:57.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:58 smithi016 conmon[32727]: 2022-03-21T19:25:58.236+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:25:59 smithi016 conmon[32727]: 2022-03-21T19:25:59.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:25:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: pgmap v4015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:25:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:25:59.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:25:59.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:25:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:25:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:25:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: pgmap v4015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:25:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:25:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:25:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:25:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:25:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:26:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:00 smithi016 conmon[32727]: 2022-03-21T19:26:00.259+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:01 smithi016 ceph-mon[22010]: pgmap v4016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:01 smithi016 conmon[32727]: 2022-03-21T19:26:01.228+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:01.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:01 smithi079 ceph-mon[23158]: pgmap v4016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:02 smithi016 conmon[32727]: 2022-03-21T19:26:02.276+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:03 smithi016 conmon[32727]: 2022-03-21T19:26:03.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:03 smithi016 ceph-mon[22010]: pgmap v4017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:03 smithi079 ceph-mon[23158]: pgmap v4017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:04 smithi016 conmon[32727]: 2022-03-21T19:26:04.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:05 smithi016 conmon[32727]: 2022-03-21T19:26:05.222+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:05 smithi016 ceph-mon[22010]: pgmap v4018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:05 smithi079 ceph-mon[23158]: pgmap v4018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:06 smithi016 conmon[32727]: 2022-03-21T19:26:06.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:07 smithi016 ceph-mon[22010]: pgmap v4019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:07 smithi016 conmon[32727]: 2022-03-21T19:26:07.196+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:07 smithi079 ceph-mon[23158]: pgmap v4019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:08 smithi016 conmon[32727]: 2022-03-21T19:26:08.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:09 smithi016 ceph-mon[22010]: pgmap v4020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:09.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:09 smithi016 conmon[32727]: 2022-03-21T19:26:09.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:09 smithi079 ceph-mon[23158]: pgmap v4020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:10 smithi016 conmon[32727]: 2022-03-21T19:26:10.230+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:11 smithi016 conmon[32727]: 2022-03-21T19:26:11.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:11 smithi016 ceph-mon[22010]: pgmap v4021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:11 smithi079 ceph-mon[23158]: pgmap v4021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:12 smithi016 conmon[32727]: 2022-03-21T19:26:12.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:13 smithi016 conmon[32727]: 2022-03-21T19:26:13.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:13 smithi016 ceph-mon[22010]: pgmap v4022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:13 smithi079 ceph-mon[23158]: pgmap v4022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:14.596 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:14.596 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:26:14.597 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:26:14.597 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:26:14.603 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:14 smithi016 conmon[32727]: 2022-03-21T19:26:14.315+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:14.603 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:14.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:26:14.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:26:14.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:26:15.292 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:15 smithi016 conmon[32727]: 2022-03-21T19:26:15.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:15 smithi016 ceph-mon[22010]: pgmap v4023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:15 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:26:15.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:15.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:15 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:26:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:15 smithi079 ceph-mon[23158]: pgmap v4023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:15 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:26:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:15 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:26:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:16 smithi016 conmon[32727]: 2022-03-21T19:26:16.321+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:17 smithi016 ceph-mon[22010]: pgmap v4024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[32727]: 2022-03-21T19:26:17.289+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:17.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: 2022-03-21T19:26:17.293+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:17.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:26:17.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:17.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: yield 2022-03-21T19:26:17.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:17.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:17.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:17.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:26:17.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:17.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:26:17.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:17.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:26:17.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:17.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:26:17.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:17.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:26:17.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:17.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:17.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:17 smithi079 ceph-mon[23158]: pgmap v4024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:18 smithi016 conmon[32727]: 2022-03-21T19:26:18.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:18.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:18.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:26:18.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:18.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:26:18.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:18.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:18.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:18.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:26:18.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:18.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:26:18.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:18.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:26:18.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:18.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:18.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:18.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:26:18.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:18.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:18.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:18.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:26:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:26:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:18.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:26:18.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:18.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:18.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:18.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:26:18.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:18.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:26:18.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:18.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:26:18.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:18.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:18.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:18.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:26:18.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:18.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:18.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:18.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:26:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:19 smithi016 ceph-mon[22010]: pgmap v4025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:19 smithi016 conmon[32727]: 2022-03-21T19:26:19.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:19 smithi079 ceph-mon[23158]: pgmap v4025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:20.617 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[32727]: 2022-03-21T19:26:20.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:20.618 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 2022-03-21T19:26:20.365+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:20.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:26:20.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:20.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: yield 2022-03-21T19:26:20.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:20.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:20.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:20.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:26:20.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:20.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:26:20.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:20.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:26:20.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:20.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:26:20.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:20.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:26:20.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:20.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:20.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:20.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 2022-03-21T19:26:20.366+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:26:20.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:26:20.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:20.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: yield 2022-03-21T19:26:20.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:20.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:20.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:20.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:26:20.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:20.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:26:20.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:20.637 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:26:20.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:20.638 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:26:20.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:20.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:26:20.640 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:20.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:20.641 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:20.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 2022-03-21T19:26:20.642 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:26:20.643 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: 2022-03-21T19:26:20.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:26:20.644 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:26:20.645 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:26:20.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:26:20.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:26:20.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:26:20.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:26:20.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:26:20.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:26:20.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:26:20.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:26:20.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:26:20.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:26:20.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:26:20.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:26:20.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:26:20.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:26:20.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:26:20.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:26:20.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:26:20.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:26:20.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:26:20.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise 2022-03-21T19:26:20.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:26:20.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:26:20.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:26:20.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:26:20.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:26:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:20 smithi016 ceph-mon[22010]: pgmap v4026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:20 smithi079 ceph-mon[23158]: pgmap v4026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:21 smithi016 conmon[32727]: 2022-03-21T19:26:21.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:26:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:26:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:26:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:26:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:26:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:26:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:26:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:26:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:26:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:26:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:26:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:26:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:26:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:26:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:26:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:26:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:26:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:26:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:26:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:26:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:26:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:26:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:26:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:26:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:26:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:26:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:26:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:26:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:26:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:26:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:26:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:26:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:26:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:26:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:26:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:26:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:26:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:26:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:26:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:26:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:21 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:26:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:26:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:26:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:26:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:26:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:26:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:26:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:26:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:26:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:26:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:26:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:26:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:26:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:26:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:26:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:26:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:26:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:26:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:26:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:26:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:26:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:26:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:26:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:26:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:26:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:26:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:26:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:26:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:26:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:26:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:26:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:26:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:26:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:26:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:26:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:26:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:26:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:26:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:26:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:26:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:26:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:26:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:26:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:26:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:26:22.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:26:22.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:26:22.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:26:22.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:26:22.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:26:22.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:26:22.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:26:22.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:26:22.002 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:21 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:22.003 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:22 smithi016 conmon[32727]: 2022-03-21T19:26:22.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:22 smithi016 ceph-mon[22010]: pgmap v4027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:22 smithi079 ceph-mon[23158]: pgmap v4027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:23 smithi016 conmon[32727]: 2022-03-21T19:26:23.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:24 smithi016 conmon[32727]: 2022-03-21T19:26:24.047+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:24 smithi016 ceph-mon[22010]: pgmap v4028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:24 smithi079 ceph-mon[23158]: pgmap v4028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:25 smithi016 conmon[32727]: 2022-03-21T19:26:25.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:26 smithi016 conmon[32727]: 2022-03-21T19:26:26.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:26 smithi016 ceph-mon[22010]: pgmap v4029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:26 smithi079 ceph-mon[23158]: pgmap v4029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:27.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:27 smithi016 conmon[32727]: 2022-03-21T19:26:27.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:28.054 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:27 smithi016 conmon[32727]: 2022-03-21T19:26:27.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:28.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:28 smithi016 ceph-mon[22010]: pgmap v4030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:28 smithi079 ceph-mon[23158]: pgmap v4030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:28 smithi016 conmon[32727]: 2022-03-21T19:26:28.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:29.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:30 smithi016 conmon[32727]: 2022-03-21T19:26:30.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:30 smithi016 ceph-mon[22010]: pgmap v4031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:30 smithi079 ceph-mon[23158]: pgmap v4031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:30 smithi016 conmon[32727]: 2022-03-21T19:26:30.977+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:31 smithi016 conmon[32727]: 2022-03-21T19:26:31.963+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:32 smithi016 ceph-mon[22010]: pgmap v4032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:32 smithi079 ceph-mon[23158]: pgmap v4032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:32 smithi016 conmon[32727]: 2022-03-21T19:26:32.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:33 smithi016 conmon[32727]: 2022-03-21T19:26:33.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:34 smithi016 ceph-mon[22010]: pgmap v4033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:34 smithi079 ceph-mon[23158]: pgmap v4033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:34 smithi016 conmon[32727]: 2022-03-21T19:26:34.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:35 smithi016 conmon[32727]: 2022-03-21T19:26:35.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:36.937 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:36.938 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:36 smithi016 ceph-mon[22010]: pgmap v4034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:36 smithi079 ceph-mon[23158]: pgmap v4034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:36 smithi016 conmon[32727]: 2022-03-21T19:26:36.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:37.951 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:37 smithi016 conmon[32727]: 2022-03-21T19:26:37.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:38.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:38.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:38 smithi016 ceph-mon[22010]: pgmap v4035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:38 smithi079 ceph-mon[23158]: pgmap v4035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:38 smithi016 conmon[32727]: 2022-03-21T19:26:38.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:39 smithi016 conmon[32727]: 2022-03-21T19:26:39.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:40 smithi079 ceph-mon[23158]: pgmap v4036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:40 smithi016 ceph-mon[22010]: pgmap v4036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:40 smithi016 conmon[32727]: 2022-03-21T19:26:40.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:41.961 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:41 smithi016 conmon[32727]: 2022-03-21T19:26:41.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:42.962 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:42.962 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:42 smithi016 ceph-mon[22010]: pgmap v4037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:42 smithi079 ceph-mon[23158]: pgmap v4037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:42 smithi016 conmon[32727]: 2022-03-21T19:26:42.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:43.991 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:43 smithi016 conmon[32727]: 2022-03-21T19:26:43.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:44.949 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:44.949 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:44 smithi016 ceph-mon[22010]: pgmap v4038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:44.949 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:44 smithi079 ceph-mon[23158]: pgmap v4038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:44 smithi016 conmon[32727]: 2022-03-21T19:26:44.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:45.959 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:45 smithi016 conmon[32727]: 2022-03-21T19:26:45.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:46 smithi079 ceph-mon[23158]: pgmap v4039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:46.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:46.990 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:46 smithi016 ceph-mon[22010]: pgmap v4039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:46 smithi016 conmon[32727]: 2022-03-21T19:26:46.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:48.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:48 smithi016 conmon[32727]: 2022-03-21T19:26:48.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:48 smithi079 ceph-mon[23158]: pgmap v4040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:49.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:49.051 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:48 smithi016 ceph-mon[22010]: pgmap v4040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:49 smithi016 conmon[32727]: 2022-03-21T19:26:49.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:50.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:50.038 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:50 smithi016 conmon[32727]: 2022-03-21T19:26:50.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:50 smithi079 ceph-mon[23158]: pgmap v4041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:51.005 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:51.005 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:50 smithi016 ceph-mon[22010]: pgmap v4041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:51.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:51 smithi016 conmon[32727]: 2022-03-21T19:26:51.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:51.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:51 smithi016 conmon[32727]: 2022-03-21T19:26:51.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:52 smithi079 ceph-mon[23158]: pgmap v4042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:53.010 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:53.011 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:52 smithi016 ceph-mon[22010]: pgmap v4042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:53 smithi016 conmon[32727]: 2022-03-21T19:26:53.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:53.988 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:53 smithi016 conmon[32727]: 2022-03-21T19:26:53.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:55.003 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:55.003 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:54 smithi016 ceph-mon[22010]: pgmap v4043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:55.004 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:54 smithi079 ceph-mon[23158]: pgmap v4043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:26:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:26:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:55 smithi016 conmon[32727]: 2022-03-21T19:26:55.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:56.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:56 smithi016 conmon[32727]: 2022-03-21T19:26:56.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:57.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:57.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:56 smithi016 ceph-mon[22010]: pgmap v4044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:56 smithi079 ceph-mon[23158]: pgmap v4044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:57.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:57 smithi016 conmon[32727]: 2022-03-21T19:26:57.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:58.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:58 smithi016 conmon[32727]: 2022-03-21T19:26:58.020+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:26:59.043 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:59.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: pgmap v4045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:59.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:26:59.044 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:26:59.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:26:59.045 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:26:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:26:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: pgmap v4045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:26:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:26:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:26:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:26:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:26:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:26:59 smithi016 conmon[32727]: 2022-03-21T19:26:59.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:00.047 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:00.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:26:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:26:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7460 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:00 smithi016 conmon[32727]: 2022-03-21T19:27:00.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:01.031 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:01.031 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:00 smithi016 ceph-mon[22010]: pgmap v4046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:00 smithi079 ceph-mon[23158]: pgmap v4046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:01.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:01 smithi016 conmon[32727]: 2022-03-21T19:27:01.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:02.060 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:02 smithi016 conmon[32727]: 2022-03-21T19:27:02.058+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:03.105 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:03.105 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:02 smithi016 ceph-mon[22010]: pgmap v4047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:02 smithi079 ceph-mon[23158]: pgmap v4047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:03 smithi016 conmon[32727]: 2022-03-21T19:27:03.102+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:04.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:04 smithi016 conmon[32727]: 2022-03-21T19:27:04.068+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:05.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:05.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:04 smithi016 ceph-mon[22010]: pgmap v4048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:05.071 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:04 smithi079 ceph-mon[23158]: pgmap v4048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:05.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:05 smithi016 conmon[32727]: 2022-03-21T19:27:05.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:06.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:06 smithi016 conmon[32727]: 2022-03-21T19:27:06.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:07.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:07.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:06 smithi016 ceph-mon[22010]: pgmap v4049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:06 smithi079 ceph-mon[23158]: pgmap v4049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:07.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:07 smithi016 conmon[32727]: 2022-03-21T19:27:07.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:08.088 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:08 smithi016 conmon[32727]: 2022-03-21T19:27:08.086+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:09.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:09.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:08 smithi016 ceph-mon[22010]: pgmap v4050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:08 smithi079 ceph-mon[23158]: pgmap v4050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:09.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:09 smithi016 conmon[32727]: 2022-03-21T19:27:09.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:10.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:10.149 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:10 smithi016 conmon[32727]: 2022-03-21T19:27:10.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:10 smithi016 ceph-mon[22010]: pgmap v4051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:10 smithi079 ceph-mon[23158]: pgmap v4051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:11 smithi016 conmon[32727]: 2022-03-21T19:27:11.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:12 smithi016 conmon[32727]: 2022-03-21T19:27:12.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:12 smithi016 ceph-mon[22010]: pgmap v4052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:12 smithi079 ceph-mon[23158]: pgmap v4052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:13 smithi016 conmon[32727]: 2022-03-21T19:27:13.162+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:14.115 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:14 smithi016 conmon[32727]: 2022-03-21T19:27:14.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:15.131 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:15.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:14 smithi016 ceph-mon[22010]: pgmap v4053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:15.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:14 smithi079 ceph-mon[23158]: pgmap v4053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:15 smithi016 conmon[32727]: 2022-03-21T19:27:15.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:16.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:16 smithi016 conmon[32727]: 2022-03-21T19:27:16.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:17.151 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:17.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:16 smithi016 ceph-mon[22010]: pgmap v4054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:16 smithi079 ceph-mon[23158]: pgmap v4054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:17 smithi016 conmon[32727]: 2022-03-21T19:27:17.149+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:18.141 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:18 smithi016 conmon[32727]: 2022-03-21T19:27:18.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:19.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:18 smithi016 ceph-mon[22010]: pgmap v4055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:18 smithi079 ceph-mon[23158]: pgmap v4055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:19 smithi016 conmon[32727]: 2022-03-21T19:27:19.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:20.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:20.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:20.413 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[32727]: 2022-03-21T19:27:20.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:20 smithi016 ceph-mon[22010]: pgmap v4056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:27:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:27:21.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:27:21.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: 2022-03-21T19:27:20.845+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:27:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:21.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: yield 2022-03-21T19:27:21.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:21.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:21.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:27:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:21.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:27:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:27:21.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:21.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:27:21.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:27:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:21.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:21.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:20 smithi079 ceph-mon[23158]: pgmap v4056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:27:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:27:21.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:27:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:21 smithi016 conmon[32727]: 2022-03-21T19:27:21.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:27:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:27:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:22.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:27:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:22.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:27:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:22.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:27:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:22.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:27:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:22.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:27:22.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:27:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:27:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:22.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:27:22.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:27:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:22.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:27:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:22.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:27:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:22.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:22.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:27:22.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:22 smithi016 conmon[32727]: 2022-03-21T19:27:22.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:22 smithi016 ceph-mon[22010]: pgmap v4057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:22 smithi079 ceph-mon[23158]: pgmap v4057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[32727]: 2022-03-21T19:27:23.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:24.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 2022-03-21T19:27:23.917+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:24.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:27:24.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:24.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: yield 2022-03-21T19:27:24.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:24.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:27:24.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:24.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:27:24.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:24.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:27:24.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:27:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:24.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:27:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:24.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:24.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 2022-03-21T19:27:23.918+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:27:24.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:27:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: yield 2022-03-21T19:27:24.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:24.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:27:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:27:24.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:27:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:24.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:27:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:27:24.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:24.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 2022-03-21T19:27:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:27:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: 2022-03-21T19:27:24.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:27:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:27:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:27:24.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:27:24.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:27:24.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:27:24.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:27:24.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:27:24.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:27:24.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:27:24.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:27:24.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:27:24.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:27:24.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:27:24.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:27:24.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:27:24.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:27:24.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:27:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:27:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:27:24.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:27:24.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:27:24.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise 2022-03-21T19:27:24.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:27:24.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:27:24.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:27:24.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:27:24.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:27:23 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:24 smithi016 conmon[32727]: 2022-03-21T19:27:24.240+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:27:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:25.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:27:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:27:25.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:27:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:25.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:27:25.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:27:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:25.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:27:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:25.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:27:25.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:27:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:25.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:27:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:25.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:27:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:25.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:27:25.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:27:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:27:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:27:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:27:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:27:25.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:27:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:27:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:27:25.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:27:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:27:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:27:25.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:27:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:27:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:27:25.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:27:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:27:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:27:25.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:27:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:27:25.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:27:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:27:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:27:25.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:27:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:27:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:27:25.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:27:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:27:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:27:25.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:24 smithi016 ceph-mon[22010]: pgmap v4058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:25.190 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:27:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:25.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:27:25.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:27:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:25.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:27:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:25.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:27:25.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:27:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:27:25.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:27:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:27:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:27:25.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:27:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:27:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:27:25.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:27:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:27:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:27:25.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:27:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:27:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:27:25.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:27:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:27:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:27:25.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:27:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:27:25.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:27:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:27:25.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:27:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:27:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:27:25.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:27:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:27:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:27:25.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:27:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:27:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:27:25.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:27:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:27:25.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:27:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:27:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:27:25.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:27:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:27:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:27:25.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:27:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:27:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:27:25.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:27:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:27:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:27:25.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:27:25.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:27:25.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:27:25.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:27:25.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:24 smithi079 ceph-mon[23158]: pgmap v4058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:25.248 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:25 smithi016 conmon[32727]: 2022-03-21T19:27:25.286+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:26 smithi016 conmon[32727]: 2022-03-21T19:27:26.322+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:26 smithi016 ceph-mon[22010]: pgmap v4059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:26 smithi079 ceph-mon[23158]: pgmap v4059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:27 smithi016 conmon[32727]: 2022-03-21T19:27:27.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:28 smithi016 conmon[32727]: 2022-03-21T19:27:28.348+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:28 smithi016 ceph-mon[22010]: pgmap v4060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:28 smithi079 ceph-mon[23158]: pgmap v4060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:29 smithi016 conmon[32727]: 2022-03-21T19:27:29.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:30 smithi016 conmon[32727]: 2022-03-21T19:27:30.322+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:30 smithi016 ceph-mon[22010]: pgmap v4061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:30 smithi079 ceph-mon[23158]: pgmap v4061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:31 smithi016 conmon[32727]: 2022-03-21T19:27:31.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:32 smithi016 conmon[32727]: 2022-03-21T19:27:32.380+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:32 smithi016 ceph-mon[22010]: pgmap v4062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:32 smithi079 ceph-mon[23158]: pgmap v4062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:33 smithi016 conmon[32727]: 2022-03-21T19:27:33.404+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:34 smithi016 conmon[32727]: 2022-03-21T19:27:34.405+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:34 smithi016 ceph-mon[22010]: pgmap v4063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:34 smithi079 ceph-mon[23158]: pgmap v4063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:35 smithi016 conmon[32727]: 2022-03-21T19:27:35.372+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:36 smithi016 conmon[32727]: 2022-03-21T19:27:36.382+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:36 smithi016 ceph-mon[22010]: pgmap v4064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:36 smithi079 ceph-mon[23158]: pgmap v4064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:37 smithi016 conmon[32727]: 2022-03-21T19:27:37.409+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:38.869 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:38 smithi016 conmon[32727]: 2022-03-21T19:27:38.416+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:38 smithi016 ceph-mon[22010]: pgmap v4065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:38 smithi079 ceph-mon[23158]: pgmap v4065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:39 smithi016 conmon[32727]: 2022-03-21T19:27:39.403+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7500 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:40 smithi016 conmon[32727]: 2022-03-21T19:27:40.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:40 smithi016 ceph-mon[22010]: pgmap v4066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:40 smithi079 ceph-mon[23158]: pgmap v4066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:41 smithi016 conmon[32727]: 2022-03-21T19:27:41.358+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:42 smithi016 conmon[32727]: 2022-03-21T19:27:42.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:43.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:42 smithi016 ceph-mon[22010]: pgmap v4067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:42 smithi079 ceph-mon[23158]: pgmap v4067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:43 smithi016 conmon[32727]: 2022-03-21T19:27:43.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:44 smithi016 conmon[32727]: 2022-03-21T19:27:44.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:44 smithi016 ceph-mon[22010]: pgmap v4068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:44 smithi079 ceph-mon[23158]: pgmap v4068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:45 smithi016 conmon[32727]: 2022-03-21T19:27:45.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:46 smithi016 conmon[32727]: 2022-03-21T19:27:46.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:46 smithi016 ceph-mon[22010]: pgmap v4069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:46 smithi079 ceph-mon[23158]: pgmap v4069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:47 smithi016 conmon[32727]: 2022-03-21T19:27:47.358+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:48 smithi016 conmon[32727]: 2022-03-21T19:27:48.383+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:48 smithi016 ceph-mon[22010]: pgmap v4070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:48 smithi079 ceph-mon[23158]: pgmap v4070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:49.902 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:49 smithi016 conmon[32727]: 2022-03-21T19:27:49.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:50 smithi016 conmon[32727]: 2022-03-21T19:27:50.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:50 smithi016 ceph-mon[22010]: pgmap v4071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:50 smithi079 ceph-mon[23158]: pgmap v4071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:51 smithi016 conmon[32727]: 2022-03-21T19:27:51.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:52 smithi016 conmon[32727]: 2022-03-21T19:27:52.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:52 smithi016 ceph-mon[22010]: pgmap v4072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:52 smithi079 ceph-mon[23158]: pgmap v4072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:53 smithi016 conmon[32727]: 2022-03-21T19:27:53.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:54 smithi016 conmon[32727]: 2022-03-21T19:27:54.368+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:54 smithi079 ceph-mon[23158]: pgmap v4073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:55.381 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:55.382 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:54 smithi016 ceph-mon[22010]: pgmap v4073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:27:55.383 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:27:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:55 smithi016 conmon[32727]: 2022-03-21T19:27:55.378+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:56.380 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:56 smithi016 conmon[32727]: 2022-03-21T19:27:56.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:56 smithi079 ceph-mon[23158]: pgmap v4074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:57.357 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:57.358 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:56 smithi016 ceph-mon[22010]: pgmap v4074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:57 smithi016 conmon[32727]: 2022-03-21T19:27:57.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:58.357 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:58 smithi016 conmon[32727]: 2022-03-21T19:27:58.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:27:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: pgmap v4075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:27:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:27:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:27:59.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:27:59.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:27:59.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: pgmap v4075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:27:59.340 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:27:59.340 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:27:59.341 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:27:59.341 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:27:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:27:59 smithi016 conmon[32727]: 2022-03-21T19:27:59.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:00.313 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:27:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:00.314 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:00.315 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:27:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:00 smithi016 conmon[32727]: 2022-03-21T19:28:00.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:00 smithi079 ceph-mon[23158]: pgmap v4076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:01.327 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:01.328 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:00 smithi016 ceph-mon[22010]: pgmap v4076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:01 smithi016 conmon[32727]: 2022-03-21T19:28:01.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:02.375 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:02 smithi016 conmon[32727]: 2022-03-21T19:28:02.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:02 smithi079 ceph-mon[23158]: pgmap v4077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:03.363 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:03.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:02 smithi016 ceph-mon[22010]: pgmap v4077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:03 smithi016 conmon[32727]: 2022-03-21T19:28:03.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:04.340 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:04 smithi016 conmon[32727]: 2022-03-21T19:28:04.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:04 smithi079 ceph-mon[23158]: pgmap v4078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:05.298 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:05.298 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:04 smithi016 ceph-mon[22010]: pgmap v4078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:05.299 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:05 smithi016 conmon[32727]: 2022-03-21T19:28:05.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:06.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:06 smithi016 conmon[32727]: 2022-03-21T19:28:06.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:06 smithi079 ceph-mon[23158]: pgmap v4079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:07.260 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:07.260 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:06 smithi016 ceph-mon[22010]: pgmap v4079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:07 smithi016 conmon[32727]: 2022-03-21T19:28:07.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:08.295 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:08 smithi016 conmon[32727]: 2022-03-21T19:28:08.293+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:08 smithi079 ceph-mon[23158]: pgmap v4080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:09.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:09.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:08 smithi016 ceph-mon[22010]: pgmap v4080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:09 smithi016 conmon[32727]: 2022-03-21T19:28:09.245+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:10.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:10 smithi016 conmon[32727]: 2022-03-21T19:28:10.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:10 smithi016 ceph-mon[22010]: pgmap v4081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:11.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:11 smithi016 conmon[32727]: 2022-03-21T19:28:11.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:10 smithi079 ceph-mon[23158]: pgmap v4081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:12.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:12 smithi016 conmon[32727]: 2022-03-21T19:28:12.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:13.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:12 smithi016 ceph-mon[22010]: pgmap v4082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:13.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:13 smithi016 conmon[32727]: 2022-03-21T19:28:13.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:12 smithi079 ceph-mon[23158]: pgmap v4082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:14.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:14 smithi016 conmon[32727]: 2022-03-21T19:28:14.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:15 smithi016 ceph-mon[22010]: pgmap v4083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:15 smithi016 conmon[32727]: 2022-03-21T19:28:15.242+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:15 smithi079 ceph-mon[23158]: pgmap v4083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:16 smithi016 conmon[32727]: 2022-03-21T19:28:16.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:17 smithi016 ceph-mon[22010]: pgmap v4084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:17 smithi016 conmon[32727]: 2022-03-21T19:28:17.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:17 smithi079 ceph-mon[23158]: pgmap v4084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:18 smithi016 conmon[32727]: 2022-03-21T19:28:18.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:19.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:19 smithi016 ceph-mon[22010]: pgmap v4085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:19 smithi016 conmon[32727]: 2022-03-21T19:28:19.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:19 smithi079 ceph-mon[23158]: pgmap v4085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:20 smithi016 conmon[32727]: 2022-03-21T19:28:20.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:21 smithi016 ceph-mon[22010]: pgmap v4086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:21 smithi016 conmon[32727]: 2022-03-21T19:28:21.241+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:21 smithi079 ceph-mon[23158]: pgmap v4086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:22 smithi016 conmon[32727]: 2022-03-21T19:28:22.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:23 smithi016 ceph-mon[22010]: pgmap v4087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:23.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:23 smithi016 conmon[32727]: 2022-03-21T19:28:23.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:23 smithi079 ceph-mon[23158]: pgmap v4087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:28:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:28:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:28:24.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:24 smithi016 conmon[32727]: 2022-03-21T19:28:24.227+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:28:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:28:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:28:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:25 smithi016 conmon[32727]: 2022-03-21T19:28:25.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:25 smithi016 ceph-mon[22010]: pgmap v4088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:25.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:25.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:25 smithi079 ceph-mon[23158]: pgmap v4088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:26 smithi016 ceph-mon[22010]: pgmap v4089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:26.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:26 smithi016 conmon[32727]: 2022-03-21T19:28:26.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:26 smithi079 ceph-mon[23158]: pgmap v4089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:27.403 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[32727]: 2022-03-21T19:28:27.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:27.404 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: 2022-03-21T19:28:27.021+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.404 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:28:27.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:27.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: yield 2022-03-21T19:28:27.405 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:27.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:27.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:27.406 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:28:27.407 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:27.407 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:28:27.407 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:27.408 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:28:27.408 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:27.409 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:28:27.409 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:27.409 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:28:27.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:27.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:27.410 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:27 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:28:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:28:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:28:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:28:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:28:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:28:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:27 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:28:27.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:27 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:28:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:28:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:28:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:28:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:28:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:28:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:27 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:28:27.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:28 smithi016 ceph-mon[22010]: pgmap v4090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:28 smithi016 conmon[32727]: 2022-03-21T19:28:28.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:28 smithi079 ceph-mon[23158]: pgmap v4090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:29.409 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:29 smithi016 conmon[32727]: 2022-03-21T19:28:29.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:30.411 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[32727]: 2022-03-21T19:28:30.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:30.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 2022-03-21T19:28:30.093+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:30.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:28:30.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:30.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: yield 2022-03-21T19:28:30.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:30.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:30.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:30.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:28:30.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:30.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:28:30.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:30.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:28:30.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:30.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:28:30.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:30.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:28:30.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:30.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:30.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:30.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 2022-03-21T19:28:30.094+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:28:30.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:28:30.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:30.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: yield 2022-03-21T19:28:30.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:30.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:28:30.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:28:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:30.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:28:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:28:30.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:28:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:30.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:30.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:30.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 2022-03-21T19:28:30.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:28:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: 2022-03-21T19:28:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:28:30.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:28:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:28:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:28:30.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:28:30.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:28:30.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:28:30.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:28:30.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:28:30.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:28:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:28:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:28:30.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:28:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:28:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:28:30.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:28:30.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:28:30.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:28:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:28:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:28:30.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:28:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:28:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise 2022-03-21T19:28:30.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:28:30.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:28:30.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:28:30.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:28:30.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:28:30 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:28:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:28:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:28:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:28:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:28:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:28:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.040 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:30 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:28:31.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:28:31.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:31.041 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:28:31.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:31.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:31.042 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:31.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:28:31.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:31.043 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:28:31.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:31.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:28:31.044 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:31.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:31.045 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:31.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:28:31.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:31.046 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:31.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:28:31.047 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:28:31.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:28:31.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:28:31.048 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:28:31.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:28:31.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:28:31.049 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:28:31.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:28:31.050 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:28:31.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:28:31.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:28:31.051 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:28:31.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:28:31.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:28:31.052 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:28:31.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:28:31.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:28:31.053 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:28:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:28:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:28:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:28:31.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:28:31.055 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:28:31.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:28:31.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:28:31.056 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:28:31.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:28:31.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:28:31.057 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:28:31.058 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:31.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:30 smithi016 ceph-mon[22010]: pgmap v4091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:31.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.060 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:28:31.060 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:31.061 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:28:31.061 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:31.062 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:31.062 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:31.062 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:28:31.063 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:31.063 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:28:31.063 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:31.064 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:28:31.064 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:31.064 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:31.065 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:31.065 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:28:31.065 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:31.066 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:31.066 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.066 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:30 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:28:31.067 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:28:31.067 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:28:31.067 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:28:31.068 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:28:31.068 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:28:31.069 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:28:31.069 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:28:31.069 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:28:31.070 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:28:31.070 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:28:31.070 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:28:31.071 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:28:31.071 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:28:31.071 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:28:31.072 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:28:31.072 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:28:31.072 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:28:31.073 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.073 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:28:31.073 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:28:31.074 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:28:31.074 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:28:31.075 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:28:31.075 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:28:31.075 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:28:31.076 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:28:31.076 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:28:31.076 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:28:31.077 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:28:31.077 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:28:31.077 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:28:31.078 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:28:31.078 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:28:31.078 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:28:31.079 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:28:31.079 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:28:31.079 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:28:31.080 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:28:31.080 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:28:31.080 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:28:31.081 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:28:31.081 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:28:31.082 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:28:31.082 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:28:31.082 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:28:31.083 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:28:31.083 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:28:31.083 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:28:31.084 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:28:31.084 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:31.084 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:30 smithi079 ceph-mon[23158]: pgmap v4091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:31.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:31 smithi016 conmon[32727]: 2022-03-21T19:28:31.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:32 smithi016 conmon[32727]: 2022-03-21T19:28:32.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:32.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:32 smithi016 ceph-mon[22010]: pgmap v4092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:32 smithi079 ceph-mon[23158]: pgmap v4092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:33 smithi016 conmon[32727]: 2022-03-21T19:28:33.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:34 smithi016 conmon[32727]: 2022-03-21T19:28:34.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:34 smithi016 ceph-mon[22010]: pgmap v4093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:34 smithi079 ceph-mon[23158]: pgmap v4093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:35 smithi016 conmon[32727]: 2022-03-21T19:28:35.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:36 smithi016 conmon[32727]: 2022-03-21T19:28:36.179+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:36 smithi079 ceph-mon[23158]: pgmap v4094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:36 smithi016 ceph-mon[22010]: pgmap v4094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:37 smithi016 conmon[32727]: 2022-03-21T19:28:37.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:38 smithi016 conmon[32727]: 2022-03-21T19:28:38.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:38 smithi079 ceph-mon[23158]: pgmap v4095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:38 smithi016 ceph-mon[22010]: pgmap v4095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:39 smithi016 conmon[32727]: 2022-03-21T19:28:39.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:40 smithi016 conmon[32727]: 2022-03-21T19:28:40.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:40 smithi079 ceph-mon[23158]: pgmap v4096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:41.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:41.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:40 smithi016 ceph-mon[22010]: pgmap v4096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:41 smithi016 conmon[32727]: 2022-03-21T19:28:41.264+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:42 smithi016 conmon[32727]: 2022-03-21T19:28:42.234+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:42.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:42 smithi079 ceph-mon[23158]: pgmap v4097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:42 smithi016 ceph-mon[22010]: pgmap v4097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:43 smithi016 conmon[32727]: 2022-03-21T19:28:43.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:44.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:44 smithi016 conmon[32727]: 2022-03-21T19:28:44.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:44 smithi079 ceph-mon[23158]: pgmap v4098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:44 smithi016 ceph-mon[22010]: pgmap v4098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:45 smithi016 conmon[32727]: 2022-03-21T19:28:45.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:46 smithi016 conmon[32727]: 2022-03-21T19:28:46.184+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:46 smithi079 ceph-mon[23158]: pgmap v4099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:46 smithi016 ceph-mon[22010]: pgmap v4099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:47 smithi016 conmon[32727]: 2022-03-21T19:28:47.210+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:48 smithi016 conmon[32727]: 2022-03-21T19:28:48.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:48 smithi079 ceph-mon[23158]: pgmap v4100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:48 smithi016 ceph-mon[22010]: pgmap v4100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:49 smithi016 conmon[32727]: 2022-03-21T19:28:49.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:50 smithi016 conmon[32727]: 2022-03-21T19:28:50.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:50 smithi079 ceph-mon[23158]: pgmap v4101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:50 smithi016 ceph-mon[22010]: pgmap v4101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:51 smithi016 conmon[32727]: 2022-03-21T19:28:51.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:52.132 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:52 smithi016 conmon[32727]: 2022-03-21T19:28:52.129+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:52 smithi016 ceph-mon[22010]: pgmap v4102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:52 smithi079 ceph-mon[23158]: pgmap v4102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:53 smithi016 conmon[32727]: 2022-03-21T19:28:53.165+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:54.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:54.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:54 smithi016 conmon[32727]: 2022-03-21T19:28:54.120+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:55.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:55.100 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:54 smithi016 ceph-mon[22010]: pgmap v4103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:55.101 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:54 smithi079 ceph-mon[23158]: pgmap v4103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:28:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:28:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:55 smithi016 conmon[32727]: 2022-03-21T19:28:55.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:56.116 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:56 smithi016 conmon[32727]: 2022-03-21T19:28:56.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:57.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:57.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:56 smithi016 ceph-mon[22010]: pgmap v4104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:56 smithi079 ceph-mon[23158]: pgmap v4104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:57 smithi016 conmon[32727]: 2022-03-21T19:28:57.133+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:58.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:58 smithi016 conmon[32727]: 2022-03-21T19:28:58.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:28:59.159 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:59.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: pgmap v4105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:59.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:28:59.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:28:59.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:28:59.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:28:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:28:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: pgmap v4105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:28:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:28:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:28:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:28:59 smithi016 conmon[32727]: 2022-03-21T19:28:59.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:28:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:28:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:00 smithi016 conmon[32727]: 2022-03-21T19:29:00.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:00 smithi016 ceph-mon[22010]: pgmap v4106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:00 smithi079 ceph-mon[23158]: pgmap v4106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:01 smithi016 conmon[32727]: 2022-03-21T19:29:01.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:02 smithi016 conmon[32727]: 2022-03-21T19:29:02.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:02 smithi016 ceph-mon[22010]: pgmap v4107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:02 smithi079 ceph-mon[23158]: pgmap v4107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:03 smithi016 conmon[32727]: 2022-03-21T19:29:03.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:04 smithi016 conmon[32727]: 2022-03-21T19:29:04.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:04.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:05 smithi016 ceph-mon[22010]: pgmap v4108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:05.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:05 smithi016 conmon[32727]: 2022-03-21T19:29:05.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:05 smithi079 ceph-mon[23158]: pgmap v4108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:06 smithi016 conmon[32727]: 2022-03-21T19:29:06.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:07 smithi016 ceph-mon[22010]: pgmap v4109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:07.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:07 smithi016 conmon[32727]: 2022-03-21T19:29:07.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:07 smithi079 ceph-mon[23158]: pgmap v4109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:08.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:08 smithi016 conmon[32727]: 2022-03-21T19:29:08.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:09 smithi016 ceph-mon[22010]: pgmap v4110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:09.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:09.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:09 smithi016 conmon[32727]: 2022-03-21T19:29:09.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:09 smithi079 ceph-mon[23158]: pgmap v4110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:09.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:10 smithi016 conmon[32727]: 2022-03-21T19:29:10.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:11 smithi016 ceph-mon[22010]: pgmap v4111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:11.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:11.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:11 smithi016 conmon[32727]: 2022-03-21T19:29:11.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:11 smithi079 ceph-mon[23158]: pgmap v4111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:12 smithi016 conmon[32727]: 2022-03-21T19:29:12.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:13 smithi016 ceph-mon[22010]: pgmap v4112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:13.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:13 smithi016 conmon[32727]: 2022-03-21T19:29:13.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:13 smithi079 ceph-mon[23158]: pgmap v4112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:14 smithi016 ceph-mon[22010]: pgmap v4113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:14.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:14 smithi016 conmon[32727]: 2022-03-21T19:29:14.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:14 smithi079 ceph-mon[23158]: pgmap v4113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:15.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:15 smithi016 conmon[32727]: 2022-03-21T19:29:15.231+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:16 smithi016 ceph-mon[22010]: pgmap v4114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:16.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:16 smithi016 conmon[32727]: 2022-03-21T19:29:16.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:16 smithi079 ceph-mon[23158]: pgmap v4114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:17.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:17 smithi016 conmon[32727]: 2022-03-21T19:29:17.181+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:18.424 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:18 smithi016 conmon[32727]: 2022-03-21T19:29:18.169+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:18 smithi079 ceph-mon[23158]: pgmap v4115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:18 smithi016 ceph-mon[22010]: pgmap v4115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:19 smithi016 conmon[32727]: 2022-03-21T19:29:19.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:20.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:20 smithi016 conmon[32727]: 2022-03-21T19:29:20.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:20 smithi016 ceph-mon[22010]: pgmap v4116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:20 smithi079 ceph-mon[23158]: pgmap v4116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:21 smithi016 conmon[32727]: 2022-03-21T19:29:21.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:21 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:22.097 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:21 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:22.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:22 smithi016 conmon[32727]: 2022-03-21T19:29:22.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:22 smithi079 ceph-mon[23158]: pgmap v4117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:23.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:23.130 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:22 smithi016 ceph-mon[22010]: pgmap v4117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:29:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:23 smithi016 conmon[32727]: 2022-03-21T19:29:23.127+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:24.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:24 smithi016 conmon[32727]: 2022-03-21T19:29:24.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:24 smithi079 ceph-mon[23158]: pgmap v4118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:25.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:25.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:24 smithi016 ceph-mon[22010]: pgmap v4118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:25 smithi016 conmon[32727]: 2022-03-21T19:29:25.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:26.104 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:26 smithi016 conmon[32727]: 2022-03-21T19:29:26.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:26.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:26 smithi079 ceph-mon[23158]: pgmap v4119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:27.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:27.062 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:26 smithi016 ceph-mon[22010]: pgmap v4119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:27 smithi016 conmon[32727]: 2022-03-21T19:29:27.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:28.064 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:28 smithi016 conmon[32727]: 2022-03-21T19:29:28.062+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:28 smithi079 ceph-mon[23158]: pgmap v4120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:29.075 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:29.076 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:28 smithi016 ceph-mon[22010]: pgmap v4120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:29 smithi016 conmon[32727]: 2022-03-21T19:29:29.073+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:30.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:30.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:30 smithi016 conmon[32727]: 2022-03-21T19:29:30.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:29:30.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:29:30.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:30 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:29:30.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:30 smithi016 ceph-mon[22010]: pgmap v4121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:29:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:29:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:30 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:29:30.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:30 smithi079 ceph-mon[23158]: pgmap v4121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:31 smithi016 conmon[32727]: 2022-03-21T19:29:31.054+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:32.018 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:32 smithi016 conmon[32727]: 2022-03-21T19:29:32.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:32.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:32 smithi079 ceph-mon[23158]: pgmap v4122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:33.064 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:33.065 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:32 smithi016 ceph-mon[22010]: pgmap v4122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[32727]: 2022-03-21T19:29:33.062+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:33.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: 2022-03-21T19:29:33.197+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:33.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:29:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: yield 2022-03-21T19:29:33.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:33.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:29:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:33.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:29:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:29:33.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:29:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:33.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:29:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:33.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:33 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:29:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:29:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:33.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:33.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:33.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:29:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:29:33.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:29:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:33.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:33.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:33.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:29:33.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:33.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:33.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:33.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:33 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:29:34.069 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:34.070 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:33 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:34.070 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:29:34.071 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:34.071 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:29:34.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:34.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:34.072 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:34.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:29:34.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:34.073 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:29:34.074 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:34.074 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:29:34.074 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:34.075 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:34.075 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:34.076 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:29:34.076 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:34.076 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:34.077 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:34.077 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:33 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:29:34.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:34 smithi016 conmon[32727]: 2022-03-21T19:29:34.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:34 smithi079 ceph-mon[23158]: pgmap v4123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:35.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:35.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:34 smithi016 ceph-mon[22010]: pgmap v4123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:35.024 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:35 smithi016 conmon[32727]: 2022-03-21T19:29:35.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:36.072 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:36.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[32727]: 2022-03-21T19:29:36.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:36.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 2022-03-21T19:29:36.269+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:29:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: yield 2022-03-21T19:29:36.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:36.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:29:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:29:36.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:36.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:29:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:29:36.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:29:36.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:36.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 2022-03-21T19:29:36.270+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:29:36.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:29:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: yield 2022-03-21T19:29:36.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:36.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:29:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:36.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:29:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:29:36.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:29:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:36.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:29:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:36.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 2022-03-21T19:29:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:29:36.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: 2022-03-21T19:29:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:29:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:29:36.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:29:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:29:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:29:36.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:29:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:29:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:29:36.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:29:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:29:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:29:36.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:29:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:29:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:29:36.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:29:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:29:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:29:36.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:29:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:29:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:29:36.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:29:36.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:29:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise 2022-03-21T19:29:36.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:29:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:29:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:29:36.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:29:36.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:29:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:29:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:29:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:29:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:29:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:29:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:29:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:36 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:29:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:29:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:29:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:29:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:29:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:29:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:29:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:29:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:29:36.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:29:36.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:29:36.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:29:36.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:29:36.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:29:36.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:29:36.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:29:36.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:29:36.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:29:36.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:29:36.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:29:36.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:29:36.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:29:36.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:29:36.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:29:36.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:29:36.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:29:36.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:29:36.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:29:36.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:29:36.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:29:36.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:29:36.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:29:36.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:29:36.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:29:36.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:29:36.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:29:36.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:29:36.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:36.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:36 smithi079 ceph-mon[23158]: pgmap v4124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:37.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:37.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:37.111 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:29:37.111 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:37.112 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:29:37.112 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:37.112 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:37.113 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:37.113 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:29:37.113 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:37.114 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:29:37.114 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:37.114 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:29:37.115 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:37.115 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:37.115 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:37.116 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:29:37.116 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:37.116 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:37.117 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:37.117 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:36 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:29:37.118 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:29:37.118 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:29:37.118 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:29:37.119 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:29:37.119 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:29:37.119 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:29:37.120 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:29:37.120 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:29:37.121 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:29:37.121 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:29:37.122 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:29:37.122 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:29:37.122 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:29:37.123 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:29:37.123 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:29:37.123 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:29:37.124 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:29:37.124 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:37.124 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:29:37.125 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:29:37.125 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:29:37.125 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:29:37.126 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:29:37.126 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:29:37.127 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:29:37.127 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:29:37.127 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:29:37.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:29:37.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:29:37.128 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:29:37.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:29:37.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:29:37.129 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:29:37.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:29:37.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:29:37.130 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:29:37.131 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:29:37.131 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:29:37.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:29:37.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:29:37.132 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:29:37.133 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:29:37.133 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:29:37.133 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:29:37.134 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:29:37.134 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:29:37.135 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:29:37.135 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:29:37.135 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:29:37.136 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:36 smithi016 ceph-mon[22010]: pgmap v4124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:37 smithi016 conmon[32727]: 2022-03-21T19:29:37.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:38.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:38 smithi016 conmon[32727]: 2022-03-21T19:29:38.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:39.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:39.099 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:38 smithi016 ceph-mon[22010]: pgmap v4125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:38 smithi079 ceph-mon[23158]: pgmap v4125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:39 smithi016 conmon[32727]: 2022-03-21T19:29:39.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:40.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:40.137 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:40.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:40 smithi016 conmon[32727]: 2022-03-21T19:29:40.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:41.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:41.153 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:40 smithi016 ceph-mon[22010]: pgmap v4126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:40 smithi079 ceph-mon[23158]: pgmap v4126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:41 smithi016 conmon[32727]: 2022-03-21T19:29:41.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:42.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:42.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:42 smithi016 conmon[32727]: 2022-03-21T19:29:42.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:43.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:43.121 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:42 smithi016 ceph-mon[22010]: pgmap v4127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:42 smithi079 ceph-mon[23158]: pgmap v4127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:43 smithi016 conmon[32727]: 2022-03-21T19:29:43.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:44.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:44 smithi016 conmon[32727]: 2022-03-21T19:29:44.096+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:45.082 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:45.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:44 smithi016 ceph-mon[22010]: pgmap v4128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:45.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:44 smithi079 ceph-mon[23158]: pgmap v4128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:45 smithi016 conmon[32727]: 2022-03-21T19:29:45.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:46.089 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:46.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:46 smithi016 conmon[32727]: 2022-03-21T19:29:46.087+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:47.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:47.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:46 smithi016 ceph-mon[22010]: pgmap v4129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:47.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:46 smithi079 ceph-mon[23158]: pgmap v4129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:47 smithi016 conmon[32727]: 2022-03-21T19:29:47.042+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:48.037 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:48 smithi016 conmon[32727]: 2022-03-21T19:29:48.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:48 smithi016 ceph-mon[22010]: pgmap v4130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:49.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:48 smithi016 conmon[32727]: 2022-03-21T19:29:48.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:48 smithi079 ceph-mon[23158]: pgmap v4130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:49.757 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:50.047 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:50.048 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:50 smithi016 conmon[32727]: 2022-03-21T19:29:50.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:51.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:51.025 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:50 smithi016 ceph-mon[22010]: pgmap v4131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:50 smithi079 ceph-mon[23158]: pgmap v4131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:51 smithi016 conmon[32727]: 2022-03-21T19:29:51.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:52 smithi016 conmon[32727]: 2022-03-21T19:29:52.009+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:53.030 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:53.031 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:52 smithi016 ceph-mon[22010]: pgmap v4132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:52 smithi079 ceph-mon[23158]: pgmap v4132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:53 smithi016 conmon[32727]: 2022-03-21T19:29:53.027+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:54.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:53 smithi016 conmon[32727]: 2022-03-21T19:29:53.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:54.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:54 smithi016 conmon[32727]: 2022-03-21T19:29:54.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:54 smithi016 ceph-mon[22010]: pgmap v4133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:54 smithi079 ceph-mon[23158]: pgmap v4133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:29:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:29:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:55 smithi016 conmon[32727]: 2022-03-21T19:29:55.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:56 smithi016 conmon[32727]: 2022-03-21T19:29:56.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:56 smithi016 ceph-mon[22010]: pgmap v4134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:56 smithi079 ceph-mon[23158]: pgmap v4134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:58.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:57 smithi016 conmon[32727]: 2022-03-21T19:29:57.981+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: pgmap v4135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:29:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:29:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:29:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:29:59.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:58 smithi016 conmon[32727]: 2022-03-21T19:29:58.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:29:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:29:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: pgmap v4135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:29:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:29:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:29:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:29:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:30:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:29:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:00.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:29:59 smithi016 conmon[32727]: 2022-03-21T19:29:59.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:29:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:00 smithi016 conmon[32727]: 2022-03-21T19:30:00.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:01.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops 2022-03-21T19:30:01.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:30:01.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:30:01.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:30:01.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:30:01.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:30:01.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops 2022-03-21T19:30:01.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:00 smithi016 ceph-mon[22010]: pgmap v4136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops 2022-03-21T19:30:01.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:30:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:30:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:30:01.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:30:01.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:30:01.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7640 sec, osd.4 has slow ops 2022-03-21T19:30:01.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:00 smithi079 ceph-mon[23158]: pgmap v4136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:01 smithi016 conmon[32727]: 2022-03-21T19:30:01.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:02 smithi016 ceph-mon[22010]: pgmap v4137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:03 smithi016 conmon[32727]: 2022-03-21T19:30:03.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:02 smithi079 ceph-mon[23158]: pgmap v4137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:04.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:04 smithi016 conmon[32727]: 2022-03-21T19:30:04.034+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:05 smithi016 conmon[32727]: 2022-03-21T19:30:05.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:05.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:04 smithi016 ceph-mon[22010]: pgmap v4138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:05.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:04 smithi079 ceph-mon[23158]: pgmap v4138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:05.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:06 smithi016 conmon[32727]: 2022-03-21T19:30:06.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:07.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:07.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:06 smithi016 ceph-mon[22010]: pgmap v4139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:07.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:07 smithi016 conmon[32727]: 2022-03-21T19:30:07.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:06 smithi079 ceph-mon[23158]: pgmap v4139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:08.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:08 smithi016 conmon[32727]: 2022-03-21T19:30:08.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:09.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:08 smithi016 ceph-mon[22010]: pgmap v4140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:09.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:09 smithi016 conmon[32727]: 2022-03-21T19:30:09.050+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:09.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:08 smithi079 ceph-mon[23158]: pgmap v4140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:10.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:10 smithi016 conmon[32727]: 2022-03-21T19:30:10.032+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:10 smithi016 ceph-mon[22010]: pgmap v4141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:11.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:11 smithi016 conmon[32727]: 2022-03-21T19:30:11.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:10 smithi079 ceph-mon[23158]: pgmap v4141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:12.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:12 smithi016 conmon[32727]: 2022-03-21T19:30:12.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:13.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:13.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:12 smithi016 ceph-mon[22010]: pgmap v4142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:13.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:12 smithi016 conmon[32727]: 2022-03-21T19:30:12.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:13.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:12 smithi079 ceph-mon[23158]: pgmap v4142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:14.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:13 smithi016 conmon[32727]: 2022-03-21T19:30:13.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:14 smithi016 ceph-mon[22010]: pgmap v4143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:15.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:14 smithi016 conmon[32727]: 2022-03-21T19:30:14.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:14 smithi079 ceph-mon[23158]: pgmap v4143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:16.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:15 smithi016 conmon[32727]: 2022-03-21T19:30:15.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:17.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:17.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:16 smithi016 ceph-mon[22010]: pgmap v4144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:17.172 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:16 smithi016 conmon[32727]: 2022-03-21T19:30:16.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:17.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:17.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:16 smithi079 ceph-mon[23158]: pgmap v4144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:17 smithi016 conmon[32727]: 2022-03-21T19:30:17.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:18 smithi016 ceph-mon[22010]: pgmap v4145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:18 smithi016 conmon[32727]: 2022-03-21T19:30:18.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:19.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:18 smithi079 ceph-mon[23158]: pgmap v4145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:19 smithi016 conmon[32727]: 2022-03-21T19:30:19.925+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:21.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:20 smithi016 ceph-mon[22010]: pgmap v4146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:21.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:20 smithi016 conmon[32727]: 2022-03-21T19:30:20.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:20 smithi079 ceph-mon[23158]: pgmap v4146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:22.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:21 smithi016 conmon[32727]: 2022-03-21T19:30:21.920+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:22 smithi016 ceph-mon[22010]: pgmap v4147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:22 smithi016 conmon[32727]: 2022-03-21T19:30:22.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:22 smithi079 ceph-mon[23158]: pgmap v4147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:23 smithi016 conmon[32727]: 2022-03-21T19:30:23.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:24 smithi016 ceph-mon[22010]: pgmap v4148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:25.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:25.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:24 smithi016 conmon[32727]: 2022-03-21T19:30:24.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:24 smithi079 ceph-mon[23158]: pgmap v4148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:26.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:25 smithi016 conmon[32727]: 2022-03-21T19:30:25.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:26 smithi016 ceph-mon[22010]: pgmap v4149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:27.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:26 smithi016 conmon[32727]: 2022-03-21T19:30:26.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:27.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:27.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:26 smithi079 ceph-mon[23158]: pgmap v4149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:27 smithi016 conmon[32727]: 2022-03-21T19:30:27.910+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:28 smithi016 conmon[32727]: 2022-03-21T19:30:28.903+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:29.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:28 smithi016 ceph-mon[22010]: pgmap v4150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:28 smithi079 ceph-mon[23158]: pgmap v4150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:29 smithi016 conmon[32727]: 2022-03-21T19:30:29.906+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:30 smithi016 conmon[32727]: 2022-03-21T19:30:30.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:31.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:30 smithi016 ceph-mon[22010]: pgmap v4151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:31.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:31.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:30 smithi079 ceph-mon[23158]: pgmap v4151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:32.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:31 smithi016 conmon[32727]: 2022-03-21T19:30:31.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:32 smithi016 conmon[32727]: 2022-03-21T19:30:32.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:33.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:33.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:32 smithi016 ceph-mon[22010]: pgmap v4152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:33.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:32 smithi079 ceph-mon[23158]: pgmap v4152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:33 smithi016 conmon[32727]: 2022-03-21T19:30:33.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:34.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:34 smithi016 conmon[32727]: 2022-03-21T19:30:34.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:35.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:34 smithi016 ceph-mon[22010]: pgmap v4153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:35.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:34 smithi079 ceph-mon[23158]: pgmap v4153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:36.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:35 smithi016 conmon[32727]: 2022-03-21T19:30:35.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:36.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[32727]: 2022-03-21T19:30:36.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:30:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:30:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:30:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:36 smithi016 ceph-mon[22010]: pgmap v4154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:37.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: 2022-03-21T19:30:36.877+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:37.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:30:37.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:37.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: yield 2022-03-21T19:30:37.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:37.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:37.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:37.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:30:37.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:37.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:30:37.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:37.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:30:37.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:37.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:30:37.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:37.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:30:37.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:37.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:37.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:30:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:30:37.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:30:37.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:36 smithi079 ceph-mon[23158]: pgmap v4154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:37 smithi016 conmon[32727]: 2022-03-21T19:30:37.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:38.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:30:38.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:38.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:30:38.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:38.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:38.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:38.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:30:38.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:38.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:30:38.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:38.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:30:38.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:38.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:38.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:38.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:30:38.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:38.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:38.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:38.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:37 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:30:38.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:30:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:38.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:30:38.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:38.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:30:38.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:38.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:30:38.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:38.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:30:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:38.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:30:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:38.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:38.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:37 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:30:38.235 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:38 smithi016 ceph-mon[22010]: pgmap v4155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:39.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:38 smithi016 conmon[32727]: 2022-03-21T19:30:38.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:38 smithi079 ceph-mon[23158]: pgmap v4155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7681 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:40.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[32727]: 2022-03-21T19:30:39.925+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:40.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 2022-03-21T19:30:39.949+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:40.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:30:40.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:40.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: yield 2022-03-21T19:30:40.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:40.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:40.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:40.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:30:40.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:40.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:30:40.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:40.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:30:40.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:40.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:30:40.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:40.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:30:40.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:40.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:40.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:40.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 2022-03-21T19:30:39.950+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:30:40.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:30:40.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:40.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: yield 2022-03-21T19:30:40.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:40.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:40.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:40.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:30:40.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:40.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:30:40.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:40.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:30:40.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:40.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:30:40.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:40.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:30:40.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:40.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:40.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:40.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 2022-03-21T19:30:40.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:30:40.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: 2022-03-21T19:30:40.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:30:40.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:30:40.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:30:40.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:30:40.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:30:40.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:30:40.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:30:40.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:30:40.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:30:40.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:30:40.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:30:40.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:30:40.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:30:40.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:30:40.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:30:40.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:30:40.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:30:40.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:30:40.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:30:40.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:30:40.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:30:40.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:30:40.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise 2022-03-21T19:30:40.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:30:40.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:30:40.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:30:40.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:30:40.196 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:30:39 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7681 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:40 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:30:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:41.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:41.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:30:41.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:41.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:30:41.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:30:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:41.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:41.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:41.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:30:41.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:41.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:41.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:40 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:30:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:41.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:30:41.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:41.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:30:41.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:41.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:30:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:30:41.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:41.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:30:41.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:41.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:41.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:30:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:30:41.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:30:41.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:30:41.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:30:41.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:30:41.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:30:41.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:30:41.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:30:41.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:30:41.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:30:41.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:30:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:30:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:30:41.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:30:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:30:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:30:41.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:30:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:30:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:30:41.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:30:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:30:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:30:41.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:30:41.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:30:41.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:30:41.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:30:41.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:30:41.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:30:41.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.251 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:40 smithi079 ceph-mon[23158]: pgmap v4156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:40 smithi016 conmon[32727]: 2022-03-21T19:30:40.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:41.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:40 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:30:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:41.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:41.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:41.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:30:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:30:41.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:41.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:30:41.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:41.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:30:41.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.424 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:40 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:30:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:30:41.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:30:41.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:30:41.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:30:41.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:30:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:30:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:30:41.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:30:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:30:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:30:41.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:30:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:30:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:30:41.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:30:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:30:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:30:41.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:30:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:30:41.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:30:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:30:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:30:41.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:30:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:30:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:30:41.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:30:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:30:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:30:41.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:30:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:30:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:30:41.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:30:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:30:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:30:41.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:30:41.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:30:41.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:30:41.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:30:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:30:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:30:41.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:30:41.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:30:41.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:30:41.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:30:41.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:30:41.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:30:41.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:30:41.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:30:41.443 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:40 smithi016 ceph-mon[22010]: pgmap v4156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:42.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:41 smithi016 conmon[32727]: 2022-03-21T19:30:41.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:42 smithi079 ceph-mon[23158]: pgmap v4157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:42 smithi016 ceph-mon[22010]: pgmap v4157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:43.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:42 smithi016 conmon[32727]: 2022-03-21T19:30:42.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:43 smithi016 conmon[32727]: 2022-03-21T19:30:43.920+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:44 smithi079 ceph-mon[23158]: pgmap v4158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:45.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:44 smithi016 ceph-mon[22010]: pgmap v4158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:45.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:44 smithi016 conmon[32727]: 2022-03-21T19:30:44.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:45 smithi016 conmon[32727]: 2022-03-21T19:30:45.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:47.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:46 smithi016 ceph-mon[22010]: pgmap v4159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:47.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:46 smithi016 conmon[32727]: 2022-03-21T19:30:46.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:46 smithi079 ceph-mon[23158]: pgmap v4159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:48.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:47 smithi016 conmon[32727]: 2022-03-21T19:30:47.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:48 smithi016 conmon[32727]: 2022-03-21T19:30:48.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:49.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:48 smithi016 ceph-mon[22010]: pgmap v4160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:49.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:48 smithi079 ceph-mon[23158]: pgmap v4160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:50.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:49 smithi016 conmon[32727]: 2022-03-21T19:30:49.805+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:51.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:50 smithi016 ceph-mon[22010]: pgmap v4161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:51.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:50 smithi016 conmon[32727]: 2022-03-21T19:30:50.788+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:51.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:50 smithi079 ceph-mon[23158]: pgmap v4161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:51 smithi016 conmon[32727]: 2022-03-21T19:30:51.793+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:53.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:53.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:52 smithi016 ceph-mon[22010]: pgmap v4162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:53.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:52 smithi016 conmon[32727]: 2022-03-21T19:30:52.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:53.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:52 smithi079 ceph-mon[23158]: pgmap v4162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:54.392 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:53 smithi016 conmon[32727]: 2022-03-21T19:30:53.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:54.394 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:54 smithi016 ceph-mon[22010]: pgmap v4163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:55.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:55.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:54 smithi016 conmon[32727]: 2022-03-21T19:30:54.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:54 smithi079 ceph-mon[23158]: pgmap v4163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:30:55.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:30:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:56.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:55 smithi016 conmon[32727]: 2022-03-21T19:30:55.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:57.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:56 smithi016 ceph-mon[22010]: pgmap v4164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:57.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:56 smithi016 conmon[32727]: 2022-03-21T19:30:56.920+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:56 smithi079 ceph-mon[23158]: pgmap v4164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:58.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:58.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:57 smithi016 conmon[32727]: 2022-03-21T19:30:57.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:58 smithi016 conmon[32727]: 2022-03-21T19:30:58.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:30:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: pgmap v4165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:30:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:30:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:30:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:30:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:30:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: pgmap v4165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:30:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:30:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:30:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:30:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:31:00.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:00.178 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:30:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:00.179 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:30:59 smithi016 conmon[32727]: 2022-03-21T19:30:59.901+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:30:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:00 smithi016 ceph-mon[22010]: pgmap v4166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:00 smithi016 conmon[32727]: 2022-03-21T19:31:00.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:00 smithi079 ceph-mon[23158]: pgmap v4166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:02.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:01 smithi016 conmon[32727]: 2022-03-21T19:31:01.901+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:02 smithi016 ceph-mon[22010]: pgmap v4167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:02 smithi016 conmon[32727]: 2022-03-21T19:31:02.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:02 smithi079 ceph-mon[23158]: pgmap v4167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:03 smithi016 conmon[32727]: 2022-03-21T19:31:03.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:04 smithi016 conmon[32727]: 2022-03-21T19:31:04.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:05 smithi016 ceph-mon[22010]: pgmap v4168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:05.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:05 smithi079 ceph-mon[23158]: pgmap v4168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:06.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:05 smithi016 conmon[32727]: 2022-03-21T19:31:05.767+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:07.012 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:06 smithi016 conmon[32727]: 2022-03-21T19:31:06.752+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:07.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:07 smithi016 ceph-mon[22010]: pgmap v4169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:07 smithi079 ceph-mon[23158]: pgmap v4169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:08.013 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:07 smithi016 conmon[32727]: 2022-03-21T19:31:07.703+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:08.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:09.018 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:08 smithi016 conmon[32727]: 2022-03-21T19:31:08.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:09.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:09 smithi016 ceph-mon[22010]: pgmap v4170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:09 smithi079 ceph-mon[23158]: pgmap v4170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:10.020 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:09 smithi016 conmon[32727]: 2022-03-21T19:31:09.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7711 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7711 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:11.029 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:10 smithi016 conmon[32727]: 2022-03-21T19:31:10.689+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:11.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:11 smithi016 ceph-mon[22010]: pgmap v4171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:11 smithi079 ceph-mon[23158]: pgmap v4171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:11 smithi016 conmon[32727]: 2022-03-21T19:31:11.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:12 smithi016 conmon[32727]: 2022-03-21T19:31:12.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:13.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:13 smithi016 ceph-mon[22010]: pgmap v4172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:13 smithi079 ceph-mon[23158]: pgmap v4172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:13 smithi016 conmon[32727]: 2022-03-21T19:31:13.569+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:14.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:14 smithi016 conmon[32727]: 2022-03-21T19:31:14.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:15 smithi016 ceph-mon[22010]: pgmap v4173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:15.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:15 smithi079 ceph-mon[23158]: pgmap v4173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:15 smithi016 conmon[32727]: 2022-03-21T19:31:15.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:16 smithi016 conmon[32727]: 2022-03-21T19:31:16.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:17.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:17 smithi016 ceph-mon[22010]: pgmap v4174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:17.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:17 smithi079 ceph-mon[23158]: pgmap v4174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:18.049 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:17 smithi016 conmon[32727]: 2022-03-21T19:31:17.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:18.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:18 smithi016 conmon[32727]: 2022-03-21T19:31:18.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:19 smithi016 ceph-mon[22010]: pgmap v4175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:19.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:19 smithi079 ceph-mon[23158]: pgmap v4175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:20.056 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:19 smithi016 conmon[32727]: 2022-03-21T19:31:19.686+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:21.061 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:20 smithi016 conmon[32727]: 2022-03-21T19:31:20.665+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:21 smithi016 ceph-mon[22010]: pgmap v4176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:21 smithi079 ceph-mon[23158]: pgmap v4176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:21 smithi016 conmon[32727]: 2022-03-21T19:31:21.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:22 smithi016 conmon[32727]: 2022-03-21T19:31:22.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:23.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:23 smithi016 ceph-mon[22010]: pgmap v4177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:23.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:23 smithi079 ceph-mon[23158]: pgmap v4177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:23.922 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:23 smithi016 conmon[32727]: 2022-03-21T19:31:23.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:24 smithi016 conmon[32727]: 2022-03-21T19:31:24.529+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:25 smithi016 ceph-mon[22010]: pgmap v4178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:25.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:25 smithi079 ceph-mon[23158]: pgmap v4178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:25 smithi016 conmon[32727]: 2022-03-21T19:31:25.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:26 smithi016 conmon[32727]: 2022-03-21T19:31:26.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:27 smithi016 ceph-mon[22010]: pgmap v4179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:27.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:27 smithi079 ceph-mon[23158]: pgmap v4179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:27 smithi016 conmon[32727]: 2022-03-21T19:31:27.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:28 smithi016 conmon[32727]: 2022-03-21T19:31:28.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:29 smithi016 ceph-mon[22010]: pgmap v4180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:29 smithi079 ceph-mon[23158]: pgmap v4180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:29 smithi016 conmon[32727]: 2022-03-21T19:31:29.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:30 smithi016 conmon[32727]: 2022-03-21T19:31:30.522+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:31 smithi016 ceph-mon[22010]: pgmap v4181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:31.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:31 smithi079 ceph-mon[23158]: pgmap v4181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:31 smithi016 conmon[32727]: 2022-03-21T19:31:31.506+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:32.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:32 smithi016 conmon[32727]: 2022-03-21T19:31:32.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:33.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:33 smithi016 ceph-mon[22010]: pgmap v4182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:33.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:33 smithi079 ceph-mon[23158]: pgmap v4182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:33 smithi016 conmon[32727]: 2022-03-21T19:31:33.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:34 smithi016 conmon[32727]: 2022-03-21T19:31:34.453+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:35 smithi016 ceph-mon[22010]: pgmap v4183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:35 smithi079 ceph-mon[23158]: pgmap v4183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:35.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:35 smithi016 conmon[32727]: 2022-03-21T19:31:35.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:36 smithi016 conmon[32727]: 2022-03-21T19:31:36.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:37 smithi016 ceph-mon[22010]: pgmap v4184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:37 smithi079 ceph-mon[23158]: pgmap v4184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:37 smithi016 conmon[32727]: 2022-03-21T19:31:37.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:38 smithi016 conmon[32727]: 2022-03-21T19:31:38.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:39 smithi016 ceph-mon[22010]: pgmap v4185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:39.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:39 smithi079 ceph-mon[23158]: pgmap v4185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:39 smithi016 conmon[32727]: 2022-03-21T19:31:39.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:40.403 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:40.404 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:40.404 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:31:40.405 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:31:40.405 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:31:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:31:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:31:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:31:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:40 smithi016 conmon[32727]: 2022-03-21T19:31:40.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:41 smithi016 ceph-mon[22010]: pgmap v4186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:41 smithi079 ceph-mon[23158]: pgmap v4186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:41 smithi016 conmon[32727]: 2022-03-21T19:31:41.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:42.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:42 smithi016 conmon[32727]: 2022-03-21T19:31:42.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:43 smithi016 ceph-mon[22010]: pgmap v4187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:43.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: 2022-03-21T19:31:43.053+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:43.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:31:43.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:43.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: yield 2022-03-21T19:31:43.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:43.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:43.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:43.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:31:43.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:43.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:31:43.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:43.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:31:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:31:43.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:43.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:31:43.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:43.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:43.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:43 smithi079 ceph-mon[23158]: pgmap v4187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:43 smithi016 conmon[32727]: 2022-03-21T19:31:43.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:44.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:44 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:31:44.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:44.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:31:44.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:44.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:44.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:44.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:31:44.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:44.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:31:44.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:44.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:31:44.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:44.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:44.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:44.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:31:44.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:44.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:44.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:44.426 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:44 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:31:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:44 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:31:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:31:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:44.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:44.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:44.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:31:44.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:44.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:31:44.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:44.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:31:44.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:44.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:44.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:44.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:31:44.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:44.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:44.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:44.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:44 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:31:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:44 smithi016 conmon[32727]: 2022-03-21T19:31:44.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:45 smithi016 ceph-mon[22010]: pgmap v4188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:45 smithi079 ceph-mon[23158]: pgmap v4188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:45 smithi016 conmon[32727]: 2022-03-21T19:31:45.424+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:46.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 2022-03-21T19:31:46.125+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:46.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:31:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: yield 2022-03-21T19:31:46.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:46.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:46.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:46.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:31:46.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:46.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:31:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:31:46.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:31:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:46.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:31:46.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:46.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 2022-03-21T19:31:46.126+0000 7f9272ee0700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:31:46.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:31:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: yield 2022-03-21T19:31:46.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:46.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:31:46.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:46.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:31:46.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:46.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:31:46.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:31:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:46.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:31:46.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:46.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:46.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:46.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 2022-03-21T19:31:46.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:31:46.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: 2022-03-21T19:31:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:31:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:31:46.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:31:46.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:31:46.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:31:46.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:31:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:31:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:31:46.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:31:46.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:31:46.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:31:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:31:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:31:46.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:31:46.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:31:46.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:31:46.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:31:46.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:31:46.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:31:46.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:31:46.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:31:46.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:31:46.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise 2022-03-21T19:31:46.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:31:46.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:31:46.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:31:46.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:31:46.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:46 smithi016 conmon[32727]: 2022-03-21T19:31:46.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:47.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:47.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:47.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:31:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:47.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:47.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:31:47.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:31:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:31:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:47.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:47.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:31:47.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:47.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:47.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:47 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:31:47.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:47.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:31:47.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:47.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:47.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:47.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:31:47.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:47.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:31:47.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:47.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:31:47.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:47.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:47.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:47.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:31:47.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:47.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:47.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:31:47.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:31:47.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:31:47.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:31:47.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:31:47.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:31:47.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:31:47.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:31:47.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:31:47.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:31:47.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:31:47.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:31:47.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:31:47.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:31:47.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:31:47.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:31:47.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:31:47.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:31:47.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:31:47.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:31:47.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:31:47.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:31:47.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:31:47.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:31:47.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:31:47.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:31:47.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:31:47.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:31:47.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:31:47.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.442 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:47 smithi016 ceph-mon[22010]: pgmap v4189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:31:47.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:47.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:47.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:47.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:31:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:31:47.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:47.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:31:47.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:47.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:31:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:47.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:47 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:31:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:31:47.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:31:47.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:31:47.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:31:47.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:31:47.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:31:47.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:31:47.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:31:47.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:31:47.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:31:47.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:31:47.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:31:47.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:31:47.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:31:47.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:31:47.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:31:47.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:31:47.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:31:47.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:31:47.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:31:47.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:31:47.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:31:47.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:31:47.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:31:47.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:31:47.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:31:47.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:31:47.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:31:47.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:31:47.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:31:47.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:31:47.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:31:47.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:31:47.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:31:47.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:31:47.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:31:47.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:31:47.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:31:47.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:31:47.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:31:47.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:31:47.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:31:47.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:31:47.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:31:47.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:31:47.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:31:47.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:31:47.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:47 smithi079 ceph-mon[23158]: pgmap v4189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:47 smithi016 conmon[32727]: 2022-03-21T19:31:47.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:48 smithi016 conmon[32727]: 2022-03-21T19:31:48.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:49.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:49 smithi016 ceph-mon[22010]: pgmap v4190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:49.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:49 smithi079 ceph-mon[23158]: pgmap v4190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:49 smithi016 conmon[32727]: 2022-03-21T19:31:49.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:50.407 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:50.407 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:50 smithi016 conmon[32727]: 2022-03-21T19:31:50.405+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:51 smithi016 conmon[32727]: 2022-03-21T19:31:51.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:51.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:51 smithi016 ceph-mon[22010]: pgmap v4191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:51.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:51 smithi079 ceph-mon[23158]: pgmap v4191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:52.410 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:52.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:52 smithi016 conmon[32727]: 2022-03-21T19:31:52.409+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:53 smithi016 conmon[32727]: 2022-03-21T19:31:53.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:53.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:53 smithi016 ceph-mon[22010]: pgmap v4192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:53.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:53 smithi079 ceph-mon[23158]: pgmap v4192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:54 smithi016 conmon[32727]: 2022-03-21T19:31:54.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:55.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:55 smithi079 ceph-mon[23158]: pgmap v4193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:55.676 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:55.677 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:55 smithi016 conmon[32727]: 2022-03-21T19:31:55.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:55.678 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:55.678 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:55 smithi016 ceph-mon[22010]: pgmap v4193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:31:55.679 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:31:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:56.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:56.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:56 smithi016 conmon[32727]: 2022-03-21T19:31:56.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:57.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:57 smithi079 ceph-mon[23158]: pgmap v4194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:57 smithi016 ceph-mon[22010]: pgmap v4194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:57 smithi016 conmon[32727]: 2022-03-21T19:31:57.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:58 smithi016 conmon[32727]: 2022-03-21T19:31:58.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: pgmap v4195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:31:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:31:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:31:59.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:31:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:31:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:31:59 smithi016 conmon[32727]: 2022-03-21T19:31:59.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:31:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:31:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: pgmap v4195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:31:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:31:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:31:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:32:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:00 smithi016 conmon[32727]: 2022-03-21T19:32:00.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:01 smithi079 ceph-mon[23158]: pgmap v4196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:01 smithi016 ceph-mon[22010]: pgmap v4196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:01.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:01 smithi016 conmon[32727]: 2022-03-21T19:32:01.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:02 smithi016 conmon[32727]: 2022-03-21T19:32:02.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:03.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:03 smithi079 ceph-mon[23158]: pgmap v4197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:03.480 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:03.480 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:03 smithi016 ceph-mon[22010]: pgmap v4197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:03 smithi016 conmon[32727]: 2022-03-21T19:32:03.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:04.472 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:04 smithi016 conmon[32727]: 2022-03-21T19:32:04.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:05 smithi079 ceph-mon[23158]: pgmap v4198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:05.493 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:05.494 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:05 smithi016 ceph-mon[22010]: pgmap v4198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:05.494 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:05 smithi016 conmon[32727]: 2022-03-21T19:32:05.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:06.476 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:06 smithi016 conmon[32727]: 2022-03-21T19:32:06.474+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:07.463 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:07.464 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:07 smithi016 ceph-mon[22010]: pgmap v4199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:07 smithi079 ceph-mon[23158]: pgmap v4199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:07 smithi016 conmon[32727]: 2022-03-21T19:32:07.461+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:08 smithi016 conmon[32727]: 2022-03-21T19:32:08.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:09 smithi016 conmon[32727]: 2022-03-21T19:32:09.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:10 smithi079 ceph-mon[23158]: pgmap v4200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:10 smithi016 ceph-mon[22010]: pgmap v4200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:10 smithi016 conmon[32727]: 2022-03-21T19:32:10.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:11.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:11.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:11 smithi016 conmon[32727]: 2022-03-21T19:32:11.406+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:12 smithi079 ceph-mon[23158]: pgmap v4201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:12 smithi016 ceph-mon[22010]: pgmap v4201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:12 smithi016 conmon[32727]: 2022-03-21T19:32:12.369+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:13.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:13 smithi016 conmon[32727]: 2022-03-21T19:32:13.323+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:14 smithi016 ceph-mon[22010]: pgmap v4202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:14.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:14 smithi016 conmon[32727]: 2022-03-21T19:32:14.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:14 smithi079 ceph-mon[23158]: pgmap v4202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:15.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:15 smithi016 conmon[32727]: 2022-03-21T19:32:15.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:16 smithi016 ceph-mon[22010]: pgmap v4203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:16 smithi016 conmon[32727]: 2022-03-21T19:32:16.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:16 smithi079 ceph-mon[23158]: pgmap v4203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:17.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:17.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:17 smithi016 conmon[32727]: 2022-03-21T19:32:17.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:18 smithi016 ceph-mon[22010]: pgmap v4204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:18 smithi016 conmon[32727]: 2022-03-21T19:32:18.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:18 smithi079 ceph-mon[23158]: pgmap v4204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:19 smithi016 conmon[32727]: 2022-03-21T19:32:19.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:20 smithi016 ceph-mon[22010]: pgmap v4205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:20 smithi016 conmon[32727]: 2022-03-21T19:32:20.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:20 smithi079 ceph-mon[23158]: pgmap v4205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:20.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:21 smithi016 conmon[32727]: 2022-03-21T19:32:21.252+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:21.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:22 smithi016 conmon[32727]: 2022-03-21T19:32:22.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:22 smithi016 ceph-mon[22010]: pgmap v4206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:22.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:22 smithi079 ceph-mon[23158]: pgmap v4206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:23 smithi016 conmon[32727]: 2022-03-21T19:32:23.224+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:24 smithi016 ceph-mon[22010]: pgmap v4207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:24.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:24 smithi016 conmon[32727]: 2022-03-21T19:32:24.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:24 smithi079 ceph-mon[23158]: pgmap v4207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:25 smithi016 conmon[32727]: 2022-03-21T19:32:25.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:25.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:26 smithi016 conmon[32727]: 2022-03-21T19:32:26.249+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:26 smithi016 ceph-mon[22010]: pgmap v4208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:26.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:26 smithi079 ceph-mon[23158]: pgmap v4208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:26.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:27.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:27 smithi016 conmon[32727]: 2022-03-21T19:32:27.250+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:28 smithi016 ceph-mon[22010]: pgmap v4209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:28 smithi016 conmon[32727]: 2022-03-21T19:32:28.243+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:28 smithi079 ceph-mon[23158]: pgmap v4209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:29.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:29 smithi016 conmon[32727]: 2022-03-21T19:32:29.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:30 smithi016 ceph-mon[22010]: pgmap v4210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:30 smithi016 conmon[32727]: 2022-03-21T19:32:30.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:30 smithi079 ceph-mon[23158]: pgmap v4210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:31.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:31.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:31 smithi016 conmon[32727]: 2022-03-21T19:32:31.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:32 smithi016 ceph-mon[22010]: pgmap v4211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:32 smithi016 conmon[32727]: 2022-03-21T19:32:32.161+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:32 smithi079 ceph-mon[23158]: pgmap v4211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:33 smithi016 conmon[32727]: 2022-03-21T19:32:33.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:34 smithi016 ceph-mon[22010]: pgmap v4212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:34 smithi016 conmon[32727]: 2022-03-21T19:32:34.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:34 smithi079 ceph-mon[23158]: pgmap v4212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:35.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:35.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:35 smithi016 conmon[32727]: 2022-03-21T19:32:35.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:36 smithi016 ceph-mon[22010]: pgmap v4213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:36 smithi016 conmon[32727]: 2022-03-21T19:32:36.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:36 smithi079 ceph-mon[23158]: pgmap v4213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:37.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:37 smithi016 conmon[32727]: 2022-03-21T19:32:37.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:38 smithi016 ceph-mon[22010]: pgmap v4214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:38 smithi016 conmon[32727]: 2022-03-21T19:32:38.114+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:38 smithi079 ceph-mon[23158]: pgmap v4214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:39.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:39.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:39 smithi016 conmon[32727]: 2022-03-21T19:32:39.070+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:40 smithi016 ceph-mon[22010]: pgmap v4215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:40.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:40 smithi016 conmon[32727]: 2022-03-21T19:32:40.086+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:40 smithi079 ceph-mon[23158]: pgmap v4215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:41.320 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:41 smithi016 conmon[32727]: 2022-03-21T19:32:41.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:42.320 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:42 smithi016 conmon[32727]: 2022-03-21T19:32:42.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:42 smithi016 ceph-mon[22010]: pgmap v4216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:42 smithi079 ceph-mon[23158]: pgmap v4216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:43.327 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:43 smithi016 conmon[32727]: 2022-03-21T19:32:43.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:44.327 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:44 smithi016 conmon[32727]: 2022-03-21T19:32:44.000+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:44 smithi016 ceph-mon[22010]: pgmap v4217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:44 smithi079 ceph-mon[23158]: pgmap v4217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:45.332 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:45 smithi016 conmon[32727]: 2022-03-21T19:32:45.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:46.334 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:45 smithi016 conmon[32727]: 2022-03-21T19:32:45.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:46 smithi079 ceph-mon[23158]: pgmap v4218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:32:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:32:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:32:46.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:46 smithi016 ceph-mon[22010]: pgmap v4218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:46.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:46.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:32:46.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:32:46.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:32:46.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: 2022-03-21T19:32:46.605+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:46.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:32:46.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:46.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: yield 2022-03-21T19:32:46.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:46.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:46.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:32:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:46.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:32:46.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:46.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:32:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:32:46.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:32:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:46.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:46.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:47.338 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:46 smithi016 conmon[32727]: 2022-03-21T19:32:46.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:48.342 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:47 smithi016 conmon[32727]: 2022-03-21T19:32:47.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:48 smithi016 ceph-mon[22010]: pgmap v4219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:32:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:32:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:48.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:48.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:32:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:32:48.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:48.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:32:48.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:48.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:48.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:48.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:32:48.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:48.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:48 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:32:48.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:48 smithi079 ceph-mon[23158]: pgmap v4219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:32:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:32:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:32:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:32:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:32:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:32:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:48 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:32:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:49.345 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:48 smithi016 conmon[32727]: 2022-03-21T19:32:48.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:49.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 2022-03-21T19:32:49.677+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:49.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:32:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: yield 2022-03-21T19:32:49.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:49.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:49.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:49.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:32:49.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:49.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:32:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:32:49.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:32:49.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:49.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:32:49.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:49.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:49.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:49.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 2022-03-21T19:32:49.678+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:32:49.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:32:49.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:49.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: yield 2022-03-21T19:32:49.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:49.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:49.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:49.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:32:49.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:49.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:32:49.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:49.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:32:49.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:49.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:32:49.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:49.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:32:49.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:49.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:49.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:49.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 2022-03-21T19:32:49.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:32:49.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: 2022-03-21T19:32:49.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:32:49.953 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:32:49.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:32:49.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:32:49.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:32:49.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:32:49.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:32:49.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:32:49.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:32:49.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:32:49.956 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:32:49.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:32:49.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:32:49.957 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:32:49.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:32:49.958 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:32:49.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:32:49.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:32:49.959 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:32:49.960 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:32:49.960 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:32:49.960 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:32:49.961 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise 2022-03-21T19:32:49.961 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:32:49.961 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:32:49.962 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:32:49.962 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:32:49.962 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:50.346 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:49 smithi016 conmon[32727]: 2022-03-21T19:32:49.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:50 smithi016 ceph-mon[22010]: pgmap v4220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:50 smithi079 ceph-mon[23158]: pgmap v4220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:51.354 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:50 smithi016 conmon[32727]: 2022-03-21T19:32:50.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:32:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:51.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:32:51.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:51.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:32:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:32:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:32:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:51 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:32:51.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:51.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:32:51.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:51.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:51.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:51.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:32:51.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:51.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:32:51.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:51.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:32:51.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:51.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:51.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:51.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:32:51.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:51.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:51.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:32:51.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:32:51.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:32:51.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:32:51.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:32:51.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:32:51.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:32:51.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:32:51.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:32:51.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:32:51.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:32:51.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:32:51.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:32:51.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:32:51.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:32:51.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:32:51.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:32:51.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:32:51.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:32:51.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:32:51.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:32:51.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:32:51.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:32:51.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:32:51.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:32:51.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:32:51.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:32:51.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:32:51.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:32:51.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.696 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:32:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:32:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:32:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:32:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:32:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:51.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:51 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:32:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:32:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:32:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:32:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:32:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:32:51.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:32:51.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:32:51.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:32:51.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:32:51.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:32:51.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:32:51.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:32:51.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:32:51.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:32:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:32:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:32:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:32:51.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:32:51.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:32:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:32:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:32:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:32:51.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:32:51.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:32:51.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:32:51.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:32:51.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:32:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:32:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:32:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:32:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:32:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:32:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:32:51.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:32:51.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:32:51.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:32:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:32:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:32:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:32:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:32:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:32:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:32:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:32:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:32:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:32:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:32:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:32:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:52.353 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:51 smithi016 conmon[32727]: 2022-03-21T19:32:51.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:52 smithi016 ceph-mon[22010]: pgmap v4221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:52 smithi079 ceph-mon[23158]: pgmap v4221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:52.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:53.358 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:52 smithi016 conmon[32727]: 2022-03-21T19:32:52.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:54.360 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:53 smithi016 conmon[32727]: 2022-03-21T19:32:53.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:54 smithi016 ceph-mon[22010]: pgmap v4222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:54 smithi079 ceph-mon[23158]: pgmap v4222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:55.365 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:54 smithi016 conmon[32727]: 2022-03-21T19:32:54.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:32:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:56.366 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:55 smithi016 conmon[32727]: 2022-03-21T19:32:55.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:56 smithi016 ceph-mon[22010]: pgmap v4223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:56 smithi079 ceph-mon[23158]: pgmap v4223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:32:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:57.371 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:56 smithi016 conmon[32727]: 2022-03-21T19:32:56.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:58.372 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:58 smithi016 conmon[32727]: 2022-03-21T19:32:58.015+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:58 smithi016 ceph-mon[22010]: pgmap v4224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:58 smithi079 ceph-mon[23158]: pgmap v4224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:32:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:59.377 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:32:59 smithi016 conmon[32727]: 2022-03-21T19:32:59.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:32:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:32:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:32:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:32:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:32:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:32:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:32:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:32:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:32:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:32:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:32:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:32:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:00.379 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:00 smithi016 conmon[32727]: 2022-03-21T19:33:00.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:00 smithi016 ceph-mon[22010]: pgmap v4225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:00 smithi079 ceph-mon[23158]: pgmap v4225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:01.384 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:01 smithi016 conmon[32727]: 2022-03-21T19:33:01.016+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:02.385 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:02 smithi016 conmon[32727]: 2022-03-21T19:33:02.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:02.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:02 smithi016 ceph-mon[22010]: pgmap v4226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:02.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:02 smithi079 ceph-mon[23158]: pgmap v4226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:03.390 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:03 smithi016 conmon[32727]: 2022-03-21T19:33:03.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:04.392 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:04 smithi016 conmon[32727]: 2022-03-21T19:33:04.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:04 smithi016 ceph-mon[22010]: pgmap v4227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:04 smithi079 ceph-mon[23158]: pgmap v4227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:05.396 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:05 smithi016 conmon[32727]: 2022-03-21T19:33:05.041+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:06.397 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:06 smithi016 conmon[32727]: 2022-03-21T19:33:06.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:06 smithi016 ceph-mon[22010]: pgmap v4228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:06 smithi079 ceph-mon[23158]: pgmap v4228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:07.402 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:07 smithi016 conmon[32727]: 2022-03-21T19:33:07.036+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:08.404 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:08 smithi016 conmon[32727]: 2022-03-21T19:33:07.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:08 smithi016 ceph-mon[22010]: pgmap v4229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:08 smithi079 ceph-mon[23158]: pgmap v4229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:09.409 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:09 smithi016 conmon[32727]: 2022-03-21T19:33:09.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:09 smithi016 ceph-mon[22010]: pgmap v4230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:09.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:09 smithi079 ceph-mon[23158]: pgmap v4230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:09.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:10.410 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:10 smithi016 conmon[32727]: 2022-03-21T19:33:10.025+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:11 smithi016 conmon[32727]: 2022-03-21T19:33:11.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:11 smithi079 ceph-mon[23158]: pgmap v4231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:11.994 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:11 smithi016 ceph-mon[22010]: pgmap v4231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:11.994 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:11 smithi016 conmon[32727]: 2022-03-21T19:33:11.992+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:12.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:12.995 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:12 smithi016 conmon[32727]: 2022-03-21T19:33:12.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:13 smithi016 conmon[32727]: 2022-03-21T19:33:13.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:13 smithi016 ceph-mon[22010]: pgmap v4232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:14.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:13 smithi079 ceph-mon[23158]: pgmap v4232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:15.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:14 smithi016 conmon[32727]: 2022-03-21T19:33:14.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:15.996 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:15 smithi016 ceph-mon[22010]: pgmap v4233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:15.997 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:15 smithi079 ceph-mon[23158]: pgmap v4233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:15 smithi016 conmon[32727]: 2022-03-21T19:33:15.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:16 smithi016 conmon[32727]: 2022-03-21T19:33:16.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:17 smithi016 ceph-mon[22010]: pgmap v4234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:17 smithi016 conmon[32727]: 2022-03-21T19:33:17.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:17 smithi079 ceph-mon[23158]: pgmap v4234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:18.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:19.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:18 smithi016 conmon[32727]: 2022-03-21T19:33:18.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:19 smithi016 ceph-mon[22010]: pgmap v4235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:19 smithi016 conmon[32727]: 2022-03-21T19:33:19.957+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:19 smithi079 ceph-mon[23158]: pgmap v4235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:21.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:21.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:20 smithi016 conmon[32727]: 2022-03-21T19:33:20.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:21 smithi016 ceph-mon[22010]: pgmap v4236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:22.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:21 smithi016 conmon[32727]: 2022-03-21T19:33:21.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:21 smithi079 ceph-mon[23158]: pgmap v4236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:22 smithi016 conmon[32727]: 2022-03-21T19:33:22.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:23 smithi016 ceph-mon[22010]: pgmap v4237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:23 smithi016 conmon[32727]: 2022-03-21T19:33:23.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:23 smithi079 ceph-mon[23158]: pgmap v4237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:25.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:24 smithi016 conmon[32727]: 2022-03-21T19:33:24.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:25 smithi016 ceph-mon[22010]: pgmap v4238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:26.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:25 smithi016 conmon[32727]: 2022-03-21T19:33:25.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:25 smithi079 ceph-mon[23158]: pgmap v4238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:27.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:26 smithi016 conmon[32727]: 2022-03-21T19:33:26.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:27 smithi016 ceph-mon[22010]: pgmap v4239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:28.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:27 smithi016 conmon[32727]: 2022-03-21T19:33:27.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:27 smithi079 ceph-mon[23158]: pgmap v4239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:29.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:28 smithi016 conmon[32727]: 2022-03-21T19:33:28.950+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:29 smithi016 ceph-mon[22010]: pgmap v4240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:30.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:29 smithi016 conmon[32727]: 2022-03-21T19:33:29.990+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:29 smithi079 ceph-mon[23158]: pgmap v4240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:30 smithi016 conmon[32727]: 2022-03-21T19:33:30.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:31 smithi016 ceph-mon[22010]: pgmap v4241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:32.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:31 smithi016 conmon[32727]: 2022-03-21T19:33:31.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:31 smithi079 ceph-mon[23158]: pgmap v4241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:33.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:33 smithi016 conmon[32727]: 2022-03-21T19:33:33.018+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:34.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:33 smithi016 ceph-mon[22010]: pgmap v4242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:34.046 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:33 smithi079 ceph-mon[23158]: pgmap v4242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:34 smithi016 conmon[32727]: 2022-03-21T19:33:34.043+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:35 smithi016 conmon[32727]: 2022-03-21T19:33:35.012+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:35 smithi016 conmon[32727]: 2022-03-21T19:33:35.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:35 smithi016 ceph-mon[22010]: pgmap v4243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:35 smithi079 ceph-mon[23158]: pgmap v4243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:37.169 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:36 smithi016 conmon[32727]: 2022-03-21T19:33:36.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:37 smithi016 ceph-mon[22010]: pgmap v4244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:38.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:37 smithi016 conmon[32727]: 2022-03-21T19:33:37.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:37 smithi079 ceph-mon[23158]: pgmap v4244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:38 smithi016 conmon[32727]: 2022-03-21T19:33:38.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:39 smithi016 ceph-mon[22010]: pgmap v4245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:40.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:39 smithi016 conmon[32727]: 2022-03-21T19:33:39.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:39 smithi079 ceph-mon[23158]: pgmap v4245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:41.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:40 smithi016 conmon[32727]: 2022-03-21T19:33:40.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:41 smithi016 ceph-mon[22010]: pgmap v4246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:41 smithi016 conmon[32727]: 2022-03-21T19:33:41.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:41 smithi079 ceph-mon[23158]: pgmap v4246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:43.211 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:42 smithi016 conmon[32727]: 2022-03-21T19:33:42.836+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:43.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:43 smithi016 ceph-mon[22010]: pgmap v4247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:44.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:43 smithi016 conmon[32727]: 2022-03-21T19:33:43.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:43 smithi079 ceph-mon[23158]: pgmap v4247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:45.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:44 smithi016 conmon[32727]: 2022-03-21T19:33:44.803+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:45 smithi016 ceph-mon[22010]: pgmap v4248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:46.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:45 smithi016 conmon[32727]: 2022-03-21T19:33:45.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:45 smithi079 ceph-mon[23158]: pgmap v4248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:47.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:46 smithi016 conmon[32727]: 2022-03-21T19:33:46.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:47 smithi016 ceph-mon[22010]: pgmap v4249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:48.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:48.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:47 smithi016 conmon[32727]: 2022-03-21T19:33:47.759+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:47 smithi079 ceph-mon[23158]: pgmap v4249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:49.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:49.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:48 smithi016 conmon[32727]: 2022-03-21T19:33:48.716+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:49 smithi016 conmon[32727]: 2022-03-21T19:33:49.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: pgmap v4250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:50.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:33:50.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:33:50.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:33:50.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7871 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: pgmap v4250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:33:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:33:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:33:50.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7871 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:51.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:50 smithi016 conmon[32727]: 2022-03-21T19:33:50.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:51 smithi016 ceph-mon[22010]: pgmap v4251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:52.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:51 smithi016 conmon[32727]: 2022-03-21T19:33:51.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:51 smithi079 ceph-mon[23158]: pgmap v4251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[32727]: 2022-03-21T19:33:52.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:53.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: 2022-03-21T19:33:52.781+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:53.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:33:53.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: yield 2022-03-21T19:33:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:53.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:53.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:53.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:33:53.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:53.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:33:53.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:53.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:33:53.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:53.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:33:53.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:53.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:33:53.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:53.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:53.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:52 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:33:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:33:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:33:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:33:53.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.729 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:53.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:33:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:53 smithi016 conmon[32727]: 2022-03-21T19:33:53.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: pgmap v4252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:33:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:33:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 194.8M 2022-03-21T19:33:54.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 204266217: error parsing value: Value '204266217' is below minimum 939524096 2022-03-21T19:33:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:54.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:33:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:33:54.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:54.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:33:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:54.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:33:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:33:54.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:54.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:33:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:54.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:54.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:54.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:33:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: pgmap v4252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:33:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:33:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 194.8M 2022-03-21T19:33:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 204266217: error parsing value: Value '204266217' is below minimum 939524096 2022-03-21T19:33:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:54.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:54.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:33:54.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:54.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:33:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:54.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:54.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:33:54.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:54.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:33:54.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:54.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:33:54.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:54.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:54.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:54.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:33:54.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:54.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:54.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:54.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:33:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:54 smithi016 conmon[32727]: 2022-03-21T19:33:54.696+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:33:56.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[32727]: 2022-03-21T19:33:55.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:56.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 2022-03-21T19:33:55.853+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:56.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:33:56.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:56.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: yield 2022-03-21T19:33:56.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:56.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:56.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:56.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:33:56.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:56.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:33:56.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:56.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:33:56.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:56.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:33:56.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:56.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:33:56.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:56.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:56.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:56.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 2022-03-21T19:33:55.854+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:33:56.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:33:56.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:56.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: yield 2022-03-21T19:33:56.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:56.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:56.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:56.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:33:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:33:56.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:33:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:56.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:33:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:33:56.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:56.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:56.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:56.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 2022-03-21T19:33:56.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:33:56.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: 2022-03-21T19:33:56.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:33:56.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:33:56.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:33:56.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:33:56.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:33:56.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:33:56.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:33:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:33:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:33:56.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:33:56.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:33:56.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:33:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:33:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:33:56.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:33:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:33:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:33:56.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:33:56.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:33:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:33:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:33:56.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:33:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise 2022-03-21T19:33:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:33:56.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:33:56.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:33:56.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:33:56.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:33:55 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:56 smithi016 ceph-mon[22010]: pgmap v4253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:56 smithi079 ceph-mon[23158]: pgmap v4253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:33:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:57.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:56 smithi016 conmon[32727]: 2022-03-21T19:33:56.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:33:57.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:33:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:33:57.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:33:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:57.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:57.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:33:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:57.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:33:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:33:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:57.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:57.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:33:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:57.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:33:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:33:57.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:57.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:33:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:57.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:33:57.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:33:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:33:57.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:33:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:33:57.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:33:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:33:57.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:33:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:33:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:33:57.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:33:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:33:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:33:57.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:33:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:33:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:33:57.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:33:57.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:33:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:33:57.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:33:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:33:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:33:57.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:33:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:33:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:33:57.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:33:57.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:33:57.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:33:57.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:33:57.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:33:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:57.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:57.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:57.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:33:57.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:57.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:33:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:33:57.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:57.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:57.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:33:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:57.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:33:57.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:33:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:33:57.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:33:57.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:33:57.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:33:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:33:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:33:57.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:33:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:33:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:33:57.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:33:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:33:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:33:57.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:33:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:33:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:33:57.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:33:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:33:57.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:33:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:33:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:33:57.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:33:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:33:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:33:57.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:33:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:33:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:33:57.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:33:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:33:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:33:57.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:33:57.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:33:57.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:33:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:33:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:33:57.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:33:57.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:33:57.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:33:57.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:33:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:33:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:33:57.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:33:57.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:33:57.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:33:57.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:33:57.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:33:57.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:33:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:57 smithi016 conmon[32727]: 2022-03-21T19:33:57.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:58 smithi016 ceph-mon[22010]: pgmap v4254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:58 smithi079 ceph-mon[23158]: pgmap v4254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:33:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:59.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:58 smithi016 conmon[32727]: 2022-03-21T19:33:58.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:33:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:33:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:33:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:33:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:33:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:33:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:33:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:33:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:33:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:33:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:33:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:34:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:33:59 smithi016 conmon[32727]: 2022-03-21T19:33:59.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:00 smithi016 ceph-mon[22010]: pgmap v4255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:00 smithi079 ceph-mon[23158]: pgmap v4255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:01.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:00 smithi016 conmon[32727]: 2022-03-21T19:34:00.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:01 smithi016 conmon[32727]: 2022-03-21T19:34:01.809+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:02 smithi016 ceph-mon[22010]: pgmap v4256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:02 smithi079 ceph-mon[23158]: pgmap v4256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:02 smithi016 conmon[32727]: 2022-03-21T19:34:02.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:04.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:03 smithi016 conmon[32727]: 2022-03-21T19:34:03.789+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:04 smithi016 ceph-mon[22010]: pgmap v4257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:04 smithi079 ceph-mon[23158]: pgmap v4257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:04 smithi016 conmon[32727]: 2022-03-21T19:34:04.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:05 smithi016 conmon[32727]: 2022-03-21T19:34:05.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:06 smithi016 ceph-mon[22010]: pgmap v4258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:06 smithi079 ceph-mon[23158]: pgmap v4258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:07.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:06 smithi016 conmon[32727]: 2022-03-21T19:34:06.788+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:07 smithi016 conmon[32727]: 2022-03-21T19:34:07.757+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:08 smithi016 ceph-mon[22010]: pgmap v4259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:08 smithi079 ceph-mon[23158]: pgmap v4259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:08 smithi016 conmon[32727]: 2022-03-21T19:34:08.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:09 smithi016 conmon[32727]: 2022-03-21T19:34:09.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:10 smithi016 ceph-mon[22010]: pgmap v4260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:10 smithi079 ceph-mon[23158]: pgmap v4260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:10.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:10 smithi016 conmon[32727]: 2022-03-21T19:34:10.711+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:11 smithi016 conmon[32727]: 2022-03-21T19:34:11.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:12.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:12 smithi016 ceph-mon[22010]: pgmap v4261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:12.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:12 smithi079 ceph-mon[23158]: pgmap v4261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:12 smithi016 conmon[32727]: 2022-03-21T19:34:12.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:13.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:14.062 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:13 smithi016 conmon[32727]: 2022-03-21T19:34:13.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:14.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:14 smithi016 ceph-mon[22010]: pgmap v4262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:14.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:14 smithi079 ceph-mon[23158]: pgmap v4262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:14 smithi016 conmon[32727]: 2022-03-21T19:34:14.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:15.615 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:15.615 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:15 smithi016 conmon[32727]: 2022-03-21T19:34:15.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:16.589 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:16 smithi016 ceph-mon[22010]: pgmap v4263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:16.590 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:16 smithi079 ceph-mon[23158]: pgmap v4263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:16 smithi016 conmon[32727]: 2022-03-21T19:34:16.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:17.611 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:17 smithi016 conmon[32727]: 2022-03-21T19:34:17.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:18.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:18 smithi016 ceph-mon[22010]: pgmap v4264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:18.631 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:18 smithi079 ceph-mon[23158]: pgmap v4264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:18 smithi016 conmon[32727]: 2022-03-21T19:34:18.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:19.624 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:19 smithi016 conmon[32727]: 2022-03-21T19:34:19.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:20 smithi016 ceph-mon[22010]: pgmap v4265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:20 smithi016 conmon[32727]: 2022-03-21T19:34:20.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:20 smithi079 ceph-mon[23158]: pgmap v4265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:21.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:21.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:21 smithi016 conmon[32727]: 2022-03-21T19:34:21.548+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:22 smithi016 ceph-mon[22010]: pgmap v4266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:22 smithi016 conmon[32727]: 2022-03-21T19:34:22.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:22 smithi079 ceph-mon[23158]: pgmap v4266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:23 smithi016 conmon[32727]: 2022-03-21T19:34:23.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:24.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:24 smithi016 ceph-mon[22010]: pgmap v4267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:24.642 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:24 smithi079 ceph-mon[23158]: pgmap v4267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:24 smithi016 conmon[32727]: 2022-03-21T19:34:24.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:25.642 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:25.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:25 smithi016 conmon[32727]: 2022-03-21T19:34:25.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:26.611 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:26 smithi016 ceph-mon[22010]: pgmap v4268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:26.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:26 smithi079 ceph-mon[23158]: pgmap v4268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:26 smithi016 conmon[32727]: 2022-03-21T19:34:26.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:27 smithi016 conmon[32727]: 2022-03-21T19:34:27.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:28.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:28 smithi016 ceph-mon[22010]: pgmap v4269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:28.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:28 smithi079 ceph-mon[23158]: pgmap v4269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:28 smithi016 conmon[32727]: 2022-03-21T19:34:28.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:29.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:29 smithi016 conmon[32727]: 2022-03-21T19:34:29.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:30 smithi016 conmon[32727]: 2022-03-21T19:34:30.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:30 smithi016 ceph-mon[22010]: pgmap v4270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:30 smithi079 ceph-mon[23158]: pgmap v4270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:31 smithi016 conmon[32727]: 2022-03-21T19:34:31.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:32 smithi016 ceph-mon[22010]: pgmap v4271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:32.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:32 smithi016 conmon[32727]: 2022-03-21T19:34:32.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:32 smithi079 ceph-mon[23158]: pgmap v4271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:33 smithi016 conmon[32727]: 2022-03-21T19:34:33.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:34 smithi016 conmon[32727]: 2022-03-21T19:34:34.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:34 smithi016 ceph-mon[22010]: pgmap v4272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:34.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:34 smithi079 ceph-mon[23158]: pgmap v4272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:35 smithi016 conmon[32727]: 2022-03-21T19:34:35.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:36 smithi016 ceph-mon[22010]: pgmap v4273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:36 smithi016 conmon[32727]: 2022-03-21T19:34:36.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:36 smithi079 ceph-mon[23158]: pgmap v4273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:37 smithi016 conmon[32727]: 2022-03-21T19:34:37.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:38 smithi016 ceph-mon[22010]: pgmap v4274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:38 smithi016 conmon[32727]: 2022-03-21T19:34:38.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:38 smithi079 ceph-mon[23158]: pgmap v4274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:39 smithi016 conmon[32727]: 2022-03-21T19:34:39.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:40 smithi016 ceph-mon[22010]: pgmap v4275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:40.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:40.674 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:40.674 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:40 smithi016 conmon[32727]: 2022-03-21T19:34:40.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:40 smithi079 ceph-mon[23158]: pgmap v4275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:41 smithi016 conmon[32727]: 2022-03-21T19:34:41.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:42.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:42 smithi079 ceph-mon[23158]: pgmap v4276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:42.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:42 smithi016 ceph-mon[22010]: pgmap v4276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:42 smithi016 conmon[32727]: 2022-03-21T19:34:42.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:43 smithi016 conmon[32727]: 2022-03-21T19:34:43.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:44 smithi079 ceph-mon[23158]: pgmap v4277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:44 smithi016 ceph-mon[22010]: pgmap v4277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:44 smithi016 conmon[32727]: 2022-03-21T19:34:44.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:45 smithi079 ceph-mon[23158]: pgmap v4278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:45.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:45 smithi016 ceph-mon[22010]: pgmap v4278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:45.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:45.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:45 smithi016 conmon[32727]: 2022-03-21T19:34:45.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:46 smithi016 conmon[32727]: 2022-03-21T19:34:46.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:47.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:47 smithi079 ceph-mon[23158]: pgmap v4279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:47.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:47 smithi016 ceph-mon[22010]: pgmap v4279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:47 smithi016 conmon[32727]: 2022-03-21T19:34:47.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:48 smithi016 conmon[32727]: 2022-03-21T19:34:48.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:49 smithi079 ceph-mon[23158]: pgmap v4280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:49.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:49 smithi016 ceph-mon[22010]: pgmap v4280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:49.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:49 smithi016 conmon[32727]: 2022-03-21T19:34:49.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:50 smithi016 conmon[32727]: 2022-03-21T19:34:50.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:51 smithi016 ceph-mon[22010]: pgmap v4281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:51.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:51 smithi016 conmon[32727]: 2022-03-21T19:34:51.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:51 smithi079 ceph-mon[23158]: pgmap v4281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:52 smithi016 conmon[32727]: 2022-03-21T19:34:52.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:53 smithi016 ceph-mon[22010]: pgmap v4282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:53.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:53 smithi016 conmon[32727]: 2022-03-21T19:34:53.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:53 smithi079 ceph-mon[23158]: pgmap v4282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:54.751 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:54 smithi016 conmon[32727]: 2022-03-21T19:34:54.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:34:55.752 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:55 smithi016 conmon[32727]: 2022-03-21T19:34:55.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:55 smithi016 ceph-mon[22010]: pgmap v4283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:56.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:55 smithi079 ceph-mon[23158]: pgmap v4283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:34:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:56 smithi016 conmon[32727]: 2022-03-21T19:34:56.440+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:34:57.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:34:57.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:34:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:57.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:34:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:34:57.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:34:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:57 smithi016 conmon[32727]: 2022-03-21T19:34:57.397+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:57 smithi016 ceph-mon[22010]: pgmap v4284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:58.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:58.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:57 smithi079 ceph-mon[23158]: pgmap v4284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:34:58.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[32727]: 2022-03-21T19:34:58.397+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:34:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:34:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:34:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:34:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:34:59.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: 2022-03-21T19:34:58.957+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:34:59.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:34:59.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:34:59.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: yield 2022-03-21T19:34:59.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:34:59.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:34:59.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:34:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:34:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:34:59.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:34:59.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:34:59.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:34:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:34:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:34:59.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:34:59.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:34:59.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:34:59.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:34:59.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:34:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:34:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:34:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:34:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:34:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:34:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:34:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:34:59 smithi016 conmon[32727]: 2022-03-21T19:34:59.348+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:59 smithi016 ceph-mon[22010]: pgmap v4285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:59 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:35:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:35:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:00.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:35:00.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:35:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:00.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:35:00.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:00.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:35:00.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:00.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:59 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:35:00.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:34:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:00.227 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:00.228 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:59 smithi079 ceph-mon[23158]: pgmap v4285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:59 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:00.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:35:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:00.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:35:00.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:00.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:00.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:35:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:00.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:35:00.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:00.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:35:00.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:00.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:00.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:00.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:35:00.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:00.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:00.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:00.241 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:59 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:35:00.242 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:34:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:00 smithi016 conmon[32727]: 2022-03-21T19:35:00.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:01 smithi016 conmon[32727]: 2022-03-21T19:35:01.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:02.031 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:02.032 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:01 smithi016 ceph-mon[22010]: pgmap v4286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:01 smithi079 ceph-mon[23158]: pgmap v4286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:02.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 2022-03-21T19:35:02.029+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:02.334 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:35:02.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:02.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: yield 2022-03-21T19:35:02.335 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:02.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:02.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:02.336 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:35:02.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:02.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:35:02.337 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:02.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:35:02.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:02.338 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:35:02.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:02.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:35:02.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:02.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:02.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:02.340 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 2022-03-21T19:35:02.030+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:35:02.341 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:35:02.341 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:02.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: yield 2022-03-21T19:35:02.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:02.342 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:02.343 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:02.343 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:35:02.343 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:02.344 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:35:02.344 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:02.344 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:35:02.345 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:02.345 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:35:02.345 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:02.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:35:02.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:02.347 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:02.347 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:02.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 2022-03-21T19:35:02.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:35:02.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: 2022-03-21T19:35:02.349 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:35:02.349 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:35:02.349 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:35:02.350 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:35:02.350 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:35:02.350 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:35:02.351 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:35:02.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:35:02.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:35:02.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:35:02.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:35:02.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:35:02.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:35:02.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:35:02.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:35:02.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:35:02.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:35:02.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:35:02.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:35:02.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:35:02.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:35:02.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:35:02.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise 2022-03-21T19:35:02.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:35:02.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:35:02.358 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:35:02.358 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:35:02.359 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:02 smithi016 conmon[32727]: 2022-03-21T19:35:02.332+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:35:03.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:03.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:35:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:35:03.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:35:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:03.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:35:03.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:02 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:35:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:03.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:35:03.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:03.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:35:03.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:35:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:03.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:35:03.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:03.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:03.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:03.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:35:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:03.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:35:03.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:35:03.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:35:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:35:03.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:35:03.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:35:03.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:35:03.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:35:03.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:35:03.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:35:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:35:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:35:03.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:35:03.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:35:03.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:35:03.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:35:03.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:35:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:35:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:35:03.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:35:03.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:35:03.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:35:03.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:35:03.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:35:03.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:35:03.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:35:03.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:35:03.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:35:03.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:35:03.193 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:35:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:03.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:35:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:35:03.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:35:03.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:03.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:35:03.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:03.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:03.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.234 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:02 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:35:03.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:35:03.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:35:03.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:35:03.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:35:03.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:35:03.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:35:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:35:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:35:03.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:35:03.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:35:03.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:35:03.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:35:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:35:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:35:03.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:35:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:35:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:35:03.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:35:03.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:35:03.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:35:03.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:35:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:35:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:35:03.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:35:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:35:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:35:03.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:35:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:35:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:35:03.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:35:03.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:35:03.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:35:03.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:35:03.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:35:03.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:35:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:35:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:35:03.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:35:03.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:35:03.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:35:03.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:35:03.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:35:03.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:35:03.251 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:35:03.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:35:03.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:35:03.252 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:35:03.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:03 smithi016 conmon[32727]: 2022-03-21T19:35:03.320+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:03 smithi016 ceph-mon[22010]: pgmap v4287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:03 smithi079 ceph-mon[23158]: pgmap v4287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:04 smithi016 conmon[32727]: 2022-03-21T19:35:04.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:05 smithi016 conmon[32727]: 2022-03-21T19:35:05.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:05 smithi016 ceph-mon[22010]: pgmap v4288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:05 smithi079 ceph-mon[23158]: pgmap v4288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:06 smithi016 conmon[32727]: 2022-03-21T19:35:06.241+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:07 smithi016 conmon[32727]: 2022-03-21T19:35:07.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:07 smithi016 ceph-mon[22010]: pgmap v4289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:07 smithi079 ceph-mon[23158]: pgmap v4289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:08 smithi016 conmon[32727]: 2022-03-21T19:35:08.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:09 smithi016 conmon[32727]: 2022-03-21T19:35:09.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:09 smithi016 ceph-mon[22010]: pgmap v4290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:10.562 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:10.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:10.564 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:09 smithi079 ceph-mon[23158]: pgmap v4290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:10.565 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:10.565 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:10 smithi016 conmon[32727]: 2022-03-21T19:35:10.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:11 smithi016 conmon[32727]: 2022-03-21T19:35:11.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:12.127 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:11 smithi016 ceph-mon[22010]: pgmap v4291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:12.128 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:11 smithi079 ceph-mon[23158]: pgmap v4291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:12.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:12.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:12 smithi016 conmon[32727]: 2022-03-21T19:35:12.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:13.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:13.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:13 smithi016 conmon[32727]: 2022-03-21T19:35:13.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:14.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:13 smithi016 ceph-mon[22010]: pgmap v4292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:14.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:13 smithi079 ceph-mon[23158]: pgmap v4292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:14.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:14 smithi016 conmon[32727]: 2022-03-21T19:35:14.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:15.090 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:15.091 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:15.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:15 smithi016 conmon[32727]: 2022-03-21T19:35:15.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:16.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:15 smithi016 ceph-mon[22010]: pgmap v4293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:16.129 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:15 smithi079 ceph-mon[23158]: pgmap v4293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:16 smithi016 conmon[32727]: 2022-03-21T19:35:16.128+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:17.089 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:17 smithi016 conmon[32727]: 2022-03-21T19:35:17.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:17 smithi016 ceph-mon[22010]: pgmap v4294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:18.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:18 smithi016 conmon[32727]: 2022-03-21T19:35:18.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:17 smithi079 ceph-mon[23158]: pgmap v4294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:19.083 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:19 smithi016 conmon[32727]: 2022-03-21T19:35:19.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:20.108 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:19 smithi016 ceph-mon[22010]: pgmap v4295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:20.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:20.109 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:19 smithi079 ceph-mon[23158]: pgmap v4295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:20 smithi016 conmon[32727]: 2022-03-21T19:35:20.107+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:21.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:21.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:21 smithi016 conmon[32727]: 2022-03-21T19:35:21.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:22.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:21 smithi016 ceph-mon[22010]: pgmap v4296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:22.148 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:21 smithi079 ceph-mon[23158]: pgmap v4296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:22.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:22 smithi016 conmon[32727]: 2022-03-21T19:35:22.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:23 smithi016 conmon[32727]: 2022-03-21T19:35:23.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:23 smithi016 ceph-mon[22010]: pgmap v4297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:24.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:23 smithi079 ceph-mon[23158]: pgmap v4297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:24 smithi016 conmon[32727]: 2022-03-21T19:35:24.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:25.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:25.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:25 smithi016 conmon[32727]: 2022-03-21T19:35:25.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:25 smithi016 ceph-mon[22010]: pgmap v4298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:25 smithi079 ceph-mon[23158]: pgmap v4298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:26 smithi016 conmon[32727]: 2022-03-21T19:35:26.179+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:27 smithi016 conmon[32727]: 2022-03-21T19:35:27.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:27 smithi016 ceph-mon[22010]: pgmap v4299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:27 smithi079 ceph-mon[23158]: pgmap v4299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:28 smithi016 conmon[32727]: 2022-03-21T19:35:28.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:29.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:29 smithi016 conmon[32727]: 2022-03-21T19:35:29.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:29 smithi016 ceph-mon[22010]: pgmap v4300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:29 smithi079 ceph-mon[23158]: pgmap v4300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:30 smithi016 conmon[32727]: 2022-03-21T19:35:30.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:31 smithi016 conmon[32727]: 2022-03-21T19:35:31.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:32.138 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:31 smithi016 ceph-mon[22010]: pgmap v4301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:32.139 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:31 smithi079 ceph-mon[23158]: pgmap v4301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:32.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:32 smithi016 conmon[32727]: 2022-03-21T19:35:32.136+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:33.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:33 smithi016 conmon[32727]: 2022-03-21T19:35:33.097+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:34.145 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:33 smithi016 ceph-mon[22010]: pgmap v4302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:34.146 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:33 smithi079 ceph-mon[23158]: pgmap v4302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:34 smithi016 conmon[32727]: 2022-03-21T19:35:34.143+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:35.133 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:35.134 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:35.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:35 smithi016 conmon[32727]: 2022-03-21T19:35:35.130+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:35 smithi016 ceph-mon[22010]: pgmap v4303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:35 smithi079 ceph-mon[23158]: pgmap v4303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:36 smithi016 conmon[32727]: 2022-03-21T19:35:36.172+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:37 smithi016 conmon[32727]: 2022-03-21T19:35:37.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:38.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:38 smithi016 conmon[32727]: 2022-03-21T19:35:38.089+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:37 smithi016 ceph-mon[22010]: pgmap v4304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:38.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:37 smithi079 ceph-mon[23158]: pgmap v4304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:39 smithi016 conmon[32727]: 2022-03-21T19:35:39.086+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:40 smithi016 conmon[32727]: 2022-03-21T19:35:40.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:39 smithi016 ceph-mon[22010]: pgmap v4305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:39 smithi079 ceph-mon[23158]: pgmap v4305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:41 smithi016 conmon[32727]: 2022-03-21T19:35:41.090+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:41 smithi016 ceph-mon[22010]: pgmap v4306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:42.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:42 smithi016 conmon[32727]: 2022-03-21T19:35:42.059+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:41 smithi079 ceph-mon[23158]: pgmap v4306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:43.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:43 smithi016 conmon[32727]: 2022-03-21T19:35:43.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:43 smithi016 ceph-mon[22010]: pgmap v4307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:44.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:44 smithi016 conmon[32727]: 2022-03-21T19:35:44.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:43 smithi079 ceph-mon[23158]: pgmap v4307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:45.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:45 smithi016 conmon[32727]: 2022-03-21T19:35:45.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:45 smithi016 ceph-mon[22010]: pgmap v4308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:46.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:46 smithi016 conmon[32727]: 2022-03-21T19:35:46.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:45 smithi079 ceph-mon[23158]: pgmap v4308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:47.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:47.421 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:47 smithi016 conmon[32727]: 2022-03-21T19:35:47.055+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:47 smithi079 ceph-mon[23158]: pgmap v4309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:47 smithi016 ceph-mon[22010]: pgmap v4309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:48 smithi016 conmon[32727]: 2022-03-21T19:35:48.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:49.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:49 smithi016 conmon[32727]: 2022-03-21T19:35:49.015+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:49.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:49 smithi079 ceph-mon[23158]: pgmap v4310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:50 smithi016 conmon[32727]: 2022-03-21T19:35:50.037+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:49 smithi016 ceph-mon[22010]: pgmap v4310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:51 smithi016 conmon[32727]: 2022-03-21T19:35:51.023+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:51 smithi079 ceph-mon[23158]: pgmap v4311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:52.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:51 smithi016 ceph-mon[22010]: pgmap v4311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:52 smithi016 conmon[32727]: 2022-03-21T19:35:52.062+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:53 smithi016 conmon[32727]: 2022-03-21T19:35:53.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:53 smithi079 ceph-mon[23158]: pgmap v4312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:54.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:54.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:53 smithi016 ceph-mon[22010]: pgmap v4312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:54.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:54.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:54 smithi016 conmon[32727]: 2022-03-21T19:35:54.039+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 7995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 7995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:35:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:55 smithi016 conmon[32727]: 2022-03-21T19:35:55.053+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:55 smithi079 ceph-mon[23158]: pgmap v4313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:55 smithi016 ceph-mon[22010]: pgmap v4313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:35:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:56.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:56 smithi016 conmon[32727]: 2022-03-21T19:35:56.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:57.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:57 smithi016 conmon[32727]: 2022-03-21T19:35:57.057+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:57 smithi079 ceph-mon[23158]: pgmap v4314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:58.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:57 smithi016 ceph-mon[22010]: pgmap v4314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:35:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:58 smithi016 conmon[32727]: 2022-03-21T19:35:58.063+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:35:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:35:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:35:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:35:59 smithi016 conmon[32727]: 2022-03-21T19:35:59.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:35:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:35:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:35:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:36:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:59 smithi079 ceph-mon[23158]: pgmap v4315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:00.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:35:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:00.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:00 smithi016 conmon[32727]: 2022-03-21T19:36:00.147+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:00.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:59 smithi016 ceph-mon[22010]: pgmap v4315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:00.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:35:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:01.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:01 smithi016 conmon[32727]: 2022-03-21T19:36:01.183+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:01 smithi079 ceph-mon[23158]: pgmap v4316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[32727]: 2022-03-21T19:36:02.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:01 smithi016 ceph-mon[22010]: pgmap v4316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:02.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:02.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: 2022-03-21T19:36:02.637+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:02.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:36:02.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:02.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: yield 2022-03-21T19:36:02.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:02.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:02.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:02.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:36:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:36:02.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:36:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:02.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:36:02.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:02.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:36:02.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:02.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:36:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:36:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:36:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:03 smithi016 conmon[32727]: 2022-03-21T19:36:03.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:36:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:36:03.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:36:03.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:03 smithi079 ceph-mon[23158]: pgmap v4317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:36:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:36:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:04.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:04.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:04.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:36:04.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:04.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:36:04.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:04.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:36:04.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:04.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:04.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:04.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:36:04.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:04.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:04.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:04.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:03 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:36:04.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:04 smithi016 conmon[32727]: 2022-03-21T19:36:04.170+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:03 smithi016 ceph-mon[22010]: pgmap v4317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:36:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:36:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:36:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:36:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:36:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:04.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:36:04.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:04.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:04.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:04.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:03 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:36:04.430 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:04 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[32727]: 2022-03-21T19:36:05.214+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:04 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:05.977 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 2022-03-21T19:36:05.710+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:05.978 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:36:05.978 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:05.979 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: yield 2022-03-21T19:36:05.979 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:05.979 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:05.980 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:05.980 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:36:05.980 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:05.981 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:36:05.981 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:05.981 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:36:05.982 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:05.982 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:36:05.983 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:05.983 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:36:05.983 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:05.984 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:05.984 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:05.984 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 2022-03-21T19:36:05.711+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:36:05.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:36:05.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:05.985 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: yield 2022-03-21T19:36:05.986 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:05.986 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:05.986 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:05.987 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:36:05.987 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:05.987 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:36:05.988 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:05.988 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:36:05.988 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:05.989 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:36:05.989 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:05.989 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:36:05.990 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:05.990 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:05.991 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:05.991 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 2022-03-21T19:36:05.991 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:36:05.992 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: 2022-03-21T19:36:05.992 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:36:05.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:36:05.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:36:05.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:36:05.994 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:36:05.994 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:36:05.994 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:36:05.995 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:36:05.995 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:36:05.995 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:36:05.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:36:05.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:36:05.996 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:36:05.997 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:36:05.997 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:36:05.997 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:36:05.998 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:36:05.998 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:36:05.998 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:36:05.999 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:36:05.999 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:36:05.999 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:36:06.000 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise 2022-03-21T19:36:06.000 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:36:06.000 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:36:06.001 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:36:06.001 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:36:06.001 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:36:05 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:06.260 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:05 smithi016 ceph-mon[22010]: pgmap v4318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:06.260 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:05 smithi079 ceph-mon[23158]: pgmap v4318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:06.779 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:06.780 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:06 smithi016 conmon[32727]: 2022-03-21T19:36:06.258+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:07.258 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:06 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.259 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.259 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:07.259 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:36:07.260 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:07.260 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:07.261 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:07.261 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:36:07.262 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:07.262 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:36:07.262 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:07.263 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:36:07.263 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:07.263 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:07.264 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:07.264 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:36:07.265 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:07.265 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:07.265 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.266 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:06 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:36:07.266 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.266 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:07.267 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:36:07.267 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:07.267 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:07.268 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:07.268 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:36:07.268 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:07.269 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:36:07.269 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:07.269 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:36:07.270 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:07.270 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:07.271 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:07.271 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:36:07.272 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:07.272 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:07.272 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.273 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:36:07.273 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:36:07.273 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:36:07.274 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.274 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:36:07.274 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:36:07.275 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:36:07.275 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:36:07.276 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:36:07.276 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:36:07.276 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:36:07.277 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:36:07.277 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:36:07.277 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:36:07.278 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:36:07.278 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:36:07.279 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:36:07.279 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:36:07.279 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:36:07.280 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:36:07.280 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:36:07.280 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:36:07.281 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:36:07.281 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:36:07.282 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:36:07.282 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:36:07.282 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:36:07.283 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:36:07.284 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:36:07.284 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:36:07.285 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.285 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:06 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:36:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:07.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:07.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:36:07.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:07.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:36:07.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:07.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:36:07.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:07.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:07.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:07.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:36:07.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:07.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:07.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:06 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:36:07.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:36:07.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:36:07.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:36:07.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:36:07.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:36:07.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:36:07.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:36:07.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:36:07.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:36:07.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:36:07.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:36:07.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:36:07.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:36:07.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:36:07.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:36:07.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:36:07.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:36:07.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:36:07.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:36:07.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:36:07.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:36:07.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:36:07.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:36:07.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:36:07.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:36:07.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:36:07.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:36:07.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:36:07.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:36:07.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:36:07.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:36:07.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:36:07.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:36:07.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:36:07.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:36:07.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:36:07.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:36:07.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:36:07.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:36:07.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:36:07.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:36:07.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:36:07.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:36:07.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:36:07.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:36:07.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:36:07.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:36:07.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:07 smithi016 conmon[32727]: 2022-03-21T19:36:07.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:08.247 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:07 smithi016 ceph-mon[22010]: pgmap v4319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:08.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:07 smithi079 ceph-mon[23158]: pgmap v4319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:08 smithi016 conmon[32727]: 2022-03-21T19:36:08.246+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:09.254 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:09 smithi016 conmon[32727]: 2022-03-21T19:36:09.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:09 smithi016 ceph-mon[22010]: pgmap v4320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:10.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:10.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:10 smithi016 conmon[32727]: 2022-03-21T19:36:10.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:09 smithi079 ceph-mon[23158]: pgmap v4320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:11.250 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:11 smithi016 conmon[32727]: 2022-03-21T19:36:11.248+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:12.289 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:11 smithi016 ceph-mon[22010]: pgmap v4321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:12.290 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:11 smithi079 ceph-mon[23158]: pgmap v4321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:12 smithi016 conmon[32727]: 2022-03-21T19:36:12.288+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:13.279 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:13 smithi016 conmon[32727]: 2022-03-21T19:36:13.277+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:14.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:14 smithi016 ceph-mon[22010]: pgmap v4322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:14.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:14 smithi079 ceph-mon[23158]: pgmap v4322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:14.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:14 smithi016 conmon[32727]: 2022-03-21T19:36:14.304+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:15.277 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:15 smithi016 conmon[32727]: 2022-03-21T19:36:15.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:16 smithi016 ceph-mon[22010]: pgmap v4323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:16.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:16 smithi016 conmon[32727]: 2022-03-21T19:36:16.226+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:16 smithi079 ceph-mon[23158]: pgmap v4323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:17 smithi016 conmon[32727]: 2022-03-21T19:36:17.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:18 smithi016 ceph-mon[22010]: pgmap v4324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:18.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:18 smithi016 conmon[32727]: 2022-03-21T19:36:18.266+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:18 smithi079 ceph-mon[23158]: pgmap v4324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:19.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:19 smithi016 conmon[32727]: 2022-03-21T19:36:19.259+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:20.287 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:20 smithi016 ceph-mon[22010]: pgmap v4325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:20.287 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:20.288 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:20 smithi079 ceph-mon[23158]: pgmap v4325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:20 smithi016 conmon[32727]: 2022-03-21T19:36:20.285+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:21.297 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:21.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:21 smithi016 conmon[32727]: 2022-03-21T19:36:21.296+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:22.315 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:22 smithi016 ceph-mon[22010]: pgmap v4326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:22.316 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:22 smithi079 ceph-mon[23158]: pgmap v4326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:22.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:22 smithi016 conmon[32727]: 2022-03-21T19:36:22.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:23.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:23 smithi016 conmon[32727]: 2022-03-21T19:36:23.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:24.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:24 smithi016 conmon[32727]: 2022-03-21T19:36:24.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:24 smithi016 ceph-mon[22010]: pgmap v4327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:24 smithi079 ceph-mon[23158]: pgmap v4327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:25.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:25 smithi016 conmon[32727]: 2022-03-21T19:36:25.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:25.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:26 smithi016 ceph-mon[22010]: pgmap v4328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:26 smithi016 conmon[32727]: 2022-03-21T19:36:26.230+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:26 smithi079 ceph-mon[23158]: pgmap v4328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:27 smithi016 conmon[32727]: 2022-03-21T19:36:27.233+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:28 smithi016 ceph-mon[22010]: pgmap v4329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:28 smithi016 conmon[32727]: 2022-03-21T19:36:28.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:28 smithi079 ceph-mon[23158]: pgmap v4329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:29 smithi016 conmon[32727]: 2022-03-21T19:36:29.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:30 smithi016 ceph-mon[22010]: pgmap v4330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:30.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:30 smithi016 conmon[32727]: 2022-03-21T19:36:30.235+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:30 smithi079 ceph-mon[23158]: pgmap v4330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:31 smithi016 conmon[32727]: 2022-03-21T19:36:31.244+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:32 smithi016 ceph-mon[22010]: pgmap v4331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:32 smithi016 conmon[32727]: 2022-03-21T19:36:32.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:32 smithi079 ceph-mon[23158]: pgmap v4331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:33 smithi016 conmon[32727]: 2022-03-21T19:36:33.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:34 smithi016 ceph-mon[22010]: pgmap v4332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:34 smithi016 conmon[32727]: 2022-03-21T19:36:34.182+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:34 smithi079 ceph-mon[23158]: pgmap v4332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:35 smithi016 conmon[32727]: 2022-03-21T19:36:35.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:36 smithi016 ceph-mon[22010]: pgmap v4333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:36 smithi016 conmon[32727]: 2022-03-21T19:36:36.195+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:36 smithi079 ceph-mon[23158]: pgmap v4333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:36.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:37.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:37 smithi016 conmon[32727]: 2022-03-21T19:36:37.171+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:38 smithi016 ceph-mon[22010]: pgmap v4334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:38 smithi016 conmon[32727]: 2022-03-21T19:36:38.125+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:38 smithi079 ceph-mon[23158]: pgmap v4334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:39 smithi016 conmon[32727]: 2022-03-21T19:36:39.126+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:40 smithi016 ceph-mon[22010]: pgmap v4335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:40.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:40.418 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:40 smithi016 conmon[32727]: 2022-03-21T19:36:40.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:40 smithi079 ceph-mon[23158]: pgmap v4335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:41 smithi016 conmon[32727]: 2022-03-21T19:36:41.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:42 smithi016 ceph-mon[22010]: pgmap v4336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:42.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:42 smithi016 conmon[32727]: 2022-03-21T19:36:42.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:42 smithi079 ceph-mon[23158]: pgmap v4336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:43 smithi016 conmon[32727]: 2022-03-21T19:36:43.216+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:44 smithi016 ceph-mon[22010]: pgmap v4337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:44.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:44 smithi016 conmon[32727]: 2022-03-21T19:36:44.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:44 smithi079 ceph-mon[23158]: pgmap v4337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:45.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:45 smithi016 conmon[32727]: 2022-03-21T19:36:45.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:46.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:46 smithi016 conmon[32727]: 2022-03-21T19:36:46.192+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:46 smithi016 ceph-mon[22010]: pgmap v4338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:46 smithi079 ceph-mon[23158]: pgmap v4338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:47 smithi016 conmon[32727]: 2022-03-21T19:36:47.193+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:48 smithi016 ceph-mon[22010]: pgmap v4339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:48.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:48 smithi016 conmon[32727]: 2022-03-21T19:36:48.198+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:48 smithi079 ceph-mon[23158]: pgmap v4339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:49 smithi016 conmon[32727]: 2022-03-21T19:36:49.157+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:50 smithi016 conmon[32727]: 2022-03-21T19:36:50.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:50 smithi016 ceph-mon[22010]: pgmap v4340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:50 smithi079 ceph-mon[23158]: pgmap v4340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:51 smithi016 conmon[32727]: 2022-03-21T19:36:51.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:52 smithi016 ceph-mon[22010]: pgmap v4341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:52.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:52 smithi016 conmon[32727]: 2022-03-21T19:36:52.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:52 smithi079 ceph-mon[23158]: pgmap v4341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:53.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:53 smithi016 conmon[32727]: 2022-03-21T19:36:53.135+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:54 smithi016 ceph-mon[22010]: pgmap v4342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:54 smithi016 conmon[32727]: 2022-03-21T19:36:54.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:54 smithi079 ceph-mon[23158]: pgmap v4342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:55.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:55 smithi016 conmon[32727]: 2022-03-21T19:36:55.139+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:36:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:56 smithi016 conmon[32727]: 2022-03-21T19:36:56.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:56 smithi016 ceph-mon[22010]: pgmap v4343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:56 smithi079 ceph-mon[23158]: pgmap v4343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:36:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:57 smithi016 conmon[32727]: 2022-03-21T19:36:57.202+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:58 smithi016 ceph-mon[22010]: pgmap v4344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:58 smithi016 conmon[32727]: 2022-03-21T19:36:58.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:58 smithi079 ceph-mon[23158]: pgmap v4344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:36:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:59.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:36:59 smithi016 conmon[32727]: 2022-03-21T19:36:59.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:36:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:36:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:36:59.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:36:59.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:36:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:36:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:36:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:36:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:36:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:36:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:36:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:37:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:00 smithi016 ceph-mon[22010]: pgmap v4345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:00 smithi016 conmon[32727]: 2022-03-21T19:37:00.250+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:00 smithi079 ceph-mon[23158]: pgmap v4345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:01 smithi016 conmon[32727]: 2022-03-21T19:37:01.215+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:02 smithi016 conmon[32727]: 2022-03-21T19:37:02.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:02 smithi016 ceph-mon[22010]: pgmap v4346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:02 smithi079 ceph-mon[23158]: pgmap v4346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:03 smithi016 conmon[32727]: 2022-03-21T19:37:03.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:03.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:04 smithi079 ceph-mon[23158]: pgmap v4347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:04 smithi016 conmon[32727]: 2022-03-21T19:37:04.249+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:04 smithi016 ceph-mon[22010]: pgmap v4347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:04.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:05 smithi016 conmon[32727]: 2022-03-21T19:37:05.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:06 smithi016 conmon[32727]: 2022-03-21T19:37:06.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: pgmap v4348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:37:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:37:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:37:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:37:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:06 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:37:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: pgmap v4348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:37:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:37:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:37:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:37:06.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:06 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:37:07.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:07 smithi016 conmon[32727]: 2022-03-21T19:37:07.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:08 smithi079 ceph-mon[23158]: pgmap v4349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:08 smithi016 ceph-mon[22010]: pgmap v4349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:08.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[32727]: 2022-03-21T19:37:08.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:09.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: 2022-03-21T19:37:08.813+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:09.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:37:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: yield 2022-03-21T19:37:09.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:09.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:37:09.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:09.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:37:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:37:09.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:37:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:09.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:37:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:09.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:08 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:09 smithi016 conmon[32727]: 2022-03-21T19:37:09.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:10 smithi079 ceph-mon[23158]: pgmap v4350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:37:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:37:10.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:10.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:10.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:37:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:10.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:37:10.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:10.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:37:10.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:10.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:37:10.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:10.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:10.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:10.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:10 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:37:10.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:10.483 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:10 smithi016 conmon[32727]: 2022-03-21T19:37:10.236+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:10 smithi016 ceph-mon[22010]: pgmap v4350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:37:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:37:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:10.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:10.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:37:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:37:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:37:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:10.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:10.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:37:10.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:10.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:10.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:10.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:10 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:37:10.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:10.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[32727]: 2022-03-21T19:37:11.232+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:12.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 2022-03-21T19:37:11.885+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:12.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:37:12.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:12.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: yield 2022-03-21T19:37:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:12.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:37:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:12.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:37:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:37:12.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:12.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:37:12.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:37:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:12.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 2022-03-21T19:37:11.887+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:37:12.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:37:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: yield 2022-03-21T19:37:12.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:12.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:12.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:12.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:37:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:37:12.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:12.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:37:12.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:37:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:12.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:37:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:12.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 2022-03-21T19:37:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:37:12.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: 2022-03-21T19:37:12.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:37:12.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:37:12.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:37:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:37:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:37:12.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:37:12.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:37:12.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:37:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:37:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:37:12.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:37:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:37:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:37:12.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:37:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:37:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:37:12.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:37:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:37:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:37:12.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:37:12.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:37:12.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:37:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise 2022-03-21T19:37:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:37:12.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:37:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:37:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:37:12.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:37:11 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:12 smithi079 ceph-mon[23158]: pgmap v4351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:12 smithi016 conmon[32727]: 2022-03-21T19:37:12.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:12 smithi016 ceph-mon[22010]: pgmap v4351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:37:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:13.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:37:13.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:37:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:13.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:37:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:13.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:37:13.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:13.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:13 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:37:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:37:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:13.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:13.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:13.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:37:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:37:13.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:13.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:37:13.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:13.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:37:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:13.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:37:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:37:13.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:37:13.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:37:13.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:37:13.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:37:13.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:37:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:37:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:37:13.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:37:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:37:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:37:13.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:37:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:37:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:37:13.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:37:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:37:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:37:13.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:37:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:37:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:37:13.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:37:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:37:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:37:13.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:37:13.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:37:13.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:37:13.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:37:13.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:37:13.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:13 smithi016 conmon[32727]: 2022-03-21T19:37:13.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:37:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:37:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:37:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:37:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:37:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:13 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:37:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:37:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:37:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:37:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:37:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:37:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:37:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:37:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:37:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:37:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:37:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:37:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:37:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:37:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:37:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:37:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:37:13.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:37:13.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:37:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:37:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:37:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:37:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:37:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:37:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:37:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:37:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:37:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:37:13.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:37:13.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:37:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:37:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:37:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:37:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:37:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:37:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:37:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:37:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:37:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:37:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:37:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:37:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:37:13.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:37:13.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:37:13.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:37:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:37:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:37:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:37:13.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:14 smithi016 ceph-mon[22010]: pgmap v4352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:14 smithi016 conmon[32727]: 2022-03-21T19:37:14.144+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:14 smithi079 ceph-mon[23158]: pgmap v4352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:15 smithi016 conmon[32727]: 2022-03-21T19:37:15.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:16 smithi016 ceph-mon[22010]: pgmap v4353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:16 smithi016 conmon[32727]: 2022-03-21T19:37:16.116+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:16 smithi079 ceph-mon[23158]: pgmap v4353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:17 smithi016 conmon[32727]: 2022-03-21T19:37:17.080+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:18 smithi016 ceph-mon[22010]: pgmap v4354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:18 smithi016 conmon[32727]: 2022-03-21T19:37:18.069+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:18 smithi079 ceph-mon[23158]: pgmap v4354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:19.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:19 smithi016 conmon[32727]: 2022-03-21T19:37:19.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:20 smithi016 conmon[32727]: 2022-03-21T19:37:20.115+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:20 smithi016 ceph-mon[22010]: pgmap v4355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:20 smithi079 ceph-mon[23158]: pgmap v4355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:21.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:21 smithi016 conmon[32727]: 2022-03-21T19:37:21.095+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:22 smithi016 ceph-mon[22010]: pgmap v4356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:22.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:22 smithi016 conmon[32727]: 2022-03-21T19:37:22.141+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:22 smithi079 ceph-mon[23158]: pgmap v4356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:23 smithi016 conmon[32727]: 2022-03-21T19:37:23.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:24 smithi016 ceph-mon[22010]: pgmap v4357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:24.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:24 smithi016 conmon[32727]: 2022-03-21T19:37:24.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:24 smithi079 ceph-mon[23158]: pgmap v4357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:25.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:25 smithi016 conmon[32727]: 2022-03-21T19:37:25.113+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:26 smithi016 ceph-mon[22010]: pgmap v4358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:26.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:26 smithi016 conmon[32727]: 2022-03-21T19:37:26.154+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:26 smithi079 ceph-mon[23158]: pgmap v4358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:27.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:27 smithi016 conmon[32727]: 2022-03-21T19:37:27.132+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:28 smithi016 ceph-mon[22010]: pgmap v4359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:28.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:28 smithi016 conmon[32727]: 2022-03-21T19:37:28.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:28 smithi079 ceph-mon[23158]: pgmap v4359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:29.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:29 smithi016 conmon[32727]: 2022-03-21T19:37:29.131+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:30.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:30 smithi016 conmon[32727]: 2022-03-21T19:37:30.150+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:30 smithi016 ceph-mon[22010]: pgmap v4360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:30 smithi079 ceph-mon[23158]: pgmap v4360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:31 smithi016 conmon[32727]: 2022-03-21T19:37:31.174+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:32 smithi016 ceph-mon[22010]: pgmap v4361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:32 smithi016 conmon[32727]: 2022-03-21T19:37:32.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:32 smithi079 ceph-mon[23158]: pgmap v4361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:33 smithi016 conmon[32727]: 2022-03-21T19:37:33.229+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:34 smithi016 ceph-mon[22010]: pgmap v4362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:34 smithi016 conmon[32727]: 2022-03-21T19:37:34.209+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:34 smithi079 ceph-mon[23158]: pgmap v4362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:35 smithi016 conmon[32727]: 2022-03-21T19:37:35.186+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:36 smithi016 ceph-mon[22010]: pgmap v4363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:36 smithi016 conmon[32727]: 2022-03-21T19:37:36.163+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:36 smithi079 ceph-mon[23158]: pgmap v4363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:37 smithi016 conmon[32727]: 2022-03-21T19:37:37.140+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:38 smithi016 ceph-mon[22010]: pgmap v4364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:38.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:38 smithi016 conmon[32727]: 2022-03-21T19:37:38.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:38 smithi079 ceph-mon[23158]: pgmap v4364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:39.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:39 smithi016 conmon[32727]: 2022-03-21T19:37:39.159+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:40 smithi016 conmon[32727]: 2022-03-21T19:37:40.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:40 smithi016 ceph-mon[22010]: pgmap v4365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:40 smithi079 ceph-mon[23158]: pgmap v4365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:41 smithi016 conmon[32727]: 2022-03-21T19:37:41.155+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:42 smithi016 ceph-mon[22010]: pgmap v4366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:42 smithi016 conmon[32727]: 2022-03-21T19:37:42.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:42 smithi079 ceph-mon[23158]: pgmap v4366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:43 smithi016 conmon[32727]: 2022-03-21T19:37:43.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:44 smithi016 ceph-mon[22010]: pgmap v4367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:44 smithi016 conmon[32727]: 2022-03-21T19:37:44.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:44 smithi079 ceph-mon[23158]: pgmap v4367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:45 smithi016 conmon[32727]: 2022-03-21T19:37:45.188+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:46 smithi016 ceph-mon[22010]: pgmap v4368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:46 smithi016 conmon[32727]: 2022-03-21T19:37:46.205+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:46 smithi079 ceph-mon[23158]: pgmap v4368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:47 smithi016 conmon[32727]: 2022-03-21T19:37:47.223+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:48 smithi016 ceph-mon[22010]: pgmap v4369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:48 smithi016 conmon[32727]: 2022-03-21T19:37:48.261+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:48 smithi079 ceph-mon[23158]: pgmap v4369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:49 smithi016 conmon[32727]: 2022-03-21T19:37:49.263+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:50 smithi016 ceph-mon[22010]: pgmap v4370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:50.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:50 smithi016 conmon[32727]: 2022-03-21T19:37:50.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:50 smithi079 ceph-mon[23158]: pgmap v4370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:51 smithi016 conmon[32727]: 2022-03-21T19:37:51.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:52 smithi016 ceph-mon[22010]: pgmap v4371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:52 smithi016 conmon[32727]: 2022-03-21T19:37:52.393+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:52 smithi079 ceph-mon[23158]: pgmap v4371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:53 smithi016 conmon[32727]: 2022-03-21T19:37:53.423+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:54 smithi016 ceph-mon[22010]: pgmap v4372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:54 smithi016 conmon[32727]: 2022-03-21T19:37:54.454+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:54 smithi079 ceph-mon[23158]: pgmap v4372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:55 smithi016 conmon[32727]: 2022-03-21T19:37:55.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:37:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:56 smithi016 ceph-mon[22010]: pgmap v4373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:56 smithi016 conmon[32727]: 2022-03-21T19:37:56.499+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:56 smithi079 ceph-mon[23158]: pgmap v4373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:37:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:57 smithi016 conmon[32727]: 2022-03-21T19:37:57.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:58 smithi016 ceph-mon[22010]: pgmap v4374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:58 smithi016 conmon[32727]: 2022-03-21T19:37:58.485+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:58 smithi079 ceph-mon[23158]: pgmap v4374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:37:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:37:59 smithi016 conmon[32727]: 2022-03-21T19:37:59.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:37:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:37:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:37:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:37:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:37:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:37:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:37:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:37:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:37:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:37:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:37:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:38:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:00 smithi016 ceph-mon[22010]: pgmap v4375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:00 smithi016 conmon[32727]: 2022-03-21T19:38:00.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:00 smithi079 ceph-mon[23158]: pgmap v4375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:01 smithi016 conmon[32727]: 2022-03-21T19:38:01.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:02 smithi016 ceph-mon[22010]: pgmap v4376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:02.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:02 smithi016 conmon[32727]: 2022-03-21T19:38:02.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:02 smithi079 ceph-mon[23158]: pgmap v4376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:03 smithi016 conmon[32727]: 2022-03-21T19:38:03.440+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:04 smithi016 ceph-mon[22010]: pgmap v4377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:04 smithi016 conmon[32727]: 2022-03-21T19:38:04.392+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:04 smithi079 ceph-mon[23158]: pgmap v4377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:05.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:05 smithi016 conmon[32727]: 2022-03-21T19:38:05.389+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:06 smithi016 conmon[32727]: 2022-03-21T19:38:06.368+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:06 smithi016 ceph-mon[22010]: pgmap v4378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:06 smithi079 ceph-mon[23158]: pgmap v4378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:07 smithi016 conmon[32727]: 2022-03-21T19:38:07.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:08 smithi016 conmon[32727]: 2022-03-21T19:38:08.395+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:08 smithi016 ceph-mon[22010]: pgmap v4379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:08 smithi079 ceph-mon[23158]: pgmap v4379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:09.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:09 smithi016 conmon[32727]: 2022-03-21T19:38:09.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:10 smithi016 ceph-mon[22010]: pgmap v4380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:10 smithi016 conmon[32727]: 2022-03-21T19:38:10.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:10 smithi079 ceph-mon[23158]: pgmap v4380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:11 smithi016 conmon[32727]: 2022-03-21T19:38:11.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[32727]: 2022-03-21T19:38:12.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:12 smithi016 ceph-mon[22010]: pgmap v4381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:38:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:38:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:12 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:38:12.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: 2022-03-21T19:38:12.365+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:12.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:38:12.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:12.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: yield 2022-03-21T19:38:12.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:12.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:12.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:12.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:38:12.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:12.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:38:12.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:12.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:38:12.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:12.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:38:12.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:12.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:38:12.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:12.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:12.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:12 smithi079 ceph-mon[23158]: pgmap v4381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:38:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:38:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:12 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:38:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:13 smithi016 conmon[32727]: 2022-03-21T19:38:13.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:38:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:38:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:38:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:38:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:38:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:38:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:13 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:38:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:38:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:38:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:13.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:38:13.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:13.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:38:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:38:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:13.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:13.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:38:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:13 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:38:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:14 smithi016 ceph-mon[22010]: pgmap v4382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:14 smithi016 conmon[32727]: 2022-03-21T19:38:14.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:14 smithi079 ceph-mon[23158]: pgmap v4382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[32727]: 2022-03-21T19:38:15.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:15.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 2022-03-21T19:38:15.437+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:15.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:38:15.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: yield 2022-03-21T19:38:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:15.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:15.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:15.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:38:15.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:15.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:38:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:38:15.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:15.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:38:15.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:38:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:15.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:15.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:15.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 2022-03-21T19:38:15.438+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:38:15.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:38:15.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: yield 2022-03-21T19:38:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:15.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:15.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:15.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:38:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:38:15.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:38:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:15.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:38:15.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:15.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:38:15.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:15.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:15.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 2022-03-21T19:38:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:38:15.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: 2022-03-21T19:38:15.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:38:15.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:38:15.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:38:15.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:38:15.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:38:15.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:38:15.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:38:15.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:38:15.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:38:15.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:38:15.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:38:15.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:38:15.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:38:15.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:38:15.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:38:15.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:38:15.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:38:15.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:38:15.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:38:15.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:38:15.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:38:15.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:38:15.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise 2022-03-21T19:38:15.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:38:15.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:38:15.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:38:15.697 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:38:15.697 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:38:15 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:16.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:16 smithi016 conmon[32727]: 2022-03-21T19:38:16.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:16 smithi016 ceph-mon[22010]: pgmap v4383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:16 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:16.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:38:16.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:16.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:16.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:16.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:38:16.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:16.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:38:16.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:16.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:38:16.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:16.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:16.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:16.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:38:16.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:16.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:16.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:16 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:38:16.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:16.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:38:16.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:16.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:16.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:16.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:38:16.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:16.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:38:16.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:16.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:38:16.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:16.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:16.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:16.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:38:16.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:16.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:16.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:38:16.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:38:16.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:38:16.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:38:16.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:38:16.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:38:16.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:38:16.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:38:16.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:38:16.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:38:16.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:38:16.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:38:16.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:38:16.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:38:16.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:38:16.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:38:16.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:38:16.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:38:16.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:38:16.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:38:16.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:38:16.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:38:16.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:38:16.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:38:16.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:38:16.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:38:16.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:38:16.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:38:16.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:38:16.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:16 smithi079 ceph-mon[23158]: pgmap v4383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:16 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:16.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:38:16.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:16.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:16.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:16.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:38:16.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:16.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:38:16.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:16.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:38:16.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:16.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:16.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:16.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:38:16.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:16.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:16.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:16 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:38:16.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:38:16.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:38:16.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:38:16.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:38:16.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:38:16.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:38:16.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:38:16.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:38:16.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:38:16.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:38:16.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:38:16.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:38:16.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:38:16.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:38:16.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:38:16.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:38:16.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:38:16.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:38:16.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:38:16.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:38:16.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:38:16.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:38:16.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:38:16.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:38:16.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:38:16.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:38:16.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:38:16.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:38:16.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:38:16.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:38:16.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:38:16.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:38:16.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:38:16.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:38:16.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:38:16.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:38:16.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:38:16.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:38:16.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:38:16.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:38:16.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:38:16.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:38:16.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:38:16.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:38:16.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:38:16.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:38:16.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:38:16.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:17 smithi016 conmon[32727]: 2022-03-21T19:38:17.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:18 smithi016 ceph-mon[22010]: pgmap v4384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:18 smithi016 conmon[32727]: 2022-03-21T19:38:18.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:18 smithi079 ceph-mon[23158]: pgmap v4384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:19.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:19.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:19 smithi016 conmon[32727]: 2022-03-21T19:38:19.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:20 smithi016 ceph-mon[22010]: pgmap v4385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:20.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:20 smithi016 conmon[32727]: 2022-03-21T19:38:20.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:20 smithi079 ceph-mon[23158]: pgmap v4385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:21 smithi016 conmon[32727]: 2022-03-21T19:38:21.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:22 smithi079 ceph-mon[23158]: pgmap v4386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:22 smithi016 ceph-mon[22010]: pgmap v4386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:22 smithi016 conmon[32727]: 2022-03-21T19:38:22.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:23 smithi016 conmon[32727]: 2022-03-21T19:38:23.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:24 smithi079 ceph-mon[23158]: pgmap v4387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:24 smithi016 ceph-mon[22010]: pgmap v4387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:24 smithi016 conmon[32727]: 2022-03-21T19:38:24.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:25 smithi016 conmon[32727]: 2022-03-21T19:38:25.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:26.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:26 smithi016 ceph-mon[22010]: pgmap v4388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:26.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:26 smithi079 ceph-mon[23158]: pgmap v4388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:26 smithi016 conmon[32727]: 2022-03-21T19:38:26.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:27 smithi016 conmon[32727]: 2022-03-21T19:38:27.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:28 smithi079 ceph-mon[23158]: pgmap v4389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:28 smithi016 conmon[32727]: 2022-03-21T19:38:28.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:28 smithi016 ceph-mon[22010]: pgmap v4389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:29 smithi016 conmon[32727]: 2022-03-21T19:38:29.581+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:30 smithi079 ceph-mon[23158]: pgmap v4390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:30 smithi016 conmon[32727]: 2022-03-21T19:38:30.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:30 smithi016 ceph-mon[22010]: pgmap v4390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:31 smithi016 conmon[32727]: 2022-03-21T19:38:31.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:32 smithi079 ceph-mon[23158]: pgmap v4391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:32 smithi016 ceph-mon[22010]: pgmap v4391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:32 smithi016 conmon[32727]: 2022-03-21T19:38:32.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:33 smithi016 conmon[32727]: 2022-03-21T19:38:33.569+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:34 smithi079 ceph-mon[23158]: pgmap v4392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:34 smithi016 ceph-mon[22010]: pgmap v4392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:34.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:34 smithi016 conmon[32727]: 2022-03-21T19:38:34.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:35.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:35 smithi016 conmon[32727]: 2022-03-21T19:38:35.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:36 smithi079 ceph-mon[23158]: pgmap v4393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:36 smithi016 ceph-mon[22010]: pgmap v4393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:36 smithi016 conmon[32727]: 2022-03-21T19:38:36.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:37 smithi079 ceph-mon[23158]: pgmap v4394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:37 smithi016 ceph-mon[22010]: pgmap v4394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:37 smithi016 conmon[32727]: 2022-03-21T19:38:37.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:38 smithi016 conmon[32727]: 2022-03-21T19:38:38.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:39 smithi016 ceph-mon[22010]: pgmap v4395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:39 smithi016 conmon[32727]: 2022-03-21T19:38:39.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:39 smithi079 ceph-mon[23158]: pgmap v4395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:40.239 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:40 smithi016 conmon[32727]: 2022-03-21T19:38:40.548+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:41 smithi016 ceph-mon[22010]: pgmap v4396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:41 smithi016 conmon[32727]: 2022-03-21T19:38:41.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:41 smithi079 ceph-mon[23158]: pgmap v4396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:42 smithi016 conmon[32727]: 2022-03-21T19:38:42.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:43 smithi016 ceph-mon[22010]: pgmap v4397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:43 smithi016 conmon[32727]: 2022-03-21T19:38:43.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:43.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:43 smithi079 ceph-mon[23158]: pgmap v4397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:44 smithi016 conmon[32727]: 2022-03-21T19:38:44.568+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:45 smithi016 ceph-mon[22010]: pgmap v4398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:45 smithi016 conmon[32727]: 2022-03-21T19:38:45.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:45 smithi079 ceph-mon[23158]: pgmap v4398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:46 smithi016 conmon[32727]: 2022-03-21T19:38:46.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:47 smithi016 ceph-mon[22010]: pgmap v4399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:47 smithi016 conmon[32727]: 2022-03-21T19:38:47.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:47 smithi079 ceph-mon[23158]: pgmap v4399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:48 smithi016 conmon[32727]: 2022-03-21T19:38:48.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:49 smithi016 ceph-mon[22010]: pgmap v4400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:49 smithi016 conmon[32727]: 2022-03-21T19:38:49.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:49 smithi079 ceph-mon[23158]: pgmap v4400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:50 smithi016 conmon[32727]: 2022-03-21T19:38:50.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:51 smithi016 ceph-mon[22010]: pgmap v4401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:51 smithi016 conmon[32727]: 2022-03-21T19:38:51.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:51 smithi079 ceph-mon[23158]: pgmap v4401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:52 smithi016 conmon[32727]: 2022-03-21T19:38:52.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:53 smithi016 conmon[32727]: 2022-03-21T19:38:53.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:53 smithi016 ceph-mon[22010]: pgmap v4402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:53 smithi079 ceph-mon[23158]: pgmap v4402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:54 smithi016 conmon[32727]: 2022-03-21T19:38:54.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:55 smithi016 conmon[32727]: 2022-03-21T19:38:55.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:55 smithi016 ceph-mon[22010]: pgmap v4403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:55 smithi079 ceph-mon[23158]: pgmap v4403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:38:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:38:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:56 smithi016 conmon[32727]: 2022-03-21T19:38:56.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:57 smithi016 conmon[32727]: 2022-03-21T19:38:57.520+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:57 smithi016 ceph-mon[22010]: pgmap v4404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:57 smithi079 ceph-mon[23158]: pgmap v4404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:58.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:58 smithi016 conmon[32727]: 2022-03-21T19:38:58.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:38:59 smithi016 conmon[32727]: 2022-03-21T19:38:59.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:38:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: pgmap v4405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:59.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:59.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:38:59.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:38:59.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:38:59.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:38:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:38:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: pgmap v4405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:38:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:38:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:38:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:38:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:38:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:38:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:39:00.801 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:00 smithi016 conmon[32727]: 2022-03-21T19:39:00.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:01.802 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:01 smithi016 conmon[32727]: 2022-03-21T19:39:01.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:01 smithi016 ceph-mon[22010]: pgmap v4406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:01 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:01 smithi079 ceph-mon[23158]: pgmap v4406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:01 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:02.807 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:02 smithi016 conmon[32727]: 2022-03-21T19:39:02.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:03 smithi016 ceph-mon[22010]: pgmap v4407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:03 smithi016 conmon[32727]: 2022-03-21T19:39:03.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:03 smithi079 ceph-mon[23158]: pgmap v4407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:04.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:04 smithi016 conmon[32727]: 2022-03-21T19:39:04.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:05 smithi016 ceph-mon[22010]: pgmap v4408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:05 smithi016 conmon[32727]: 2022-03-21T19:39:05.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:05 smithi079 ceph-mon[23158]: pgmap v4408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:06 smithi016 conmon[32727]: 2022-03-21T19:39:06.584+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:07.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:07 smithi016 ceph-mon[22010]: pgmap v4409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:07.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:07 smithi016 conmon[32727]: 2022-03-21T19:39:07.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:07 smithi079 ceph-mon[23158]: pgmap v4409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:08.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:08 smithi016 conmon[32727]: 2022-03-21T19:39:08.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:09 smithi016 ceph-mon[22010]: pgmap v4410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:09.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:09 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:09 smithi016 conmon[32727]: 2022-03-21T19:39:09.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:09 smithi079 ceph-mon[23158]: pgmap v4410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:10.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:10.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:09 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:10 smithi016 conmon[32727]: 2022-03-21T19:39:10.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:11 smithi016 ceph-mon[22010]: pgmap v4411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:11.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:11 smithi016 conmon[32727]: 2022-03-21T19:39:11.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:11 smithi079 ceph-mon[23158]: pgmap v4411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:12 smithi016 conmon[32727]: 2022-03-21T19:39:12.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:13 smithi016 ceph-mon[22010]: pgmap v4412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:13 smithi016 conmon[32727]: 2022-03-21T19:39:13.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:13 smithi079 ceph-mon[23158]: pgmap v4412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:14 smithi016 conmon[32727]: 2022-03-21T19:39:14.631+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:15.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:15.845 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:15 smithi016 conmon[32727]: 2022-03-21T19:39:15.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:15 smithi016 ceph-mon[22010]: pgmap v4413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:39:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:39:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:15 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:39:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:15 smithi079 ceph-mon[23158]: pgmap v4413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:39:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:39:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:15 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:39:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:16 smithi016 conmon[32727]: 2022-03-21T19:39:16.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:17.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:17 smithi016 conmon[32727]: 2022-03-21T19:39:17.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:17.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:17 smithi016 ceph-mon[22010]: pgmap v4414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:17.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:17 smithi079 ceph-mon[23158]: pgmap v4414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:18.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:18.857 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[32727]: 2022-03-21T19:39:18.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:18.858 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: 2022-03-21T19:39:18.541+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:18.858 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:39:18.859 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:18.859 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: yield 2022-03-21T19:39:18.859 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:18.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:18.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:18.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:39:18.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:18.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:39:18.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:18.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:39:18.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:18.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:39:18.863 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:18.863 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:39:18.863 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:18.864 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:18.864 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:18 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:19 smithi016 conmon[32727]: 2022-03-21T19:39:19.622+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:19 smithi016 ceph-mon[22010]: pgmap v4415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:19 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:39:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:39:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:39:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:39:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:39:19.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:19.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:19.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:39:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:19.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:19.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:19 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:39:19.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:19.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:19 smithi079 ceph-mon[23158]: pgmap v4415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:19 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:39:20.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:20.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:39:20.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:20.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:20.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:20.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:39:20.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:20.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:39:20.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:20.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:39:20.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:20.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:20.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:20.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:39:20.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:20.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:20.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:20.232 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:19 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:39:20.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:20.233 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:20 smithi016 conmon[32727]: 2022-03-21T19:39:20.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:21.865 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[32727]: 2022-03-21T19:39:21.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:21.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 2022-03-21T19:39:21.613+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:21.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:39:21.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:21.867 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: yield 2022-03-21T19:39:21.867 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:21.868 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:21.868 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:21.868 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:39:21.869 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:21.869 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:39:21.869 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:21.870 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:39:21.870 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:21.870 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:39:21.871 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:21.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:39:21.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:21.872 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:21.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:21.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 2022-03-21T19:39:21.614+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:39:21.873 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:39:21.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:21.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: yield 2022-03-21T19:39:21.874 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:21.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:21.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:21.875 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:39:21.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:21.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:39:21.876 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:21.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:39:21.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:21.877 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:39:21.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:21.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:39:21.878 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:21.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:21.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:21.879 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 2022-03-21T19:39:21.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:39:21.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: 2022-03-21T19:39:21.880 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:39:21.881 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:39:21.881 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:39:21.881 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:39:21.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:39:21.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:39:21.882 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:39:21.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:39:21.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:39:21.883 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:39:21.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:39:21.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:39:21.884 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:39:21.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:39:21.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:39:21.885 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:39:21.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:39:21.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:39:21.886 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:39:21.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:39:21.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:39:21.887 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:39:21.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise 2022-03-21T19:39:21.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:39:21.888 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:39:21.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:39:21.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:39:21.889 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:39:21 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:21 smithi016 ceph-mon[22010]: pgmap v4416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:21 smithi079 ceph-mon[23158]: pgmap v4416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:23.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:22 smithi016 conmon[32727]: 2022-03-21T19:39:22.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:22 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:39:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:23.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:23.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:23.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:39:23.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:23.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:39:23.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:39:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:23.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:39:23.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:23.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:23.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.173 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:22 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:39:23.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:23.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:39:23.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:23.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:23.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:23.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:39:23.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:23.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:39:23.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:23.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:39:23.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:23.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:23.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:23.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:39:23.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:23.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:23.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:39:23.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:39:23.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:39:23.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:39:23.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:39:23.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:39:23.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:39:23.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:39:23.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:39:23.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:39:23.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:39:23.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:39:23.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:39:23.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:39:23.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:39:23.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:39:23.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:39:23.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:39:23.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:39:23.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:39:23.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:39:23.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:39:23.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:39:23.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:39:23.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:39:23.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:39:23.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:39:23.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:39:23.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:39:23.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.191 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:22 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:39:23.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:23.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:39:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:39:23.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:23.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:39:23.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:23.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:23.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:23.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:39:23.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:23.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:23.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:22 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:39:23.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:39:23.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:39:23.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:39:23.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:39:23.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:39:23.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:39:23.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:39:23.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:39:23.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:39:23.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:39:23.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:39:23.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:39:23.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:39:23.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:39:23.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:39:23.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:39:23.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:39:23.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:39:23.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:39:23.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:39:23.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:39:23.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:39:23.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:39:23.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:39:23.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:39:23.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:39:23.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:39:23.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:39:23.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:39:23.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:39:23.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:39:23.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:39:23.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:39:23.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:39:23.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:39:23.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:39:23.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:39:23.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:39:23.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:39:23.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:39:23.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:39:23.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:39:23.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:39:23.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:39:23.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:39:23.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:39:23.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:39:23.249 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:23 smithi016 ceph-mon[22010]: pgmap v4417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:23.919 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:23.919 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:23 smithi016 conmon[32727]: 2022-03-21T19:39:23.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:23 smithi079 ceph-mon[23158]: pgmap v4417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:24.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:24 smithi016 conmon[32727]: 2022-03-21T19:39:24.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:25.879 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:25 smithi016 conmon[32727]: 2022-03-21T19:39:25.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:25 smithi016 ceph-mon[22010]: pgmap v4418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:25 smithi079 ceph-mon[23158]: pgmap v4418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:26.884 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:26 smithi016 conmon[32727]: 2022-03-21T19:39:26.622+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:27.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:27.886 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:27 smithi016 conmon[32727]: 2022-03-21T19:39:27.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:27 smithi016 ceph-mon[22010]: pgmap v4419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:27 smithi079 ceph-mon[23158]: pgmap v4419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:28.890 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:28 smithi016 conmon[32727]: 2022-03-21T19:39:28.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:29.892 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:29 smithi016 conmon[32727]: 2022-03-21T19:39:29.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:29 smithi016 ceph-mon[22010]: pgmap v4420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:30.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:29 smithi079 ceph-mon[23158]: pgmap v4420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:30.899 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:30 smithi016 conmon[32727]: 2022-03-21T19:39:30.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:31.902 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:31 smithi016 conmon[32727]: 2022-03-21T19:39:31.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:31 smithi016 ceph-mon[22010]: pgmap v4421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:31 smithi079 ceph-mon[23158]: pgmap v4421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:32.906 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:32 smithi016 conmon[32727]: 2022-03-21T19:39:32.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:33.907 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:33 smithi016 conmon[32727]: 2022-03-21T19:39:33.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:33 smithi016 ceph-mon[22010]: pgmap v4422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:33 smithi079 ceph-mon[23158]: pgmap v4422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:34.912 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:34 smithi016 conmon[32727]: 2022-03-21T19:39:34.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:35 smithi016 conmon[32727]: 2022-03-21T19:39:35.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:35 smithi016 ceph-mon[22010]: pgmap v4423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:35 smithi079 ceph-mon[23158]: pgmap v4423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:36.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:36 smithi016 conmon[32727]: 2022-03-21T19:39:36.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:37 smithi016 conmon[32727]: 2022-03-21T19:39:37.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:37 smithi079 ceph-mon[23158]: pgmap v4424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:37 smithi016 ceph-mon[22010]: pgmap v4424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:38 smithi016 conmon[32727]: 2022-03-21T19:39:38.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:39 smithi016 conmon[32727]: 2022-03-21T19:39:39.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:39 smithi079 ceph-mon[23158]: pgmap v4425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:39 smithi016 ceph-mon[22010]: pgmap v4425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:40 smithi016 conmon[32727]: 2022-03-21T19:39:40.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:41.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:41 smithi016 conmon[32727]: 2022-03-21T19:39:41.648+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:41 smithi079 ceph-mon[23158]: pgmap v4426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:41 smithi016 ceph-mon[22010]: pgmap v4426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:42.937 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:42 smithi016 conmon[32727]: 2022-03-21T19:39:42.680+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:43 smithi016 conmon[32727]: 2022-03-21T19:39:43.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:43 smithi079 ceph-mon[23158]: pgmap v4427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:44.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:43 smithi016 ceph-mon[22010]: pgmap v4427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:44.942 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:44 smithi016 conmon[32727]: 2022-03-21T19:39:44.665+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:45.943 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:45 smithi016 conmon[32727]: 2022-03-21T19:39:45.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:45 smithi079 ceph-mon[23158]: pgmap v4428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:45 smithi016 ceph-mon[22010]: pgmap v4428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:46.948 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:46 smithi016 conmon[32727]: 2022-03-21T19:39:46.664+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:47 smithi016 conmon[32727]: 2022-03-21T19:39:47.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:47 smithi079 ceph-mon[23158]: pgmap v4429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:47 smithi016 ceph-mon[22010]: pgmap v4429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:48.955 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:48 smithi016 conmon[32727]: 2022-03-21T19:39:48.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:49.956 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:49 smithi016 conmon[32727]: 2022-03-21T19:39:49.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:49 smithi079 ceph-mon[23158]: pgmap v4430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:50.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:49 smithi016 ceph-mon[22010]: pgmap v4430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:50 smithi016 conmon[32727]: 2022-03-21T19:39:50.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:52.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:51 smithi016 conmon[32727]: 2022-03-21T19:39:51.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:52.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:51 smithi016 ceph-mon[22010]: pgmap v4431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:52.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:51 smithi079 ceph-mon[23158]: pgmap v4431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:52.970 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:52 smithi016 conmon[32727]: 2022-03-21T19:39:52.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:53.972 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:53 smithi016 conmon[32727]: 2022-03-21T19:39:53.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:53 smithi079 ceph-mon[23158]: pgmap v4432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:53 smithi016 ceph-mon[22010]: pgmap v4432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:54.979 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:54 smithi016 conmon[32727]: 2022-03-21T19:39:54.684+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:39:55.980 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:55 smithi016 conmon[32727]: 2022-03-21T19:39:55.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:55 smithi016 ceph-mon[22010]: pgmap v4433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:55 smithi079 ceph-mon[23158]: pgmap v4433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:39:56.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:56 smithi016 conmon[32727]: 2022-03-21T19:39:56.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:57.987 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:57 smithi016 conmon[32727]: 2022-03-21T19:39:57.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:57 smithi016 ceph-mon[22010]: pgmap v4434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:57 smithi079 ceph-mon[23158]: pgmap v4434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:39:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:58 smithi016 conmon[32727]: 2022-03-21T19:39:58.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:39:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:39:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:39:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:39:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:39:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:39:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:39:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:39:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:39:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:39:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:39:59 smithi016 conmon[32727]: 2022-03-21T19:39:59.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:59 smithi016 ceph-mon[22010]: pgmap v4435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:39:59 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:59 smithi079 ceph-mon[23158]: pgmap v4435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:39:59 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:00 smithi016 conmon[32727]: 2022-03-21T19:40:00.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops 2022-03-21T19:40:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:40:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:40:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:40:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:40:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:40:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops 2022-03-21T19:40:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops 2022-03-21T19:40:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:40:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:40:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:40:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:40:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:40:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8240 sec, osd.4 has slow ops 2022-03-21T19:40:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:01 smithi016 conmon[32727]: 2022-03-21T19:40:01.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:02 smithi016 ceph-mon[22010]: pgmap v4436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:02 smithi079 ceph-mon[23158]: pgmap v4436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:02.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:02 smithi016 conmon[32727]: 2022-03-21T19:40:02.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:03 smithi016 conmon[32727]: 2022-03-21T19:40:03.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:04 smithi016 ceph-mon[22010]: pgmap v4437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:04 smithi079 ceph-mon[23158]: pgmap v4437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:04 smithi016 conmon[32727]: 2022-03-21T19:40:04.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:05 smithi016 conmon[32727]: 2022-03-21T19:40:05.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:06 smithi016 ceph-mon[22010]: pgmap v4438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:06 smithi079 ceph-mon[23158]: pgmap v4438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:06 smithi016 conmon[32727]: 2022-03-21T19:40:06.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:07 smithi016 conmon[32727]: 2022-03-21T19:40:07.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:08 smithi016 ceph-mon[22010]: pgmap v4439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:08 smithi079 ceph-mon[23158]: pgmap v4439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:09.026 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:08 smithi016 conmon[32727]: 2022-03-21T19:40:08.677+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:09 smithi016 conmon[32727]: 2022-03-21T19:40:09.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:10 smithi016 ceph-mon[22010]: pgmap v4440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:10 smithi079 ceph-mon[23158]: pgmap v4440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:40:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:11.032 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:10 smithi016 conmon[32727]: 2022-03-21T19:40:10.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:11 smithi016 conmon[32727]: 2022-03-21T19:40:11.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:12 smithi016 ceph-mon[22010]: pgmap v4441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:12 smithi079 ceph-mon[23158]: pgmap v4441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:12 smithi016 conmon[32727]: 2022-03-21T19:40:12.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:13 smithi016 conmon[32727]: 2022-03-21T19:40:13.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:14 smithi016 ceph-mon[22010]: pgmap v4442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:14 smithi079 ceph-mon[23158]: pgmap v4442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:14 smithi016 conmon[32727]: 2022-03-21T19:40:14.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:15.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:15 smithi016 conmon[32727]: 2022-03-21T19:40:15.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:16 smithi016 ceph-mon[22010]: pgmap v4443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:16.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:16 smithi079 ceph-mon[23158]: pgmap v4443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:16 smithi016 conmon[32727]: 2022-03-21T19:40:16.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:17 smithi016 conmon[32727]: 2022-03-21T19:40:17.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:18 smithi016 ceph-mon[22010]: pgmap v4444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:18 smithi079 ceph-mon[23158]: pgmap v4444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:18 smithi016 conmon[32727]: 2022-03-21T19:40:18.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:19 smithi016 conmon[32727]: 2022-03-21T19:40:19.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:20 smithi016 ceph-mon[22010]: pgmap v4445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:20 smithi079 ceph-mon[23158]: pgmap v4445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:20 smithi016 conmon[32727]: 2022-03-21T19:40:20.508+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:21 smithi016 conmon[32727]: 2022-03-21T19:40:21.494+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:22 smithi016 ceph-mon[22010]: pgmap v4446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:40:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:40:22.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:22 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:40:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:22 smithi079 ceph-mon[23158]: pgmap v4446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:40:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:40:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:22 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:40:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:22 smithi016 conmon[32727]: 2022-03-21T19:40:22.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:23 smithi016 conmon[32727]: 2022-03-21T19:40:23.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:24 smithi016 ceph-mon[22010]: pgmap v4447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:24 smithi079 ceph-mon[23158]: pgmap v4447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[32727]: 2022-03-21T19:40:24.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:24.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: 2022-03-21T19:40:24.717+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:24.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:40:24.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:24.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: yield 2022-03-21T19:40:24.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:24.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:24.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:24.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:40:24.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:24.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:40:24.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:24.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:40:24.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:24.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:40:24.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:24.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:40:24.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:24.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:24.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:25 smithi016 conmon[32727]: 2022-03-21T19:40:25.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:26.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:26 smithi016 ceph-mon[22010]: pgmap v4448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:26 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:40:26.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:26.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:40:26.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:26.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:40:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:40:26.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:40:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:26.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:26.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:26.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:40:26.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:26.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:26.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:26.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:26 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:40:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:26 smithi079 ceph-mon[23158]: pgmap v4448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:26 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:40:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:40:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:26.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:40:26.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:40:26.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:40:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:26.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:40:26.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:26.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:26.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:26 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:40:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:26 smithi016 conmon[32727]: 2022-03-21T19:40:26.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:27.791 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[32727]: 2022-03-21T19:40:27.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:28.088 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 2022-03-21T19:40:27.789+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:28.088 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:40:28.089 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:28.089 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: yield 2022-03-21T19:40:28.089 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:28.090 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:28.090 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:28.090 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:40:28.091 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:28.091 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:40:28.092 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:28.092 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:40:28.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:28.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:40:28.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:28.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:40:28.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:28.094 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:28.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:28.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 2022-03-21T19:40:27.790+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:40:28.095 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:40:28.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:28.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: yield 2022-03-21T19:40:28.096 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:28.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:28.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:28.099 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:40:28.099 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:28.099 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:40:28.100 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:28.100 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:40:28.100 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:28.101 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:40:28.101 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:28.101 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:40:28.102 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:28.102 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:28.102 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:28.103 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 2022-03-21T19:40:28.103 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:40:28.103 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: 2022-03-21T19:40:28.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:40:28.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:40:28.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:40:28.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:40:28.105 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:40:28.106 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:40:28.106 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:40:28.107 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:40:28.107 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:40:28.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:40:28.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:40:28.108 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:40:28.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:40:28.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:40:28.109 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:40:28.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:40:28.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:40:28.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:40:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:40:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:40:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:40:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:40:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise 2022-03-21T19:40:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:40:28.113 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:40:28.113 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:40:28.113 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:40:28.114 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:40:27 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:28 smithi016 ceph-mon[22010]: pgmap v4449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:28 smithi079 ceph-mon[23158]: pgmap v4449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:28 smithi016 conmon[32727]: 2022-03-21T19:40:28.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:40:29.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:29.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:29.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:29.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:40:29.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:29.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:40:29.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:29.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:40:29.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:29.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:29.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:29.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:40:29.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:29.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:29.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.423 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:29 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:40:29.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:40:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:29.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:29.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:29.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:40:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:40:29.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:29.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:40:29.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:29.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:29.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:29.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:40:29.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:29.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:29.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:40:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:40:29.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:40:29.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:40:29.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:40:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:40:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:40:29.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:40:29.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:40:29.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:40:29.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:40:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:40:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:40:29.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:40:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:40:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:40:29.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:40:29.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:40:29.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:40:29.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:40:29.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:40:29.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:40:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:40:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:40:29.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:40:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:40:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:40:29.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:40:29.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:40:29.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:29.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:29.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:40:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:29.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:40:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:29.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:40:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:40:29.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:29.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:40:29.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:29.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:29.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:29 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:40:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:40:29.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:40:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:40:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:40:29.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:40:29.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:40:29.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:40:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:40:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:40:29.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:40:29.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:40:29.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:40:29.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:40:29.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:40:29.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:40:29.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:40:29.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:40:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:40:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:40:29.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:40:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:40:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:40:29.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:40:29.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:40:29.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:40:29.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:40:29.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:40:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:40:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:40:29.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:40:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:40:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:40:29.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:40:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:40:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:40:29.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:40:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:40:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:40:29.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:40:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:40:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:40:29.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:40:29.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:40:29.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:40:29.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:40:29.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:40:29.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:40:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:29 smithi016 conmon[32727]: 2022-03-21T19:40:29.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:30 smithi016 ceph-mon[22010]: pgmap v4450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:30 smithi079 ceph-mon[23158]: pgmap v4450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:30 smithi016 conmon[32727]: 2022-03-21T19:40:30.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:31 smithi016 conmon[32727]: 2022-03-21T19:40:31.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:32 smithi016 ceph-mon[22010]: pgmap v4451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:32 smithi079 ceph-mon[23158]: pgmap v4451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:32 smithi016 conmon[32727]: 2022-03-21T19:40:32.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:33 smithi016 conmon[32727]: 2022-03-21T19:40:33.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:34 smithi016 ceph-mon[22010]: pgmap v4452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:34 smithi079 ceph-mon[23158]: pgmap v4452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:34 smithi016 conmon[32727]: 2022-03-21T19:40:34.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:35 smithi016 conmon[32727]: 2022-03-21T19:40:35.474+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:36 smithi016 ceph-mon[22010]: pgmap v4453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:36 smithi079 ceph-mon[23158]: pgmap v4453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:36 smithi016 conmon[32727]: 2022-03-21T19:40:36.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:37 smithi016 conmon[32727]: 2022-03-21T19:40:37.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:38 smithi016 ceph-mon[22010]: pgmap v4454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:38 smithi079 ceph-mon[23158]: pgmap v4454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:38 smithi016 conmon[32727]: 2022-03-21T19:40:38.459+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:39 smithi016 conmon[32727]: 2022-03-21T19:40:39.432+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:40.395 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:40.396 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:40 smithi016 ceph-mon[22010]: pgmap v4455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:40.396 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:40 smithi079 ceph-mon[23158]: pgmap v4455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:40 smithi016 conmon[32727]: 2022-03-21T19:40:40.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:41.397 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:41.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:41.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:41 smithi016 conmon[32727]: 2022-03-21T19:40:41.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:42 smithi016 ceph-mon[22010]: pgmap v4456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:42.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:42 smithi016 conmon[32727]: 2022-03-21T19:40:42.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:42 smithi079 ceph-mon[23158]: pgmap v4456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:43 smithi016 conmon[32727]: 2022-03-21T19:40:43.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:44.408 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:44.409 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:44 smithi016 ceph-mon[22010]: pgmap v4457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:44 smithi079 ceph-mon[23158]: pgmap v4457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:44 smithi016 conmon[32727]: 2022-03-21T19:40:44.407+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:45.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:45 smithi016 conmon[32727]: 2022-03-21T19:40:45.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:46 smithi016 ceph-mon[22010]: pgmap v4458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:46 smithi079 ceph-mon[23158]: pgmap v4458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:46 smithi016 conmon[32727]: 2022-03-21T19:40:46.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:47 smithi016 conmon[32727]: 2022-03-21T19:40:47.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:48 smithi016 ceph-mon[22010]: pgmap v4459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:48 smithi079 ceph-mon[23158]: pgmap v4459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:48 smithi016 conmon[32727]: 2022-03-21T19:40:48.426+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:49 smithi016 conmon[32727]: 2022-03-21T19:40:49.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:50 smithi016 ceph-mon[22010]: pgmap v4460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:50 smithi079 ceph-mon[23158]: pgmap v4460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:50 smithi016 conmon[32727]: 2022-03-21T19:40:50.442+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:51 smithi016 conmon[32727]: 2022-03-21T19:40:51.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:52 smithi079 ceph-mon[23158]: pgmap v4461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:52.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:52.489 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:52 smithi016 ceph-mon[22010]: pgmap v4461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:52 smithi016 conmon[32727]: 2022-03-21T19:40:52.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:53.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:53 smithi016 conmon[32727]: 2022-03-21T19:40:53.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:54 smithi079 ceph-mon[23158]: pgmap v4462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:54.487 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:54.487 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:54 smithi016 ceph-mon[22010]: pgmap v4462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:54 smithi016 conmon[32727]: 2022-03-21T19:40:54.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:55.498 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:55.498 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:40:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:55 smithi016 conmon[32727]: 2022-03-21T19:40:55.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:56 smithi079 ceph-mon[23158]: pgmap v4463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:56.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:56.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:56 smithi016 ceph-mon[22010]: pgmap v4463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:40:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:56 smithi016 conmon[32727]: 2022-03-21T19:40:56.505+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:57.484 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:57 smithi016 conmon[32727]: 2022-03-21T19:40:57.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:58 smithi079 ceph-mon[23158]: pgmap v4464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:58.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:58.495 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:58 smithi016 ceph-mon[22010]: pgmap v4464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:40:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:58 smithi016 conmon[32727]: 2022-03-21T19:40:58.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:40:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:40:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:40:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:40:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:40:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:40:59.484 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:40:59.485 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:40:59.485 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:40:59.486 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:40:59.486 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:40:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:40:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:40:59 smithi016 conmon[32727]: 2022-03-21T19:40:59.482+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:00 smithi079 ceph-mon[23158]: pgmap v4465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:00.530 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:00.531 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:00 smithi016 ceph-mon[22010]: pgmap v4465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:00.532 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:00 smithi016 conmon[32727]: 2022-03-21T19:41:00.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:01.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:01 smithi016 conmon[32727]: 2022-03-21T19:41:01.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:02 smithi079 ceph-mon[23158]: pgmap v4466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:02.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:02.482 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:02 smithi016 ceph-mon[22010]: pgmap v4466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:02.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:02 smithi016 conmon[32727]: 2022-03-21T19:41:02.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:03.468 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:03 smithi016 conmon[32727]: 2022-03-21T19:41:03.466+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:04 smithi079 ceph-mon[23158]: pgmap v4467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:04.478 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:04.479 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:04 smithi016 ceph-mon[22010]: pgmap v4467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:04.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:04 smithi016 conmon[32727]: 2022-03-21T19:41:04.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:05.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:05 smithi016 conmon[32727]: 2022-03-21T19:41:05.436+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:06 smithi079 ceph-mon[23158]: pgmap v4468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:06 smithi016 conmon[32727]: 2022-03-21T19:41:06.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:06 smithi016 ceph-mon[22010]: pgmap v4468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:07.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:07 smithi016 conmon[32727]: 2022-03-21T19:41:07.402+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:08 smithi079 ceph-mon[23158]: pgmap v4469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:08 smithi016 ceph-mon[22010]: pgmap v4469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:08 smithi016 conmon[32727]: 2022-03-21T19:41:08.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:09 smithi016 conmon[32727]: 2022-03-21T19:41:09.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:10 smithi016 ceph-mon[22010]: pgmap v4470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:10 smithi016 conmon[32727]: 2022-03-21T19:41:10.460+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:10 smithi079 ceph-mon[23158]: pgmap v4470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:11.497 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:11 smithi016 conmon[32727]: 2022-03-21T19:41:11.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:12 smithi016 ceph-mon[22010]: pgmap v4471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:12.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:12 smithi016 conmon[32727]: 2022-03-21T19:41:12.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:12 smithi079 ceph-mon[23158]: pgmap v4471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:13.750 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:13 smithi016 conmon[32727]: 2022-03-21T19:41:13.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:13.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:14.503 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:14.504 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:14 smithi016 ceph-mon[22010]: pgmap v4472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:14 smithi079 ceph-mon[23158]: pgmap v4472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:14 smithi016 conmon[32727]: 2022-03-21T19:41:14.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:15 smithi016 conmon[32727]: 2022-03-21T19:41:15.454+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:16 smithi016 ceph-mon[22010]: pgmap v4473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:16 smithi016 conmon[32727]: 2022-03-21T19:41:16.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:16 smithi079 ceph-mon[23158]: pgmap v4473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:17 smithi016 conmon[32727]: 2022-03-21T19:41:17.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:18 smithi016 ceph-mon[22010]: pgmap v4474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:18.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:18 smithi016 conmon[32727]: 2022-03-21T19:41:18.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:18 smithi079 ceph-mon[23158]: pgmap v4474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:19 smithi016 conmon[32727]: 2022-03-21T19:41:19.406+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:20 smithi016 ceph-mon[22010]: pgmap v4475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:20.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:20 smithi016 conmon[32727]: 2022-03-21T19:41:20.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:20 smithi079 ceph-mon[23158]: pgmap v4475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8320 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:21 smithi016 conmon[32727]: 2022-03-21T19:41:21.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:22 smithi016 ceph-mon[22010]: pgmap v4476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:22.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:22 smithi016 conmon[32727]: 2022-03-21T19:41:22.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:22 smithi079 ceph-mon[23158]: pgmap v4476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:23 smithi016 conmon[32727]: 2022-03-21T19:41:23.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:24 smithi016 ceph-mon[22010]: pgmap v4477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:24 smithi016 conmon[32727]: 2022-03-21T19:41:24.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:24 smithi079 ceph-mon[23158]: pgmap v4477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:25.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:25 smithi016 conmon[32727]: 2022-03-21T19:41:25.448+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:26 smithi016 ceph-mon[22010]: pgmap v4478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:26.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:26 smithi016 conmon[32727]: 2022-03-21T19:41:26.496+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:26 smithi079 ceph-mon[23158]: pgmap v4478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:27 smithi016 conmon[32727]: 2022-03-21T19:41:27.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[32727]: 2022-03-21T19:41:28.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:28 smithi016 ceph-mon[22010]: pgmap v4479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:41:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:41:28.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:28 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:41:28.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: 2022-03-21T19:41:28.397+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:28.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:41:28.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:28.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: yield 2022-03-21T19:41:28.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:28.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:28.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:28.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:41:28.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:28.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:41:28.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:28.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:41:28.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:28.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:41:28.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:28.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:41:28.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:28.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:28.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:28 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:28 smithi079 ceph-mon[23158]: pgmap v4479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:41:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:41:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:28 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:41:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:29 smithi016 conmon[32727]: 2022-03-21T19:41:29.415+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:29.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:29 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:41:29.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:29.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:41:29.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:29.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:29.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:29.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:41:29.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:29.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:41:29.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:29.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:41:29.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:29.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:29.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:29.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:41:29.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:29.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:29.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:29.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:29 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:41:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:29 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:41:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:41:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:29.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:41:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:29.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:41:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:41:29.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:29.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:41:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:29.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:29.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:29 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:41:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:30 smithi016 ceph-mon[22010]: pgmap v4480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:30 smithi016 conmon[32727]: 2022-03-21T19:41:30.365+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:30 smithi079 ceph-mon[23158]: pgmap v4480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:31.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:31.339 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:31 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:41:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[32727]: 2022-03-21T19:41:31.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:31.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 2022-03-21T19:41:31.469+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:31.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:41:31.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:31.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: yield 2022-03-21T19:41:31.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:31.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:31.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:31.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:41:31.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:31.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:41:31.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:31.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:41:31.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:31.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:41:31.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:31.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:41:31.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:31.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:31.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:31.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 2022-03-21T19:41:31.470+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:41:31.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:41:31.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:31.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: yield 2022-03-21T19:41:31.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:31.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:31.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:31.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:41:31.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:31.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:41:31.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:31.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:41:31.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:31.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:41:31.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:31.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:41:31.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:31.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:31.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:31.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 2022-03-21T19:41:31.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:41:31.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: 2022-03-21T19:41:31.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:41:31.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:41:31.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:41:31.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:41:31.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:41:31.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:41:31.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:41:31.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:41:31.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:41:31.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:41:31.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:41:31.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:41:31.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:41:31.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:41:31.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:41:31.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:41:31.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:41:31.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:41:31.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:41:31.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:41:31.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:41:31.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:41:31.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise 2022-03-21T19:41:31.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:41:31.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:41:31.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:41:31.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:41:31.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:41:31 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:31 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:41:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:32 smithi016 conmon[32727]: 2022-03-21T19:41:32.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: pgmap v4481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T19:41:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:32.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:41:32.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:32.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:32.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:32.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:41:32.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:32.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:41:32.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:32.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:41:32.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:32.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:32.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:32.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:41:32.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:32.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:32.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:32 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:41:32.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:32.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:41:32.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:32.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:32.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:32.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:41:32.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:32.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:41:32.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:32.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:41:32.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:32.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:32.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:32.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:41:32.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:32.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:32.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:41:32.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:41:32.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:41:32.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:41:32.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:41:32.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:41:32.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:41:32.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:41:32.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:41:32.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:41:32.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:41:32.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:41:32.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:41:32.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:41:32.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:41:32.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:41:32.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:41:32.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:41:32.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:41:32.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:41:32.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:41:32.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:41:32.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:41:32.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:41:32.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:41:32.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:41:32.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:41:32.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:41:32.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:41:32.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: pgmap v4481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T19:41:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:32.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:41:32.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:32.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:32.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:32.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:41:32.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:32.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:41:32.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:32.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:41:32.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:32.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:32.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:32.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:41:32.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:32.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:32.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:32 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:41:32.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:41:32.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:41:32.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:41:32.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:41:32.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:41:32.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:41:32.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:41:32.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:41:32.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:41:32.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:41:32.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:41:32.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:41:32.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:41:32.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:41:32.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:41:32.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:41:32.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:32.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:41:32.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:41:32.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:41:32.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:41:32.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:41:32.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:41:32.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:41:32.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:41:32.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:41:32.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:41:32.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:41:32.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:41:32.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:41:32.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:41:32.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:41:32.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:41:32.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:41:32.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:41:32.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:41:32.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:41:32.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:41:32.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:41:32.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:41:32.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:41:32.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:41:32.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:41:32.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:41:32.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:41:32.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:41:32.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:41:32.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:41:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:33.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:33 smithi016 conmon[32727]: 2022-03-21T19:41:33.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:34 smithi016 ceph-mon[22010]: pgmap v4482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:34.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:34 smithi016 conmon[32727]: 2022-03-21T19:41:34.378+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:34 smithi079 ceph-mon[23158]: pgmap v4482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:35 smithi016 conmon[32727]: 2022-03-21T19:41:35.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:36 smithi016 ceph-mon[22010]: pgmap v4483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:36 smithi016 conmon[32727]: 2022-03-21T19:41:36.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:36 smithi079 ceph-mon[23158]: pgmap v4483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:37 smithi016 conmon[32727]: 2022-03-21T19:41:37.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:38 smithi016 ceph-mon[22010]: pgmap v4484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:38 smithi016 conmon[32727]: 2022-03-21T19:41:38.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:38 smithi079 ceph-mon[23158]: pgmap v4484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:39 smithi016 conmon[32727]: 2022-03-21T19:41:39.390+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:40 smithi016 ceph-mon[22010]: pgmap v4485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:40 smithi016 conmon[32727]: 2022-03-21T19:41:40.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:40 smithi079 ceph-mon[23158]: pgmap v4485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8340 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:41 smithi016 conmon[32727]: 2022-03-21T19:41:41.418+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:42 smithi016 ceph-mon[22010]: pgmap v4486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:42 smithi016 conmon[32727]: 2022-03-21T19:41:42.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:42 smithi079 ceph-mon[23158]: pgmap v4486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:43.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:43 smithi016 conmon[32727]: 2022-03-21T19:41:43.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:44 smithi016 ceph-mon[22010]: pgmap v4487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:44 smithi016 conmon[32727]: 2022-03-21T19:41:44.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:44 smithi079 ceph-mon[23158]: pgmap v4487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:45 smithi016 conmon[32727]: 2022-03-21T19:41:45.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:46 smithi016 conmon[32727]: 2022-03-21T19:41:46.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:46 smithi016 ceph-mon[22010]: pgmap v4488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:46 smithi079 ceph-mon[23158]: pgmap v4488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:47 smithi016 conmon[32727]: 2022-03-21T19:41:47.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:48 smithi016 conmon[32727]: 2022-03-21T19:41:48.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:48 smithi016 ceph-mon[22010]: pgmap v4489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:48 smithi079 ceph-mon[23158]: pgmap v4489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:49 smithi016 conmon[32727]: 2022-03-21T19:41:49.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:50 smithi016 conmon[32727]: 2022-03-21T19:41:50.494+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:50 smithi016 ceph-mon[22010]: pgmap v4490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:50 smithi079 ceph-mon[23158]: pgmap v4490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:51 smithi016 conmon[32727]: 2022-03-21T19:41:51.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:52 smithi016 ceph-mon[22010]: pgmap v4491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:52 smithi016 conmon[32727]: 2022-03-21T19:41:52.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:52 smithi079 ceph-mon[23158]: pgmap v4491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:53 smithi016 conmon[32727]: 2022-03-21T19:41:53.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:54 smithi016 ceph-mon[22010]: pgmap v4492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:54.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:54 smithi016 conmon[32727]: 2022-03-21T19:41:54.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:54 smithi079 ceph-mon[23158]: pgmap v4492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:55 smithi016 conmon[32727]: 2022-03-21T19:41:55.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:41:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:56 smithi016 ceph-mon[22010]: pgmap v4493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:56 smithi016 conmon[32727]: 2022-03-21T19:41:56.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:56 smithi079 ceph-mon[23158]: pgmap v4493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:41:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:57 smithi016 conmon[32727]: 2022-03-21T19:41:57.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:58 smithi016 ceph-mon[22010]: pgmap v4494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:58 smithi016 conmon[32727]: 2022-03-21T19:41:58.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:58 smithi079 ceph-mon[23158]: pgmap v4494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:41:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:41:59 smithi016 conmon[32727]: 2022-03-21T19:41:59.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:41:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:41:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:41:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:41:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:41:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:41:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:41:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:41:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:41:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:41:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:41:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:41:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:00 smithi016 ceph-mon[22010]: pgmap v4495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:00 smithi016 conmon[32727]: 2022-03-21T19:42:00.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:00 smithi079 ceph-mon[23158]: pgmap v4495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8360 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:01 smithi016 conmon[32727]: 2022-03-21T19:42:01.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:02 smithi016 ceph-mon[22010]: pgmap v4496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:02 smithi016 conmon[32727]: 2022-03-21T19:42:02.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:02 smithi079 ceph-mon[23158]: pgmap v4496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:03 smithi016 conmon[32727]: 2022-03-21T19:42:03.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:04 smithi016 ceph-mon[22010]: pgmap v4497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:04 smithi016 conmon[32727]: 2022-03-21T19:42:04.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:04 smithi079 ceph-mon[23158]: pgmap v4497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:05 smithi016 conmon[32727]: 2022-03-21T19:42:05.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:06 smithi016 ceph-mon[22010]: pgmap v4498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:06.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:06 smithi016 conmon[32727]: 2022-03-21T19:42:06.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:06 smithi079 ceph-mon[23158]: pgmap v4498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:07 smithi016 conmon[32727]: 2022-03-21T19:42:07.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:08 smithi016 ceph-mon[22010]: pgmap v4499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:08 smithi016 conmon[32727]: 2022-03-21T19:42:08.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:08 smithi079 ceph-mon[23158]: pgmap v4499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:09 smithi016 conmon[32727]: 2022-03-21T19:42:09.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:10 smithi079 ceph-mon[23158]: pgmap v4500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:10 smithi016 conmon[32727]: 2022-03-21T19:42:10.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:10 smithi016 ceph-mon[22010]: pgmap v4500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:11 smithi016 conmon[32727]: 2022-03-21T19:42:11.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:12 smithi079 ceph-mon[23158]: pgmap v4501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:12 smithi016 conmon[32727]: 2022-03-21T19:42:12.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:12 smithi016 ceph-mon[22010]: pgmap v4501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:13 smithi016 conmon[32727]: 2022-03-21T19:42:13.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:14 smithi079 ceph-mon[23158]: pgmap v4502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:14 smithi016 ceph-mon[22010]: pgmap v4502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:14 smithi016 conmon[32727]: 2022-03-21T19:42:14.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:15 smithi016 conmon[32727]: 2022-03-21T19:42:15.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:16 smithi079 ceph-mon[23158]: pgmap v4503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:16 smithi016 ceph-mon[22010]: pgmap v4503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:16.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:16 smithi016 conmon[32727]: 2022-03-21T19:42:16.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:17 smithi016 conmon[32727]: 2022-03-21T19:42:17.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:18 smithi079 ceph-mon[23158]: pgmap v4504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:18 smithi016 ceph-mon[22010]: pgmap v4504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:18 smithi016 conmon[32727]: 2022-03-21T19:42:18.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:19 smithi016 conmon[32727]: 2022-03-21T19:42:19.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:20 smithi079 ceph-mon[23158]: pgmap v4505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:20 smithi016 ceph-mon[22010]: pgmap v4505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8380 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:20 smithi016 conmon[32727]: 2022-03-21T19:42:20.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:21 smithi016 conmon[32727]: 2022-03-21T19:42:21.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:22 smithi079 ceph-mon[23158]: pgmap v4506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:22 smithi016 ceph-mon[22010]: pgmap v4506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:22 smithi016 conmon[32727]: 2022-03-21T19:42:22.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:23 smithi016 conmon[32727]: 2022-03-21T19:42:23.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:24 smithi079 ceph-mon[23158]: pgmap v4507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:24 smithi016 ceph-mon[22010]: pgmap v4507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:24.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:24 smithi016 conmon[32727]: 2022-03-21T19:42:24.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:25 smithi016 conmon[32727]: 2022-03-21T19:42:25.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:26 smithi079 ceph-mon[23158]: pgmap v4508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:26 smithi016 conmon[32727]: 2022-03-21T19:42:26.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:26 smithi016 ceph-mon[22010]: pgmap v4508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:27 smithi016 conmon[32727]: 2022-03-21T19:42:27.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:28 smithi016 ceph-mon[22010]: pgmap v4509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:28 smithi016 conmon[32727]: 2022-03-21T19:42:28.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:28 smithi079 ceph-mon[23158]: pgmap v4509: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:29 smithi016 conmon[32727]: 2022-03-21T19:42:29.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:30 smithi016 conmon[32727]: 2022-03-21T19:42:30.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:30 smithi016 ceph-mon[22010]: pgmap v4510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:30.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:30 smithi079 ceph-mon[23158]: pgmap v4510: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:31.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:31.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:42:31.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:42:31.909 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:42:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:31 smithi016 conmon[32727]: 2022-03-21T19:42:31.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:42:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:42:31.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:42:32.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:32 smithi079 ceph-mon[23158]: pgmap v4511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:32.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:32 smithi016 ceph-mon[22010]: pgmap v4511: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:32 smithi016 conmon[32727]: 2022-03-21T19:42:32.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:33 smithi016 conmon[32727]: 2022-03-21T19:42:33.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[32727]: 2022-03-21T19:42:34.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:34.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:34 smithi016 ceph-mon[22010]: pgmap v4512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:34.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:34.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:34 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:42:34.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: 2022-03-21T19:42:34.573+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:34.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:42:34.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:34.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: yield 2022-03-21T19:42:34.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:34.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:34.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:34.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:42:34.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:34.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:42:34.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:34.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:42:34.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:34.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:42:34.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:34.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:42:34.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:34.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:34.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:34 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:34 smithi079 ceph-mon[23158]: pgmap v4512: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:34 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:42:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:35 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T19:42:35.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:35.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:35.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:35 smithi016 conmon[32727]: 2022-03-21T19:42:35.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:35 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T19:42:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:36 smithi016 conmon[32727]: 2022-03-21T19:42:36.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:42:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:42:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:36.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:42:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:36.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:42:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:42:36.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:36.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:42:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:36.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:36.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:42:36.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:36 smithi016 ceph-mon[22010]: pgmap v4513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:36.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:42:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:42:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:42:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:42:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:42:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:42:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:42:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:36 smithi079 ceph-mon[23158]: pgmap v4513: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:37 smithi016 ceph-mon[22010]: pgmap v4514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:37.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[32727]: 2022-03-21T19:42:37.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:37.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 2022-03-21T19:42:37.645+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:37.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:42:37.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:37.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: yield 2022-03-21T19:42:37.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:37.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:37.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:37.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:42:37.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:37.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:42:37.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:37.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:42:37.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:37.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:42:37.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:37.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:42:37.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:37.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:37.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:37.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 2022-03-21T19:42:37.646+0000 7f9277eaa700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:42:37.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:42:37.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:37.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: yield 2022-03-21T19:42:37.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:37.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:37.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:37.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:42:37.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:37.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:42:37.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:37.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:42:37.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:37.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:42:37.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:37.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:42:37.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:37.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:37.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:37.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 2022-03-21T19:42:37.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:42:37.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: 2022-03-21T19:42:37.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:42:37.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:42:37.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:42:37.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:42:37.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:42:37.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:42:37.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:42:37.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:42:37.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:42:37.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:42:37.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:42:37.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:42:37.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:42:37.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:42:37.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:42:37.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:42:37.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:42:37.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:42:37.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:42:37.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:42:37.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:42:37.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:42:37.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise 2022-03-21T19:42:37.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:42:37.942 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:42:37.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:42:37.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:42:37.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:42:37 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:37 smithi079 ceph-mon[23158]: pgmap v4514: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:38 smithi016 conmon[32727]: 2022-03-21T19:42:38.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:42:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:38.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:42:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:38.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:42:38.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:38.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:42:38.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:38.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:38.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:38.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:42:38.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:38.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:38.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.930 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:38 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:42:38.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:38.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:42:38.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:38.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:38.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:38.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:42:38.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:38.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:42:38.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:38.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:42:38.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:38.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:38.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:38.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:42:38.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:38.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:38.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:42:38.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:42:38.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:42:38.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:42:38.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:42:38.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:42:38.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:42:38.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:42:38.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:42:38.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:42:38.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:42:38.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:42:38.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:42:38.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:42:38.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:42:38.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:42:38.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:42:38.950 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:42:38.950 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:42:38.951 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:42:38.951 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:42:38.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:42:38.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:42:38.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:42:38.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:42:38.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:42:38.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:42:38.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:42:38.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:42:38.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:38.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:42:38.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:38.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:38.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:38.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:42:38.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:38.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:42:38.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:38.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:42:38.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:38.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:38.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:38.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:42:38.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:38.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:38.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:38 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:42:38.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:42:38.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:42:38.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:42:38.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:42:38.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:42:38.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:42:38.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:42:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:42:38.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:42:38.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:42:38.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:42:38.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:42:38.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:42:38.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:42:38.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:42:38.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:42:38.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:38.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:42:38.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:42:38.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:42:38.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:42:38.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:42:38.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:42:38.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:42:38.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:42:38.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:42:38.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:42:38.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:42:38.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:42:38.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:42:39.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:42:39.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:42:39.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:42:39.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:42:39.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:42:39.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:42:39.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:42:39.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:42:39.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:42:39.003 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:42:39.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:42:39.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:42:39.004 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:42:39.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:42:39.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:42:39.005 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:42:39.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:42:39.006 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:42:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:39 smithi016 ceph-mon[22010]: pgmap v4515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:39 smithi016 conmon[32727]: 2022-03-21T19:42:39.694+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:39 smithi079 ceph-mon[23158]: pgmap v4515: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:40 smithi016 conmon[32727]: 2022-03-21T19:42:40.736+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8400 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:41.774 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:41 smithi016 ceph-mon[22010]: pgmap v4516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:41.775 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:41 smithi079 ceph-mon[23158]: pgmap v4516: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:41 smithi016 conmon[32727]: 2022-03-21T19:42:41.772+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:42 smithi016 conmon[32727]: 2022-03-21T19:42:42.752+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:43.783 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:43 smithi016 ceph-mon[22010]: pgmap v4517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:43.783 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:43 smithi079 ceph-mon[23158]: pgmap v4517: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:43 smithi016 conmon[32727]: 2022-03-21T19:42:43.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:44.790 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:44 smithi016 conmon[32727]: 2022-03-21T19:42:44.789+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:45.794 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:45 smithi016 ceph-mon[22010]: pgmap v4518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:45.794 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:45.795 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:45 smithi079 ceph-mon[23158]: pgmap v4518: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:45 smithi016 conmon[32727]: 2022-03-21T19:42:45.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:46 smithi016 conmon[32727]: 2022-03-21T19:42:46.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:47 smithi016 ceph-mon[22010]: pgmap v4519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:47.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:47.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:47 smithi016 conmon[32727]: 2022-03-21T19:42:47.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:47 smithi079 ceph-mon[23158]: pgmap v4519: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:48 smithi016 conmon[32727]: 2022-03-21T19:42:48.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:49 smithi016 ceph-mon[22010]: pgmap v4520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:49 smithi016 conmon[32727]: 2022-03-21T19:42:49.659+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:49 smithi079 ceph-mon[23158]: pgmap v4520: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:50 smithi016 conmon[32727]: 2022-03-21T19:42:50.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:50.960 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:50.960 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:51 smithi016 conmon[32727]: 2022-03-21T19:42:51.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:51 smithi016 ceph-mon[22010]: pgmap v4521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:51 smithi079 ceph-mon[23158]: pgmap v4521: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:52 smithi016 conmon[32727]: 2022-03-21T19:42:52.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:53 smithi016 conmon[32727]: 2022-03-21T19:42:53.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:53 smithi016 ceph-mon[22010]: pgmap v4522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:53 smithi079 ceph-mon[23158]: pgmap v4522: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:54 smithi016 conmon[32727]: 2022-03-21T19:42:54.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:55 smithi016 conmon[32727]: 2022-03-21T19:42:55.605+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:55.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:55 smithi016 ceph-mon[22010]: pgmap v4523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:55.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:55 smithi079 ceph-mon[23158]: pgmap v4523: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:42:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:42:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:56 smithi016 conmon[32727]: 2022-03-21T19:42:56.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:57 smithi016 conmon[32727]: 2022-03-21T19:42:57.605+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:57 smithi016 ceph-mon[22010]: pgmap v4524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:57 smithi079 ceph-mon[23158]: pgmap v4524: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:42:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:42:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:42:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:42:58.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:58 smithi016 conmon[32727]: 2022-03-21T19:42:58.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:42:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:42:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:42:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:42:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:59 smithi016 ceph-mon[22010]: pgmap v4525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:42:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:42:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:42:59 smithi016 conmon[32727]: 2022-03-21T19:42:59.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:42:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:59 smithi079 ceph-mon[23158]: pgmap v4525: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:42:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:42:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:00 smithi016 conmon[32727]: 2022-03-21T19:43:00.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8420 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:01 smithi016 ceph-mon[22010]: pgmap v4526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:01.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:01 smithi016 conmon[32727]: 2022-03-21T19:43:01.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:01 smithi079 ceph-mon[23158]: pgmap v4526: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:02 smithi016 conmon[32727]: 2022-03-21T19:43:02.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:03 smithi016 ceph-mon[22010]: pgmap v4527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:03 smithi016 conmon[32727]: 2022-03-21T19:43:03.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:03 smithi079 ceph-mon[23158]: pgmap v4527: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:04 smithi016 conmon[32727]: 2022-03-21T19:43:04.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:05 smithi016 ceph-mon[22010]: pgmap v4528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:05 smithi016 conmon[32727]: 2022-03-21T19:43:05.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:05 smithi079 ceph-mon[23158]: pgmap v4528: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:06 smithi016 conmon[32727]: 2022-03-21T19:43:06.708+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:07 smithi016 ceph-mon[22010]: pgmap v4529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:07 smithi016 conmon[32727]: 2022-03-21T19:43:07.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:07 smithi079 ceph-mon[23158]: pgmap v4529: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:08 smithi016 conmon[32727]: 2022-03-21T19:43:08.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:09 smithi016 ceph-mon[22010]: pgmap v4530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:09 smithi016 conmon[32727]: 2022-03-21T19:43:09.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:09 smithi079 ceph-mon[23158]: pgmap v4530: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:10 smithi016 conmon[32727]: 2022-03-21T19:43:10.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:11 smithi016 ceph-mon[22010]: pgmap v4531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:11 smithi016 conmon[32727]: 2022-03-21T19:43:11.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:11 smithi079 ceph-mon[23158]: pgmap v4531: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:12 smithi016 conmon[32727]: 2022-03-21T19:43:12.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:13 smithi016 ceph-mon[22010]: pgmap v4532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:13 smithi016 conmon[32727]: 2022-03-21T19:43:13.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:13 smithi079 ceph-mon[23158]: pgmap v4532: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:14 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:14 smithi016 conmon[32727]: 2022-03-21T19:43:14.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:14 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:15 smithi016 ceph-mon[22010]: pgmap v4533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:15 smithi016 conmon[32727]: 2022-03-21T19:43:15.787+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:15 smithi079 ceph-mon[23158]: pgmap v4533: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:16 smithi016 conmon[32727]: 2022-03-21T19:43:16.784+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:17 smithi016 ceph-mon[22010]: pgmap v4534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:17 smithi016 conmon[32727]: 2022-03-21T19:43:17.789+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:17 smithi079 ceph-mon[23158]: pgmap v4534: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:18 smithi016 conmon[32727]: 2022-03-21T19:43:18.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:19 smithi016 ceph-mon[22010]: pgmap v4535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:19 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:20.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:19 smithi016 conmon[32727]: 2022-03-21T19:43:19.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:19 smithi079 ceph-mon[23158]: pgmap v4535: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:19 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8440 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:21.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:21.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:20 smithi016 conmon[32727]: 2022-03-21T19:43:20.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:21 smithi016 ceph-mon[22010]: pgmap v4536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:21 smithi016 conmon[32727]: 2022-03-21T19:43:21.849+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:21 smithi079 ceph-mon[23158]: pgmap v4536: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:23.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:22 smithi016 conmon[32727]: 2022-03-21T19:43:22.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:23 smithi016 ceph-mon[22010]: pgmap v4537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:23 smithi016 conmon[32727]: 2022-03-21T19:43:23.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:23 smithi079 ceph-mon[23158]: pgmap v4537: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:24 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8444 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:24 smithi016 conmon[32727]: 2022-03-21T19:43:24.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:24 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8444 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:25 smithi016 ceph-mon[22010]: pgmap v4538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:25 smithi016 conmon[32727]: 2022-03-21T19:43:25.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:25 smithi079 ceph-mon[23158]: pgmap v4538: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:26 smithi016 conmon[32727]: 2022-03-21T19:43:26.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:27 smithi016 ceph-mon[22010]: pgmap v4539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:27 smithi016 conmon[32727]: 2022-03-21T19:43:27.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:27 smithi079 ceph-mon[23158]: pgmap v4539: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:28 smithi016 conmon[32727]: 2022-03-21T19:43:28.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:29 smithi079 ceph-mon[23158]: pgmap v4540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:30.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:29 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:29 smithi016 conmon[32727]: 2022-03-21T19:43:29.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:29 smithi016 ceph-mon[22010]: pgmap v4540: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:29 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:30 smithi016 conmon[32727]: 2022-03-21T19:43:30.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:31 smithi016 ceph-mon[22010]: pgmap v4541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:31 smithi016 conmon[32727]: 2022-03-21T19:43:31.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:31 smithi079 ceph-mon[23158]: pgmap v4541: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:32 smithi016 conmon[32727]: 2022-03-21T19:43:32.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:33 smithi016 ceph-mon[22010]: pgmap v4542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:33 smithi016 conmon[32727]: 2022-03-21T19:43:33.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:33 smithi079 ceph-mon[23158]: pgmap v4542: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:35.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:34 smithi016 conmon[32727]: 2022-03-21T19:43:34.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:35 smithi016 ceph-mon[22010]: pgmap v4543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:35 smithi016 conmon[32727]: 2022-03-21T19:43:35.810+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:35 smithi079 ceph-mon[23158]: pgmap v4543: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:37.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:36 smithi016 conmon[32727]: 2022-03-21T19:43:36.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:37.947 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:37 smithi016 conmon[32727]: 2022-03-21T19:43:37.831+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:37 smithi079 ceph-mon[23158]: pgmap v4544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:43:37.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:43:37.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:43:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:37 smithi016 ceph-mon[22010]: pgmap v4544: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:43:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:43:38.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:43:38.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: 2022-03-21T19:43:38.125+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:38.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:43:38.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:38.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: yield 2022-03-21T19:43:38.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:38.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:38.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:38.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:43:38.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:38.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:43:38.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:38.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:43:38.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:38.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:43:38.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:43:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:38.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:38.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:38 smithi016 conmon[32727]: 2022-03-21T19:43:38.812+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:43:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:39.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:43:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:43:39.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:43:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:39.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:43:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:39.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:43:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:39.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:39.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:38 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:43:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:43:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:43:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:39.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:43:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:39.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:43:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:43:39.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:39.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:43:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:39.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:39.230 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:38 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:43:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:39 smithi016 conmon[32727]: 2022-03-21T19:43:39.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:39 smithi016 ceph-mon[22010]: pgmap v4545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8459 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:39 smithi079 ceph-mon[23158]: pgmap v4545: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8459 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:40 smithi016 conmon[32727]: 2022-03-21T19:43:40.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:41.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 2022-03-21T19:43:41.197+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:41.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:43:41.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:41.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: yield 2022-03-21T19:43:41.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:41.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:43:41.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:41.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:43:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:43:41.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:43:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:41.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:43:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:41.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:41.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 2022-03-21T19:43:41.198+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:43:41.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:43:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: yield 2022-03-21T19:43:41.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:41.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:41.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:41.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:43:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:43:41.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:41.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:43:41.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:41.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:43:41.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:41.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:43:41.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:41.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:41.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 2022-03-21T19:43:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:43:41.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: 2022-03-21T19:43:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:43:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:43:41.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:43:41.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:43:41.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:43:41.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:43:41.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:43:41.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:43:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:43:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:43:41.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:43:41.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:43:41.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:43:41.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:43:41.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:43:41.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:43:41.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:43:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:43:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:43:41.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:43:41.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:43:41.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:43:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise 2022-03-21T19:43:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:43:41.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:43:41.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:43:41.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:43:41.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:41 smithi016 conmon[32727]: 2022-03-21T19:43:41.912+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:41 smithi016 ceph-mon[22010]: pgmap v4546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:42.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:41 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:42.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:43:42.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:42.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:42.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:42.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:43:42.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:42.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:43:42.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:42.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:43:42.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:42.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:42.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:42.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:43:42.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:42.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:42.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.174 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:41 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:43:42.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:42.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:43:42.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:42.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:42.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:42.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:43:42.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:42.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:43:42.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:42.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:43:42.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:42.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:42.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:42.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:43:42.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:42.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:42.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:43:42.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:43:42.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:43:42.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:43:42.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:43:42.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:43:42.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:43:42.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:43:42.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:43:42.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:43:42.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:43:42.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:43:42.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:43:42.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:43:42.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:43:42.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:43:42.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:43:42.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:43:42.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:43:42.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:43:42.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:43:42.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:43:42.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:43:42.190 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:43:42.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:43:42.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:43:42.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:43:42.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:43:42.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:43:42.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:41 smithi079 ceph-mon[23158]: pgmap v4546: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:41 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:43:42.225 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:42.226 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:43:42.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:42.227 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:43:42.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:42.228 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:43:42.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:42.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:42.229 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:42.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:43:42.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:42.230 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:42.231 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.231 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:41 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:43:42.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:43:42.232 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:43:42.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:43:42.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:43:42.233 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:43:42.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:43:42.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:43:42.234 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:43:42.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:43:42.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:43:42.235 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:43:42.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:43:42.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:43:42.236 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:43:42.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:43:42.237 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:43:42.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:42.238 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:43:42.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:43:42.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:43:42.239 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:43:42.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:43:42.240 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:43:42.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:43:42.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:43:42.241 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:43:42.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:43:42.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:43:42.242 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:43:42.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:43:42.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:43:42.243 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:43:42.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:43:42.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:43:42.244 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:43:42.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:43:42.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:43:42.245 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:43:42.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:43:42.246 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:43:42.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:43:42.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:43:42.247 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:43:42.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:43:42.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:43:42.248 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:43:42.249 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:43:42.250 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:43:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:42 smithi016 conmon[32727]: 2022-03-21T19:43:42.940+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:43 smithi079 ceph-mon[23158]: pgmap v4547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:43 smithi016 ceph-mon[22010]: pgmap v4547: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:44.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:43 smithi016 conmon[32727]: 2022-03-21T19:43:43.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8464 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8464 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:45.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:44 smithi016 conmon[32727]: 2022-03-21T19:43:44.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:45 smithi079 ceph-mon[23158]: pgmap v4548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:45 smithi016 ceph-mon[22010]: pgmap v4548: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:45 smithi016 conmon[32727]: 2022-03-21T19:43:45.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:47 smithi016 conmon[32727]: 2022-03-21T19:43:47.003+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:47 smithi016 ceph-mon[22010]: pgmap v4549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:48 smithi016 conmon[32727]: 2022-03-21T19:43:48.001+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:47 smithi079 ceph-mon[23158]: pgmap v4549: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:48 smithi016 conmon[32727]: 2022-03-21T19:43:48.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:49 smithi016 ceph-mon[22010]: pgmap v4550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:49 smithi016 conmon[32727]: 2022-03-21T19:43:49.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:49 smithi079 ceph-mon[23158]: pgmap v4550: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:51.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:50 smithi016 conmon[32727]: 2022-03-21T19:43:50.891+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:51 smithi016 ceph-mon[22010]: pgmap v4551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:52.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:52.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:51 smithi016 conmon[32727]: 2022-03-21T19:43:51.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:51 smithi079 ceph-mon[23158]: pgmap v4551: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:52.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:53.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:52 smithi016 conmon[32727]: 2022-03-21T19:43:52.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:53 smithi016 ceph-mon[22010]: pgmap v4552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:53 smithi016 conmon[32727]: 2022-03-21T19:43:53.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:53 smithi079 ceph-mon[23158]: pgmap v4552: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:54.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:55.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:55.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:55.423 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:54 smithi016 conmon[32727]: 2022-03-21T19:43:54.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:55.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:43:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:56 smithi016 ceph-mon[22010]: pgmap v4553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:55 smithi016 conmon[32727]: 2022-03-21T19:43:55.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:56 smithi079 ceph-mon[23158]: pgmap v4553: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:43:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:56 smithi016 conmon[32727]: 2022-03-21T19:43:56.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:58 smithi016 ceph-mon[22010]: pgmap v4554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:57 smithi016 conmon[32727]: 2022-03-21T19:43:57.925+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:58 smithi079 ceph-mon[23158]: pgmap v4554: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:43:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:58 smithi016 conmon[32727]: 2022-03-21T19:43:58.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:43:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:43:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:43:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:43:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:43:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:43:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:43:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:43:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:43:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:43:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:43:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:44:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:43:59 smithi016 conmon[32727]: 2022-03-21T19:43:59.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:00 smithi016 ceph-mon[22010]: pgmap v4555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:00.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:00 smithi079 ceph-mon[23158]: pgmap v4555: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8480 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:01.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:00 smithi016 conmon[32727]: 2022-03-21T19:44:00.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:01 smithi016 conmon[32727]: 2022-03-21T19:44:01.875+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:02 smithi016 ceph-mon[22010]: pgmap v4556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:02 smithi079 ceph-mon[23158]: pgmap v4556: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:03.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:02 smithi016 conmon[32727]: 2022-03-21T19:44:02.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:03 smithi016 conmon[32727]: 2022-03-21T19:44:03.819+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:04 smithi016 ceph-mon[22010]: pgmap v4557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:04 smithi079 ceph-mon[23158]: pgmap v4557: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:04 smithi016 conmon[32727]: 2022-03-21T19:44:04.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:05 smithi016 conmon[32727]: 2022-03-21T19:44:05.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:06 smithi016 ceph-mon[22010]: pgmap v4558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:06.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:06 smithi079 ceph-mon[23158]: pgmap v4558: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:06 smithi016 conmon[32727]: 2022-03-21T19:44:06.867+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:07 smithi016 conmon[32727]: 2022-03-21T19:44:07.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:08 smithi016 ceph-mon[22010]: pgmap v4559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:08.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:08 smithi079 ceph-mon[23158]: pgmap v4559: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:08 smithi016 conmon[32727]: 2022-03-21T19:44:08.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:09 smithi016 conmon[32727]: 2022-03-21T19:44:09.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:10 smithi016 ceph-mon[22010]: pgmap v4560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:10.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:10.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:10 smithi079 ceph-mon[23158]: pgmap v4560: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:11.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:10 smithi016 conmon[32727]: 2022-03-21T19:44:10.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:12 smithi016 ceph-mon[22010]: pgmap v4561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:11 smithi016 conmon[32727]: 2022-03-21T19:44:11.827+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:12 smithi079 ceph-mon[23158]: pgmap v4561: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:12 smithi016 conmon[32727]: 2022-03-21T19:44:12.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:14 smithi016 ceph-mon[22010]: pgmap v4562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:14.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:13 smithi016 conmon[32727]: 2022-03-21T19:44:13.830+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:14 smithi079 ceph-mon[23158]: pgmap v4562: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:15.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:14 smithi016 conmon[32727]: 2022-03-21T19:44:14.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:16 smithi016 ceph-mon[22010]: pgmap v4563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:15 smithi016 conmon[32727]: 2022-03-21T19:44:15.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:16 smithi079 ceph-mon[23158]: pgmap v4563: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:16 smithi016 conmon[32727]: 2022-03-21T19:44:16.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:17.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:18.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:18 smithi016 ceph-mon[22010]: pgmap v4564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:18.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:18.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:17 smithi016 conmon[32727]: 2022-03-21T19:44:17.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:18 smithi079 ceph-mon[23158]: pgmap v4564: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:19.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:19.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:18 smithi016 conmon[32727]: 2022-03-21T19:44:18.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:20 smithi016 ceph-mon[22010]: pgmap v4565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8499 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:20.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:19 smithi016 conmon[32727]: 2022-03-21T19:44:19.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:20 smithi079 ceph-mon[23158]: pgmap v4565: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8499 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:21.088 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:20 smithi016 conmon[32727]: 2022-03-21T19:44:20.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:22 smithi016 ceph-mon[22010]: pgmap v4566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:21 smithi016 conmon[32727]: 2022-03-21T19:44:21.843+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:22 smithi079 ceph-mon[23158]: pgmap v4566: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:23.094 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:22 smithi016 conmon[32727]: 2022-03-21T19:44:22.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:24.096 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:23 smithi016 conmon[32727]: 2022-03-21T19:44:23.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:24 smithi016 ceph-mon[22010]: pgmap v4567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:24 smithi079 ceph-mon[23158]: pgmap v4567: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:25.101 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:24 smithi016 conmon[32727]: 2022-03-21T19:44:24.737+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:26.102 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:25 smithi016 conmon[32727]: 2022-03-21T19:44:25.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:26 smithi016 ceph-mon[22010]: pgmap v4568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:26 smithi079 ceph-mon[23158]: pgmap v4568: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:26 smithi016 conmon[32727]: 2022-03-21T19:44:26.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:28.108 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:27 smithi016 conmon[32727]: 2022-03-21T19:44:27.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:28 smithi016 ceph-mon[22010]: pgmap v4569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:28 smithi079 ceph-mon[23158]: pgmap v4569: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:29.113 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:28 smithi016 conmon[32727]: 2022-03-21T19:44:28.665+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:29 smithi016 conmon[32727]: 2022-03-21T19:44:29.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:30 smithi016 ceph-mon[22010]: pgmap v4570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:30 smithi079 ceph-mon[23158]: pgmap v4570: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:30.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:30.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:30 smithi016 conmon[32727]: 2022-03-21T19:44:30.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:31 smithi016 conmon[32727]: 2022-03-21T19:44:31.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:32 smithi016 ceph-mon[22010]: pgmap v4571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:32.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:32 smithi079 ceph-mon[23158]: pgmap v4571: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:33.125 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:32 smithi016 conmon[32727]: 2022-03-21T19:44:32.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:33.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:34.127 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:33 smithi016 conmon[32727]: 2022-03-21T19:44:33.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:34 smithi016 ceph-mon[22010]: pgmap v4572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:34.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:34 smithi079 ceph-mon[23158]: pgmap v4572: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:35.131 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:34 smithi016 conmon[32727]: 2022-03-21T19:44:34.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:36.133 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:35 smithi016 conmon[32727]: 2022-03-21T19:44:35.663+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:36 smithi016 ceph-mon[22010]: pgmap v4573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:36 smithi079 ceph-mon[23158]: pgmap v4573: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:37.138 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:36 smithi016 conmon[32727]: 2022-03-21T19:44:36.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:38.139 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:37 smithi016 conmon[32727]: 2022-03-21T19:44:37.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:38.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:38 smithi016 ceph-mon[22010]: pgmap v4574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:38.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:38 smithi079 ceph-mon[23158]: pgmap v4574: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:39.145 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:38 smithi016 conmon[32727]: 2022-03-21T19:44:38.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:40.147 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:39 smithi016 conmon[32727]: 2022-03-21T19:44:39.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:40 smithi016 ceph-mon[22010]: pgmap v4575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:40.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:40 smithi079 ceph-mon[23158]: pgmap v4575: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8520 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:41.151 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:40 smithi016 conmon[32727]: 2022-03-21T19:44:40.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:41.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:42.153 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:41 smithi016 conmon[32727]: 2022-03-21T19:44:41.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:42 smithi016 ceph-mon[22010]: pgmap v4576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:44:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:44:42.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:42 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:44:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:42 smithi079 ceph-mon[23158]: pgmap v4576: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:44:42.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:44:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:42 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:44:43.157 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:42 smithi016 conmon[32727]: 2022-03-21T19:44:42.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:44.158 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:43 smithi016 conmon[32727]: 2022-03-21T19:44:43.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:44 smithi016 ceph-mon[22010]: pgmap v4577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:44 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:44:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:44 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:44:44.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: 2022-03-21T19:44:44.301+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:44.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:44:44.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:44.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: yield 2022-03-21T19:44:44.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:44.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:44.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:44.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:44:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:44:44.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:44:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:44.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:44:44.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:44.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:44:44.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:44.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:44.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:44 smithi079 ceph-mon[23158]: pgmap v4577: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:44 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:44:44.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:44 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:44:44.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:44 smithi016 conmon[32727]: 2022-03-21T19:44:44.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:44:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:44:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.420 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.421 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 183.2M 2022-03-21T19:44:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 192186622: error parsing value: Value '192186622' is below minimum 939524096 2022-03-21T19:44:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:45.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:44:45.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:45.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:44:45.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:45.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:45.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:45.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:44:45.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:45.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:44:45.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:45.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:44:45.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:45.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:45.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:45.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:44:45.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:45.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:45.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:45.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:44:45.429 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:44:45.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:44:45.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:44:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 183.2M 2022-03-21T19:44:45.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 192186622: error parsing value: Value '192186622' is below minimum 939524096 2022-03-21T19:44:45.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:45.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:44:45.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:45.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:44:45.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:45.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:45.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:45.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:44:45.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:45.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:44:45.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:45.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:44:45.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:45.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:45.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:45.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:44:45.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:45.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:45.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:45.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:44:45.488 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:45 smithi016 conmon[32727]: 2022-03-21T19:44:45.601+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:46 smithi016 ceph-mon[22010]: pgmap v4578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:46 smithi079 ceph-mon[23158]: pgmap v4578: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:46.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:46 smithi016 conmon[32727]: 2022-03-21T19:44:46.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:47.600 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:47.601 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 2022-03-21T19:44:47.373+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:47.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:44:47.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:47.602 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: yield 2022-03-21T19:44:47.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:47.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:47.603 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:47.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:44:47.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:47.604 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:44:47.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:47.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:44:47.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:47.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:44:47.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:47.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:44:47.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:47.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:47.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:47.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 2022-03-21T19:44:47.374+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:44:47.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:44:47.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:47.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: yield 2022-03-21T19:44:47.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:47.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:47.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:47.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:44:47.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:47.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:44:47.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:47.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:44:47.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:47.613 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:44:47.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:47.614 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:44:47.615 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:47.616 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:47.616 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:47.617 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 2022-03-21T19:44:47.617 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:44:47.618 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: 2022-03-21T19:44:47.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:44:47.619 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:44:47.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:44:47.620 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:44:47.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:44:47.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:44:47.622 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:44:47.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:44:47.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:44:47.623 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:44:47.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:44:47.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:44:47.624 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:44:47.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:44:47.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:44:47.625 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:44:47.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:44:47.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:44:47.626 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:44:47.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:44:47.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:44:47.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:44:47.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise 2022-03-21T19:44:47.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:44:47.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:44:47.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:44:47.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:44:47.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:47 smithi016 conmon[32727]: 2022-03-21T19:44:47.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:48 smithi079 ceph-mon[23158]: pgmap v4579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:44:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:48.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:44:48.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:48.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:44:48.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:48.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:44:48.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:48.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:48.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:48.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:44:48.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:48.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:48.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:48 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:44:48.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:48.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:44:48.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:48.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:48.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:48.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:44:48.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:48.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:44:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:44:48.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:48.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:48.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:48.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:44:48.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:48.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:48.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:44:48.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:44:48.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:44:48.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:44:48.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:44:48.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:44:48.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:44:48.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:44:48.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:44:48.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:44:48.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:44:48.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:44:48.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:44:48.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:44:48.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:44:48.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:44:48.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:44:48.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:44:48.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:44:48.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:44:48.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:44:48.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:44:48.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:44:48.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:44:48.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:44:48.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:44:48.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:44:48.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:44:48.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:44:48.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:48.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:48 smithi016 ceph-mon[22010]: pgmap v4579: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:48.578 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.578 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.578 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:48.579 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:44:48.579 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:48.579 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:48.580 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:48.580 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:44:48.580 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:48.581 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:44:48.581 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:48.581 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:44:48.582 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:48.582 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:48.582 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:48.583 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:44:48.583 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:48.583 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:48.584 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.584 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:48 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:44:48.585 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.585 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:44:48.585 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:44:48.586 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:44:48.586 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:44:48.586 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:44:48.587 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:44:48.587 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:44:48.587 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:44:48.588 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:44:48.588 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:44:48.588 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:44:48.589 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:44:48.589 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:44:48.589 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:44:48.590 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:44:48.590 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:44:48.591 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.591 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:44:48.591 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:44:48.592 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:44:48.592 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:44:48.592 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:44:48.593 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:44:48.593 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:44:48.593 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:44:48.594 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:44:48.594 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:44:48.594 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:44:48.595 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:44:48.595 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:44:48.595 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:44:48.596 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:44:48.596 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:44:48.596 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:44:48.597 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:44:48.597 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:44:48.597 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:44:48.598 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:44:48.598 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:44:48.599 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:44:48.600 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:44:48.602 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:44:48.602 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:44:48.603 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:44:48.603 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:44:48.603 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:44:48.604 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:44:48.604 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:44:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:48 smithi016 conmon[32727]: 2022-03-21T19:44:48.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:49.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:49 smithi016 conmon[32727]: 2022-03-21T19:44:49.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:50 smithi079 ceph-mon[23158]: pgmap v4580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:50.517 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:50.517 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:50 smithi016 ceph-mon[22010]: pgmap v4580: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:50.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:50 smithi016 conmon[32727]: 2022-03-21T19:44:50.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:51.488 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:51 smithi016 conmon[32727]: 2022-03-21T19:44:51.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:52 smithi079 ceph-mon[23158]: pgmap v4581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:52.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:52.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:52 smithi016 ceph-mon[22010]: pgmap v4581: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:52 smithi016 conmon[32727]: 2022-03-21T19:44:52.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:53.460 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:53 smithi016 conmon[32727]: 2022-03-21T19:44:53.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:54 smithi079 ceph-mon[23158]: pgmap v4582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:54 smithi016 conmon[32727]: 2022-03-21T19:44:54.424+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:54 smithi016 ceph-mon[22010]: pgmap v4582: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:44:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:55 smithi016 conmon[32727]: 2022-03-21T19:44:55.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:56 smithi079 ceph-mon[23158]: pgmap v4583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:56 smithi016 ceph-mon[22010]: pgmap v4583: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:44:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:56 smithi016 conmon[32727]: 2022-03-21T19:44:56.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:57.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:57.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:57 smithi016 conmon[32727]: 2022-03-21T19:44:57.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:58 smithi079 ceph-mon[23158]: pgmap v4584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:58 smithi016 ceph-mon[22010]: pgmap v4584: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:44:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:44:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:58 smithi016 conmon[32727]: 2022-03-21T19:44:58.409+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:44:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:44:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:44:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:44:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:44:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:44:59 smithi016 conmon[32727]: 2022-03-21T19:44:59.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:44:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:44:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:44:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:44:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:44:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:45:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:00 smithi079 ceph-mon[23158]: pgmap v4585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:00 smithi016 ceph-mon[22010]: pgmap v4585: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8540 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:00 smithi016 conmon[32727]: 2022-03-21T19:45:00.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:01 smithi016 conmon[32727]: 2022-03-21T19:45:01.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:02 smithi079 ceph-mon[23158]: pgmap v4586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:02 smithi016 ceph-mon[22010]: pgmap v4586: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:02 smithi016 conmon[32727]: 2022-03-21T19:45:02.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:03 smithi016 conmon[32727]: 2022-03-21T19:45:03.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:04 smithi079 ceph-mon[23158]: pgmap v4587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:04 smithi016 ceph-mon[22010]: pgmap v4587: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:04 smithi016 conmon[32727]: 2022-03-21T19:45:04.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:05 smithi016 conmon[32727]: 2022-03-21T19:45:05.468+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:06 smithi016 ceph-mon[22010]: pgmap v4588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:06 smithi016 conmon[32727]: 2022-03-21T19:45:06.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:06 smithi079 ceph-mon[23158]: pgmap v4588: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:07 smithi016 conmon[32727]: 2022-03-21T19:45:07.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:08 smithi016 conmon[32727]: 2022-03-21T19:45:08.453+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:08 smithi016 ceph-mon[22010]: pgmap v4589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:08 smithi079 ceph-mon[23158]: pgmap v4589: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:09 smithi016 conmon[32727]: 2022-03-21T19:45:09.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:10 smithi016 ceph-mon[22010]: pgmap v4590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:10.671 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:10 smithi016 conmon[32727]: 2022-03-21T19:45:10.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:10 smithi079 ceph-mon[23158]: pgmap v4590: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:11 smithi016 conmon[32727]: 2022-03-21T19:45:11.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:12 smithi016 conmon[32727]: 2022-03-21T19:45:12.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:12 smithi016 ceph-mon[22010]: pgmap v4591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:12 smithi079 ceph-mon[23158]: pgmap v4591: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:13 smithi016 conmon[32727]: 2022-03-21T19:45:13.471+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:14 smithi016 ceph-mon[22010]: pgmap v4592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:14 smithi016 conmon[32727]: 2022-03-21T19:45:14.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:14 smithi079 ceph-mon[23158]: pgmap v4592: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:15.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:15.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:15 smithi016 conmon[32727]: 2022-03-21T19:45:15.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:16 smithi016 ceph-mon[22010]: pgmap v4593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:16 smithi016 conmon[32727]: 2022-03-21T19:45:16.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:16 smithi079 ceph-mon[23158]: pgmap v4593: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:17.540 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:17 smithi016 conmon[32727]: 2022-03-21T19:45:17.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:18.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:18.571 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:18 smithi016 ceph-mon[22010]: pgmap v4594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:18 smithi079 ceph-mon[23158]: pgmap v4594: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:18 smithi016 conmon[32727]: 2022-03-21T19:45:18.568+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:19.560 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:19 smithi016 conmon[32727]: 2022-03-21T19:45:19.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:20.535 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:20.536 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:20 smithi016 ceph-mon[22010]: pgmap v4595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:20.536 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:20 smithi079 ceph-mon[23158]: pgmap v4595: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8560 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:20 smithi016 conmon[32727]: 2022-03-21T19:45:20.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:21.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:21 smithi016 conmon[32727]: 2022-03-21T19:45:21.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:22 smithi016 ceph-mon[22010]: pgmap v4596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:22 smithi016 conmon[32727]: 2022-03-21T19:45:22.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:22 smithi079 ceph-mon[23158]: pgmap v4596: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:23 smithi016 conmon[32727]: 2022-03-21T19:45:23.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:24.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:24 smithi016 ceph-mon[22010]: pgmap v4597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:24.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:24 smithi016 conmon[32727]: 2022-03-21T19:45:24.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:24.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:24 smithi079 ceph-mon[23158]: pgmap v4597: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:25.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:25 smithi016 conmon[32727]: 2022-03-21T19:45:25.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:26 smithi016 ceph-mon[22010]: pgmap v4598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:26.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:26 smithi016 conmon[32727]: 2022-03-21T19:45:26.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:26 smithi079 ceph-mon[23158]: pgmap v4598: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:27 smithi016 conmon[32727]: 2022-03-21T19:45:27.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:28 smithi016 ceph-mon[22010]: pgmap v4599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:28.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:28 smithi016 conmon[32727]: 2022-03-21T19:45:28.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:28 smithi079 ceph-mon[23158]: pgmap v4599: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:29 smithi016 conmon[32727]: 2022-03-21T19:45:29.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:30 smithi016 ceph-mon[22010]: pgmap v4600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:30.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:30 smithi016 conmon[32727]: 2022-03-21T19:45:30.488+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:30 smithi079 ceph-mon[23158]: pgmap v4600: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:31 smithi016 conmon[32727]: 2022-03-21T19:45:31.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:32 smithi016 ceph-mon[22010]: pgmap v4601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:32 smithi016 conmon[32727]: 2022-03-21T19:45:32.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:32 smithi079 ceph-mon[23158]: pgmap v4601: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:33.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:33 smithi016 conmon[32727]: 2022-03-21T19:45:33.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:34 smithi016 ceph-mon[22010]: pgmap v4602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:34 smithi016 conmon[32727]: 2022-03-21T19:45:34.471+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:34 smithi079 ceph-mon[23158]: pgmap v4602: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:35 smithi016 conmon[32727]: 2022-03-21T19:45:35.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:36 smithi016 ceph-mon[22010]: pgmap v4603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:36 smithi016 conmon[32727]: 2022-03-21T19:45:36.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:36 smithi079 ceph-mon[23158]: pgmap v4603: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:37 smithi016 conmon[32727]: 2022-03-21T19:45:37.455+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:38 smithi016 ceph-mon[22010]: pgmap v4604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:38 smithi016 conmon[32727]: 2022-03-21T19:45:38.409+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:38 smithi079 ceph-mon[23158]: pgmap v4604: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:39.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:39 smithi016 conmon[32727]: 2022-03-21T19:45:39.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:40 smithi016 ceph-mon[22010]: pgmap v4605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:40 smithi016 conmon[32727]: 2022-03-21T19:45:40.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:40 smithi079 ceph-mon[23158]: pgmap v4605: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:40.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8580 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:41 smithi016 conmon[32727]: 2022-03-21T19:45:41.400+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:42 smithi016 conmon[32727]: 2022-03-21T19:45:42.430+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:42 smithi016 ceph-mon[22010]: pgmap v4606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:42 smithi079 ceph-mon[23158]: pgmap v4606: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:43 smithi016 conmon[32727]: 2022-03-21T19:45:43.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:44 smithi016 ceph-mon[22010]: pgmap v4607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:44 smithi016 conmon[32727]: 2022-03-21T19:45:44.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:44 smithi079 ceph-mon[23158]: pgmap v4607: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:45 smithi016 conmon[32727]: 2022-03-21T19:45:45.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:46 smithi016 conmon[32727]: 2022-03-21T19:45:46.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:46 smithi016 ceph-mon[22010]: pgmap v4608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:46 smithi079 ceph-mon[23158]: pgmap v4608: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:47 smithi016 conmon[32727]: 2022-03-21T19:45:47.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: pgmap v4609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:45:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:45:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:45:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:45:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:48 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:45:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:48 smithi016 conmon[32727]: 2022-03-21T19:45:48.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: pgmap v4609: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:45:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:45:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:45:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:45:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:48 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:45:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:49 smithi016 conmon[32727]: 2022-03-21T19:45:49.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[32727]: 2022-03-21T19:45:50.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:50 smithi016 ceph-mon[22010]: pgmap v4610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:50.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: 2022-03-21T19:45:50.477+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:50.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:45:50.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:50.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: yield 2022-03-21T19:45:50.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:50.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:50.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:50.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:45:50.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:50.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:45:50.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:50.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:45:50.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:50.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:45:50.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:50.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:45:50.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:50.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:50.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:50 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:50 smithi079 ceph-mon[23158]: pgmap v4610: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:51 smithi016 conmon[32727]: 2022-03-21T19:45:51.407+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:45:51.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:45:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:51.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:51.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:51.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:45:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:45:51.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:45:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:51.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:45:51.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:51.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:51.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:51.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:51 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:45:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:45:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:45:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:45:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:45:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:45:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:45:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:51 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:45:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:52 smithi016 ceph-mon[22010]: pgmap v4611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:52 smithi016 conmon[32727]: 2022-03-21T19:45:52.424+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:52 smithi079 ceph-mon[23158]: pgmap v4611: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:53.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[32727]: 2022-03-21T19:45:53.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:53.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 2022-03-21T19:45:53.549+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:53.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:45:53.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:53.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: yield 2022-03-21T19:45:53.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:53.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:53.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:53.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:45:53.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:53.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:45:53.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:53.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:45:53.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:53.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:45:53.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:53.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:45:53.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:53.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:53.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:53.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 2022-03-21T19:45:53.550+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:45:53.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:45:53.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:53.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: yield 2022-03-21T19:45:53.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:53.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:53.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:53.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:45:53.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:53.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:45:53.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:53.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:45:53.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:53.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:45:53.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:53.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:45:53.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:53.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:53.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:53.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 2022-03-21T19:45:53.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:45:53.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: 2022-03-21T19:45:53.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:45:53.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:45:53.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:45:53.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:45:53.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:45:53.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:45:53.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:45:53.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:45:53.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:45:53.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:45:53.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:45:53.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:45:53.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:45:53.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:45:53.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:45:53.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:45:53.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:45:53.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:45:53.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:45:53.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:45:53.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:45:53.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:45:53.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise 2022-03-21T19:45:53.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:45:53.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:45:53.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:45:53.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:45:53.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:45:53 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:54 smithi016 conmon[32727]: 2022-03-21T19:45:54.456+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:54 smithi016 ceph-mon[22010]: pgmap v4612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:54 smithi079 ceph-mon[23158]: pgmap v4612: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:55.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:55 smithi016 conmon[32727]: 2022-03-21T19:45:55.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:55 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:55.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:45:55.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:55.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:55.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:55.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:45:55.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:55.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:45:55.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:55.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:45:55.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:55.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:55.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:55.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:45:55.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:55.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:55.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:55 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:45:55.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:55.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:45:55.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:55.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:55.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:55.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:45:55.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:55.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:45:55.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:55.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:45:55.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:55.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:55.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:55.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:45:55.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:55.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:55.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:45:55.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:45:55.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:45:55.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:45:55.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:45:55.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:45:55.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:45:55.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:45:55.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:45:55.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:45:55.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:45:55.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:45:55.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:45:55.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:45:55.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:45:55.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:45:55.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:45:55.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:45:55.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:45:55.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:45:55.694 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:45:55.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:45:55.695 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:45:55.696 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:45:55.696 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:45:55.697 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:45:55.697 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:45:55.698 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:45:55.699 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:45:55.699 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.700 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:55 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:45:55.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:55.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:55.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:45:55.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:55.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:45:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:45:55.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:55.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:45:55.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:55.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:55.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:55 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:45:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:45:55.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:45:55.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:45:55.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:45:55.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:45:55.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:45:55.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:45:55.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:45:55.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:45:55.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:45:55.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:45:55.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:45:55.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:45:55.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:45:55.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:45:55.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:45:55.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:45:55.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:45:55.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:45:55.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:45:55.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:45:55.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:45:55.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:45:55.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:45:55.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:45:55.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:45:55.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:45:55.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:45:55.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:45:55.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:45:55.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:45:55.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:45:55.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:45:55.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:45:55.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:45:55.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:45:55.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:45:55.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:45:55.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:45:55.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:45:55.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:45:55.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:45:55.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:45:55.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:45:55.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:45:55.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:45:55.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:45:55.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:45:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:56 smithi016 ceph-mon[22010]: pgmap v4613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:56 smithi016 conmon[32727]: 2022-03-21T19:45:56.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:56 smithi079 ceph-mon[23158]: pgmap v4613: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:45:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:57 smithi016 conmon[32727]: 2022-03-21T19:45:57.513+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:58 smithi016 ceph-mon[22010]: pgmap v4614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:58 smithi016 conmon[32727]: 2022-03-21T19:45:58.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:58 smithi079 ceph-mon[23158]: pgmap v4614: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:45:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:45:59 smithi016 conmon[32727]: 2022-03-21T19:45:59.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:45:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:45:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:45:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:45:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:45:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:45:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:45:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:45:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:45:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:45:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:45:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:46:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:00 smithi016 ceph-mon[22010]: pgmap v4615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:00 smithi016 conmon[32727]: 2022-03-21T19:46:00.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:00 smithi079 ceph-mon[23158]: pgmap v4615: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8600 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:01 smithi016 conmon[32727]: 2022-03-21T19:46:01.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:02 smithi016 ceph-mon[22010]: pgmap v4616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:02 smithi016 conmon[32727]: 2022-03-21T19:46:02.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:02 smithi079 ceph-mon[23158]: pgmap v4616: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:03 smithi016 conmon[32727]: 2022-03-21T19:46:03.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:04 smithi079 ceph-mon[23158]: pgmap v4617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:04 smithi016 ceph-mon[22010]: pgmap v4617: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:04 smithi016 conmon[32727]: 2022-03-21T19:46:04.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:05.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:05.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:05 smithi016 conmon[32727]: 2022-03-21T19:46:05.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:06 smithi079 ceph-mon[23158]: pgmap v4618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:06 smithi016 conmon[32727]: 2022-03-21T19:46:06.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:06 smithi016 ceph-mon[22010]: pgmap v4618: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:07 smithi016 conmon[32727]: 2022-03-21T19:46:07.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:08 smithi079 ceph-mon[23158]: pgmap v4619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:08 smithi016 ceph-mon[22010]: pgmap v4619: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:08 smithi016 conmon[32727]: 2022-03-21T19:46:08.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:09 smithi016 conmon[32727]: 2022-03-21T19:46:09.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:10 smithi079 ceph-mon[23158]: pgmap v4620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:10.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:10 smithi016 conmon[32727]: 2022-03-21T19:46:10.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:10 smithi016 ceph-mon[22010]: pgmap v4620: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:11 smithi016 conmon[32727]: 2022-03-21T19:46:11.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:12 smithi079 ceph-mon[23158]: pgmap v4621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:12 smithi016 ceph-mon[22010]: pgmap v4621: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:12 smithi016 conmon[32727]: 2022-03-21T19:46:12.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:13 smithi016 conmon[32727]: 2022-03-21T19:46:13.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:14 smithi079 ceph-mon[23158]: pgmap v4622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:14 smithi016 ceph-mon[22010]: pgmap v4622: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:14 smithi016 conmon[32727]: 2022-03-21T19:46:14.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:15 smithi016 conmon[32727]: 2022-03-21T19:46:15.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:16 smithi079 ceph-mon[23158]: pgmap v4623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:16 smithi016 ceph-mon[22010]: pgmap v4623: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:16 smithi016 conmon[32727]: 2022-03-21T19:46:16.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:17 smithi016 conmon[32727]: 2022-03-21T19:46:17.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:18 smithi079 ceph-mon[23158]: pgmap v4624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:18 smithi016 ceph-mon[22010]: pgmap v4624: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:18 smithi016 conmon[32727]: 2022-03-21T19:46:18.521+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:19 smithi016 conmon[32727]: 2022-03-21T19:46:19.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:20 smithi079 ceph-mon[23158]: pgmap v4625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:20 smithi016 conmon[32727]: 2022-03-21T19:46:20.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:20 smithi016 ceph-mon[22010]: pgmap v4625: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8620 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:21 smithi016 conmon[32727]: 2022-03-21T19:46:21.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:22 smithi016 ceph-mon[22010]: pgmap v4626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:22 smithi016 conmon[32727]: 2022-03-21T19:46:22.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:22 smithi079 ceph-mon[23158]: pgmap v4626: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:23 smithi016 conmon[32727]: 2022-03-21T19:46:23.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:24 smithi016 ceph-mon[22010]: pgmap v4627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:24 smithi016 conmon[32727]: 2022-03-21T19:46:24.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:24 smithi079 ceph-mon[23158]: pgmap v4627: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:25.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:25 smithi016 conmon[32727]: 2022-03-21T19:46:25.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:26 smithi016 ceph-mon[22010]: pgmap v4628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:26.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:26 smithi016 conmon[32727]: 2022-03-21T19:46:26.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:26 smithi079 ceph-mon[23158]: pgmap v4628: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:27 smithi016 conmon[32727]: 2022-03-21T19:46:27.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:28 smithi016 ceph-mon[22010]: pgmap v4629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:28 smithi016 conmon[32727]: 2022-03-21T19:46:28.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:28 smithi079 ceph-mon[23158]: pgmap v4629: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:29 smithi016 conmon[32727]: 2022-03-21T19:46:29.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:30 smithi016 conmon[32727]: 2022-03-21T19:46:30.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:30 smithi016 ceph-mon[22010]: pgmap v4630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:30 smithi079 ceph-mon[23158]: pgmap v4630: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:31 smithi016 conmon[32727]: 2022-03-21T19:46:31.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:32 smithi016 conmon[32727]: 2022-03-21T19:46:32.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:32 smithi016 ceph-mon[22010]: pgmap v4631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:32 smithi079 ceph-mon[23158]: pgmap v4631: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:33 smithi016 conmon[32727]: 2022-03-21T19:46:33.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:34 smithi016 ceph-mon[22010]: pgmap v4632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:34 smithi016 conmon[32727]: 2022-03-21T19:46:34.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:34 smithi079 ceph-mon[23158]: pgmap v4632: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:35 smithi016 conmon[32727]: 2022-03-21T19:46:35.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:36 smithi016 ceph-mon[22010]: pgmap v4633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:36.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:36.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:36 smithi016 conmon[32727]: 2022-03-21T19:46:36.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:36 smithi079 ceph-mon[23158]: pgmap v4633: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:37 smithi016 conmon[32727]: 2022-03-21T19:46:37.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:38 smithi016 ceph-mon[22010]: pgmap v4634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:38.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:38 smithi016 conmon[32727]: 2022-03-21T19:46:38.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:38 smithi079 ceph-mon[23158]: pgmap v4634: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:39 smithi016 conmon[32727]: 2022-03-21T19:46:39.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:40 smithi016 ceph-mon[22010]: pgmap v4635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:40.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:40 smithi016 conmon[32727]: 2022-03-21T19:46:40.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:40 smithi079 ceph-mon[23158]: pgmap v4635: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8640 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:41 smithi016 conmon[32727]: 2022-03-21T19:46:41.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:42 smithi016 ceph-mon[22010]: pgmap v4636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:42 smithi016 conmon[32727]: 2022-03-21T19:46:42.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:42 smithi079 ceph-mon[23158]: pgmap v4636: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:43 smithi016 conmon[32727]: 2022-03-21T19:46:43.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:44 smithi016 ceph-mon[22010]: pgmap v4637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:44 smithi016 conmon[32727]: 2022-03-21T19:46:44.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:44 smithi079 ceph-mon[23158]: pgmap v4637: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:45 smithi016 ceph-mon[22010]: pgmap v4638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:45.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:45.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:45 smithi016 conmon[32727]: 2022-03-21T19:46:45.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:45 smithi079 ceph-mon[23158]: pgmap v4638: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:46 smithi016 conmon[32727]: 2022-03-21T19:46:46.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:47.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:47 smithi016 ceph-mon[22010]: pgmap v4639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:47 smithi016 conmon[32727]: 2022-03-21T19:46:47.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:47 smithi079 ceph-mon[23158]: pgmap v4639: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:48 smithi016 conmon[32727]: 2022-03-21T19:46:48.670+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:49 smithi016 ceph-mon[22010]: pgmap v4640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:49.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:49 smithi016 conmon[32727]: 2022-03-21T19:46:49.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:49 smithi079 ceph-mon[23158]: pgmap v4640: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:50 smithi016 conmon[32727]: 2022-03-21T19:46:50.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:51 smithi016 ceph-mon[22010]: pgmap v4641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:51 smithi016 conmon[32727]: 2022-03-21T19:46:51.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:51.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:51 smithi079 ceph-mon[23158]: pgmap v4641: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:52 smithi016 conmon[32727]: 2022-03-21T19:46:52.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:53 smithi016 conmon[32727]: 2022-03-21T19:46:53.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:53 smithi016 ceph-mon[22010]: pgmap v4642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:46:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:46:53.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:53 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:46:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:53 smithi079 ceph-mon[23158]: pgmap v4642: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:46:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:46:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:53 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:46:54.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: 2022-03-21T19:46:54.093+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:46:54.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:54.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: yield 2022-03-21T19:46:54.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:54.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:54.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:54.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:46:54.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:54.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:46:54.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:46:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:54.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:46:54.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:54.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:46:54.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:54.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:54.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:54 smithi016 conmon[32727]: 2022-03-21T19:46:54.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:46:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:46:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:46:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:46:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:46:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:46:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:46:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:46:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:46:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:46:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:46:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:46:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:46:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:46:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:55 smithi016 ceph-mon[22010]: pgmap v4643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:55.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:55 smithi016 conmon[32727]: 2022-03-21T19:46:55.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:55 smithi079 ceph-mon[23158]: pgmap v4643: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:46:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:46:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:56 smithi016 conmon[32727]: 2022-03-21T19:46:56.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:57.474 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 2022-03-21T19:46:57.165+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.475 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:46:57.475 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.476 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: yield 2022-03-21T19:46:57.476 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.476 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.477 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.477 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:46:57.478 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.478 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:46:57.478 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.479 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:46:57.479 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.479 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.480 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.480 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:46:57.480 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.481 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.481 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.481 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 2022-03-21T19:46:57.166+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:46:57.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:46:57.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: yield 2022-03-21T19:46:57.483 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.483 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:46:57.484 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.485 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:46:57.485 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.485 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:46:57.486 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:46:57.488 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.488 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 2022-03-21T19:46:57.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:46:57.490 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: 2022-03-21T19:46:57.490 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:46:57.490 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:46:57.491 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:46:57.491 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:46:57.491 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:46:57.492 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:46:57.492 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:46:57.493 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:46:57.493 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:46:57.493 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:46:57.494 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:46:57.494 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:46:57.494 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:46:57.495 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:46:57.495 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:46:57.495 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:46:57.496 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:46:57.496 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:46:57.497 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:46:57.497 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:46:57.497 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:46:57.498 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:46:57.498 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise 2022-03-21T19:46:57.498 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:46:57.499 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:46:57.499 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:46:57.499 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:46:57.500 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:57 smithi016 conmon[32727]: 2022-03-21T19:46:57.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:57 smithi016 ceph-mon[22010]: pgmap v4644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:46:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:46:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:46:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:46:57.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:46:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:46:57.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:46:57.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:46:57.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:46:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:46:57.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:46:57.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:46:57.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:46:57.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:46:57.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:46:57.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:46:57.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:46:57.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:46:57.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:46:57.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:46:57.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:46:57.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:46:57.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:46:57.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:46:57.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:46:57.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:46:57.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:46:57.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:46:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:46:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:46:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:46:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:46:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:46:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:46:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:46:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:46:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:46:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:46:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:46:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:46:57.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:57 smithi079 ceph-mon[23158]: pgmap v4644: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:46:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:46:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:46:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:46:57.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:46:57.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:46:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:46:57.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:46:57.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:46:57.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:46:57.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:46:57.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:46:57.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:46:57.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:46:57.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:46:57.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:46:57.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:46:57.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:46:57.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:46:57.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:46:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:46:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:46:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:46:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:46:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:46:57.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:46:57.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:46:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:46:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:46:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:46:57.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:46:57.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:46:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:46:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:46:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:46:57.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:46:57.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:46:57.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:46:57.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:46:57.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:46:57.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:46:57.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:46:57.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:46:57.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:46:57.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:46:57.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:46:57.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:46:57.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:46:57.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:46:57.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:46:57.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:46:57.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:46:57.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:46:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:58 smithi016 conmon[32727]: 2022-03-21T19:46:58.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:46:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:46:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:46:58.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:46:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:46:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:46:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:46:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:46:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:46:59 smithi016 ceph-mon[22010]: pgmap v4645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:46:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:46:59 smithi016 conmon[32727]: 2022-03-21T19:46:59.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:46:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:46:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:46:59 smithi079 ceph-mon[23158]: pgmap v4645: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:00 smithi016 conmon[32727]: 2022-03-21T19:47:00.513+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8660 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:01 smithi016 ceph-mon[22010]: pgmap v4646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:01 smithi016 conmon[32727]: 2022-03-21T19:47:01.497+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:01 smithi079 ceph-mon[23158]: pgmap v4646: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:02 smithi016 conmon[32727]: 2022-03-21T19:47:02.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:03 smithi016 conmon[32727]: 2022-03-21T19:47:03.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:03.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:03 smithi016 ceph-mon[22010]: pgmap v4647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:03 smithi079 ceph-mon[23158]: pgmap v4647: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:04 smithi016 conmon[32727]: 2022-03-21T19:47:04.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:05 smithi016 ceph-mon[22010]: pgmap v4648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:05.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:05 smithi016 conmon[32727]: 2022-03-21T19:47:05.609+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:05 smithi079 ceph-mon[23158]: pgmap v4648: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:06.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:06 smithi016 conmon[32727]: 2022-03-21T19:47:06.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:07 smithi016 ceph-mon[22010]: pgmap v4649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:07 smithi016 conmon[32727]: 2022-03-21T19:47:07.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:07 smithi079 ceph-mon[23158]: pgmap v4649: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:08 smithi016 conmon[32727]: 2022-03-21T19:47:08.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:09 smithi016 ceph-mon[22010]: pgmap v4650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:09 smithi016 conmon[32727]: 2022-03-21T19:47:09.561+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:09 smithi079 ceph-mon[23158]: pgmap v4650: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:10 smithi016 conmon[32727]: 2022-03-21T19:47:10.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8670 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:11 smithi016 ceph-mon[22010]: pgmap v4651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:11 smithi016 conmon[32727]: 2022-03-21T19:47:11.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:11 smithi079 ceph-mon[23158]: pgmap v4651: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:12 smithi016 conmon[32727]: 2022-03-21T19:47:12.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:13.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:13 smithi016 ceph-mon[22010]: pgmap v4652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:13 smithi016 conmon[32727]: 2022-03-21T19:47:13.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:13 smithi079 ceph-mon[23158]: pgmap v4652: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:14 smithi016 conmon[32727]: 2022-03-21T19:47:14.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:15 smithi016 ceph-mon[22010]: pgmap v4653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:15.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:15 smithi016 conmon[32727]: 2022-03-21T19:47:15.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:15 smithi079 ceph-mon[23158]: pgmap v4653: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:16 smithi016 conmon[32727]: 2022-03-21T19:47:16.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:17 smithi016 ceph-mon[22010]: pgmap v4654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:17 smithi016 conmon[32727]: 2022-03-21T19:47:17.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:17.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:17 smithi079 ceph-mon[23158]: pgmap v4654: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:18 smithi016 conmon[32727]: 2022-03-21T19:47:18.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:19 smithi016 ceph-mon[22010]: pgmap v4655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:19.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:19 smithi016 conmon[32727]: 2022-03-21T19:47:19.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:19 smithi079 ceph-mon[23158]: pgmap v4655: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:20 smithi016 conmon[32727]: 2022-03-21T19:47:20.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8680 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:21 smithi016 ceph-mon[22010]: pgmap v4656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:21.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:21 smithi016 conmon[32727]: 2022-03-21T19:47:21.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:21 smithi079 ceph-mon[23158]: pgmap v4656: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:22 smithi016 conmon[32727]: 2022-03-21T19:47:22.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:23.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:23 smithi079 ceph-mon[23158]: pgmap v4657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:23 smithi016 ceph-mon[22010]: pgmap v4657: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:23 smithi016 conmon[32727]: 2022-03-21T19:47:23.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:24 smithi016 conmon[32727]: 2022-03-21T19:47:24.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:25 smithi079 ceph-mon[23158]: pgmap v4658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:25 smithi016 ceph-mon[22010]: pgmap v4658: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8685 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:26.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:25 smithi016 conmon[32727]: 2022-03-21T19:47:25.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:26 smithi016 conmon[32727]: 2022-03-21T19:47:26.733+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:27 smithi079 ceph-mon[23158]: pgmap v4659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:28.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:27 smithi016 conmon[32727]: 2022-03-21T19:47:27.689+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:27 smithi016 ceph-mon[22010]: pgmap v4659: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:28 smithi016 conmon[32727]: 2022-03-21T19:47:28.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:29 smithi079 ceph-mon[23158]: pgmap v4660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:29.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:29 smithi016 conmon[32727]: 2022-03-21T19:47:29.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:30.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:29 smithi016 ceph-mon[22010]: pgmap v4660: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:30.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:30 smithi016 conmon[32727]: 2022-03-21T19:47:30.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:31 smithi016 ceph-mon[22010]: pgmap v4661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:32.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:31 smithi016 conmon[32727]: 2022-03-21T19:47:31.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:31 smithi079 ceph-mon[23158]: pgmap v4661: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:32.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:32 smithi016 conmon[32727]: 2022-03-21T19:47:32.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:33.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:33 smithi016 conmon[32727]: 2022-03-21T19:47:33.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:34.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:34.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:33 smithi079 ceph-mon[23158]: pgmap v4662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:33 smithi016 ceph-mon[22010]: pgmap v4662: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:34 smithi016 conmon[32727]: 2022-03-21T19:47:34.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:35.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:35.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:34 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:34 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:35 smithi016 conmon[32727]: 2022-03-21T19:47:35.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:36.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:35 smithi079 ceph-mon[23158]: pgmap v4663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:36.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:35 smithi016 ceph-mon[22010]: pgmap v4663: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:36 smithi016 conmon[32727]: 2022-03-21T19:47:36.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:37.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:37.419 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:37 smithi016 conmon[32727]: 2022-03-21T19:47:37.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:38.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:38.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:37 smithi079 ceph-mon[23158]: pgmap v4664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:37 smithi016 ceph-mon[22010]: pgmap v4664: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:38 smithi016 conmon[32727]: 2022-03-21T19:47:38.605+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:39 smithi016 conmon[32727]: 2022-03-21T19:47:39.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:40.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:39 smithi079 ceph-mon[23158]: pgmap v4665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:40.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:39 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:39 smithi016 ceph-mon[22010]: pgmap v4665: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:39 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8700 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:40 smithi016 conmon[32727]: 2022-03-21T19:47:40.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:41.955 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:41 smithi016 conmon[32727]: 2022-03-21T19:47:41.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:41 smithi079 ceph-mon[23158]: pgmap v4666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:41 smithi016 ceph-mon[22010]: pgmap v4666: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:42.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:42 smithi016 conmon[32727]: 2022-03-21T19:47:42.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:43 smithi016 conmon[32727]: 2022-03-21T19:47:43.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:43 smithi079 ceph-mon[23158]: pgmap v4667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:43 smithi016 ceph-mon[22010]: pgmap v4667: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:44.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:44 smithi016 conmon[32727]: 2022-03-21T19:47:44.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:44 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:44 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:45 smithi016 conmon[32727]: 2022-03-21T19:47:45.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:45 smithi079 ceph-mon[23158]: pgmap v4668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:45 smithi016 ceph-mon[22010]: pgmap v4668: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:46.974 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:46 smithi016 conmon[32727]: 2022-03-21T19:47:46.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:47 smithi016 conmon[32727]: 2022-03-21T19:47:47.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:47 smithi016 ceph-mon[22010]: pgmap v4669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:47 smithi079 ceph-mon[23158]: pgmap v4669: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:48.981 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:48 smithi016 conmon[32727]: 2022-03-21T19:47:48.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:49.982 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:49 smithi016 conmon[32727]: 2022-03-21T19:47:49.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:49 smithi016 ceph-mon[22010]: pgmap v4670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:49 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:49 smithi079 ceph-mon[23158]: pgmap v4670: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:49 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:50 smithi016 conmon[32727]: 2022-03-21T19:47:50.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:51 smithi016 conmon[32727]: 2022-03-21T19:47:51.598+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:51 smithi016 ceph-mon[22010]: pgmap v4671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:51 smithi079 ceph-mon[23158]: pgmap v4671: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:52 smithi016 conmon[32727]: 2022-03-21T19:47:52.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:53 smithi016 conmon[32727]: 2022-03-21T19:47:53.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:54.475 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:53 smithi016 ceph-mon[22010]: pgmap v4672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:54.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:54.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:53 smithi079 ceph-mon[23158]: pgmap v4672: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:54 smithi016 conmon[32727]: 2022-03-21T19:47:54.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:54 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:54 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:47:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:55 smithi016 conmon[32727]: 2022-03-21T19:47:55.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:56 smithi016 ceph-mon[22010]: pgmap v4673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:56 smithi079 ceph-mon[23158]: pgmap v4673: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:47:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:56 smithi016 conmon[32727]: 2022-03-21T19:47:56.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:57 smithi016 conmon[32727]: 2022-03-21T19:47:57.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: pgmap v4674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:47:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:47:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:47:58.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:47:58.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: pgmap v4674: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:47:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:47:58.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:47:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:47:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:47:58.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:47:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:58 smithi016 conmon[32727]: 2022-03-21T19:47:58.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:47:59.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:47:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:47:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:47:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:47:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:47:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:47:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:47:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:47:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:47:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:47:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:47:59 smithi016 conmon[32727]: 2022-03-21T19:47:59.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:00.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:00.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:00 smithi016 ceph-mon[22010]: pgmap v4675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:00.272 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:00 smithi079 ceph-mon[23158]: pgmap v4675: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8720 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:00.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: 2022-03-21T19:48:00.269+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:00.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:48:00.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:00.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: yield 2022-03-21T19:48:00.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:00.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:00.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:00.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:48:00.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:00.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:48:00.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:00.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:48:00.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:00.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:48:00.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:00.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:48:00.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:00.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:00.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:01.019 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:00 smithi016 conmon[32727]: 2022-03-21T19:48:00.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:01.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:01 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:48:01.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:48:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:01.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:48:01.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:48:01.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:48:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:01.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:01.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:48:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:01.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:01.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:01 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:48:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:01 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:48:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:01.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:48:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:01.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:48:01.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:01.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:48:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:01.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:48:01.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:01.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:48:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:01.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:01.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:01.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:01 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:48:02.021 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:01 smithi016 conmon[32727]: 2022-03-21T19:48:01.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:02 smithi016 ceph-mon[22010]: pgmap v4676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:02 smithi079 ceph-mon[23158]: pgmap v4676: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:03.025 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:02 smithi016 conmon[32727]: 2022-03-21T19:48:02.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:03.343 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:03.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 2022-03-21T19:48:03.341+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:03.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:48:03.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:03.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: yield 2022-03-21T19:48:03.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:03.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:03.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:03.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:48:03.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:03.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:48:03.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:03.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:48:03.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:03.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:48:03.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:03.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:48:03.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:03.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:03.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:03.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 2022-03-21T19:48:03.342+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:48:03.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:48:03.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:03.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: yield 2022-03-21T19:48:03.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:03.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:03.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:03.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:48:03.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:03.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:48:03.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:03.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:48:03.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:03.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:48:03.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:03.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:48:03.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:03.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:03.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:03.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 2022-03-21T19:48:03.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:48:03.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: 2022-03-21T19:48:03.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:48:03.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:48:03.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:48:03.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:48:03.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:48:03.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:48:03.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:48:03.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:48:03.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:48:03.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:48:03.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:48:03.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:48:03.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:48:03.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:48:03.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:48:03.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:48:03.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:48:03.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:48:03.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:48:03.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:48:03.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:48:03.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:48:03.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise 2022-03-21T19:48:03.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:48:03.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:48:03.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:48:03.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:48:03.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.027 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:03 smithi016 conmon[32727]: 2022-03-21T19:48:03.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:04 smithi016 ceph-mon[22010]: pgmap v4677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:48:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:04.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:48:04.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:48:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:04.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:48:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:04.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:48:04.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:04 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:48:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:48:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:04.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:04.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:48:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:04.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:48:04.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:04.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:48:04.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:04.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:04.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:04.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:48:04.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:04.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:04.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:48:04.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:48:04.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:48:04.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:48:04.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:48:04.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:48:04.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:48:04.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:48:04.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:48:04.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:48:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:48:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:48:04.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:48:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:48:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:48:04.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:48:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:48:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:48:04.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:48:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:48:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:48:04.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:48:04.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:48:04.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:48:04.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:48:04.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:48:04.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:48:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:48:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:48:04.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:04 smithi079 ceph-mon[23158]: pgmap v4677: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:48:04.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:04.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:04.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:04.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:48:04.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:04.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:48:04.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:04.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:48:04.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:04.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:04.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:04.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:48:04.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:04.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:04.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:04 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:48:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:48:04.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:48:04.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:48:04.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:48:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:48:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:48:04.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:48:04.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:48:04.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:48:04.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:48:04.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:48:04.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:48:04.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:48:04.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:48:04.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:48:04.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:48:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:48:04.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:48:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:48:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:48:04.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:48:04.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:48:04.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:48:04.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:48:04.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:48:04.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:48:04.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:48:04.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:48:04.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:48:04.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:48:04.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:48:04.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:48:04.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:48:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:48:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:48:04.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:48:04.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:48:04.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:48:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:48:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:48:04.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:48:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:48:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:48:04.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:48:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:48:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:48:04.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:48:05.032 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:04 smithi016 conmon[32727]: 2022-03-21T19:48:04.671+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:06.034 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:05 smithi016 conmon[32727]: 2022-03-21T19:48:05.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:06 smithi016 ceph-mon[22010]: pgmap v4678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:06 smithi079 ceph-mon[23158]: pgmap v4678: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:07.039 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:06 smithi016 conmon[32727]: 2022-03-21T19:48:06.682+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:07.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:07.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:08.040 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:07 smithi016 conmon[32727]: 2022-03-21T19:48:07.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:08 smithi016 ceph-mon[22010]: pgmap v4679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:08 smithi079 ceph-mon[23158]: pgmap v4679: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:09.046 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:08 smithi016 conmon[32727]: 2022-03-21T19:48:08.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:10.047 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:09 smithi016 conmon[32727]: 2022-03-21T19:48:09.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:10 smithi016 ceph-mon[22010]: pgmap v4680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:10 smithi079 ceph-mon[23158]: pgmap v4680: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:11.053 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:10 smithi016 conmon[32727]: 2022-03-21T19:48:10.751+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:11.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:12.055 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:11 smithi016 conmon[32727]: 2022-03-21T19:48:11.773+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:12 smithi016 ceph-mon[22010]: pgmap v4681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:12 smithi079 ceph-mon[23158]: pgmap v4681: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:13.060 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:12 smithi016 conmon[32727]: 2022-03-21T19:48:12.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:14.061 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:13 smithi016 conmon[32727]: 2022-03-21T19:48:13.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:14 smithi016 ceph-mon[22010]: pgmap v4682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:14 smithi079 ceph-mon[23158]: pgmap v4682: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:14.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:15.067 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:14 smithi016 conmon[32727]: 2022-03-21T19:48:14.806+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:15.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:15.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:15 smithi016 conmon[32727]: 2022-03-21T19:48:15.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:16 smithi016 ceph-mon[22010]: pgmap v4683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:16.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:16 smithi079 ceph-mon[23158]: pgmap v4683: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:16 smithi016 conmon[32727]: 2022-03-21T19:48:16.827+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:18.074 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:17 smithi016 conmon[32727]: 2022-03-21T19:48:17.809+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:18 smithi016 ceph-mon[22010]: pgmap v4684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:18.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:18 smithi079 ceph-mon[23158]: pgmap v4684: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:19.079 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:18 smithi016 conmon[32727]: 2022-03-21T19:48:18.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:19.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:19 smithi016 conmon[32727]: 2022-03-21T19:48:19.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:20.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:20 smithi016 ceph-mon[22010]: pgmap v4685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:20.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:20 smithi079 ceph-mon[23158]: pgmap v4685: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8740 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:21.086 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:20 smithi016 conmon[32727]: 2022-03-21T19:48:20.793+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:22.087 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:21 smithi016 conmon[32727]: 2022-03-21T19:48:21.827+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:22 smithi016 ceph-mon[22010]: pgmap v4686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:22 smithi079 ceph-mon[23158]: pgmap v4686: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:23.091 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:22 smithi016 conmon[32727]: 2022-03-21T19:48:22.821+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:24 smithi016 ceph-mon[22010]: pgmap v4687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:24.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:23 smithi016 conmon[32727]: 2022-03-21T19:48:23.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:24 smithi079 ceph-mon[23158]: pgmap v4687: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:25.237 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:25.238 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:24 smithi016 conmon[32727]: 2022-03-21T19:48:24.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8744 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:26 smithi016 ceph-mon[22010]: pgmap v4688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:26.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:25 smithi016 conmon[32727]: 2022-03-21T19:48:25.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:26 smithi079 ceph-mon[23158]: pgmap v4688: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:26 smithi016 conmon[32727]: 2022-03-21T19:48:26.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:28 smithi016 ceph-mon[22010]: pgmap v4689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:27 smithi016 conmon[32727]: 2022-03-21T19:48:27.931+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:28 smithi079 ceph-mon[23158]: pgmap v4689: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:29.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:28 smithi016 conmon[32727]: 2022-03-21T19:48:28.920+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:29 smithi016 conmon[32727]: 2022-03-21T19:48:29.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:30 smithi016 ceph-mon[22010]: pgmap v4690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:30.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:30.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8749 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:30 smithi079 ceph-mon[23158]: pgmap v4690: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8749 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:30 smithi016 conmon[32727]: 2022-03-21T19:48:30.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:32 smithi016 ceph-mon[22010]: pgmap v4691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:32.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:31 smithi016 conmon[32727]: 2022-03-21T19:48:31.944+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:32 smithi079 ceph-mon[23158]: pgmap v4691: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:33.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:32 smithi016 conmon[32727]: 2022-03-21T19:48:32.916+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:34 smithi016 ceph-mon[22010]: pgmap v4692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:34.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:33 smithi016 conmon[32727]: 2022-03-21T19:48:33.892+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:34 smithi079 ceph-mon[23158]: pgmap v4692: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:35.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:34 smithi016 conmon[32727]: 2022-03-21T19:48:34.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:35.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:35.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:36 smithi016 ceph-mon[22010]: pgmap v4693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:35 smithi016 conmon[32727]: 2022-03-21T19:48:35.880+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:36 smithi079 ceph-mon[23158]: pgmap v4693: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:37.134 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:36 smithi016 conmon[32727]: 2022-03-21T19:48:36.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:38.136 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:37 smithi016 conmon[32727]: 2022-03-21T19:48:37.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:38 smithi016 ceph-mon[22010]: pgmap v4694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:38 smithi079 ceph-mon[23158]: pgmap v4694: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:39.141 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:38 smithi016 conmon[32727]: 2022-03-21T19:48:38.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:40.142 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:39 smithi016 conmon[32727]: 2022-03-21T19:48:39.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:40 smithi016 ceph-mon[22010]: pgmap v4695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:40 smithi079 ceph-mon[23158]: pgmap v4695: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8760 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:41.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:40 smithi016 conmon[32727]: 2022-03-21T19:48:40.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:41 smithi016 conmon[32727]: 2022-03-21T19:48:41.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:42 smithi016 ceph-mon[22010]: pgmap v4696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:42 smithi079 ceph-mon[23158]: pgmap v4696: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:43.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:42 smithi016 conmon[32727]: 2022-03-21T19:48:42.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:44.155 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:43 smithi016 conmon[32727]: 2022-03-21T19:48:43.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:44 smithi016 ceph-mon[22010]: pgmap v4697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:44 smithi079 ceph-mon[23158]: pgmap v4697: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:48:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:45.162 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:44 smithi016 conmon[32727]: 2022-03-21T19:48:44.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:45 smithi016 conmon[32727]: 2022-03-21T19:48:45.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:46 smithi016 ceph-mon[22010]: pgmap v4698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:46 smithi079 ceph-mon[23158]: pgmap v4698: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:46 smithi016 conmon[32727]: 2022-03-21T19:48:46.829+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:47 smithi016 conmon[32727]: 2022-03-21T19:48:47.817+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:48 smithi079 ceph-mon[23158]: pgmap v4699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:48 smithi016 ceph-mon[22010]: pgmap v4699: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:48 smithi016 conmon[32727]: 2022-03-21T19:48:48.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:50.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:49 smithi016 conmon[32727]: 2022-03-21T19:48:49.857+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:50 smithi079 ceph-mon[23158]: pgmap v4700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:50 smithi016 ceph-mon[22010]: pgmap v4700: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:50 smithi016 conmon[32727]: 2022-03-21T19:48:50.893+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:51 smithi016 conmon[32727]: 2022-03-21T19:48:51.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:52 smithi079 ceph-mon[23158]: pgmap v4701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:52 smithi016 ceph-mon[22010]: pgmap v4701: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:52 smithi016 conmon[32727]: 2022-03-21T19:48:52.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:53 smithi016 conmon[32727]: 2022-03-21T19:48:53.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:54 smithi079 ceph-mon[23158]: pgmap v4702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:54 smithi016 ceph-mon[22010]: pgmap v4702: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:54 smithi016 conmon[32727]: 2022-03-21T19:48:54.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8774 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:55.733 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8774 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:48:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:55 smithi016 conmon[32727]: 2022-03-21T19:48:55.901+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:56 smithi079 ceph-mon[23158]: pgmap v4703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:56 smithi016 ceph-mon[22010]: pgmap v4703: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:48:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:56 smithi016 conmon[32727]: 2022-03-21T19:48:56.869+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:57 smithi016 conmon[32727]: 2022-03-21T19:48:57.831+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:58 smithi079 ceph-mon[23158]: pgmap v4704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:58 smithi016 ceph-mon[22010]: pgmap v4704: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:48:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:58 smithi016 conmon[32727]: 2022-03-21T19:48:58.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:48:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:48:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:48:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:48:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:48:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:48:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:48:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:48:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:48:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:48:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:48:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:49:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:48:59 smithi016 conmon[32727]: 2022-03-21T19:48:59.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:00 smithi079 ceph-mon[23158]: pgmap v4705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:00 smithi016 ceph-mon[22010]: pgmap v4705: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8780 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:00 smithi016 conmon[32727]: 2022-03-21T19:49:00.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:01 smithi016 conmon[32727]: 2022-03-21T19:49:01.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:02 smithi079 ceph-mon[23158]: pgmap v4706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:02 smithi016 ceph-mon[22010]: pgmap v4706: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:02 smithi016 conmon[32727]: 2022-03-21T19:49:02.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[32727]: 2022-03-21T19:49:03.814+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:04.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: 2022-03-21T19:49:03.821+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:04.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:49:04.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:04.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: yield 2022-03-21T19:49:04.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:04.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:04.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:49:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:04.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:49:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:49:04.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:49:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:04.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:49:04.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:04.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:04.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:03 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:04 smithi079 ceph-mon[23158]: pgmap v4707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:49:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:49:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:04 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:49:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:04 smithi016 ceph-mon[22010]: pgmap v4707: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:49:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:49:04.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:04 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:49:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:04 smithi016 conmon[32727]: 2022-03-21T19:49:04.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:49:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:49:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:05.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:05.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:05.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:49:05.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:05.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:49:05.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:05.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:49:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:05.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:05.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:49:05.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:05.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:05.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:05.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:05 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:49:05.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8784 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:49:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:49:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:49:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:49:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:49:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:49:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:05 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:49:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8784 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:05 smithi016 conmon[32727]: 2022-03-21T19:49:05.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:06 smithi016 ceph-mon[22010]: pgmap v4708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:06 smithi079 ceph-mon[23158]: pgmap v4708: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:07.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[32727]: 2022-03-21T19:49:06.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:07.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 2022-03-21T19:49:06.893+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:07.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:49:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: yield 2022-03-21T19:49:07.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:07.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:07.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:49:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:07.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:49:07.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:07.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:49:07.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:49:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:07.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:49:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:07.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 2022-03-21T19:49:06.894+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:49:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:49:07.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: yield 2022-03-21T19:49:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:07.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:07.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:07.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:49:07.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:49:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:07.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:49:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:49:07.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:07.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:49:07.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:07.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:07.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:07.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 2022-03-21T19:49:07.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:49:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: 2022-03-21T19:49:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:49:07.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:49:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:49:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:49:07.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:49:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:49:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:49:07.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:49:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:49:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:49:07.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:49:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:49:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:49:07.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:49:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:49:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:49:07.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:49:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:49:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:49:07.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:49:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:49:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:49:07.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise 2022-03-21T19:49:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:49:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:49:07.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:49:07.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:49:07.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:49:06 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:07 smithi016 conmon[32727]: 2022-03-21T19:49:07.890+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:08 smithi016 ceph-mon[22010]: pgmap v4709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:49:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:49:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:49:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:49:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:49:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:08 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:49:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:08.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:49:08.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:08.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:49:08.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:49:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:08.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:49:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:08.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:49:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:08.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:49:08.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:49:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:49:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:49:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:49:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:49:08.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:49:08.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:49:08.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:49:08.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:49:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:49:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:49:08.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:49:08.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:49:08.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:49:08.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:49:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:49:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:49:08.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:49:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:49:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:49:08.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:49:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:49:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:49:08.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:49:08.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:49:08.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:49:08.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:49:08.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:49:08.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:08 smithi079 ceph-mon[23158]: pgmap v4709: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:49:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:08.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:49:08.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:49:08.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:49:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:08.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:49:08.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:08 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:49:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:49:08.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:49:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:49:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:49:08.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:49:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:49:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:49:08.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:49:08.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:49:08.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:49:08.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:49:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:49:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:49:08.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:49:08.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:49:08.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:49:08.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:08.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:49:08.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:49:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:49:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:49:08.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:49:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:49:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:49:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:49:08.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:49:08.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:49:08.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:49:08.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:49:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:49:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:49:08.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:49:08.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:49:08.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:49:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:49:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:49:08.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:49:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:49:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:49:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:49:08.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:49:08.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:49:08.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:49:08.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:49:08.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:49:08.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:49:08.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:49:08.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:49:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:08 smithi016 conmon[32727]: 2022-03-21T19:49:08.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:09 smithi016 conmon[32727]: 2022-03-21T19:49:09.884+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:10 smithi016 ceph-mon[22010]: pgmap v4710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:10 smithi079 ceph-mon[23158]: pgmap v4710: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:10 smithi016 conmon[32727]: 2022-03-21T19:49:10.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:12.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:11 smithi016 conmon[32727]: 2022-03-21T19:49:11.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:12 smithi016 ceph-mon[22010]: pgmap v4711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:12 smithi079 ceph-mon[23158]: pgmap v4711: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:13.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:12 smithi016 conmon[32727]: 2022-03-21T19:49:12.866+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:13 smithi016 conmon[32727]: 2022-03-21T19:49:13.845+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:14 smithi016 ceph-mon[22010]: pgmap v4712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:14 smithi079 ceph-mon[23158]: pgmap v4712: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:14 smithi016 conmon[32727]: 2022-03-21T19:49:14.835+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:15 smithi016 conmon[32727]: 2022-03-21T19:49:15.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:16 smithi016 ceph-mon[22010]: pgmap v4713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:16 smithi079 ceph-mon[23158]: pgmap v4713: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:16 smithi016 conmon[32727]: 2022-03-21T19:49:16.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:17 smithi016 conmon[32727]: 2022-03-21T19:49:17.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:18 smithi016 ceph-mon[22010]: pgmap v4714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:18 smithi079 ceph-mon[23158]: pgmap v4714: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:18 smithi016 conmon[32727]: 2022-03-21T19:49:18.900+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:19 smithi016 conmon[32727]: 2022-03-21T19:49:19.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:20 smithi016 ceph-mon[22010]: pgmap v4715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:20 smithi079 ceph-mon[23158]: pgmap v4715: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8800 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:20 smithi016 conmon[32727]: 2022-03-21T19:49:20.892+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:21 smithi016 conmon[32727]: 2022-03-21T19:49:21.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:22 smithi016 ceph-mon[22010]: pgmap v4716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:22 smithi079 ceph-mon[23158]: pgmap v4716: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:23.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:22 smithi016 conmon[32727]: 2022-03-21T19:49:22.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:23 smithi016 conmon[32727]: 2022-03-21T19:49:23.836+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:24 smithi016 ceph-mon[22010]: pgmap v4717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:24 smithi079 ceph-mon[23158]: pgmap v4717: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:24 smithi016 conmon[32727]: 2022-03-21T19:49:24.831+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:25 smithi016 conmon[32727]: 2022-03-21T19:49:25.856+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:26 smithi016 ceph-mon[22010]: pgmap v4718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:26 smithi079 ceph-mon[23158]: pgmap v4718: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:26 smithi016 conmon[32727]: 2022-03-21T19:49:26.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:28.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:27 smithi016 conmon[32727]: 2022-03-21T19:49:27.812+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:28 smithi016 ceph-mon[22010]: pgmap v4719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:28 smithi079 ceph-mon[23158]: pgmap v4719: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:29.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:28 smithi016 conmon[32727]: 2022-03-21T19:49:28.793+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:29 smithi016 conmon[32727]: 2022-03-21T19:49:29.784+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:30 smithi016 ceph-mon[22010]: pgmap v4720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:30 smithi079 ceph-mon[23158]: pgmap v4720: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:30 smithi016 conmon[32727]: 2022-03-21T19:49:30.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:31 smithi016 conmon[32727]: 2022-03-21T19:49:31.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:32 smithi016 ceph-mon[22010]: pgmap v4721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:32 smithi079 ceph-mon[23158]: pgmap v4721: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:33.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:32 smithi016 conmon[32727]: 2022-03-21T19:49:32.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:34.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:33 smithi016 conmon[32727]: 2022-03-21T19:49:33.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:34 smithi016 ceph-mon[22010]: pgmap v4722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:34 smithi079 ceph-mon[23158]: pgmap v4722: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:35.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:34 smithi016 conmon[32727]: 2022-03-21T19:49:34.725+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:36.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:35 smithi016 conmon[32727]: 2022-03-21T19:49:35.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:36 smithi016 ceph-mon[22010]: pgmap v4723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:36 smithi079 ceph-mon[23158]: pgmap v4723: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:37.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:36 smithi016 conmon[32727]: 2022-03-21T19:49:36.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:38.168 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:37 smithi016 conmon[32727]: 2022-03-21T19:49:37.703+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:38 smithi016 ceph-mon[22010]: pgmap v4724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:38 smithi079 ceph-mon[23158]: pgmap v4724: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:38 smithi016 conmon[32727]: 2022-03-21T19:49:38.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:39 smithi016 conmon[32727]: 2022-03-21T19:49:39.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:40 smithi016 ceph-mon[22010]: pgmap v4725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:40 smithi079 ceph-mon[23158]: pgmap v4725: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8820 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:40 smithi016 conmon[32727]: 2022-03-21T19:49:40.705+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:41 smithi016 conmon[32727]: 2022-03-21T19:49:41.685+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:42.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:42.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:42 smithi016 ceph-mon[22010]: pgmap v4726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:42 smithi079 ceph-mon[23158]: pgmap v4726: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:42 smithi016 conmon[32727]: 2022-03-21T19:49:42.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:43 smithi016 conmon[32727]: 2022-03-21T19:49:43.783+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:44 smithi016 ceph-mon[22010]: pgmap v4727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:44 smithi079 ceph-mon[23158]: pgmap v4727: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:44 smithi016 conmon[32727]: 2022-03-21T19:49:44.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:45 smithi016 conmon[32727]: 2022-03-21T19:49:45.796+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:46 smithi016 ceph-mon[22010]: pgmap v4728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:46 smithi079 ceph-mon[23158]: pgmap v4728: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:46 smithi016 conmon[32727]: 2022-03-21T19:49:46.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:47 smithi016 conmon[32727]: 2022-03-21T19:49:47.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:48 smithi016 ceph-mon[22010]: pgmap v4729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:48 smithi079 ceph-mon[23158]: pgmap v4729: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:48 smithi016 conmon[32727]: 2022-03-21T19:49:48.759+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:49 smithi016 conmon[32727]: 2022-03-21T19:49:49.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:50 smithi016 ceph-mon[22010]: pgmap v4730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:50 smithi079 ceph-mon[23158]: pgmap v4730: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:50 smithi016 conmon[32727]: 2022-03-21T19:49:50.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:51 smithi016 conmon[32727]: 2022-03-21T19:49:51.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:52 smithi016 ceph-mon[22010]: pgmap v4731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:52 smithi079 ceph-mon[23158]: pgmap v4731: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:52 smithi016 conmon[32727]: 2022-03-21T19:49:52.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:53 smithi016 conmon[32727]: 2022-03-21T19:49:53.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:54 smithi016 ceph-mon[22010]: pgmap v4732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:54 smithi079 ceph-mon[23158]: pgmap v4732: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:54 smithi016 conmon[32727]: 2022-03-21T19:49:54.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:49:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:55 smithi016 conmon[32727]: 2022-03-21T19:49:55.715+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:56 smithi016 ceph-mon[22010]: pgmap v4733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:56 smithi079 ceph-mon[23158]: pgmap v4733: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:49:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:56 smithi016 conmon[32727]: 2022-03-21T19:49:56.670+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:57 smithi016 conmon[32727]: 2022-03-21T19:49:57.694+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:58 smithi016 ceph-mon[22010]: pgmap v4734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:58 smithi079 ceph-mon[23158]: pgmap v4734: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:49:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:58 smithi016 conmon[32727]: 2022-03-21T19:49:58.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:49:59 smithi016 conmon[32727]: 2022-03-21T19:49:59.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:49:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:49:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:49:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:49:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:49:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:49:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:49:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:49:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:49:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:49:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:49:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:50:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:00 smithi016 conmon[32727]: 2022-03-21T19:50:00.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: pgmap v4735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops 2022-03-21T19:50:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:50:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:50:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:50:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:50:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:50:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops 2022-03-21T19:50:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: pgmap v4735: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops 2022-03-21T19:50:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T19:50:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T19:50:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T19:50:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:50:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T19:50:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8840 sec, osd.4 has slow ops 2022-03-21T19:50:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:01 smithi016 conmon[32727]: 2022-03-21T19:50:01.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:01.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:02 smithi016 conmon[32727]: 2022-03-21T19:50:02.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:02 smithi016 ceph-mon[22010]: pgmap v4736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:02 smithi079 ceph-mon[23158]: pgmap v4736: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:03 smithi016 conmon[32727]: 2022-03-21T19:50:03.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:04 smithi079 ceph-mon[23158]: pgmap v4737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:04.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:04 smithi016 conmon[32727]: 2022-03-21T19:50:04.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:04 smithi016 ceph-mon[22010]: pgmap v4737: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:05 smithi016 conmon[32727]: 2022-03-21T19:50:05.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:06 smithi079 ceph-mon[23158]: pgmap v4738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:06 smithi016 conmon[32727]: 2022-03-21T19:50:06.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:06 smithi016 ceph-mon[22010]: pgmap v4738: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:07 smithi016 conmon[32727]: 2022-03-21T19:50:07.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:50:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:50:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:07 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:50:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:50:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:50:07.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:07 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:50:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:08 smithi079 ceph-mon[23158]: pgmap v4739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:08 smithi016 ceph-mon[22010]: pgmap v4739: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:08 smithi016 conmon[32727]: 2022-03-21T19:50:08.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[32727]: 2022-03-21T19:50:09.615+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:10.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: 2022-03-21T19:50:09.997+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:10.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:50:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: yield 2022-03-21T19:50:10.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:10.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:10.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:10.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:50:10.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:10.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:50:10.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:10.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:50:10.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:10.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:50:10.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:10.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:50:10.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:10.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:10.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:09 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:10 smithi079 ceph-mon[23158]: pgmap v4740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:10 smithi016 conmon[32727]: 2022-03-21T19:50:10.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:10 smithi016 ceph-mon[22010]: pgmap v4740: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:11 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:50:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:50:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:11.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:50:11.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:50:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:11.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:50:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:11.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:11.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:50:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:11.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:11.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:11 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:50:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:11 smithi016 conmon[32727]: 2022-03-21T19:50:11.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:11 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:50:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:11.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:50:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:11.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:50:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:11.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:50:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:50:11.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:11.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:50:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:11.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:11.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:11 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:50:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:12 smithi079 ceph-mon[23158]: pgmap v4741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:12 smithi016 conmon[32727]: 2022-03-21T19:50:12.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:12 smithi016 ceph-mon[22010]: pgmap v4741: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:13.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 2022-03-21T19:50:13.068+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:13.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:50:13.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:13.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: yield 2022-03-21T19:50:13.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:13.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:50:13.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:13.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:50:13.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:50:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:13.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:50:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:50:13.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:13.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:13.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:13.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 2022-03-21T19:50:13.069+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:50:13.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:50:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: yield 2022-03-21T19:50:13.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:13.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:13.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:13.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:50:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:50:13.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:13.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:50:13.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:50:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:13.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:50:13.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:13.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 2022-03-21T19:50:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:50:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: 2022-03-21T19:50:13.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:50:13.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:50:13.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:50:13.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:50:13.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:50:13.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:50:13.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:50:13.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:50:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:50:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:50:13.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:50:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:50:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:50:13.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:50:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:50:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:50:13.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:50:13.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:50:13.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:50:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:50:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:50:13.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:50:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise 2022-03-21T19:50:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:50:13.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:50:13.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:50:13.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:50:13.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:13 smithi016 conmon[32727]: 2022-03-21T19:50:13.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:14 smithi079 ceph-mon[23158]: pgmap v4742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:14 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:50:14.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:14.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:14.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:14.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:50:14.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:14.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:50:14.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:14.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:50:14.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:14.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:14.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:14.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:50:14.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:14.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:14.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:14 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:50:14.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:14.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:50:14.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:14.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:14.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:14.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:50:14.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:14.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:50:14.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:14.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:50:14.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:14.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:14.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:14.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:50:14.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:14.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:14.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:50:14.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:50:14.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:50:14.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:50:14.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:50:14.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:50:14.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:50:14.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:50:14.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:50:14.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:50:14.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:50:14.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:50:14.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:50:14.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:50:14.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:50:14.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:50:14.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:50:14.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:50:14.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:50:14.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:50:14.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:50:14.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:50:14.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:50:14.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:50:14.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:50:14.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:50:14.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:50:14.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:50:14.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:50:14.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:14 smithi016 conmon[32727]: 2022-03-21T19:50:14.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:14 smithi016 ceph-mon[22010]: pgmap v4742: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:14 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:50:14.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:14.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:50:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:50:14.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:50:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:14.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:50:14.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:14.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:14 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:50:14.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:50:14.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:50:14.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:50:14.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:50:14.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:50:14.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:50:14.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:50:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:50:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:50:14.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:50:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:50:14.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:50:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:50:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:50:14.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:50:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:50:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:50:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:50:14.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:50:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:50:14.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:50:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:50:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:50:14.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:50:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:50:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:50:14.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:50:14.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:50:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:50:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:50:14.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:50:14.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:50:14.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:50:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:50:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:50:14.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:50:14.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:50:14.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:50:14.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:50:14.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:50:14.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:50:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:50:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:50:14.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:50:14.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:50:14.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:50:14.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:50:14.943 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:15 smithi016 conmon[32727]: 2022-03-21T19:50:15.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:16 smithi079 ceph-mon[23158]: pgmap v4743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:16 smithi016 ceph-mon[22010]: pgmap v4743: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:16 smithi016 conmon[32727]: 2022-03-21T19:50:16.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:17 smithi016 conmon[32727]: 2022-03-21T19:50:17.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:18 smithi079 ceph-mon[23158]: pgmap v4744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:18 smithi016 ceph-mon[22010]: pgmap v4744: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:18 smithi016 conmon[32727]: 2022-03-21T19:50:18.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:19 smithi016 conmon[32727]: 2022-03-21T19:50:19.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:20 smithi079 ceph-mon[23158]: pgmap v4745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:20 smithi016 ceph-mon[22010]: pgmap v4745: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8860 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:20 smithi016 conmon[32727]: 2022-03-21T19:50:20.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:21 smithi016 conmon[32727]: 2022-03-21T19:50:21.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:22 smithi016 ceph-mon[22010]: pgmap v4746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:22.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:22 smithi016 conmon[32727]: 2022-03-21T19:50:22.449+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:22 smithi079 ceph-mon[23158]: pgmap v4746: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:23 smithi016 conmon[32727]: 2022-03-21T19:50:23.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:24 smithi016 ceph-mon[22010]: pgmap v4747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:24 smithi016 conmon[32727]: 2022-03-21T19:50:24.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:24 smithi079 ceph-mon[23158]: pgmap v4747: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:25.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:25 smithi016 conmon[32727]: 2022-03-21T19:50:25.425+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:26 smithi016 ceph-mon[22010]: pgmap v4748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:26 smithi016 conmon[32727]: 2022-03-21T19:50:26.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:26 smithi079 ceph-mon[23158]: pgmap v4748: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:27 smithi016 conmon[32727]: 2022-03-21T19:50:27.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:28 smithi016 ceph-mon[22010]: pgmap v4749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:28 smithi016 conmon[32727]: 2022-03-21T19:50:28.482+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:28 smithi079 ceph-mon[23158]: pgmap v4749: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:29 smithi016 conmon[32727]: 2022-03-21T19:50:29.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:30 smithi016 conmon[32727]: 2022-03-21T19:50:30.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:30 smithi016 ceph-mon[22010]: pgmap v4750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:30 smithi079 ceph-mon[23158]: pgmap v4750: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:31 smithi016 conmon[32727]: 2022-03-21T19:50:31.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:32 smithi016 ceph-mon[22010]: pgmap v4751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:32 smithi016 conmon[32727]: 2022-03-21T19:50:32.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:32 smithi079 ceph-mon[23158]: pgmap v4751: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:33 smithi016 conmon[32727]: 2022-03-21T19:50:33.368+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:34 smithi016 ceph-mon[22010]: pgmap v4752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:34 smithi016 conmon[32727]: 2022-03-21T19:50:34.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:34 smithi079 ceph-mon[23158]: pgmap v4752: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:35 smithi016 conmon[32727]: 2022-03-21T19:50:35.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:36 smithi016 ceph-mon[22010]: pgmap v4753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:36 smithi016 conmon[32727]: 2022-03-21T19:50:36.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:36 smithi079 ceph-mon[23158]: pgmap v4753: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:37 smithi016 conmon[32727]: 2022-03-21T19:50:37.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:38 smithi016 ceph-mon[22010]: pgmap v4754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:38.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:38 smithi016 conmon[32727]: 2022-03-21T19:50:38.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:38 smithi079 ceph-mon[23158]: pgmap v4754: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:39 smithi016 conmon[32727]: 2022-03-21T19:50:39.358+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:40 smithi016 ceph-mon[22010]: pgmap v4755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:40.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:40 smithi016 conmon[32727]: 2022-03-21T19:50:40.393+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:40 smithi079 ceph-mon[23158]: pgmap v4755: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8880 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:41 smithi016 conmon[32727]: 2022-03-21T19:50:41.370+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:42 smithi016 ceph-mon[22010]: pgmap v4756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:42 smithi016 conmon[32727]: 2022-03-21T19:50:42.397+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:42 smithi079 ceph-mon[23158]: pgmap v4756: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:43 smithi016 conmon[32727]: 2022-03-21T19:50:43.393+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:44 smithi016 ceph-mon[22010]: pgmap v4757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:44.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:44 smithi016 conmon[32727]: 2022-03-21T19:50:44.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:44 smithi079 ceph-mon[23158]: pgmap v4757: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:45.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:45.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:45 smithi016 conmon[32727]: 2022-03-21T19:50:45.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:46 smithi016 ceph-mon[22010]: pgmap v4758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:46 smithi016 conmon[32727]: 2022-03-21T19:50:46.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:46 smithi079 ceph-mon[23158]: pgmap v4758: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:47 smithi016 conmon[32727]: 2022-03-21T19:50:47.380+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:48 smithi016 ceph-mon[22010]: pgmap v4759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:48 smithi016 conmon[32727]: 2022-03-21T19:50:48.418+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:48 smithi079 ceph-mon[23158]: pgmap v4759: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:49 smithi016 conmon[32727]: 2022-03-21T19:50:49.452+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:50 smithi016 conmon[32727]: 2022-03-21T19:50:50.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:50 smithi016 ceph-mon[22010]: pgmap v4760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:50.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:50 smithi079 ceph-mon[23158]: pgmap v4760: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:50.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:51 smithi016 conmon[32727]: 2022-03-21T19:50:51.426+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:52 smithi016 ceph-mon[22010]: pgmap v4761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:52 smithi016 conmon[32727]: 2022-03-21T19:50:52.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:52 smithi079 ceph-mon[23158]: pgmap v4761: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:53 smithi016 conmon[32727]: 2022-03-21T19:50:53.395+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:54 smithi016 ceph-mon[22010]: pgmap v4762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:54 smithi016 conmon[32727]: 2022-03-21T19:50:54.392+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:54 smithi079 ceph-mon[23158]: pgmap v4762: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:55 smithi016 conmon[32727]: 2022-03-21T19:50:55.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:50:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:56 smithi016 conmon[32727]: 2022-03-21T19:50:56.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:56 smithi016 ceph-mon[22010]: pgmap v4763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:56 smithi079 ceph-mon[23158]: pgmap v4763: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:50:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:57.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:57 smithi016 conmon[32727]: 2022-03-21T19:50:57.399+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:57.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:58 smithi016 conmon[32727]: 2022-03-21T19:50:58.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:58 smithi016 ceph-mon[22010]: pgmap v4764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:58 smithi079 ceph-mon[23158]: pgmap v4764: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:50:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:50:59 smithi016 conmon[32727]: 2022-03-21T19:50:59.432+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:50:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:50:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:50:59.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:50:59.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:50:59.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:50:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:50:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:50:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:50:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:50:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:50:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:50:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:00 smithi016 ceph-mon[22010]: pgmap v4765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:00.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:00 smithi016 conmon[32727]: 2022-03-21T19:51:00.401+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:00 smithi079 ceph-mon[23158]: pgmap v4765: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8900 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:01 smithi016 ceph-mon[22010]: pgmap v4766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:01.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:01 smithi016 conmon[32727]: 2022-03-21T19:51:01.386+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:01 smithi079 ceph-mon[23158]: pgmap v4766: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:02 smithi016 conmon[32727]: 2022-03-21T19:51:02.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:03 smithi016 ceph-mon[22010]: pgmap v4767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:03.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:03 smithi016 conmon[32727]: 2022-03-21T19:51:03.404+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:03 smithi079 ceph-mon[23158]: pgmap v4767: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:04 smithi016 conmon[32727]: 2022-03-21T19:51:04.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:05.613 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:05 smithi016 conmon[32727]: 2022-03-21T19:51:05.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:05 smithi016 ceph-mon[22010]: pgmap v4768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:05.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:05 smithi079 ceph-mon[23158]: pgmap v4768: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:06.615 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:06 smithi016 conmon[32727]: 2022-03-21T19:51:06.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:07.620 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:07 smithi016 conmon[32727]: 2022-03-21T19:51:07.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:07 smithi016 ceph-mon[22010]: pgmap v4769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:07.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:07 smithi079 ceph-mon[23158]: pgmap v4769: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:08.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:08 smithi016 conmon[32727]: 2022-03-21T19:51:08.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:09.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:09 smithi016 conmon[32727]: 2022-03-21T19:51:09.349+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:09 smithi016 ceph-mon[22010]: pgmap v4770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:09.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:09 smithi079 ceph-mon[23158]: pgmap v4770: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:10 smithi016 conmon[32727]: 2022-03-21T19:51:10.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:11.633 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:11 smithi016 conmon[32727]: 2022-03-21T19:51:11.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:11 smithi016 ceph-mon[22010]: pgmap v4771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:11 smithi079 ceph-mon[23158]: pgmap v4771: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:12.634 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:12 smithi016 conmon[32727]: 2022-03-21T19:51:12.376+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:13 smithi016 conmon[32727]: 2022-03-21T19:51:13.399+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:13 smithi016 ceph-mon[22010]: pgmap v4772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:51:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:51:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:13 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:51:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:13 smithi079 ceph-mon[23158]: pgmap v4772: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:51:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:51:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:13 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:51:13.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:14 smithi016 conmon[32727]: 2022-03-21T19:51:14.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:15 smithi016 conmon[32727]: 2022-03-21T19:51:15.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:15 smithi016 ceph-mon[22010]: pgmap v4773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:15 smithi079 ceph-mon[23158]: pgmap v4773: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:16.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[32727]: 2022-03-21T19:51:16.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:16.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: 2022-03-21T19:51:16.172+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:51:16.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:16.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: yield 2022-03-21T19:51:16.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:16.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:16.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:16.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:51:16.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:16.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:51:16.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:16.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:51:16.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:16.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:51:16.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:16.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:51:16.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:16.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:16.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:16 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:16 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:51:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:51:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:16.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:16.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:51:16.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:16.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:51:16.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:16.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:51:16.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:16.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:16.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:16.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:51:16.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:16.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:16.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:16 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:51:16.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:16 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:51:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:51:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:16.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:51:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:16.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:51:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:51:16.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:16.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:51:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:16.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:16.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:16 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:51:16.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:17.651 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:17 smithi016 conmon[32727]: 2022-03-21T19:51:17.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:17 smithi016 ceph-mon[22010]: pgmap v4774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:17 smithi079 ceph-mon[23158]: pgmap v4774: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:18.652 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:18 smithi016 conmon[32727]: 2022-03-21T19:51:18.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:19.657 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[32727]: 2022-03-21T19:51:19.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:19.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 2022-03-21T19:51:19.244+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:51:19.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: yield 2022-03-21T19:51:19.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:51:19.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:51:19.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:51:19.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:51:19.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 2022-03-21T19:51:19.245+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:51:19.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:51:19.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: yield 2022-03-21T19:51:19.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:51:19.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:51:19.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:51:19.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:51:19.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 2022-03-21T19:51:19.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:51:19.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: 2022-03-21T19:51:19.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:51:19.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:51:19.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:51:19.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:51:19.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:51:19.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:51:19.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:51:19.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:51:19.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:51:19.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:51:19.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:51:19.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:51:19.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:51:19.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:51:19.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:51:19.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:51:19.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:51:19.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:51:19.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:51:19.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:51:19.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:51:19.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:51:19.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise 2022-03-21T19:51:19.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:51:19.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:51:19.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:51:19.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:51:19.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:51:19 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:19 smithi016 ceph-mon[22010]: pgmap v4775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:19 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:51:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:51:19.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:51:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:51:19.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:51:19.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:19 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:51:19.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:51:19.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:51:19.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:51:19.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:51:19.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:51:19.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:51:19.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:51:19.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:51:19.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:51:19.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:51:19.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:51:19.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:51:19.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:51:19.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:51:19.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:51:19.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:51:19.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:51:19.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:51:19.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:51:19.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:51:19.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:51:19.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:51:19.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:51:19.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:51:19.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:51:19.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:51:19.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:51:19.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:51:19.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:51:19.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:51:19.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:51:19.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:51:19.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:51:19.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:51:19.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.939 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:19 smithi079 ceph-mon[23158]: pgmap v4775: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:19 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:51:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:51:19.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:51:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:51:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:51:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:19 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:51:19.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:51:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:51:19.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:51:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:51:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:51:19.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:51:19.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:51:19.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:51:19.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:51:19.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:51:19.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:51:19.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:51:19.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:51:19.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:51:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:51:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:51:19.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:51:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:51:19.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:51:19.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:51:19.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:51:19.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:51:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:51:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:51:19.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:51:19.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:51:19.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:51:19.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:51:19.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:51:19.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:51:19.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:51:19.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:51:19.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:51:19.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:51:19.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:51:19.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:51:19.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:51:19.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:51:19.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:51:19.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:51:19.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:51:19.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:51:19.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:51:19.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:51:19.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:51:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:51:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:51:19.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:20.658 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:20 smithi016 conmon[32727]: 2022-03-21T19:51:20.283+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8920 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:21.660 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:21 smithi016 conmon[32727]: 2022-03-21T19:51:21.233+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:21 smithi016 ceph-mon[22010]: pgmap v4776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:21 smithi079 ceph-mon[23158]: pgmap v4776: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:22.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:22 smithi016 conmon[32727]: 2022-03-21T19:51:22.229+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:23.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:23.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:23 smithi016 conmon[32727]: 2022-03-21T19:51:23.240+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:23 smithi079 ceph-mon[23158]: pgmap v4777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:23 smithi016 ceph-mon[22010]: pgmap v4777: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:24.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:24.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:24 smithi016 conmon[32727]: 2022-03-21T19:51:24.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:25.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:25.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:25 smithi016 conmon[32727]: 2022-03-21T19:51:25.203+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:25 smithi079 ceph-mon[23158]: pgmap v4778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:25.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:25 smithi016 ceph-mon[22010]: pgmap v4778: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:26.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:26 smithi016 conmon[32727]: 2022-03-21T19:51:26.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:27.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:27 smithi016 conmon[32727]: 2022-03-21T19:51:27.152+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:27 smithi079 ceph-mon[23158]: pgmap v4779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:28.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:27 smithi016 ceph-mon[22010]: pgmap v4779: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:28.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:28 smithi016 conmon[32727]: 2022-03-21T19:51:28.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:29.162 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:29 smithi016 conmon[32727]: 2022-03-21T19:51:29.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:29 smithi079 ceph-mon[23158]: pgmap v4780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:29 smithi016 ceph-mon[22010]: pgmap v4780: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:30 smithi016 conmon[32727]: 2022-03-21T19:51:30.177+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:31 smithi016 conmon[32727]: 2022-03-21T19:51:31.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:31 smithi079 ceph-mon[23158]: pgmap v4781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:32.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:31 smithi016 ceph-mon[22010]: pgmap v4781: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:32.160 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:32 smithi016 conmon[32727]: 2022-03-21T19:51:32.158+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:33 smithi016 conmon[32727]: 2022-03-21T19:51:33.197+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:33 smithi079 ceph-mon[23158]: pgmap v4782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:33 smithi016 ceph-mon[22010]: pgmap v4782: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:34 smithi016 conmon[32727]: 2022-03-21T19:51:34.173+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:35 smithi016 conmon[32727]: 2022-03-21T19:51:35.168+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:35 smithi079 ceph-mon[23158]: pgmap v4783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:35.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:35 smithi016 ceph-mon[22010]: pgmap v4783: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:36.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:36 smithi016 conmon[32727]: 2022-03-21T19:51:36.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:37 smithi016 conmon[32727]: 2022-03-21T19:51:37.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:37 smithi079 ceph-mon[23158]: pgmap v4784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:37 smithi016 ceph-mon[22010]: pgmap v4784: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:38 smithi016 conmon[32727]: 2022-03-21T19:51:38.199+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:39 smithi016 conmon[32727]: 2022-03-21T19:51:39.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:39 smithi079 ceph-mon[23158]: pgmap v4785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:39 smithi016 ceph-mon[22010]: pgmap v4785: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:40 smithi016 conmon[32727]: 2022-03-21T19:51:40.191+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8940 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:41.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:41 smithi016 conmon[32727]: 2022-03-21T19:51:41.213+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:41 smithi016 ceph-mon[22010]: pgmap v4786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:42.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:41 smithi079 ceph-mon[23158]: pgmap v4786: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:42.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:42 smithi016 conmon[32727]: 2022-03-21T19:51:42.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:43.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:43 smithi016 conmon[32727]: 2022-03-21T19:51:43.208+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:43 smithi016 ceph-mon[22010]: pgmap v4787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:43 smithi079 ceph-mon[23158]: pgmap v4787: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:44.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:44 smithi016 conmon[32727]: 2022-03-21T19:51:44.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:45.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:45 smithi016 conmon[32727]: 2022-03-21T19:51:45.221+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:45 smithi016 ceph-mon[22010]: pgmap v4788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:45 smithi079 ceph-mon[23158]: pgmap v4788: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:46.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:46 smithi016 conmon[32727]: 2022-03-21T19:51:46.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:47.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:47.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:47 smithi016 conmon[32727]: 2022-03-21T19:51:47.265+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:48.022 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:47 smithi016 ceph-mon[22010]: pgmap v4789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:48.023 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:47 smithi079 ceph-mon[23158]: pgmap v4789: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:48.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:48.378 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:48 smithi016 conmon[32727]: 2022-03-21T19:51:48.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:49 smithi016 conmon[32727]: 2022-03-21T19:51:49.207+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:49 smithi016 ceph-mon[22010]: pgmap v4790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:50.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:49 smithi079 ceph-mon[23158]: pgmap v4790: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:50.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:50 smithi016 conmon[32727]: 2022-03-21T19:51:50.211+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:51.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:51.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:51.310 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:51 smithi016 conmon[32727]: 2022-03-21T19:51:51.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:51 smithi016 ceph-mon[22010]: pgmap v4791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:52.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:51 smithi079 ceph-mon[23158]: pgmap v4791: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:52.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:52.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:52 smithi016 conmon[32727]: 2022-03-21T19:51:52.264+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:53.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:53 smithi016 conmon[32727]: 2022-03-21T19:51:53.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:53 smithi016 ceph-mon[22010]: pgmap v4792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:53 smithi079 ceph-mon[23158]: pgmap v4792: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:54 smithi016 conmon[32727]: 2022-03-21T19:51:54.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:55 smithi016 conmon[32727]: 2022-03-21T19:51:55.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:56.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:55 smithi016 ceph-mon[22010]: pgmap v4793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:56.119 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:56.120 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:55 smithi079 ceph-mon[23158]: pgmap v4793: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:51:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8960 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:51:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:56 smithi016 conmon[32727]: 2022-03-21T19:51:56.284+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:57.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:57.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:57 smithi016 conmon[32727]: 2022-03-21T19:51:57.328+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:57 smithi016 ceph-mon[22010]: pgmap v4794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:58.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:57 smithi079 ceph-mon[23158]: pgmap v4794: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:51:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:58 smithi016 conmon[32727]: 2022-03-21T19:51:58.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:51:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:51:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:51:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:51:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:51:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:51:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:51:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:51:59.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:51:59 smithi016 conmon[32727]: 2022-03-21T19:51:59.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:59 smithi016 ceph-mon[22010]: pgmap v4795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:51:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:59 smithi079 ceph-mon[23158]: pgmap v4795: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:51:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:00 smithi016 conmon[32727]: 2022-03-21T19:52:00.358+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:01.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:01 smithi016 conmon[32727]: 2022-03-21T19:52:01.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:02.372 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:01 smithi016 ceph-mon[22010]: pgmap v4796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:02.373 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:01 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:02.373 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:01 smithi079 ceph-mon[23158]: pgmap v4796: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:01 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:02.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:02 smithi016 conmon[32727]: 2022-03-21T19:52:02.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:03.250 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:03.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:03 smithi016 conmon[32727]: 2022-03-21T19:52:03.348+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:04.361 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:03 smithi016 ceph-mon[22010]: pgmap v4797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:04.362 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:03 smithi079 ceph-mon[23158]: pgmap v4797: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:04.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:04 smithi016 conmon[32727]: 2022-03-21T19:52:04.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:05.384 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:05.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:05 smithi016 conmon[32727]: 2022-03-21T19:52:05.383+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:06.342 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:06 smithi016 ceph-mon[22010]: pgmap v4798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:06.342 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:06 smithi079 ceph-mon[23158]: pgmap v4798: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:06 smithi016 conmon[32727]: 2022-03-21T19:52:06.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:07.335 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:07 smithi016 conmon[32727]: 2022-03-21T19:52:07.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:08.364 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:08 smithi016 ceph-mon[22010]: pgmap v4799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:08.365 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:08 smithi079 ceph-mon[23158]: pgmap v4799: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:08 smithi016 conmon[32727]: 2022-03-21T19:52:08.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:09.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:09 smithi016 conmon[32727]: 2022-03-21T19:52:09.400+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:10 smithi016 ceph-mon[22010]: pgmap v4800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:10 smithi079 ceph-mon[23158]: pgmap v4800: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:10 smithi016 conmon[32727]: 2022-03-21T19:52:10.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:11 smithi016 conmon[32727]: 2022-03-21T19:52:11.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:12 smithi016 ceph-mon[22010]: pgmap v4801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:12 smithi079 ceph-mon[23158]: pgmap v4801: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:12.800 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:12 smithi016 conmon[32727]: 2022-03-21T19:52:12.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:13 smithi016 conmon[32727]: 2022-03-21T19:52:13.474+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:14 smithi016 ceph-mon[22010]: pgmap v4802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:14 smithi079 ceph-mon[23158]: pgmap v4802: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:14 smithi016 conmon[32727]: 2022-03-21T19:52:14.426+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:15.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:15.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:15.581 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:15.582 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:15.582 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:15 smithi016 conmon[32727]: 2022-03-21T19:52:15.456+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:16 smithi016 ceph-mon[22010]: pgmap v4803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:16 smithi079 ceph-mon[23158]: pgmap v4803: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:16 smithi016 conmon[32727]: 2022-03-21T19:52:16.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:17 smithi016 conmon[32727]: 2022-03-21T19:52:17.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:18 smithi016 ceph-mon[22010]: pgmap v4804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:18 smithi079 ceph-mon[23158]: pgmap v4804: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:18 smithi016 conmon[32727]: 2022-03-21T19:52:18.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:19.301 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:19.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[32727]: 2022-03-21T19:52:19.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:20.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: 2022-03-21T19:52:19.852+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:20.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:52:20.043 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:20.044 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: yield 2022-03-21T19:52:20.044 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:20.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:20.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:20.045 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:52:20.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:20.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:52:20.046 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:20.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:52:20.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:20.047 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:52:20.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:20.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:52:20.048 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:20.049 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:20.049 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:19 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: pgmap v4805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:52:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:52:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:52:20.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: pgmap v4805: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:52:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:52:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:52:20.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8980 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:20 smithi016 conmon[32727]: 2022-03-21T19:52:20.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:52:21.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:52:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:21.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:52:21.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:21.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:52:21.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:21.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:52:21.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:21.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:21.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:21.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:52:21.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:21.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:21.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:21.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:21 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:52:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:52:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:21.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:52:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:21.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:52:21.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:52:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:21.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:52:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:21.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:52:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:21.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:21.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:21 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:52:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:21 smithi016 conmon[32727]: 2022-03-21T19:52:21.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:22.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:22 smithi016 ceph-mon[22010]: pgmap v4806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:22.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:22 smithi079 ceph-mon[23158]: pgmap v4806: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:22.807 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[32727]: 2022-03-21T19:52:22.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:23.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 2022-03-21T19:52:22.924+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:52:23.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:23.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: yield 2022-03-21T19:52:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:23.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:23.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:52:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:52:23.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:52:23.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:52:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:23.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:52:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:23.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 2022-03-21T19:52:22.925+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:52:23.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:52:23.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: yield 2022-03-21T19:52:23.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:23.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:52:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:52:23.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:52:23.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:52:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:23.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:52:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:23.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 2022-03-21T19:52:23.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:52:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: 2022-03-21T19:52:23.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:52:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:52:23.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:52:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:52:23.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:52:23.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:52:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:52:23.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:52:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:52:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:52:23.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:52:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:52:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:52:23.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:52:23.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:52:23.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:52:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:52:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:52:23.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:52:23.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:52:23.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:52:23.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:52:23.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise 2022-03-21T19:52:23.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:52:23.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:52:23.450 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:52:23.451 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:52:23.452 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:52:22 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:23 smithi016 conmon[32727]: 2022-03-21T19:52:23.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:24 smithi016 ceph-mon[22010]: pgmap v4807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:24.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:24 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:24.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:52:24.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:24.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:24.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:24.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:52:24.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:24.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:52:24.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:24.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:52:24.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:24.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:24.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:24.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:52:24.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:24.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:24.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.428 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:24 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:52:24.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:24.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:52:24.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:24.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:24.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:24.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:52:24.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:24.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:52:24.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:24.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:52:24.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:24.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:24.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:24.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:52:24.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:24.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:24.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:52:24.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:52:24.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:52:24.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:52:24.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:52:24.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:52:24.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:52:24.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:52:24.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:52:24.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:52:24.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:52:24.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:52:24.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:52:24.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:52:24.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:52:24.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:52:24.451 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:52:24.452 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:52:24.453 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:52:24.453 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:52:24.454 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:52:24.455 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:52:24.455 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:52:24.456 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:52:24.457 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:52:24.457 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:52:24.458 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:52:24.458 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:52:24.459 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:52:24.460 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:24 smithi079 ceph-mon[23158]: pgmap v4807: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:24 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:52:24.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:24.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:52:24.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:52:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:24.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:52:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:24.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:24.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:52:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:24.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.484 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:24 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:52:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:52:24.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:52:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:52:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:52:24.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:52:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:52:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:52:24.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:52:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:52:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:52:24.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:52:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:52:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:52:24.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:52:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:52:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:52:24.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:52:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:52:24.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:52:24.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:52:24.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:52:24.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:52:24.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:52:24.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:52:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:52:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:52:24.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:52:24.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:52:24.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:52:24.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:52:24.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:52:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:52:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:52:24.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:52:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:52:24.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:52:24.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:52:24.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:52:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:52:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:52:24.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:52:24.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:52:24.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:52:24.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:52:24.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:52:24.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:52:24.504 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:52:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:24 smithi016 conmon[32727]: 2022-03-21T19:52:24.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:25.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:25 smithi016 conmon[32727]: 2022-03-21T19:52:25.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:26 smithi016 ceph-mon[22010]: pgmap v4808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:26 smithi079 ceph-mon[23158]: pgmap v4808: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:26 smithi016 conmon[32727]: 2022-03-21T19:52:26.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:27 smithi016 conmon[32727]: 2022-03-21T19:52:27.459+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:28 smithi016 ceph-mon[22010]: pgmap v4809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:28 smithi079 ceph-mon[23158]: pgmap v4809: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:28.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:28 smithi016 conmon[32727]: 2022-03-21T19:52:28.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:29 smithi016 conmon[32727]: 2022-03-21T19:52:29.468+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:30 smithi016 ceph-mon[22010]: pgmap v4810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:30 smithi079 ceph-mon[23158]: pgmap v4810: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:30 smithi016 conmon[32727]: 2022-03-21T19:52:30.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:31.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:31 smithi016 conmon[32727]: 2022-03-21T19:52:31.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:32.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:32 smithi016 ceph-mon[22010]: pgmap v4811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:32 smithi079 ceph-mon[23158]: pgmap v4811: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:32 smithi016 conmon[32727]: 2022-03-21T19:52:32.575+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:33 smithi016 conmon[32727]: 2022-03-21T19:52:33.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:34.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:34 smithi016 ceph-mon[22010]: pgmap v4812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:34 smithi079 ceph-mon[23158]: pgmap v4812: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:34 smithi016 conmon[32727]: 2022-03-21T19:52:34.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 8995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:35.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 8995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:35 smithi016 conmon[32727]: 2022-03-21T19:52:35.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:36 smithi016 ceph-mon[22010]: pgmap v4813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:36 smithi079 ceph-mon[23158]: pgmap v4813: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:36 smithi016 conmon[32727]: 2022-03-21T19:52:36.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:37 smithi016 conmon[32727]: 2022-03-21T19:52:37.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:38 smithi016 ceph-mon[22010]: pgmap v4814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:38 smithi079 ceph-mon[23158]: pgmap v4814: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:38 smithi016 conmon[32727]: 2022-03-21T19:52:38.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:39.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:39 smithi016 conmon[32727]: 2022-03-21T19:52:39.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:40 smithi016 ceph-mon[22010]: pgmap v4815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:40 smithi079 ceph-mon[23158]: pgmap v4815: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9000 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:40 smithi016 conmon[32727]: 2022-03-21T19:52:40.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:41.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:41 smithi016 conmon[32727]: 2022-03-21T19:52:41.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:42.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:42.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:42 smithi016 ceph-mon[22010]: pgmap v4816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:42.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:42 smithi079 ceph-mon[23158]: pgmap v4816: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:42.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:42 smithi016 conmon[32727]: 2022-03-21T19:52:42.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:43.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:43 smithi016 conmon[32727]: 2022-03-21T19:52:43.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:44.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:44 smithi016 ceph-mon[22010]: pgmap v4817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:44.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:44 smithi079 ceph-mon[23158]: pgmap v4817: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:44.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:44 smithi016 conmon[32727]: 2022-03-21T19:52:44.488+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:45 smithi016 conmon[32727]: 2022-03-21T19:52:45.506+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:46 smithi016 ceph-mon[22010]: pgmap v4818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:46 smithi079 ceph-mon[23158]: pgmap v4818: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:46.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:46 smithi016 conmon[32727]: 2022-03-21T19:52:46.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:47 smithi016 conmon[32727]: 2022-03-21T19:52:47.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:48 smithi016 ceph-mon[22010]: pgmap v4819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:48 smithi079 ceph-mon[23158]: pgmap v4819: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:48 smithi016 conmon[32727]: 2022-03-21T19:52:48.459+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:49.413 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:49 smithi016 conmon[32727]: 2022-03-21T19:52:49.412+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:50 smithi016 ceph-mon[22010]: pgmap v4820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:50.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:50 smithi079 ceph-mon[23158]: pgmap v4820: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:50 smithi016 conmon[32727]: 2022-03-21T19:52:50.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:51.409 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:51.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:51 smithi016 conmon[32727]: 2022-03-21T19:52:51.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:52 smithi016 conmon[32727]: 2022-03-21T19:52:52.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:52 smithi016 ceph-mon[22010]: pgmap v4821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:52 smithi079 ceph-mon[23158]: pgmap v4821: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:53 smithi016 conmon[32727]: 2022-03-21T19:52:53.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:53.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:54.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:54 smithi016 conmon[32727]: 2022-03-21T19:52:54.370+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:54 smithi016 ceph-mon[22010]: pgmap v4822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:54 smithi079 ceph-mon[23158]: pgmap v4822: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:55.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:55 smithi016 conmon[32727]: 2022-03-21T19:52:55.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:52:56.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:56 smithi016 conmon[32727]: 2022-03-21T19:52:56.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:56 smithi016 ceph-mon[22010]: pgmap v4823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:56 smithi079 ceph-mon[23158]: pgmap v4823: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:52:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:57 smithi016 conmon[32727]: 2022-03-21T19:52:57.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:58 smithi016 ceph-mon[22010]: pgmap v4824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:58.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:58 smithi016 conmon[32727]: 2022-03-21T19:52:58.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:58 smithi079 ceph-mon[23158]: pgmap v4824: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:52:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:52:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:52:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:52:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:52:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:52:59.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:52:59 smithi016 conmon[32727]: 2022-03-21T19:52:59.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:52:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:52:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:52:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:52:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:52:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:52:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:53:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:00 smithi079 ceph-mon[23158]: pgmap v4825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:00 smithi016 conmon[32727]: 2022-03-21T19:53:00.361+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:00 smithi016 ceph-mon[22010]: pgmap v4825: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9020 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:01 smithi016 conmon[32727]: 2022-03-21T19:53:01.366+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:02 smithi079 ceph-mon[23158]: pgmap v4826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:02 smithi016 ceph-mon[22010]: pgmap v4826: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:02 smithi016 conmon[32727]: 2022-03-21T19:53:02.399+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:03 smithi016 conmon[32727]: 2022-03-21T19:53:03.376+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:04 smithi079 ceph-mon[23158]: pgmap v4827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:04 smithi016 ceph-mon[22010]: pgmap v4827: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:04 smithi016 conmon[32727]: 2022-03-21T19:53:04.423+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:05 smithi016 conmon[32727]: 2022-03-21T19:53:05.431+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:06.447 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:06 smithi016 ceph-mon[22010]: pgmap v4828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:06.448 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:06 smithi079 ceph-mon[23158]: pgmap v4828: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:06 smithi016 conmon[32727]: 2022-03-21T19:53:06.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:07 smithi016 conmon[32727]: 2022-03-21T19:53:07.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:08 smithi079 ceph-mon[23158]: pgmap v4829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:08 smithi016 conmon[32727]: 2022-03-21T19:53:08.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:08 smithi016 ceph-mon[22010]: pgmap v4829: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:09.454 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:09 smithi016 conmon[32727]: 2022-03-21T19:53:09.453+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:10 smithi079 ceph-mon[23158]: pgmap v4830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:10 smithi016 conmon[32727]: 2022-03-21T19:53:10.407+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:10 smithi016 ceph-mon[22010]: pgmap v4830: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:11.846 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:11 smithi016 conmon[32727]: 2022-03-21T19:53:11.441+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:12 smithi079 ceph-mon[23158]: pgmap v4831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:12 smithi016 ceph-mon[22010]: pgmap v4831: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:12 smithi016 conmon[32727]: 2022-03-21T19:53:12.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:13.670 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:13 smithi016 conmon[32727]: 2022-03-21T19:53:13.452+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:14 smithi079 ceph-mon[23158]: pgmap v4832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:14.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:14 smithi016 ceph-mon[22010]: pgmap v4832: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:14.482 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:14 smithi016 conmon[32727]: 2022-03-21T19:53:14.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:15 smithi016 conmon[32727]: 2022-03-21T19:53:15.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:15.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:16 smithi079 ceph-mon[23158]: pgmap v4833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:16.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:16 smithi016 ceph-mon[22010]: pgmap v4833: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:16 smithi016 conmon[32727]: 2022-03-21T19:53:16.417+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:17 smithi016 conmon[32727]: 2022-03-21T19:53:17.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:17.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:18 smithi016 ceph-mon[22010]: pgmap v4834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:18 smithi016 conmon[32727]: 2022-03-21T19:53:18.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:18 smithi079 ceph-mon[23158]: pgmap v4834: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:19 smithi016 conmon[32727]: 2022-03-21T19:53:19.425+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:20 smithi016 conmon[32727]: 2022-03-21T19:53:20.428+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:20 smithi016 ceph-mon[22010]: pgmap v4835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:20.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:20 smithi079 ceph-mon[23158]: pgmap v4835: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9040 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:21 smithi016 conmon[32727]: 2022-03-21T19:53:21.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:22 smithi016 ceph-mon[22010]: pgmap v4836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:22 smithi016 conmon[32727]: 2022-03-21T19:53:22.379+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:22 smithi079 ceph-mon[23158]: pgmap v4836: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:23.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:23 smithi016 conmon[32727]: 2022-03-21T19:53:23.380+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:53:23.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:53:23.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:23 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:53:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:53:23.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:23 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:53:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:24 smithi016 ceph-mon[22010]: pgmap v4837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:24 smithi016 conmon[32727]: 2022-03-21T19:53:24.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:24 smithi079 ceph-mon[23158]: pgmap v4837: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:25 smithi016 conmon[32727]: 2022-03-21T19:53:25.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:26.351 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:26 smithi016 ceph-mon[22010]: pgmap v4838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:26.352 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:26.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: 2022-03-21T19:53:26.028+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:26.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:53:26.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:26.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: yield 2022-03-21T19:53:26.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:26.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:26.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:26.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:53:26.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:26.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:53:26.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:26.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:53:26.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:26.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:53:26.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:26.357 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:53:26.358 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:26.358 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:26.358 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:26 smithi016 conmon[32727]: 2022-03-21T19:53:26.349+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:26.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:26 smithi079 ceph-mon[23158]: pgmap v4838: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:27 smithi016 conmon[32727]: 2022-03-21T19:53:27.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:27 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:53:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:53:27.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:53:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:27.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:53:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:53:27.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:27.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:53:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:27.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:27 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:53:27.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:27 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:53:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:53:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:27.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:53:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:53:27.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:53:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:27.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:27.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:53:27.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:27.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:27 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:53:27.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:28 smithi016 conmon[32727]: 2022-03-21T19:53:28.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:28 smithi016 ceph-mon[22010]: pgmap v4839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:28 smithi079 ceph-mon[23158]: pgmap v4839: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:29.369 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:29.370 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 2022-03-21T19:53:29.100+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:29.371 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:53:29.371 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:29.371 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: yield 2022-03-21T19:53:29.372 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:29.372 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:29.372 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:29.373 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:53:29.373 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:29.373 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:53:29.374 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:29.375 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:53:29.375 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:29.375 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:53:29.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:29.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:53:29.376 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:29.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:29.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:29.377 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 2022-03-21T19:53:29.101+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:53:29.378 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:53:29.378 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:29.379 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: yield 2022-03-21T19:53:29.379 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:29.379 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:29.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:29.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:53:29.380 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:29.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:53:29.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:29.381 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:53:29.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:29.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:53:29.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:29.384 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:53:29.384 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:29.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:29.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:29.385 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 2022-03-21T19:53:29.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:53:29.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: 2022-03-21T19:53:29.386 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:53:29.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:53:29.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:53:29.387 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:53:29.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:53:29.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:53:29.388 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:53:29.389 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:53:29.390 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:53:29.390 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:53:29.390 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:53:29.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:53:29.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:53:29.391 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:53:29.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:53:29.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:53:29.392 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:53:29.393 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:53:29.393 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:53:29.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:53:29.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:53:29.394 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:53:29.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise 2022-03-21T19:53:29.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:53:29.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:53:29.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:53:29.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:53:29.396 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:29.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:29 smithi016 conmon[32727]: 2022-03-21T19:53:29.367+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:30 smithi016 conmon[32727]: 2022-03-21T19:53:30.342+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:30 smithi016 ceph-mon[22010]: pgmap v4840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:53:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:53:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:53:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:53:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:30.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:53:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:30.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:30 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:53:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:30.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:53:30.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:30.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:30.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:30.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:53:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:53:30.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:30.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:53:30.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:53:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:53:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:53:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:53:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:53:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:53:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:53:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:53:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:53:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:53:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:53:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:53:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:53:30.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:53:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:53:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:53:30.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:53:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:53:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:53:30.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:53:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:53:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:53:30.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:53:30.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:53:30.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:53:30.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:53:30.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:53:30.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:53:30.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:53:30.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:53:30.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:30.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:30 smithi079 ceph-mon[23158]: pgmap v4840: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:53:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:53:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:53:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:53:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:53:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:30 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:53:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:53:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:53:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:53:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:53:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:53:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:53:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:53:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:53:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:53:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:53:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:53:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:53:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:53:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:53:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:53:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:53:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:53:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:53:30.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:53:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:53:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:53:30.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:53:30.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:53:30.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:53:30.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:53:30.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:53:30.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:53:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:53:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:53:30.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:53:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:53:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:53:30.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:53:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:53:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:53:30.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:53:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:53:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:53:30.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:53:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:53:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:53:30.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:53:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:53:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:53:30.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:53:30.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:53:30.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:53:30.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:30.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:31 smithi016 conmon[32727]: 2022-03-21T19:53:31.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:32 smithi016 conmon[32727]: 2022-03-21T19:53:32.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:32 smithi016 ceph-mon[22010]: pgmap v4841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:32 smithi079 ceph-mon[23158]: pgmap v4841: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:33 smithi016 conmon[32727]: 2022-03-21T19:53:33.397+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:34 smithi016 conmon[32727]: 2022-03-21T19:53:34.423+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:34 smithi016 ceph-mon[22010]: pgmap v4842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:34 smithi079 ceph-mon[23158]: pgmap v4842: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:35 smithi016 conmon[32727]: 2022-03-21T19:53:35.378+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:35.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:36.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:36 smithi016 conmon[32727]: 2022-03-21T19:53:36.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:36 smithi016 ceph-mon[22010]: pgmap v4843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:36 smithi079 ceph-mon[23158]: pgmap v4843: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:37 smithi016 conmon[32727]: 2022-03-21T19:53:37.386+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:38 smithi016 conmon[32727]: 2022-03-21T19:53:38.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:38 smithi016 ceph-mon[22010]: pgmap v4844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:38 smithi079 ceph-mon[23158]: pgmap v4844: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:39 smithi016 conmon[32727]: 2022-03-21T19:53:39.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:40 smithi016 ceph-mon[22010]: pgmap v4845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:40.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:40 smithi016 conmon[32727]: 2022-03-21T19:53:40.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:40 smithi079 ceph-mon[23158]: pgmap v4845: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9060 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:41 smithi016 conmon[32727]: 2022-03-21T19:53:41.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:42 smithi016 ceph-mon[22010]: pgmap v4846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:43.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:43.096 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:42 smithi016 conmon[32727]: 2022-03-21T19:53:42.366+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:43.098 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:42 smithi079 ceph-mon[23158]: pgmap v4846: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:43.098 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:43 smithi016 conmon[32727]: 2022-03-21T19:53:43.342+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:44 smithi016 ceph-mon[22010]: pgmap v4847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:44 smithi016 conmon[32727]: 2022-03-21T19:53:44.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:44 smithi079 ceph-mon[23158]: pgmap v4847: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:45 smithi016 conmon[32727]: 2022-03-21T19:53:45.337+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:46 smithi016 ceph-mon[22010]: pgmap v4848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:46 smithi016 conmon[32727]: 2022-03-21T19:53:46.319+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:46 smithi079 ceph-mon[23158]: pgmap v4848: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:47 smithi016 conmon[32727]: 2022-03-21T19:53:47.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:48 smithi016 ceph-mon[22010]: pgmap v4849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:48 smithi016 conmon[32727]: 2022-03-21T19:53:48.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:48 smithi079 ceph-mon[23158]: pgmap v4849: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:49 smithi016 conmon[32727]: 2022-03-21T19:53:49.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:50 smithi016 ceph-mon[22010]: pgmap v4850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:50 smithi016 conmon[32727]: 2022-03-21T19:53:50.388+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:50 smithi079 ceph-mon[23158]: pgmap v4850: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:51 smithi016 conmon[32727]: 2022-03-21T19:53:51.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:52 smithi016 ceph-mon[22010]: pgmap v4851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:52 smithi016 conmon[32727]: 2022-03-21T19:53:52.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:52 smithi079 ceph-mon[23158]: pgmap v4851: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:53 smithi016 conmon[32727]: 2022-03-21T19:53:53.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:54 smithi016 ceph-mon[22010]: pgmap v4852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:54 smithi016 conmon[32727]: 2022-03-21T19:53:54.319+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:54 smithi079 ceph-mon[23158]: pgmap v4852: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:55 smithi016 conmon[32727]: 2022-03-21T19:53:55.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:53:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:56 smithi016 ceph-mon[22010]: pgmap v4853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:56 smithi016 conmon[32727]: 2022-03-21T19:53:56.395+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:56 smithi079 ceph-mon[23158]: pgmap v4853: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:53:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:57.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:57.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:57 smithi016 conmon[32727]: 2022-03-21T19:53:57.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:58 smithi016 ceph-mon[22010]: pgmap v4854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:58.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:58 smithi016 conmon[32727]: 2022-03-21T19:53:58.359+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:58 smithi079 ceph-mon[23158]: pgmap v4854: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:53:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:53:59 smithi016 conmon[32727]: 2022-03-21T19:53:59.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:53:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:53:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:53:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:53:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:53:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:53:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:53:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:53:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:53:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:53:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:53:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:54:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:00 smithi016 ceph-mon[22010]: pgmap v4855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:00 smithi016 conmon[32727]: 2022-03-21T19:54:00.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:00 smithi079 ceph-mon[23158]: pgmap v4855: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9080 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:01 smithi016 conmon[32727]: 2022-03-21T19:54:01.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:02 smithi016 ceph-mon[22010]: pgmap v4856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:02 smithi016 conmon[32727]: 2022-03-21T19:54:02.368+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:02 smithi079 ceph-mon[23158]: pgmap v4856: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:03 smithi016 conmon[32727]: 2022-03-21T19:54:03.323+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:04 smithi016 ceph-mon[22010]: pgmap v4857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:04 smithi016 conmon[32727]: 2022-03-21T19:54:04.334+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:04 smithi079 ceph-mon[23158]: pgmap v4857: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:05 smithi016 conmon[32727]: 2022-03-21T19:54:05.298+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:06 smithi016 ceph-mon[22010]: pgmap v4858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:06 smithi016 conmon[32727]: 2022-03-21T19:54:06.332+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:06 smithi079 ceph-mon[23158]: pgmap v4858: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:07 smithi016 conmon[32727]: 2022-03-21T19:54:07.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:08 smithi016 ceph-mon[22010]: pgmap v4859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:08 smithi016 conmon[32727]: 2022-03-21T19:54:08.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:08 smithi079 ceph-mon[23158]: pgmap v4859: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:09 smithi016 conmon[32727]: 2022-03-21T19:54:09.326+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:10 smithi016 ceph-mon[22010]: pgmap v4860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:10 smithi016 conmon[32727]: 2022-03-21T19:54:10.361+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:10 smithi079 ceph-mon[23158]: pgmap v4860: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:11 smithi016 conmon[32727]: 2022-03-21T19:54:11.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:12 smithi016 ceph-mon[22010]: pgmap v4861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:12 smithi016 conmon[32727]: 2022-03-21T19:54:12.297+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:12 smithi079 ceph-mon[23158]: pgmap v4861: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:13.907 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:13 smithi016 conmon[32727]: 2022-03-21T19:54:13.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:14 smithi016 ceph-mon[22010]: pgmap v4862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:14.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:14 smithi016 conmon[32727]: 2022-03-21T19:54:14.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:14 smithi079 ceph-mon[23158]: pgmap v4862: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:15 smithi016 conmon[32727]: 2022-03-21T19:54:15.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:16 smithi016 ceph-mon[22010]: pgmap v4863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:16 smithi016 conmon[32727]: 2022-03-21T19:54:16.273+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:16 smithi079 ceph-mon[23158]: pgmap v4863: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:17 smithi016 conmon[32727]: 2022-03-21T19:54:17.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:18 smithi016 ceph-mon[22010]: pgmap v4864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:18 smithi016 conmon[32727]: 2022-03-21T19:54:18.315+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:18 smithi079 ceph-mon[23158]: pgmap v4864: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:19 smithi016 conmon[32727]: 2022-03-21T19:54:19.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:20 smithi016 ceph-mon[22010]: pgmap v4865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:20 smithi016 conmon[32727]: 2022-03-21T19:54:20.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:20 smithi079 ceph-mon[23158]: pgmap v4865: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9100 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:21 smithi016 conmon[32727]: 2022-03-21T19:54:21.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:22 smithi016 ceph-mon[22010]: pgmap v4866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:22 smithi016 conmon[32727]: 2022-03-21T19:54:22.350+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:22 smithi079 ceph-mon[23158]: pgmap v4866: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:23 smithi016 conmon[32727]: 2022-03-21T19:54:23.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:24 smithi016 ceph-mon[22010]: pgmap v4867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:24 smithi016 conmon[32727]: 2022-03-21T19:54:24.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:24 smithi079 ceph-mon[23158]: pgmap v4867: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:25 smithi016 conmon[32727]: 2022-03-21T19:54:25.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:26 smithi016 ceph-mon[22010]: pgmap v4868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:26 smithi016 conmon[32727]: 2022-03-21T19:54:26.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:26 smithi079 ceph-mon[23158]: pgmap v4868: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:27 smithi016 conmon[32727]: 2022-03-21T19:54:27.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:28 smithi016 ceph-mon[22010]: pgmap v4869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:28 smithi016 conmon[32727]: 2022-03-21T19:54:28.340+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:28 smithi079 ceph-mon[23158]: pgmap v4869: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[32727]: 2022-03-21T19:54:29.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:54:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:54:29.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:29 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:54:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:54:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:54:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:29 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:54:29.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: 2022-03-21T19:54:29.580+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:29.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:54:29.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:29.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: yield 2022-03-21T19:54:29.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:29.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:29.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:54:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:29.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:54:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:54:29.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:54:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:29.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:54:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:29.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:29 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:30 smithi016 conmon[32727]: 2022-03-21T19:54:30.290+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:30 smithi016 ceph-mon[22010]: pgmap v4870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:30.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:30.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:30 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:54:30.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:54:30.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:30.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:30.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:54:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:54:30.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:54:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:30.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:54:30.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:30.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:30 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:54:30.683 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:30 smithi079 ceph-mon[23158]: pgmap v4870: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:30.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:30 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:30.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:54:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:30.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:54:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:30.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:30.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:54:30.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:30.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:54:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:30.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:54:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:30.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:30.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:54:30.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:30.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:30.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:30 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:54:30.737 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:31 smithi016 conmon[32727]: 2022-03-21T19:54:31.308+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:32.654 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[32727]: 2022-03-21T19:54:32.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:32.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:32 smithi016 ceph-mon[22010]: pgmap v4871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:32.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:32 smithi079 ceph-mon[23158]: pgmap v4871: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:32.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 2022-03-21T19:54:32.652+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:32.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:54:32.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:32.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: yield 2022-03-21T19:54:32.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:32.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:32.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:32.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:54:32.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:32.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:54:32.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:32.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:54:32.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:32.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:54:32.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:32.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:54:32.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:32.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:32.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:32.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 2022-03-21T19:54:32.653+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:54:32.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:54:32.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:32.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: yield 2022-03-21T19:54:32.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:32.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:32.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:32.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:54:32.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:32.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:54:32.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:32.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:54:32.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:32.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:54:32.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:32.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:54:32.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:32.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:32.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:32.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 2022-03-21T19:54:32.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:54:32.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: 2022-03-21T19:54:32.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:54:32.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:54:32.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:54:32.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:54:32.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:54:32.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:54:32.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:54:32.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:54:32.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:54:32.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:54:32.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:54:32.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:54:32.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:54:32.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:54:32.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:54:32.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:54:32.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:54:32.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:54:32.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:54:32.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:54:32.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:54:32.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:54:32.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise 2022-03-21T19:54:32.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:54:32.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:54:32.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:54:32.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:54:32.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:54:32 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:33 smithi016 conmon[32727]: 2022-03-21T19:54:33.234+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:33.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:34 smithi016 conmon[32727]: 2022-03-21T19:54:34.225+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:34 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:54:34.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:34.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:34.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:34.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:54:34.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:34.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:54:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:54:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:54:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:34 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:54:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:34.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:54:34.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:34.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:34.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:34.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:54:34.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:34.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:54:34.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:34.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:54:34.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:34.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:34.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:34.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:54:34.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:34.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:34.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:54:34.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:54:34.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:54:34.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:54:34.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:54:34.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:54:34.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:54:34.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:54:34.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:54:34.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:54:34.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:54:34.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:54:34.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:54:34.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:54:34.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:54:34.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:54:34.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:54:34.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:54:34.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:54:34.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:54:34.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:54:34.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:54:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:54:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:54:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:54:34.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:54:34.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:54:34.692 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:54:34.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:54:34.693 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.693 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:34 smithi016 ceph-mon[22010]: pgmap v4872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:34.694 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:34 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:54:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:54:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:54:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:54:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:54:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:34 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:54:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:54:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:54:34.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:54:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:54:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:54:34.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:54:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:54:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:54:34.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:54:34.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:54:34.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:54:34.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:54:34.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:54:34.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:54:34.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:54:34.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:54:34.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:54:34.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:54:34.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:54:34.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:54:34.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:54:34.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:54:34.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:54:34.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:54:34.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:54:34.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:54:34.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:54:34.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:54:34.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:54:34.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:54:34.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:54:34.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:54:34.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:54:34.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:54:34.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:54:34.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:54:34.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:54:34.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:54:34.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:54:34.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:54:34.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:54:34.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:54:34.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:54:34.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:54:34.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:54:34.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:54:34.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:54:34.749 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:34 smithi079 ceph-mon[23158]: pgmap v4872: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:34.750 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:35.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:35 smithi016 conmon[32727]: 2022-03-21T19:54:35.255+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:35.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:36.478 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:36 smithi016 conmon[32727]: 2022-03-21T19:54:36.206+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:36 smithi016 ceph-mon[22010]: pgmap v4873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:36 smithi079 ceph-mon[23158]: pgmap v4873: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:37.483 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:37 smithi016 conmon[32727]: 2022-03-21T19:54:37.164+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:38.484 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:38 smithi016 conmon[32727]: 2022-03-21T19:54:38.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:38 smithi016 ceph-mon[22010]: pgmap v4874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:38 smithi079 ceph-mon[23158]: pgmap v4874: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:39.489 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:39 smithi016 conmon[32727]: 2022-03-21T19:54:39.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:40 smithi016 ceph-mon[22010]: pgmap v4875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:40 smithi016 conmon[32727]: 2022-03-21T19:54:40.242+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:40.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:40 smithi079 ceph-mon[23158]: pgmap v4875: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:40.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:40.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9120 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:41 smithi016 conmon[32727]: 2022-03-21T19:54:41.280+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:42.497 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:42 smithi079 ceph-mon[23158]: pgmap v4876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:42.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:42 smithi016 ceph-mon[22010]: pgmap v4876: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:42 smithi016 conmon[32727]: 2022-03-21T19:54:42.259+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:43.501 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:43 smithi016 conmon[32727]: 2022-03-21T19:54:43.243+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:44.503 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:44 smithi016 conmon[32727]: 2022-03-21T19:54:44.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:44 smithi016 ceph-mon[22010]: pgmap v4877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:44 smithi079 ceph-mon[23158]: pgmap v4877: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:45.509 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:45 smithi016 conmon[32727]: 2022-03-21T19:54:45.217+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:46.508 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:46 smithi016 conmon[32727]: 2022-03-21T19:54:46.253+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:46 smithi016 ceph-mon[22010]: pgmap v4878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:46 smithi079 ceph-mon[23158]: pgmap v4878: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:47.513 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:47 smithi016 conmon[32727]: 2022-03-21T19:54:47.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:48.515 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:48 smithi016 conmon[32727]: 2022-03-21T19:54:48.234+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:48 smithi016 ceph-mon[22010]: pgmap v4879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:48 smithi079 ceph-mon[23158]: pgmap v4879: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:49 smithi016 conmon[32727]: 2022-03-21T19:54:49.276+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:50 smithi016 ceph-mon[22010]: pgmap v4880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:50.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:50 smithi016 conmon[32727]: 2022-03-21T19:54:50.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:50 smithi079 ceph-mon[23158]: pgmap v4880: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:51 smithi016 conmon[32727]: 2022-03-21T19:54:51.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:52 smithi016 ceph-mon[22010]: pgmap v4881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:52 smithi016 conmon[32727]: 2022-03-21T19:54:52.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:52 smithi079 ceph-mon[23158]: pgmap v4881: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:53 smithi016 conmon[32727]: 2022-03-21T19:54:53.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:54 smithi016 conmon[32727]: 2022-03-21T19:54:54.324+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:54 smithi016 ceph-mon[22010]: pgmap v4882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:54 smithi079 ceph-mon[23158]: pgmap v4882: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:55.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:55 smithi016 conmon[32727]: 2022-03-21T19:54:55.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:54:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:56 smithi016 ceph-mon[22010]: pgmap v4883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:56.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:56 smithi016 conmon[32727]: 2022-03-21T19:54:56.331+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:56 smithi079 ceph-mon[23158]: pgmap v4883: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:54:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:57 smithi016 conmon[32727]: 2022-03-21T19:54:57.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:58 smithi016 ceph-mon[22010]: pgmap v4884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:58 smithi016 conmon[32727]: 2022-03-21T19:54:58.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:58 smithi079 ceph-mon[23158]: pgmap v4884: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:54:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:54:59 smithi016 conmon[32727]: 2022-03-21T19:54:59.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:54:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:54:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:54:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:54:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:54:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:54:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:54:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:54:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:54:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:54:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:54:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:54:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:00 smithi016 ceph-mon[22010]: pgmap v4885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:00 smithi016 conmon[32727]: 2022-03-21T19:55:00.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:00 smithi079 ceph-mon[23158]: pgmap v4885: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9140 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:01 smithi016 conmon[32727]: 2022-03-21T19:55:01.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:02 smithi016 ceph-mon[22010]: pgmap v4886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:02 smithi016 conmon[32727]: 2022-03-21T19:55:02.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:02 smithi079 ceph-mon[23158]: pgmap v4886: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:03 smithi016 conmon[32727]: 2022-03-21T19:55:03.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:04 smithi016 ceph-mon[22010]: pgmap v4887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:04 smithi016 conmon[32727]: 2022-03-21T19:55:04.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:04 smithi079 ceph-mon[23158]: pgmap v4887: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:05.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:05 smithi016 conmon[32727]: 2022-03-21T19:55:05.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:05.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:06.575 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:06 smithi016 conmon[32727]: 2022-03-21T19:55:06.306+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:06 smithi016 ceph-mon[22010]: pgmap v4888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:06 smithi079 ceph-mon[23158]: pgmap v4888: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:07.580 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:07 smithi016 conmon[32727]: 2022-03-21T19:55:07.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:08.582 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:08 smithi016 conmon[32727]: 2022-03-21T19:55:08.276+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:08 smithi016 ceph-mon[22010]: pgmap v4889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:08 smithi079 ceph-mon[23158]: pgmap v4889: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:09.587 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:09 smithi016 conmon[32727]: 2022-03-21T19:55:09.241+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:10.588 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:10 smithi016 conmon[32727]: 2022-03-21T19:55:10.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:10 smithi016 ceph-mon[22010]: pgmap v4890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:10 smithi079 ceph-mon[23158]: pgmap v4890: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:11.594 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:11 smithi016 conmon[32727]: 2022-03-21T19:55:11.265+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:12.595 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:12 smithi016 conmon[32727]: 2022-03-21T19:55:12.238+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:12 smithi016 ceph-mon[22010]: pgmap v4891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:12 smithi079 ceph-mon[23158]: pgmap v4891: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:13.601 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:13 smithi016 conmon[32727]: 2022-03-21T19:55:13.254+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:14.602 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:14 smithi016 conmon[32727]: 2022-03-21T19:55:14.219+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:14 smithi016 ceph-mon[22010]: pgmap v4892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:14 smithi079 ceph-mon[23158]: pgmap v4892: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:15.611 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:15 smithi016 conmon[32727]: 2022-03-21T19:55:15.218+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:16.610 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:16 smithi016 conmon[32727]: 2022-03-21T19:55:16.220+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:16 smithi016 ceph-mon[22010]: pgmap v4893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:16.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:16 smithi079 ceph-mon[23158]: pgmap v4893: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:17.615 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:17 smithi016 conmon[32727]: 2022-03-21T19:55:17.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:18.616 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:18 smithi016 conmon[32727]: 2022-03-21T19:55:18.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:18 smithi016 ceph-mon[22010]: pgmap v4894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:18 smithi079 ceph-mon[23158]: pgmap v4894: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:19.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:19 smithi016 conmon[32727]: 2022-03-21T19:55:19.349+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:20.622 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:20 smithi016 conmon[32727]: 2022-03-21T19:55:20.308+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:20 smithi016 ceph-mon[22010]: pgmap v4895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:20 smithi079 ceph-mon[23158]: pgmap v4895: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9160 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:21.637 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:21 smithi016 conmon[32727]: 2022-03-21T19:55:21.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:22.629 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:22 smithi016 conmon[32727]: 2022-03-21T19:55:22.329+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:22 smithi016 ceph-mon[22010]: pgmap v4896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:22 smithi079 ceph-mon[23158]: pgmap v4896: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:23.635 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:23 smithi016 conmon[32727]: 2022-03-21T19:55:23.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:23.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:24.636 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:24 smithi016 conmon[32727]: 2022-03-21T19:55:24.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:24 smithi016 ceph-mon[22010]: pgmap v4897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:24 smithi079 ceph-mon[23158]: pgmap v4897: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:25.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:25 smithi016 conmon[32727]: 2022-03-21T19:55:25.402+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:26.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:26 smithi016 conmon[32727]: 2022-03-21T19:55:26.356+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:26 smithi016 ceph-mon[22010]: pgmap v4898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:26 smithi079 ceph-mon[23158]: pgmap v4898: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:27.644 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:27 smithi016 conmon[32727]: 2022-03-21T19:55:27.316+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:27 smithi016 ceph-mon[22010]: pgmap v4899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:27 smithi079 ceph-mon[23158]: pgmap v4899: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:28.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:28 smithi016 conmon[32727]: 2022-03-21T19:55:28.341+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:29.650 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:29 smithi016 conmon[32727]: 2022-03-21T19:55:29.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:29 smithi016 ceph-mon[22010]: pgmap v4900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:29.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:29 smithi079 ceph-mon[23158]: pgmap v4900: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:30.652 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:30 smithi016 conmon[32727]: 2022-03-21T19:55:30.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:31.657 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:31 smithi016 conmon[32727]: 2022-03-21T19:55:31.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:31 smithi016 ceph-mon[22010]: pgmap v4901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:31 smithi079 ceph-mon[23158]: pgmap v4901: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:32.658 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:32 smithi016 conmon[32727]: 2022-03-21T19:55:32.284+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:33 smithi016 conmon[32727]: 2022-03-21T19:55:33.264+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:55:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:55:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:33 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:55:33.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:33 smithi016 ceph-mon[22010]: pgmap v4902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:33.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:55:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:55:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:33 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:55:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:33 smithi079 ceph-mon[23158]: pgmap v4902: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:33.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:34 smithi016 conmon[32727]: 2022-03-21T19:55:34.249+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:35.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[32727]: 2022-03-21T19:55:35.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:35 smithi079 ceph-mon[23158]: pgmap v4903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:35 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:55:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:35.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:35 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:55:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:35 smithi016 ceph-mon[22010]: pgmap v4903: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:36.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:35 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:55:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:36.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:35 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:55:36.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: 2022-03-21T19:55:35.756+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:36.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:55:36.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:36.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: yield 2022-03-21T19:55:36.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:36.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:36.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:36.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:55:36.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:36.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:55:36.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:36.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:55:36.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:36.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:55:36.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:36.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:55:36.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:36.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:36.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:35 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:36.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:36 smithi016 conmon[32727]: 2022-03-21T19:55:36.264+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 171.7M 2022-03-21T19:55:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 180107026: error parsing value: Value '180107026' is below minimum 939524096 2022-03-21T19:55:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:55:36.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:55:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:55:36.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:55:36.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:36.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:55:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:55:36.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:55:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:36.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:55:36.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:36.987 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:55:36.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 171.7M 2022-03-21T19:55:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 180107026: error parsing value: Value '180107026' is below minimum 939524096 2022-03-21T19:55:37.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.169 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T19:55:37.171 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T19:55:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T19:55:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:37.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:55:37.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:37.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:55:37.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:37.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:37.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:37.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:55:37.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:37.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:55:37.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:37.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:55:37.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:37.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:37.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:37.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:55:37.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:37.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:37.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:37.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:55:37.179 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:37 smithi016 conmon[32727]: 2022-03-21T19:55:37.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:37 smithi079 ceph-mon[23158]: pgmap v4904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:37 smithi016 ceph-mon[22010]: pgmap v4904: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:38.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[32727]: 2022-03-21T19:55:38.356+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:39.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 2022-03-21T19:55:38.828+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:55:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:39.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: yield 2022-03-21T19:55:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:39.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:55:39.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:55:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:39.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:55:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:55:39.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:55:39.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 2022-03-21T19:55:38.829+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:55:39.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:55:39.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:39.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: yield 2022-03-21T19:55:39.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:39.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:39.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:39.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:55:39.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:39.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:55:39.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:39.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:55:39.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:39.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:55:39.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:39.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:55:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:39.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 2022-03-21T19:55:39.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:55:39.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: 2022-03-21T19:55:39.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:55:39.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:55:39.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:55:39.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:55:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:55:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:55:39.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:55:39.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:55:39.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:55:39.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:55:39.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:55:39.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:55:39.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:55:39.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:55:39.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:55:39.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:55:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:55:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:55:39.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:55:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:55:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:55:39.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:55:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise 2022-03-21T19:55:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:55:39.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:55:39.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:55:39.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:55:39.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:55:38 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:39 smithi016 conmon[32727]: 2022-03-21T19:55:39.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:39 smithi079 ceph-mon[23158]: pgmap v4905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:39 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:55:39.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:55:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:39.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:55:39.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:55:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:39.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:55:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:39.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:55:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:39.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:39 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:55:39.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:55:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:55:39.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:39.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:55:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:55:39.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:55:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:39.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:55:39.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:55:39.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:55:39.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:55:39.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:55:39.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:55:39.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:55:39.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:55:39.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:55:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:55:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:55:39.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:55:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:55:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:55:39.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:55:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:55:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:55:39.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:55:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:55:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:55:39.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:55:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:55:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:55:39.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:55:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:55:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:55:39.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:55:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:55:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:55:39.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:55:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:55:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:39.997 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:39 smithi016 ceph-mon[22010]: pgmap v4905: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:39 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:40.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:55:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:55:40.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:40.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:55:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:40.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:55:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:55:40.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:40.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:55:40.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:40.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:39 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:55:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:55:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:55:40.173 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:55:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:55:40.174 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:55:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:55:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:55:40.175 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:55:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:55:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:55:40.176 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:55:40.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:55:40.177 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:55:40.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:55:40.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:55:40.178 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:55:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:55:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:40.179 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:55:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:55:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:55:40.180 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:55:40.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:55:40.181 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:55:40.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:55:40.182 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:55:40.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:55:40.183 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:55:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:55:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:55:40.184 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:55:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:55:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:55:40.185 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:55:40.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:55:40.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:55:40.186 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:55:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:55:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:55:40.187 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:55:40.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:55:40.188 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:55:40.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:55:40.189 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:55:40.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:55:40.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:55:40.191 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:55:40.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:55:40.192 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:55:40.192 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:40 smithi016 conmon[32727]: 2022-03-21T19:55:40.356+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:41.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9180 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:41.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:41 smithi016 conmon[32727]: 2022-03-21T19:55:41.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:41 smithi079 ceph-mon[23158]: pgmap v4906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:41 smithi016 ceph-mon[22010]: pgmap v4906: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:42.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:42 smithi016 conmon[32727]: 2022-03-21T19:55:42.326+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:43.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:43 smithi016 conmon[32727]: 2022-03-21T19:55:43.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:43 smithi079 ceph-mon[23158]: pgmap v4907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:44.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:43 smithi016 ceph-mon[22010]: pgmap v4907: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:44.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:44 smithi016 conmon[32727]: 2022-03-21T19:55:44.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:45 smithi016 conmon[32727]: 2022-03-21T19:55:45.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:45 smithi079 ceph-mon[23158]: pgmap v4908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:45.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:46.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:45 smithi016 ceph-mon[22010]: pgmap v4908: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:46.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:46.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:46 smithi016 conmon[32727]: 2022-03-21T19:55:46.304+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:47.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:47.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:47 smithi016 conmon[32727]: 2022-03-21T19:55:47.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:47 smithi079 ceph-mon[23158]: pgmap v4909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:47.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:47 smithi016 ceph-mon[22010]: pgmap v4909: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:48.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:48.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:48 smithi016 conmon[32727]: 2022-03-21T19:55:48.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:49.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:49 smithi016 conmon[32727]: 2022-03-21T19:55:49.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:49 smithi079 ceph-mon[23158]: pgmap v4910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:49 smithi016 ceph-mon[22010]: pgmap v4910: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:50.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:50 smithi016 conmon[32727]: 2022-03-21T19:55:50.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:51.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:51 smithi016 conmon[32727]: 2022-03-21T19:55:51.307+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:51 smithi079 ceph-mon[23158]: pgmap v4911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:51 smithi016 ceph-mon[22010]: pgmap v4911: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:52.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:52.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:52 smithi016 conmon[32727]: 2022-03-21T19:55:52.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:53.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:53.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:53 smithi016 conmon[32727]: 2022-03-21T19:55:53.279+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:54.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:53 smithi016 ceph-mon[22010]: pgmap v4912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:54.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:53 smithi079 ceph-mon[23158]: pgmap v4912: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:54.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:54 smithi016 conmon[32727]: 2022-03-21T19:55:54.260+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:55.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:55.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:55 smithi016 conmon[32727]: 2022-03-21T19:55:55.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:56.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:55 smithi016 ceph-mon[22010]: pgmap v4913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:56.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:55 smithi079 ceph-mon[23158]: pgmap v4913: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:55:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:55:56.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:56.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:56 smithi016 conmon[32727]: 2022-03-21T19:55:56.300+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:57.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:57.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:57 smithi016 conmon[32727]: 2022-03-21T19:55:57.262+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:57 smithi016 ceph-mon[22010]: pgmap v4914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:58.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:57 smithi079 ceph-mon[23158]: pgmap v4914: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:55:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:58 smithi016 conmon[32727]: 2022-03-21T19:55:58.257+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:55:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:55:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:55:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:55:59.168 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:55:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:55:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:55:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:55:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:55:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:55:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:55:59 smithi016 conmon[32727]: 2022-03-21T19:55:59.269+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:00.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:59 smithi016 ceph-mon[22010]: pgmap v4915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:00.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:55:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:59 smithi079 ceph-mon[23158]: pgmap v4915: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:00.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:55:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:00 smithi016 conmon[32727]: 2022-03-21T19:56:00.251+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9200 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:01.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:01 smithi016 conmon[32727]: 2022-03-21T19:56:01.270+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:02.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:01 smithi016 ceph-mon[22010]: pgmap v4916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:02.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:02.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:01 smithi079 ceph-mon[23158]: pgmap v4916: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:02.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:02 smithi016 conmon[32727]: 2022-03-21T19:56:02.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:03.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:03.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:03 smithi016 conmon[32727]: 2022-03-21T19:56:03.190+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:03 smithi016 ceph-mon[22010]: pgmap v4917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:04.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:03 smithi079 ceph-mon[23158]: pgmap v4917: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:04.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:04 smithi016 conmon[32727]: 2022-03-21T19:56:04.166+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:05.152 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:05 smithi016 conmon[32727]: 2022-03-21T19:56:05.151+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:06.122 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:05 smithi016 ceph-mon[22010]: pgmap v4918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:06.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:06.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:05 smithi079 ceph-mon[23158]: pgmap v4918: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:06.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:06.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:06 smithi016 conmon[32727]: 2022-03-21T19:56:06.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:07.163 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:07.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:07.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:07 smithi016 conmon[32727]: 2022-03-21T19:56:07.162+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:07 smithi016 ceph-mon[22010]: pgmap v4919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:08.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:07 smithi079 ceph-mon[23158]: pgmap v4919: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:08.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:08 smithi016 conmon[32727]: 2022-03-21T19:56:08.175+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:09.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:09 smithi016 conmon[32727]: 2022-03-21T19:56:09.180+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:10.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:09 smithi016 ceph-mon[22010]: pgmap v4920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:10.147 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:09 smithi079 ceph-mon[23158]: pgmap v4920: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:10.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:10 smithi016 conmon[32727]: 2022-03-21T19:56:10.146+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:11.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:11.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:11 smithi016 conmon[32727]: 2022-03-21T19:56:11.185+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:11 smithi016 ceph-mon[22010]: pgmap v4921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:12.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:11 smithi079 ceph-mon[23158]: pgmap v4921: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:12.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:12 smithi016 conmon[32727]: 2022-03-21T19:56:12.194+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:13.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:13 smithi016 conmon[32727]: 2022-03-21T19:56:13.212+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:13 smithi016 ceph-mon[22010]: pgmap v4922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:14.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:13 smithi079 ceph-mon[23158]: pgmap v4922: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:14.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:14 smithi016 conmon[32727]: 2022-03-21T19:56:14.189+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:15.154 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:15.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:15 smithi016 conmon[32727]: 2022-03-21T19:56:15.153+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:16.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:15 smithi016 ceph-mon[22010]: pgmap v4923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:16.110 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:16.111 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:16.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:15 smithi079 ceph-mon[23158]: pgmap v4923: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:16.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:16 smithi016 conmon[32727]: 2022-03-21T19:56:16.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:17.087 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:17 smithi016 conmon[32727]: 2022-03-21T19:56:17.086+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:18.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:17 smithi016 ceph-mon[22010]: pgmap v4924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:18.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:17 smithi079 ceph-mon[23158]: pgmap v4924: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:18 smithi016 conmon[32727]: 2022-03-21T19:56:18.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:19.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:19.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:19 smithi016 conmon[32727]: 2022-03-21T19:56:19.022+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:20.066 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:19 smithi016 ceph-mon[22010]: pgmap v4925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:20.066 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:19 smithi079 ceph-mon[23158]: pgmap v4925: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:20.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:20 smithi016 conmon[32727]: 2022-03-21T19:56:20.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:21.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:21.170 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:21.171 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:21 smithi016 conmon[32727]: 2022-03-21T19:56:21.040+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9220 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:21 smithi016 ceph-mon[22010]: pgmap v4926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:22.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:22.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:21 smithi016 conmon[32727]: 2022-03-21T19:56:21.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:21 smithi079 ceph-mon[23158]: pgmap v4926: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:22.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:23.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:23.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:23 smithi016 conmon[32727]: 2022-03-21T19:56:23.044+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:24.093 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:23 smithi016 ceph-mon[22010]: pgmap v4927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:24.094 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:23 smithi079 ceph-mon[23158]: pgmap v4927: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:24 smithi016 conmon[32727]: 2022-03-21T19:56:24.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:25 smithi016 conmon[32727]: 2022-03-21T19:56:25.065+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:25.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:26.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:26 smithi016 conmon[32727]: 2022-03-21T19:56:26.056+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:26.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:25 smithi016 ceph-mon[22010]: pgmap v4928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:26.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:26.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:25 smithi079 ceph-mon[23158]: pgmap v4928: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:26.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:27.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:27.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:27 smithi016 conmon[32727]: 2022-03-21T19:56:27.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:27.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:28.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:28 smithi016 conmon[32727]: 2022-03-21T19:56:28.083+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:28.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:27 smithi016 ceph-mon[22010]: pgmap v4929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:28.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:28.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:27 smithi079 ceph-mon[23158]: pgmap v4929: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:28.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:29.095 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:29.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:29 smithi016 conmon[32727]: 2022-03-21T19:56:29.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:30.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:29 smithi016 ceph-mon[22010]: pgmap v4930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:30.098 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:29 smithi079 ceph-mon[23158]: pgmap v4930: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:30.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:30 smithi016 conmon[32727]: 2022-03-21T19:56:30.092+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:31.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:31.123 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:31.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:31.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:31 smithi016 conmon[32727]: 2022-03-21T19:56:31.121+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:32.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:32 smithi016 ceph-mon[22010]: pgmap v4931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:32.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:32.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:32 smithi016 conmon[32727]: 2022-03-21T19:56:32.156+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:32 smithi079 ceph-mon[23158]: pgmap v4931: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:33.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:33.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:33 smithi016 conmon[32727]: 2022-03-21T19:56:33.134+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:33.482 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:34 smithi016 ceph-mon[22010]: pgmap v4932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:34.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:34.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:34 smithi016 conmon[32727]: 2022-03-21T19:56:34.176+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:34 smithi079 ceph-mon[23158]: pgmap v4932: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:35.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:35.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:35.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:35 smithi016 conmon[32727]: 2022-03-21T19:56:35.142+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:36.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:36 smithi016 ceph-mon[22010]: pgmap v4933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:36.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:36.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:36 smithi016 conmon[32727]: 2022-03-21T19:56:36.160+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:36.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:36 smithi079 ceph-mon[23158]: pgmap v4933: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:36.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:37.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:37.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:37 smithi016 conmon[32727]: 2022-03-21T19:56:37.137+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:38.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:38 smithi016 ceph-mon[22010]: pgmap v4934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:38.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:38.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:38 smithi016 conmon[32727]: 2022-03-21T19:56:38.187+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:38 smithi079 ceph-mon[23158]: pgmap v4934: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:38.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:56:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:56:39.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:39 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:56:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:39.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:56:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:56:39.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:39 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:56:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:39 smithi016 conmon[32727]: 2022-03-21T19:56:39.200+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:40.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:40 smithi016 ceph-mon[22010]: pgmap v4935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:40.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:40.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:40 smithi016 conmon[32727]: 2022-03-21T19:56:40.250+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:40 smithi079 ceph-mon[23158]: pgmap v4935: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9240 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:41.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:41.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[32727]: 2022-03-21T19:56:41.274+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:42.270 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:42 smithi016 ceph-mon[22010]: pgmap v4936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:42.271 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:42.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: 2022-03-21T19:56:41.932+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:42.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:56:42.272 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:42.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: yield 2022-03-21T19:56:42.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:42.273 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:56:42.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:42.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:56:42.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:42.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:56:42.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:42.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:56:42.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:42.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:56:42.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:42.278 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:42.278 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:41 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:42.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:42 smithi079 ceph-mon[23158]: pgmap v4936: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:42.479 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:42 smithi016 conmon[32727]: 2022-03-21T19:56:42.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:43.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:43 smithi016 conmon[32727]: 2022-03-21T19:56:43.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:43 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:43.415 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:56:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:56:43.416 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:43.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:56:43.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:56:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:43.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:56:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:43.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:56:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:43.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:43 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:56:43.422 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:43 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:56:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:56:43.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:43.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:56:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:43.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:56:43.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:43.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:56:43.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:43.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:56:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:43.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:43.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:43 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:56:43.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:44.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:44 smithi016 conmon[32727]: 2022-03-21T19:56:44.247+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:44 smithi016 ceph-mon[22010]: pgmap v4937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:44.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:44 smithi079 ceph-mon[23158]: pgmap v4937: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:45.417 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[32727]: 2022-03-21T19:56:45.237+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:45.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 2022-03-21T19:56:45.004+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:56:45.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:45.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: yield 2022-03-21T19:56:45.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:45.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:45.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:56:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:45.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:56:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:56:45.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:45.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:56:45.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:45.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:56:45.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:45.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 2022-03-21T19:56:45.005+0000 7f92786ab700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:56:45.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:56:45.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:45.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: yield 2022-03-21T19:56:45.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:45.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:56:45.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:56:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:45.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:56:45.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:45.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:56:45.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:45.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:56:45.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:45.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:45.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 2022-03-21T19:56:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:56:45.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: 2022-03-21T19:56:45.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:56:45.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:56:45.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:56:45.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:56:45.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:56:45.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:56:45.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:56:45.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:56:45.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:56:45.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:56:45.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:56:45.443 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:56:45.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:56:45.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:56:45.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:56:45.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:56:45.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:56:45.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:56:45.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:56:45.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:56:45.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:56:45.447 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:56:45.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise 2022-03-21T19:56:45.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:56:45.448 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:56:45.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:56:45.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:56:45.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:56:45 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:45.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:46.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:46 smithi016 conmon[32727]: 2022-03-21T19:56:46.278+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:46 smithi016 ceph-mon[22010]: pgmap v4938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:46.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.417 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:46.418 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:56:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:46.419 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:46.420 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:56:46.421 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:46.422 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:56:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:46.423 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:56:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:46.424 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:46.425 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:56:46.426 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:46.427 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.428 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:46 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:56:46.429 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:46.430 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:56:46.431 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:46.432 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:46.433 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:56:46.434 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:46.435 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:56:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:46.436 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:56:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:46.437 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:56:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:46.438 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:56:46.439 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:56:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:56:46.440 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:56:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:56:46.441 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:56:46.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:56:46.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:56:46.442 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:56:46.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:56:46.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:56:46.443 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:56:46.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:56:46.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:56:46.444 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:56:46.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:56:46.445 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:56:46.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:56:46.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:56:46.446 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:56:46.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:56:46.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:56:46.447 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:56:46.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:56:46.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:56:46.448 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:56:46.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:56:46.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:56:46.449 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:56:46.450 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.450 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:46 smithi079 ceph-mon[23158]: pgmap v4938: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:56:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:46.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:56:46.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:46.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:56:46.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:56:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:46.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:46.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:46.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:56:46.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:46.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:46 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:56:46.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:56:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:56:46.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:56:46.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:56:46.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:56:46.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:56:46.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:56:46.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:56:46.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:56:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:56:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:56:46.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:56:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:56:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:56:46.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:56:46.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:56:46.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:56:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:56:46.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:56:46.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:56:46.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:56:46.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:56:46.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:56:46.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:56:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:56:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:56:46.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:56:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:56:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:56:46.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:56:46.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:56:46.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:56:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:56:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:56:46.499 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:56:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:56:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:56:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:56:46.500 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:56:46.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:56:46.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:56:46.501 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:56:46.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:56:46.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:56:46.502 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:56:46.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:56:46.503 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:56:46.503 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:47 smithi016 conmon[32727]: 2022-03-21T19:56:47.325+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:48.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:48 smithi016 conmon[32727]: 2022-03-21T19:56:48.313+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:48 smithi016 ceph-mon[22010]: pgmap v4939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:48 smithi079 ceph-mon[23158]: pgmap v4939: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:48.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:49.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:49 smithi016 conmon[32727]: 2022-03-21T19:56:49.274+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:49.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:50 smithi016 ceph-mon[22010]: pgmap v4940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:50.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:50.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:50 smithi016 conmon[32727]: 2022-03-21T19:56:50.256+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:50 smithi079 ceph-mon[23158]: pgmap v4940: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:50.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:51 smithi016 conmon[32727]: 2022-03-21T19:56:51.228+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:52 smithi016 ceph-mon[22010]: pgmap v4941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:52.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:52 smithi016 conmon[32727]: 2022-03-21T19:56:52.271+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:52 smithi079 ceph-mon[23158]: pgmap v4941: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:53.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:53 smithi016 conmon[32727]: 2022-03-21T19:56:53.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:53.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:54 smithi016 ceph-mon[22010]: pgmap v4942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:54.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:54.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:54 smithi016 conmon[32727]: 2022-03-21T19:56:54.279+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:54 smithi079 ceph-mon[23158]: pgmap v4942: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:55.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:55.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:55 smithi016 conmon[32727]: 2022-03-21T19:56:55.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:56:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:56 smithi016 ceph-mon[22010]: pgmap v4943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:56.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:56 smithi016 conmon[32727]: 2022-03-21T19:56:56.335+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:56 smithi079 ceph-mon[23158]: pgmap v4943: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:56:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:57.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:57 smithi016 conmon[32727]: 2022-03-21T19:56:57.363+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:58.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:58 smithi016 ceph-mon[22010]: pgmap v4944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:58.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:58.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:58 smithi016 conmon[32727]: 2022-03-21T19:56:58.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:58.477 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:58 smithi079 ceph-mon[23158]: pgmap v4944: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:56:58.478 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:56:59 smithi016 conmon[32727]: 2022-03-21T19:56:59.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:56:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:59.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:56:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:56:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:56:59.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:56:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:56:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:56:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:56:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:56:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:56:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:56:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:57:00.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:00 smithi016 ceph-mon[22010]: pgmap v4945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:00.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:00.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:00 smithi016 conmon[32727]: 2022-03-21T19:57:00.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:00 smithi079 ceph-mon[23158]: pgmap v4945: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9260 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:01.403 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:01.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:01 smithi016 conmon[32727]: 2022-03-21T19:57:01.402+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:02.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:02 smithi016 ceph-mon[22010]: pgmap v4946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:02.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:02.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:02 smithi016 conmon[32727]: 2022-03-21T19:57:02.365+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:02 smithi079 ceph-mon[23158]: pgmap v4946: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:03.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:03.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:03 smithi016 conmon[32727]: 2022-03-21T19:57:03.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:04 smithi016 ceph-mon[22010]: pgmap v4947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:04.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:04.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:04 smithi016 conmon[32727]: 2022-03-21T19:57:04.375+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:04 smithi079 ceph-mon[23158]: pgmap v4947: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:04.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:05.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:05.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:05 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:05.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:05 smithi016 conmon[32727]: 2022-03-21T19:57:05.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:05 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:06.417 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:06 smithi016 ceph-mon[22010]: pgmap v4948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:06.418 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:06.419 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:06 smithi016 conmon[32727]: 2022-03-21T19:57:06.357+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:06 smithi079 ceph-mon[23158]: pgmap v4948: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:07 smithi016 conmon[32727]: 2022-03-21T19:57:07.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:08 smithi079 ceph-mon[23158]: pgmap v4949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:08 smithi016 ceph-mon[22010]: pgmap v4949: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:08 smithi016 conmon[32727]: 2022-03-21T19:57:08.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:09 smithi016 conmon[32727]: 2022-03-21T19:57:09.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:10 smithi079 ceph-mon[23158]: pgmap v4950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:10.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:10 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:10 smithi016 ceph-mon[22010]: pgmap v4950: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:10 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:10 smithi016 conmon[32727]: 2022-03-21T19:57:10.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:11 smithi016 conmon[32727]: 2022-03-21T19:57:11.291+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:12 smithi079 ceph-mon[23158]: pgmap v4951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:12 smithi016 ceph-mon[22010]: pgmap v4951: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:12 smithi016 conmon[32727]: 2022-03-21T19:57:12.271+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:13 smithi016 conmon[32727]: 2022-03-21T19:57:13.292+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:14 smithi079 ceph-mon[23158]: pgmap v4952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:14 smithi016 ceph-mon[22010]: pgmap v4952: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:14 smithi016 conmon[32727]: 2022-03-21T19:57:14.293+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:15 smithi016 conmon[32727]: 2022-03-21T19:57:15.272+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:16 smithi079 ceph-mon[23158]: pgmap v4953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:16 smithi016 ceph-mon[22010]: pgmap v4953: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:16 smithi016 conmon[32727]: 2022-03-21T19:57:16.294+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:17 smithi016 conmon[32727]: 2022-03-21T19:57:17.327+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:18 smithi079 ceph-mon[23158]: pgmap v4954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:18 smithi016 ceph-mon[22010]: pgmap v4954: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:18 smithi016 conmon[32727]: 2022-03-21T19:57:18.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:19.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:19 smithi016 conmon[32727]: 2022-03-21T19:57:19.359+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:20 smithi079 ceph-mon[23158]: pgmap v4955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:20 smithi016 ceph-mon[22010]: pgmap v4955: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9280 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:20 smithi016 conmon[32727]: 2022-03-21T19:57:20.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:21 smithi016 conmon[32727]: 2022-03-21T19:57:21.347+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:22 smithi079 ceph-mon[23158]: pgmap v4956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:22 smithi016 ceph-mon[22010]: pgmap v4956: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:22 smithi016 conmon[32727]: 2022-03-21T19:57:22.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:23 smithi016 conmon[32727]: 2022-03-21T19:57:23.410+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:24 smithi016 ceph-mon[22010]: pgmap v4957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:24.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:24 smithi016 conmon[32727]: 2022-03-21T19:57:24.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:24 smithi079 ceph-mon[23158]: pgmap v4957: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:25.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:25 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:25.670 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:25 smithi016 conmon[32727]: 2022-03-21T19:57:25.440+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:25 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:26.485 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:26 smithi016 ceph-mon[22010]: pgmap v4958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:26.486 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:26 smithi079 ceph-mon[23158]: pgmap v4958: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:26 smithi016 conmon[32727]: 2022-03-21T19:57:26.482+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:27.513 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:27 smithi016 conmon[32727]: 2022-03-21T19:57:27.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:28.518 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:28 smithi016 ceph-mon[22010]: pgmap v4959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:28.519 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:28 smithi079 ceph-mon[23158]: pgmap v4959: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:28 smithi016 conmon[32727]: 2022-03-21T19:57:28.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:29.524 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:29 smithi016 conmon[32727]: 2022-03-21T19:57:29.521+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:30 smithi016 ceph-mon[22010]: pgmap v4960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:30 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:30 smithi016 conmon[32727]: 2022-03-21T19:57:30.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:30 smithi079 ceph-mon[23158]: pgmap v4960: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:30 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:31.521 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:31.521 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:31 smithi016 conmon[32727]: 2022-03-21T19:57:31.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:32.565 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:32 smithi016 ceph-mon[22010]: pgmap v4961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:32.566 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:32 smithi079 ceph-mon[23158]: pgmap v4961: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:32 smithi016 conmon[32727]: 2022-03-21T19:57:32.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:33 smithi016 conmon[32727]: 2022-03-21T19:57:33.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:34.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:34.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:34 smithi016 ceph-mon[22010]: pgmap v4962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:34 smithi079 ceph-mon[23158]: pgmap v4962: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:34 smithi016 conmon[32727]: 2022-03-21T19:57:34.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:35.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:35.579 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:35 smithi016 conmon[32727]: 2022-03-21T19:57:35.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:36.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:36.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:36 smithi016 ceph-mon[22010]: pgmap v4963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:36 smithi079 ceph-mon[23158]: pgmap v4963: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:36 smithi016 conmon[32727]: 2022-03-21T19:57:36.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:37.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:37 smithi016 conmon[32727]: 2022-03-21T19:57:37.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:38.523 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:38.524 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:38 smithi016 ceph-mon[22010]: pgmap v4964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:38 smithi079 ceph-mon[23158]: pgmap v4964: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:38 smithi016 conmon[32727]: 2022-03-21T19:57:38.521+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:39.549 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:39.549 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:39 smithi016 conmon[32727]: 2022-03-21T19:57:39.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:40 smithi016 conmon[32727]: 2022-03-21T19:57:40.513+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:40 smithi016 ceph-mon[22010]: pgmap v4965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:40 smithi079 ceph-mon[23158]: pgmap v4965: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9300 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:41 smithi016 conmon[32727]: 2022-03-21T19:57:41.478+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:41.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:42 smithi016 ceph-mon[22010]: pgmap v4966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:42 smithi016 conmon[32727]: 2022-03-21T19:57:42.465+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:42 smithi079 ceph-mon[23158]: pgmap v4966: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:43 smithi016 conmon[32727]: 2022-03-21T19:57:43.469+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:43.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:44 smithi016 ceph-mon[22010]: pgmap v4967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:44 smithi016 conmon[32727]: 2022-03-21T19:57:44.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:44 smithi079 ceph-mon[23158]: pgmap v4967: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:45.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:57:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:57:45.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:45 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:57:45.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:45 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:45.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:45.462 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:57:45.462 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:57:45.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:45 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:57:45.463 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:45 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[32727]: 2022-03-21T19:57:45.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: 2022-03-21T19:57:45.612+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:57:45.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: yield 2022-03-21T19:57:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:45.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:45.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:57:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:57:45.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:57:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:45.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:57:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:57:45.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:45.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:45 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:46 smithi016 conmon[32727]: 2022-03-21T19:57:46.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:46 smithi016 ceph-mon[22010]: pgmap v4968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:46 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:57:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:57:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:46.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:57:46.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:57:46.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:57:46.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:46.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:57:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:46.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:46.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:46 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:57:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:46 smithi079 ceph-mon[23158]: pgmap v4968: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:46 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:57:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:57:46.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:46.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:57:46.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:57:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:46.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:57:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:46.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:57:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:46.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:46.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:46 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:57:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:47 smithi016 conmon[32727]: 2022-03-21T19:57:47.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:48 smithi016 ceph-mon[22010]: pgmap v4969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[32727]: 2022-03-21T19:57:48.399+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:48 smithi079 ceph-mon[23158]: pgmap v4969: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:49.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 2022-03-21T19:57:48.685+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:49.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:57:49.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:49.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: yield 2022-03-21T19:57:49.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:49.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:49.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:49.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:57:49.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:49.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:57:49.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:49.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:57:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:57:49.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:57:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:49.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:49.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:49.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 2022-03-21T19:57:48.685+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:57:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:57:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:49.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: yield 2022-03-21T19:57:49.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:49.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:49.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:49.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:57:49.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:49.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:57:49.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:49.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:57:49.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:49.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:57:49.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:49.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:57:49.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:49.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:49.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:49.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 2022-03-21T19:57:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:57:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: 2022-03-21T19:57:49.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:57:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:57:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:57:49.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:57:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:57:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:57:49.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:57:49.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:57:49.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:57:49.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:57:49.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:57:49.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:57:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:57:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:57:49.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:57:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:57:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:57:49.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:57:49.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:57:49.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:57:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:57:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:57:49.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise 2022-03-21T19:57:49.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:57:49.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:57:49.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:57:49.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:57:49.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:57:48 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:49 smithi016 conmon[32727]: 2022-03-21T19:57:49.380+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:50.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:50 smithi016 conmon[32727]: 2022-03-21T19:57:50.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:50 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:57:50.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:50.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:50.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:50.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:57:50.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:50.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:57:50.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:50.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:57:50.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:50.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:50.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:50.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:57:50.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:50.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:50.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:50 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:57:50.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:50.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:57:50.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:50.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:50.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:50.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:57:50.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:50.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:57:50.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:50.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:57:50.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:50.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:50.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:50.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:57:50.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:50.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:50.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:57:50.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:57:50.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:57:50.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:57:50.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:57:50.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:57:50.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:57:50.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:57:50.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:57:50.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:57:50.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:57:50.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:57:50.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:57:50.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:57:50.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:57:50.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:57:50.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:57:50.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:57:50.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:57:50.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:57:50.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:57:50.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:57:50.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:57:50.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:57:50.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:57:50.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:57:50.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:57:50.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:57:50.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:57:50.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:50 smithi016 ceph-mon[22010]: pgmap v4970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:50.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:50.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:50 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:50 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:57:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:50.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:57:50.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:50.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:57:50.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:50.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:57:50.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:50.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:50.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:50.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:57:50.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:50.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:50.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:50 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:57:50.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:57:50.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:57:50.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:57:50.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:57:50.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:57:50.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:57:50.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:57:50.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:57:50.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:57:50.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:57:50.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:57:50.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:57:50.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:57:50.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:57:50.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:57:50.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:57:50.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:57:50.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:57:50.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:57:50.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:57:50.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:57:50.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:57:50.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:57:50.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:57:50.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:57:50.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:57:50.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:57:50.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:57:50.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:57:50.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:57:50.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:57:50.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:57:50.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:57:50.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:57:50.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:57:50.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:57:50.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:57:50.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:57:50.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:57:50.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:57:50.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:57:50.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:57:50.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:57:50.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:57:50.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:57:50.758 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:57:50.758 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:57:50.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:50 smithi079 ceph-mon[23158]: pgmap v4970: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:50.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:50.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:50 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:51.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:51 smithi016 conmon[32727]: 2022-03-21T19:57:51.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:51.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:52.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:52 smithi016 conmon[32727]: 2022-03-21T19:57:52.410+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:52 smithi016 ceph-mon[22010]: pgmap v4971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:52 smithi079 ceph-mon[23158]: pgmap v4971: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:53.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:53 smithi016 conmon[32727]: 2022-03-21T19:57:53.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:54 smithi016 ceph-mon[22010]: pgmap v4972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:54 smithi016 conmon[32727]: 2022-03-21T19:57:54.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:54 smithi079 ceph-mon[23158]: pgmap v4972: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:55.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:55.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:55 smithi016 conmon[32727]: 2022-03-21T19:57:55.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:57:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:56 smithi016 ceph-mon[22010]: pgmap v4973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:56 smithi016 conmon[32727]: 2022-03-21T19:57:56.403+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:56 smithi079 ceph-mon[23158]: pgmap v4973: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:57:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:57.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:57 smithi016 conmon[32727]: 2022-03-21T19:57:57.415+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:58 smithi016 conmon[32727]: 2022-03-21T19:57:58.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:58 smithi016 ceph-mon[22010]: pgmap v4974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:58 smithi079 ceph-mon[23158]: pgmap v4974: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:57:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:57:59 smithi016 conmon[32727]: 2022-03-21T19:57:59.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:57:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:57:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:57:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:57:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:57:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:57:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:57:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:57:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:57:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:57:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:57:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:58:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:00 smithi016 ceph-mon[22010]: pgmap v4975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:00.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:00 smithi016 conmon[32727]: 2022-03-21T19:58:00.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:00 smithi079 ceph-mon[23158]: pgmap v4975: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:01 smithi016 conmon[32727]: 2022-03-21T19:58:01.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:02 smithi016 conmon[32727]: 2022-03-21T19:58:02.391+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:02 smithi016 ceph-mon[22010]: pgmap v4976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:02 smithi079 ceph-mon[23158]: pgmap v4976: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:03 smithi016 conmon[32727]: 2022-03-21T19:58:03.349+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:04 smithi016 ceph-mon[22010]: pgmap v4977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:04.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:04 smithi016 conmon[32727]: 2022-03-21T19:58:04.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:04 smithi079 ceph-mon[23158]: pgmap v4977: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:05 smithi016 conmon[32727]: 2022-03-21T19:58:05.368+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:06 smithi016 conmon[32727]: 2022-03-21T19:58:06.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:06 smithi016 ceph-mon[22010]: pgmap v4978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:06 smithi079 ceph-mon[23158]: pgmap v4978: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:07 smithi016 conmon[32727]: 2022-03-21T19:58:07.311+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:08 smithi016 conmon[32727]: 2022-03-21T19:58:08.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:08 smithi016 ceph-mon[22010]: pgmap v4979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:08 smithi079 ceph-mon[23158]: pgmap v4979: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:09 smithi016 conmon[32727]: 2022-03-21T19:58:09.284+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:09.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:10 smithi016 conmon[32727]: 2022-03-21T19:58:10.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:10 smithi016 ceph-mon[22010]: pgmap v4980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:10 smithi079 ceph-mon[23158]: pgmap v4980: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:11.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:11 smithi016 conmon[32727]: 2022-03-21T19:58:11.275+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:11.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:12 smithi016 ceph-mon[22010]: pgmap v4981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:12.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:12 smithi016 conmon[32727]: 2022-03-21T19:58:12.265+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:12 smithi079 ceph-mon[23158]: pgmap v4981: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:13 smithi016 conmon[32727]: 2022-03-21T19:58:13.303+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:14 smithi016 ceph-mon[22010]: pgmap v4982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:14.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:14.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:14 smithi016 conmon[32727]: 2022-03-21T19:58:14.286+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:14 smithi079 ceph-mon[23158]: pgmap v4982: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:15.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:15 smithi016 conmon[32727]: 2022-03-21T19:58:15.330+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:16 smithi016 ceph-mon[22010]: pgmap v4983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:16.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:16 smithi016 conmon[32727]: 2022-03-21T19:58:16.299+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:16 smithi079 ceph-mon[23158]: pgmap v4983: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:17 smithi016 conmon[32727]: 2022-03-21T19:58:17.285+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:18 smithi016 ceph-mon[22010]: pgmap v4984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:18 smithi016 conmon[32727]: 2022-03-21T19:58:18.306+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:18 smithi079 ceph-mon[23158]: pgmap v4984: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:19 smithi016 conmon[32727]: 2022-03-21T19:58:19.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:20 smithi016 conmon[32727]: 2022-03-21T19:58:20.353+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:20 smithi016 ceph-mon[22010]: pgmap v4985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:20 smithi079 ceph-mon[23158]: pgmap v4985: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:21 smithi016 conmon[32727]: 2022-03-21T19:58:21.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:22 smithi016 ceph-mon[22010]: pgmap v4986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:22.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:22 smithi016 conmon[32727]: 2022-03-21T19:58:22.312+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:22 smithi079 ceph-mon[23158]: pgmap v4986: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:23 smithi016 conmon[32727]: 2022-03-21T19:58:23.295+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:24 smithi016 ceph-mon[22010]: pgmap v4987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:24 smithi016 conmon[32727]: 2022-03-21T19:58:24.308+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:24 smithi079 ceph-mon[23158]: pgmap v4987: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T19:58:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:25 smithi016 conmon[32727]: 2022-03-21T19:58:25.321+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:26 smithi016 ceph-mon[22010]: pgmap v4988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:26.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:26 smithi016 conmon[32727]: 2022-03-21T19:58:26.344+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:26 smithi079 ceph-mon[23158]: pgmap v4988: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:27 smithi016 conmon[32727]: 2022-03-21T19:58:27.336+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:28 smithi016 ceph-mon[22010]: pgmap v4989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:28 smithi016 conmon[32727]: 2022-03-21T19:58:28.383+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:28 smithi079 ceph-mon[23158]: pgmap v4989: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:29 smithi016 conmon[32727]: 2022-03-21T19:58:29.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:30 smithi016 ceph-mon[22010]: pgmap v4990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:30 smithi016 conmon[32727]: 2022-03-21T19:58:30.354+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:30 smithi079 ceph-mon[23158]: pgmap v4990: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:31 smithi016 conmon[32727]: 2022-03-21T19:58:31.338+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:32 smithi016 ceph-mon[22010]: pgmap v4991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:32 smithi016 conmon[32727]: 2022-03-21T19:58:32.323+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:32 smithi079 ceph-mon[23158]: pgmap v4991: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:33 smithi016 conmon[32727]: 2022-03-21T19:58:33.281+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:34 smithi016 ceph-mon[22010]: pgmap v4992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:34 smithi016 conmon[32727]: 2022-03-21T19:58:34.305+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:34 smithi079 ceph-mon[23158]: pgmap v4992: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:35 smithi016 conmon[32727]: 2022-03-21T19:58:35.287+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:36 smithi016 ceph-mon[22010]: pgmap v4993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:36 smithi016 conmon[32727]: 2022-03-21T19:58:36.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:36 smithi079 ceph-mon[23158]: pgmap v4993: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:37 smithi016 conmon[32727]: 2022-03-21T19:58:37.383+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:38 smithi016 conmon[32727]: 2022-03-21T19:58:38.381+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:38 smithi016 ceph-mon[22010]: pgmap v4994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:38 smithi079 ceph-mon[23158]: pgmap v4994: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:39.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:39 smithi016 conmon[32727]: 2022-03-21T19:58:39.416+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:40 smithi016 conmon[32727]: 2022-03-21T19:58:40.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:40 smithi016 ceph-mon[22010]: pgmap v4995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:40 smithi079 ceph-mon[23158]: pgmap v4995: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:41 smithi016 conmon[32727]: 2022-03-21T19:58:41.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:42 smithi016 ceph-mon[22010]: pgmap v4996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:42.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:42 smithi016 conmon[32727]: 2022-03-21T19:58:42.459+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:42 smithi079 ceph-mon[23158]: pgmap v4996: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:43.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:43 smithi016 conmon[32727]: 2022-03-21T19:58:43.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:44 smithi016 ceph-mon[22010]: pgmap v4997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:44 smithi016 conmon[32727]: 2022-03-21T19:58:44.454+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:44 smithi079 ceph-mon[23158]: pgmap v4997: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:45 smithi016 conmon[32727]: 2022-03-21T19:58:45.415+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:46 smithi016 ceph-mon[22010]: pgmap v4998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:46 smithi016 conmon[32727]: 2022-03-21T19:58:46.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:46 smithi079 ceph-mon[23158]: pgmap v4998: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:47.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:47.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:47 smithi016 conmon[32727]: 2022-03-21T19:58:47.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:48.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:48 smithi016 ceph-mon[22010]: pgmap v4999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:48.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:48.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:48 smithi016 conmon[32727]: 2022-03-21T19:58:48.346+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:48.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:48 smithi079 ceph-mon[23158]: pgmap v4999: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:48.761 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:49.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:49 smithi016 conmon[32727]: 2022-03-21T19:58:49.380+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:58:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:58:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:58:49.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:58:49.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:49 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:58:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:58:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:58:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:58:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:58:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:49 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T19:58:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:50 smithi016 ceph-mon[22010]: pgmap v5000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:50 smithi016 conmon[32727]: 2022-03-21T19:58:50.392+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:50 smithi079 ceph-mon[23158]: pgmap v5000: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:50.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:51.790 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:51.791 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[32727]: 2022-03-21T19:58:51.418+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:52.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: 2022-03-21T19:58:51.789+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:52.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:58:52.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:52.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: yield 2022-03-21T19:58:52.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:52.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:52.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:52.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:58:52.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:52.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:58:52.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:52.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:58:52.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:52.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:58:52.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:52.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:58:52.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:52.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:52.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:51 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:52 smithi016 ceph-mon[22010]: pgmap v5001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:52 smithi016 conmon[32727]: 2022-03-21T19:58:52.414+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:52 smithi079 ceph-mon[23158]: pgmap v5001: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:53.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:53 smithi016 conmon[32727]: 2022-03-21T19:58:53.389+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:53 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:53.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:58:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:53.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:58:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:53.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:53.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:58:53.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:53.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:58:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:58:53.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:53.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:58:53.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:53.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:53.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:53.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:53 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:58:53.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:53 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:58:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:53.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:58:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:53.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:53.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:58:53.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:53.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:58:53.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:53.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:58:53.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:53.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:53.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:53.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:58:53.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:53.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:53.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:53.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:53 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:58:53.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:54.863 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[32727]: 2022-03-21T19:58:54.425+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:54.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:54 smithi016 ceph-mon[22010]: pgmap v5002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:54.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:54 smithi079 ceph-mon[23158]: pgmap v5002: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:55.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 2022-03-21T19:58:54.861+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:55.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:58:55.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:55.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: yield 2022-03-21T19:58:55.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:55.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:58:55.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:55.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:58:55.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:55.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:58:55.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:55.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:58:55.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:55.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:58:55.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:55.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:55.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:55.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 2022-03-21T19:58:54.862+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:58:55.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:58:55.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:55.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: yield 2022-03-21T19:58:55.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:55.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:55.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:55.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:58:55.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:55.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:58:55.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:55.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:58:55.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:55.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:58:55.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:55.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:58:55.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:55.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:55.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:55.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 2022-03-21T19:58:55.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:58:55.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: 2022-03-21T19:58:55.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:58:55.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:58:55.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:58:55.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:58:55.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:58:55.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:58:55.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:58:55.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:58:55.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:58:55.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:58:55.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:58:55.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:58:55.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:58:55.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:58:55.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:58:55.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:58:55.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:58:55.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:58:55.191 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:58:55.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:58:55.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:58:55.192 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:58:55.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise 2022-03-21T19:58:55.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:58:55.193 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:58:55.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:58:55.194 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:58:55.195 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:58:54 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:55 smithi016 conmon[32727]: 2022-03-21T19:58:55.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:58:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:56 smithi016 conmon[32727]: 2022-03-21T19:58:56.465+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:56.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:58:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:56.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:58:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:58:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:56.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:58:56.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:56.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:56.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:56.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:58:56.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:56.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:56.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.929 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:56 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:58:56.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:56.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:58:56.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:56.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:56.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:56.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:58:56.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:56.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:58:56.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:56.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:58:56.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:56.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:56.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:56.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:58:56.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:56.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:56.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:58:56.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:58:56.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:58:56.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:58:56.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:58:56.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:58:56.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:58:56.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:58:56.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:58:56.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:58:56.948 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:58:56.949 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:58:56.950 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:58:56.951 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:58:56.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:58:56.952 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:58:56.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:58:56.953 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:58:56.954 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:58:56.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:58:56.955 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:58:56.956 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:58:56.957 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:58:56.957 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:58:56.958 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:58:56.958 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:58:56.961 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:58:56.961 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:58:56.961 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:58:56.962 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.962 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:56 smithi016 ceph-mon[22010]: pgmap v5003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:56.963 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:58:56.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:56.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:58:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:58:56.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:58:56.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:56.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:58:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:56.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:56 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:58:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:58:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:58:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:58:56.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:58:56.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:58:56.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:58:56.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:58:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:58:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:58:56.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:58:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:58:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:58:56.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:58:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:58:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:58:56.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:58:56.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:58:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:58:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:58:56.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:58:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:58:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:58:56.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:58:56.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:58:56.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:58:56.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:58:56.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:58:56.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:58:56.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:58:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:58:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:58:56.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:58:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:58:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:58:56.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:58:56.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:58:56.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:58:56.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:58:56.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:58:56.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:58:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:58:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:58:56.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:58:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:58:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:58:56.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:58:56.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:58:56.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:56 smithi079 ceph-mon[23158]: pgmap v5003: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:58:56.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:57 smithi016 conmon[32727]: 2022-03-21T19:58:57.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:58 smithi016 ceph-mon[22010]: pgmap v5004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:58 smithi016 conmon[32727]: 2022-03-21T19:58:58.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:58 smithi079 ceph-mon[23158]: pgmap v5004: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:58:58.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:58:59 smithi016 conmon[32727]: 2022-03-21T19:58:59.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:58:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:58:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:58:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:58:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:58:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:58:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:58:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:58:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:58:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:58:59.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:58:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:59:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:00 smithi016 conmon[32727]: 2022-03-21T19:59:00.584+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:00.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:00 smithi016 ceph-mon[22010]: pgmap v5005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:00.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:00.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:00 smithi079 ceph-mon[23158]: pgmap v5005: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:01 smithi016 conmon[32727]: 2022-03-21T19:59:01.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:02 smithi016 ceph-mon[22010]: pgmap v5006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:02 smithi016 conmon[32727]: 2022-03-21T19:59:02.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:02 smithi079 ceph-mon[23158]: pgmap v5006: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:03 smithi016 conmon[32727]: 2022-03-21T19:59:03.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:04 smithi016 ceph-mon[22010]: pgmap v5007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:04 smithi016 conmon[32727]: 2022-03-21T19:59:04.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:04 smithi079 ceph-mon[23158]: pgmap v5007: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:05 smithi016 conmon[32727]: 2022-03-21T19:59:05.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:06.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:06 smithi016 ceph-mon[22010]: pgmap v5008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:06 smithi079 ceph-mon[23158]: pgmap v5008: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:06 smithi016 conmon[32727]: 2022-03-21T19:59:06.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:07 smithi016 conmon[32727]: 2022-03-21T19:59:07.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:08.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:08 smithi016 ceph-mon[22010]: pgmap v5009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:08.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:08 smithi079 ceph-mon[23158]: pgmap v5009: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:08 smithi016 conmon[32727]: 2022-03-21T19:59:08.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:09.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:09.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:09 smithi016 conmon[32727]: 2022-03-21T19:59:09.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:10 smithi016 ceph-mon[22010]: pgmap v5010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:10 smithi079 ceph-mon[23158]: pgmap v5010: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:10 smithi016 conmon[32727]: 2022-03-21T19:59:10.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:11 smithi016 conmon[32727]: 2022-03-21T19:59:11.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:12 smithi016 ceph-mon[22010]: pgmap v5011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:12 smithi079 ceph-mon[23158]: pgmap v5011: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:12 smithi016 conmon[32727]: 2022-03-21T19:59:12.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:13.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:13 smithi016 conmon[32727]: 2022-03-21T19:59:13.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:14 smithi016 ceph-mon[22010]: pgmap v5012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:14 smithi079 ceph-mon[23158]: pgmap v5012: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:14 smithi016 conmon[32727]: 2022-03-21T19:59:14.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9395 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:15 smithi016 conmon[32727]: 2022-03-21T19:59:15.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:16 smithi016 ceph-mon[22010]: pgmap v5013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:16 smithi079 ceph-mon[23158]: pgmap v5013: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:16 smithi016 conmon[32727]: 2022-03-21T19:59:16.570+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:17.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:17.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:17 smithi016 conmon[32727]: 2022-03-21T19:59:17.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:18 smithi016 ceph-mon[22010]: pgmap v5014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:18.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:18 smithi079 ceph-mon[23158]: pgmap v5014: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:18.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:18 smithi016 conmon[32727]: 2022-03-21T19:59:18.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:19.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:19.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:19 smithi016 conmon[32727]: 2022-03-21T19:59:19.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:20 smithi016 ceph-mon[22010]: pgmap v5015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:20.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:20 smithi079 ceph-mon[23158]: pgmap v5015: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:20.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:20.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9405 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:20 smithi016 conmon[32727]: 2022-03-21T19:59:20.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:21.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:21.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:21 smithi016 conmon[32727]: 2022-03-21T19:59:21.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:22.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:22 smithi016 ceph-mon[22010]: pgmap v5016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:22.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:22 smithi079 ceph-mon[23158]: pgmap v5016: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:22 smithi016 conmon[32727]: 2022-03-21T19:59:22.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:23.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:23.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:23.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:23 smithi016 conmon[32727]: 2022-03-21T19:59:23.513+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:24.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:24 smithi016 ceph-mon[22010]: pgmap v5017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:24.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:24 smithi079 ceph-mon[23158]: pgmap v5017: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:24.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:24 smithi016 conmon[32727]: 2022-03-21T19:59:24.538+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:25.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:25.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:25 smithi016 conmon[32727]: 2022-03-21T19:59:25.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:26.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:26 smithi016 ceph-mon[22010]: pgmap v5018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:26.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:26 smithi079 ceph-mon[23158]: pgmap v5018: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9410 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:26 smithi016 conmon[32727]: 2022-03-21T19:59:26.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:27 smithi016 conmon[32727]: 2022-03-21T19:59:27.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:28 smithi016 ceph-mon[22010]: pgmap v5019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:28 smithi079 ceph-mon[23158]: pgmap v5019: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:28 smithi016 conmon[32727]: 2022-03-21T19:59:28.548+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:29.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:29 smithi016 conmon[32727]: 2022-03-21T19:59:29.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:30.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:30 smithi016 ceph-mon[22010]: pgmap v5020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:30.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:30 smithi079 ceph-mon[23158]: pgmap v5020: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:30.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:30 smithi016 conmon[32727]: 2022-03-21T19:59:30.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:31.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:31 smithi016 conmon[32727]: 2022-03-21T19:59:31.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:32 smithi079 ceph-mon[23158]: pgmap v5021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:32.511 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:32 smithi016 ceph-mon[22010]: pgmap v5021: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:32.512 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:32 smithi016 conmon[32727]: 2022-03-21T19:59:32.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:33.559 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:33 smithi016 conmon[32727]: 2022-03-21T19:59:33.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:34.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:34 smithi079 ceph-mon[23158]: pgmap v5022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:34.593 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:34 smithi016 ceph-mon[22010]: pgmap v5022: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:34 smithi016 conmon[32727]: 2022-03-21T19:59:34.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:35.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:35.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9415 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:35 smithi016 conmon[32727]: 2022-03-21T19:59:35.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:36 smithi079 ceph-mon[23158]: pgmap v5023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:36.634 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:36.634 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:36 smithi016 ceph-mon[22010]: pgmap v5023: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:36 smithi016 conmon[32727]: 2022-03-21T19:59:36.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:37.623 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:37 smithi016 conmon[32727]: 2022-03-21T19:59:37.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:38 smithi079 ceph-mon[23158]: pgmap v5024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:38.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:38.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:38 smithi016 ceph-mon[22010]: pgmap v5024: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:38 smithi016 conmon[32727]: 2022-03-21T19:59:38.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:39.584 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:39 smithi016 conmon[32727]: 2022-03-21T19:59:39.583+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:40.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:40 smithi079 ceph-mon[23158]: pgmap v5025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:40.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:40.573 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:40.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:40 smithi016 ceph-mon[22010]: pgmap v5025: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:40.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9425 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:40 smithi016 conmon[32727]: 2022-03-21T19:59:40.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:41.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:41.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:41 smithi016 conmon[32727]: 2022-03-21T19:59:41.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:42 smithi079 ceph-mon[23158]: pgmap v5026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:42 smithi016 ceph-mon[22010]: pgmap v5026: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:42 smithi016 conmon[32727]: 2022-03-21T19:59:42.669+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:43.647 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:43 smithi016 conmon[32727]: 2022-03-21T19:59:43.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:44.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:44 smithi079 ceph-mon[23158]: pgmap v5027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:44.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:44.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:44 smithi016 ceph-mon[22010]: pgmap v5027: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:44.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:44 smithi016 conmon[32727]: 2022-03-21T19:59:44.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:45.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:45.634 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:45 smithi016 conmon[32727]: 2022-03-21T19:59:45.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:46 smithi079 ceph-mon[23158]: pgmap v5028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:46.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:46.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:46.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:46 smithi016 ceph-mon[22010]: pgmap v5028: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:46.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9430 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:46.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:46 smithi016 conmon[32727]: 2022-03-21T19:59:46.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:47.577 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:47 smithi016 conmon[32727]: 2022-03-21T19:59:47.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:48.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:48 smithi079 ceph-mon[23158]: pgmap v5029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:48.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:48.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:48 smithi016 ceph-mon[22010]: pgmap v5029: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:48 smithi016 conmon[32727]: 2022-03-21T19:59:48.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:49.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:49 smithi016 conmon[32727]: 2022-03-21T19:59:49.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:50.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:50.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:50 smithi016 ceph-mon[22010]: pgmap v5030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:50 smithi079 ceph-mon[23158]: pgmap v5030: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:50 smithi016 conmon[32727]: 2022-03-21T19:59:50.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:51.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:51 smithi016 conmon[32727]: 2022-03-21T19:59:51.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:52.589 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:52.589 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:52 smithi016 ceph-mon[22010]: pgmap v5031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:52 smithi079 ceph-mon[23158]: pgmap v5031: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:52 smithi016 conmon[32727]: 2022-03-21T19:59:52.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:53.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:53 smithi016 conmon[32727]: 2022-03-21T19:59:53.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:54.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:54.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:54 smithi016 ceph-mon[22010]: pgmap v5032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:54 smithi079 ceph-mon[23158]: pgmap v5032: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:54.908 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:54 smithi016 conmon[32727]: 2022-03-21T19:59:54.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:55.603 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:55.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:59:55.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:59:55.604 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:55 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:59:55.605 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:55.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: 2022-03-21T19:59:55.341+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:55.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:59:55.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:55.606 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: yield 2022-03-21T19:59:55.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:55.607 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:55.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:55.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:59:55.608 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:55.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:59:55.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:55.609 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:59:55.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:55.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:59:55.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:55.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:59:55.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:55.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:55.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T19:59:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T19:59:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:55 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T19:59:55.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9435 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T19:59:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:55 smithi016 conmon[32727]: 2022-03-21T19:59:55.602+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:56.574 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:56.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:56 smithi016 ceph-mon[22010]: pgmap v5033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:56.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:56 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:56.576 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:59:56.576 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:56.577 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:59:56.578 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:56.578 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:56.579 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:56.579 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:59:56.580 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:56.581 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:59:56.581 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:56.582 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:59:56.583 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:56.583 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:56.584 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:56.584 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:59:56.584 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:56.585 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:56.585 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:56.585 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:56 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:59:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:56 smithi079 ceph-mon[23158]: pgmap v5033: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T19:59:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:56 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:59:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:59:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:56.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:59:56.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:56.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:59:56.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:56.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:59:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:56.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:56.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:59:56.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:56.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:56.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:56.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:56 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T19:59:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:56 smithi016 conmon[32727]: 2022-03-21T19:59:56.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:57.543 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:57 smithi016 conmon[32727]: 2022-03-21T19:59:57.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:58.559 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:58.560 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:58 smithi016 ceph-mon[22010]: pgmap v5034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:58.560 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:58.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 2022-03-21T19:59:58.413+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:58.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:59:58.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:58.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: yield 2022-03-21T19:59:58.564 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:58.565 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:58.565 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:58.566 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:59:58.566 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:58.567 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:59:58.567 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:58.567 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:59:58.568 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:58.568 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:59:58.568 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:58.569 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:59:58.569 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:58.569 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:58.570 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:58.570 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 2022-03-21T19:59:58.414+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:59:58.571 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:59:58.571 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:58.571 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: yield 2022-03-21T19:59:58.572 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:58.572 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:58.572 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:58.573 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T19:59:58.573 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:58.573 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T19:59:58.574 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:58.574 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T19:59:58.574 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:58.575 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T19:59:58.575 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:58.575 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T19:59:58.576 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:58.576 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:58.577 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:58.577 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 2022-03-21T19:59:58.577 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T19:59:58.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: 2022-03-21T19:59:58.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T19:59:58.578 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:59:58.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return f(*arg) 2022-03-21T19:59:58.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:59:58.579 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T19:59:58.580 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:59:58.580 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:59:58.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:59:58.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:59:58.581 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:59:58.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T19:59:58.582 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:59:58.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:59:58.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:59:58.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T19:59:58.584 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:59:58.584 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise self._exception 2022-03-21T19:59:58.584 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:59:58.585 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:59:58.585 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:59:58.585 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T19:59:58.586 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:59:58.586 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise 2022-03-21T19:59:58.587 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:59:58.587 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T19:59:58.587 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:59:58.588 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T19:59:58.588 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:58 smithi079 ceph-mon[23158]: pgmap v5034: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T19:59:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T19:59:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:58 smithi016 conmon[32727]: 2022-03-21T19:59:58.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T19:59:59.512 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.512 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:59.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:59:59.513 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:59.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:59.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:59.514 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:59:59.515 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:59.515 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:59:59.516 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:59.516 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:59:59.516 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:59.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:59.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:59.517 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:59:59.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:59.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:59.518 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.519 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:59:59.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.519 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:59.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T19:59:59.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:59.520 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:59.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:59.521 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T19:59:59.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:59.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T19:59:59.522 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:59.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T19:59:59.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:59.523 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:59.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:59.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T19:59:59.524 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:59.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:59.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.525 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:59:59.526 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:59:59.526 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T19:59:59.527 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.527 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:59:59.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T19:59:59.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:59:59.528 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T19:59:59.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:59:59.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:59:59.529 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:59:59.530 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:59:59.530 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:59:59.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:59:59.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:59:59.531 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:59:59.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:59:59.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T19:59:59.532 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:59:59.533 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T19:59:59.533 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:59:59.533 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:59:59.534 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:59:59.534 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:59:59.534 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:59:59.535 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T19:59:59.535 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:59:59.535 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:59:59.536 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:59:59.536 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T19:59:59.536 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.537 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:59:59.537 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:59:59.537 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:59:59.538 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 19:59:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:59:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:59.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:59:59.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:59.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:59.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:59.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:59:59.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:59.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:59:59.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:59.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:59:59.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:59.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:59.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:59.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:59:59.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:59.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:59.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.736 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T19:59:59.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T19:59:59.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T19:59:59.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T19:59:59.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T19:59:59.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T19:59:59.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T19:59:59.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T19:59:59.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T19:59:59.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T19:59:59.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T19:59:59.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T19:59:59.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T19:59:59.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T19:59:59.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T19:59:59.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T19:59:59.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T19:59:59.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:59:59.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T19:59:59.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T19:59:59.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T19:59:59.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T19:59:59.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T19:59:59.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T19:59:59.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T19:59:59.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T19:59:59.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T19:59:59.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T19:59:59.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T19:59:59.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T19:59:59.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T19:59:59.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T19:59:59.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T19:59:59.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T19:59:59.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T19:59:59.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T19:59:59.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T19:59:59.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T19:59:59.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T19:59:59.753 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T19:59:59.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T19:59:59.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T19:59:59.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T19:59:59.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T19:59:59.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T19:59:59.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T19:59:59.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T19:59:59.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T19:59:59.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:59:59.758 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:59:59.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T19:59:59.759 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 19:59:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T19:59:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 19:59:59 smithi016 conmon[32727]: 2022-03-21T19:59:59.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:00 smithi016 conmon[32727]: 2022-03-21T20:00:00.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: pgmap v5035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops 2022-03-21T20:00:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T20:00:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T20:00:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T20:00:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T20:00:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T20:00:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops 2022-03-21T20:00:00.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: pgmap v5035: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops 2022-03-21T20:00:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T20:00:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T20:00:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T20:00:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T20:00:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 5h, current state unknown, last acting [] 2022-03-21T20:00:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops 2022-03-21T20:00:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9445 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:01.537 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:01 smithi016 conmon[32727]: 2022-03-21T20:00:01.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:02.531 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:02 smithi016 ceph-mon[22010]: pgmap v5036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:02.532 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:02 smithi079 ceph-mon[23158]: pgmap v5036: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:02.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:02 smithi016 conmon[32727]: 2022-03-21T20:00:02.530+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:03.546 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:03 smithi016 conmon[32727]: 2022-03-21T20:00:03.545+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:04.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:04 smithi016 conmon[32727]: 2022-03-21T20:00:04.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:04 smithi016 ceph-mon[22010]: pgmap v5037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:04 smithi079 ceph-mon[23158]: pgmap v5037: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:05.550 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:05 smithi016 conmon[32727]: 2022-03-21T20:00:05.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:06.585 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:06 smithi016 ceph-mon[22010]: pgmap v5038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:06.585 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:06.586 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:06 smithi079 ceph-mon[23158]: pgmap v5038: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9450 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:06 smithi016 conmon[32727]: 2022-03-21T20:00:06.584+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:07 smithi016 conmon[32727]: 2022-03-21T20:00:07.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:08.560 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:08.560 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:08 smithi016 ceph-mon[22010]: pgmap v5039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:08 smithi079 ceph-mon[23158]: pgmap v5039: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:08 smithi016 conmon[32727]: 2022-03-21T20:00:08.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:09.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:09 smithi016 conmon[32727]: 2022-03-21T20:00:09.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:10.563 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:10.564 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:10 smithi016 ceph-mon[22010]: pgmap v5040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:10.564 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:10 smithi079 ceph-mon[23158]: pgmap v5040: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:10 smithi016 conmon[32727]: 2022-03-21T20:00:10.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:11 smithi016 conmon[32727]: 2022-03-21T20:00:11.524+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:12.567 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:12.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:12 smithi016 ceph-mon[22010]: pgmap v5041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:12.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:12 smithi079 ceph-mon[23158]: pgmap v5041: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:12 smithi016 conmon[32727]: 2022-03-21T20:00:12.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:13 smithi016 conmon[32727]: 2022-03-21T20:00:13.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:14.567 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:14.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:14 smithi016 ceph-mon[22010]: pgmap v5042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:14 smithi079 ceph-mon[23158]: pgmap v5042: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:14 smithi016 conmon[32727]: 2022-03-21T20:00:14.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:15.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:15.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:15 smithi016 conmon[32727]: 2022-03-21T20:00:15.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9455 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:16.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:16 smithi016 conmon[32727]: 2022-03-21T20:00:16.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:16 smithi016 ceph-mon[22010]: pgmap v5043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:16 smithi079 ceph-mon[23158]: pgmap v5043: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:16.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:17 smithi016 conmon[32727]: 2022-03-21T20:00:17.483+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:18 smithi016 ceph-mon[22010]: pgmap v5044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:18 smithi016 conmon[32727]: 2022-03-21T20:00:18.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:18 smithi079 ceph-mon[23158]: pgmap v5044: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:19 smithi016 conmon[32727]: 2022-03-21T20:00:19.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:20 smithi016 ceph-mon[22010]: pgmap v5045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:20.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:20 smithi016 conmon[32727]: 2022-03-21T20:00:20.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:20 smithi079 ceph-mon[23158]: pgmap v5045: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9465 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:21.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:21 smithi016 conmon[32727]: 2022-03-21T20:00:21.468+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:22.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:22 smithi016 conmon[32727]: 2022-03-21T20:00:22.452+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:22 smithi016 ceph-mon[22010]: pgmap v5046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:22.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:22 smithi079 ceph-mon[23158]: pgmap v5046: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:23.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:23 smithi016 conmon[32727]: 2022-03-21T20:00:23.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:24 smithi016 ceph-mon[22010]: pgmap v5047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:24.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:24.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:24 smithi016 conmon[32727]: 2022-03-21T20:00:24.443+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:24 smithi079 ceph-mon[23158]: pgmap v5047: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:25.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:25 smithi016 conmon[32727]: 2022-03-21T20:00:25.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:26 smithi016 conmon[32727]: 2022-03-21T20:00:26.433+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:26 smithi016 ceph-mon[22010]: pgmap v5048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:26 smithi079 ceph-mon[23158]: pgmap v5048: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9470 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:27.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:27.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:27 smithi016 conmon[32727]: 2022-03-21T20:00:27.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:28 smithi016 ceph-mon[22010]: pgmap v5049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:28.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:28 smithi016 conmon[32727]: 2022-03-21T20:00:28.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:28 smithi079 ceph-mon[23158]: pgmap v5049: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:29 smithi016 conmon[32727]: 2022-03-21T20:00:29.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:30 smithi016 ceph-mon[22010]: pgmap v5050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:30 smithi016 conmon[32727]: 2022-03-21T20:00:30.471+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:30 smithi079 ceph-mon[23158]: pgmap v5050: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:31 smithi016 conmon[32727]: 2022-03-21T20:00:31.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:32 smithi016 ceph-mon[22010]: pgmap v5051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:32 smithi016 conmon[32727]: 2022-03-21T20:00:32.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:32 smithi079 ceph-mon[23158]: pgmap v5051: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:33 smithi016 conmon[32727]: 2022-03-21T20:00:33.439+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:34 smithi016 ceph-mon[22010]: pgmap v5052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:34 smithi016 conmon[32727]: 2022-03-21T20:00:34.464+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:34 smithi079 ceph-mon[23158]: pgmap v5052: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:35.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:35 smithi016 conmon[32727]: 2022-03-21T20:00:35.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9475 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:36 smithi016 ceph-mon[22010]: pgmap v5053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:36 smithi016 conmon[32727]: 2022-03-21T20:00:36.444+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:36 smithi079 ceph-mon[23158]: pgmap v5053: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:37 smithi016 conmon[32727]: 2022-03-21T20:00:37.493+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:38 smithi016 ceph-mon[22010]: pgmap v5054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:38.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:38.668 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:38 smithi016 conmon[32727]: 2022-03-21T20:00:38.492+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:38 smithi079 ceph-mon[23158]: pgmap v5054: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:38.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:39 smithi016 conmon[32727]: 2022-03-21T20:00:39.508+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:40 smithi016 ceph-mon[22010]: pgmap v5055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:40 smithi016 conmon[32727]: 2022-03-21T20:00:40.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:40 smithi079 ceph-mon[23158]: pgmap v5055: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9485 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:41 smithi016 conmon[32727]: 2022-03-21T20:00:41.463+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:42 smithi016 ceph-mon[22010]: pgmap v5056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:42.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:42 smithi016 conmon[32727]: 2022-03-21T20:00:42.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:42 smithi079 ceph-mon[23158]: pgmap v5056: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:43 smithi016 conmon[32727]: 2022-03-21T20:00:43.441+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:44 smithi016 ceph-mon[22010]: pgmap v5057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:44 smithi016 conmon[32727]: 2022-03-21T20:00:44.441+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:44 smithi079 ceph-mon[23158]: pgmap v5057: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:45 smithi016 conmon[32727]: 2022-03-21T20:00:45.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:46 smithi016 ceph-mon[22010]: pgmap v5058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:46.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:46 smithi016 conmon[32727]: 2022-03-21T20:00:46.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:46 smithi079 ceph-mon[23158]: pgmap v5058: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9490 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:47 smithi016 conmon[32727]: 2022-03-21T20:00:47.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:48 smithi016 ceph-mon[22010]: pgmap v5059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:48 smithi016 conmon[32727]: 2022-03-21T20:00:48.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:48 smithi079 ceph-mon[23158]: pgmap v5059: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:49 smithi016 conmon[32727]: 2022-03-21T20:00:49.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:50 smithi016 ceph-mon[22010]: pgmap v5060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:50 smithi079 ceph-mon[23158]: pgmap v5060: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:50 smithi016 conmon[32727]: 2022-03-21T20:00:50.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:51.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:51 smithi016 conmon[32727]: 2022-03-21T20:00:51.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:52.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:52.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:52 smithi016 ceph-mon[22010]: pgmap v5061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:52 smithi079 ceph-mon[23158]: pgmap v5061: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:52 smithi016 conmon[32727]: 2022-03-21T20:00:52.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:53 smithi016 conmon[32727]: 2022-03-21T20:00:53.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:54 smithi079 ceph-mon[23158]: pgmap v5062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:54 smithi016 ceph-mon[22010]: pgmap v5062: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:54 smithi016 conmon[32727]: 2022-03-21T20:00:54.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:55 smithi016 conmon[32727]: 2022-03-21T20:00:55.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9495 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:00:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:56 smithi079 ceph-mon[23158]: pgmap v5063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:56 smithi016 ceph-mon[22010]: pgmap v5063: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:00:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:56 smithi016 conmon[32727]: 2022-03-21T20:00:56.541+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:57.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:57.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:57 smithi016 conmon[32727]: 2022-03-21T20:00:57.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:58 smithi079 ceph-mon[23158]: pgmap v5064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:00:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:00:58.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:58 smithi016 ceph-mon[22010]: pgmap v5064: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:00:58.885 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:00:58.885 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:00:58.885 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:00:58.886 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:58 smithi016 conmon[32727]: 2022-03-21T20:00:58.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:00:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:00:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:00:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:00:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:00:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:00:59 smithi016 conmon[32727]: 2022-03-21T20:00:59.560+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:00:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:00:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:00:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:00:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:00:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:00:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:01:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:00 smithi079 ceph-mon[23158]: pgmap v5065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:00 smithi016 ceph-mon[22010]: pgmap v5065: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9505 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:00.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:00 smithi016 conmon[32727]: 2022-03-21T20:01:00.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:01.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: 2022-03-21T20:01:01.517+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:01.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:01:01.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:01.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: yield 2022-03-21T20:01:01.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:01.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:01:01.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:01.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:01:01.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:01.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:01:01.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:01:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:01.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:01:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:01.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:01.922 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:01 smithi016 conmon[32727]: 2022-03-21T20:01:01.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:02 smithi079 ceph-mon[23158]: pgmap v5066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:02 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:01:02.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:02.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:01:02.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:02.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:02.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:02.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:01:02.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:02.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:01:02.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:02.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:01:02.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:02.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:02.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:02.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:01:02.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:02.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:02.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:02.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:02 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:01:02.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:02 smithi016 conmon[32727]: 2022-03-21T20:01:02.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:02.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:02 smithi016 ceph-mon[22010]: pgmap v5066: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:02 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:01:02.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:02.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:01:02.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:02.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:02.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:02.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:01:02.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:02.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:01:02.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:02.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:01:02.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:02.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:02.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:02.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:01:02.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:02.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:02.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:02.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:02 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:01:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:03 smithi016 conmon[32727]: 2022-03-21T20:01:03.597+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:04 smithi079 ceph-mon[23158]: pgmap v5067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:04 smithi016 ceph-mon[22010]: pgmap v5067: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[32727]: 2022-03-21T20:01:04.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:04.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 2022-03-21T20:01:04.589+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:04.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:01:04.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:04.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: yield 2022-03-21T20:01:04.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:04.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:04.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:04.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:01:04.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:04.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:01:04.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:04.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:01:04.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:04.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:01:04.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:04.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:01:04.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:04.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:04.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:04.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 2022-03-21T20:01:04.590+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:01:04.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:01:04.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:04.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: yield 2022-03-21T20:01:04.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:04.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:04.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:04.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:01:04.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:04.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:01:04.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:04.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:01:04.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:04.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:01:04.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:04.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:01:04.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:04.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:04.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:04.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 2022-03-21T20:01:04.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:01:04.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: 2022-03-21T20:01:04.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:01:04.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:01:04.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:01:04.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:01:04.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:01:04.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:01:04.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:01:04.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:01:04.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:01:04.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:01:04.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:01:04.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:01:04.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:01:04.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:01:04.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:01:04.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:01:04.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:01:04.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:01:04.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:01:04.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:01:04.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:01:04.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:01:04.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise 2022-03-21T20:01:04.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:01:04.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:01:04.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:01:04.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:01:04.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:01:04 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:05 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:05.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:01:05.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:05.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:01:05.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:05.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:05.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:05.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:05 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:05.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:05.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:01:05.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:05.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:01:05.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:05.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:01:05.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:01:05.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:01:05.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:01:05.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:01:05.788 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:01:05.789 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:01:05.790 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:01:05.791 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:01:05.792 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:01:05.793 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:01:05.794 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:01:05.795 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:01:05.796 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:05 smithi016 conmon[32727]: 2022-03-21T20:01:05.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:05 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:05.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:01:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:05.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:01:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:05.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:01:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:01:05.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:05.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:05.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:05.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:01:05.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:05.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:05.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.927 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:05 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:01:05.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:01:05.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:01:05.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:01:05.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:01:05.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:01:05.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:01:05.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:01:05.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:01:05.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:01:05.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:01:05.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:01:05.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:01:05.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:01:05.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:01:05.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:01:05.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:01:05.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:05.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:01:05.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:01:05.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:01:05.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:01:05.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:01:05.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:01:05.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:01:05.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:01:05.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:01:05.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:01:05.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:01:05.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:01:05.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:01:05.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:01:05.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:01:05.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:01:05.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:01:05.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:01:05.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:01:05.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:01:05.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:01:05.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:01:05.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:01:05.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:01:05.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:01:05.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:01:05.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:01:05.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:01:05.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:01:05.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:01:05.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:01:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:06 smithi079 ceph-mon[23158]: pgmap v5068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:06 smithi016 conmon[32727]: 2022-03-21T20:01:06.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:06.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:06 smithi016 ceph-mon[22010]: pgmap v5068: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:06.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9510 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:07 smithi016 conmon[32727]: 2022-03-21T20:01:07.664+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:08 smithi079 ceph-mon[23158]: pgmap v5069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:08 smithi016 ceph-mon[22010]: pgmap v5069: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:08.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:08 smithi016 conmon[32727]: 2022-03-21T20:01:08.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:09.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:09.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:09 smithi016 conmon[32727]: 2022-03-21T20:01:09.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:10 smithi079 ceph-mon[23158]: pgmap v5070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:10 smithi016 conmon[32727]: 2022-03-21T20:01:10.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:10.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:10 smithi016 ceph-mon[22010]: pgmap v5070: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:11 smithi016 conmon[32727]: 2022-03-21T20:01:11.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:12 smithi016 ceph-mon[22010]: pgmap v5071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:12.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:12 smithi016 conmon[32727]: 2022-03-21T20:01:12.615+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:12 smithi079 ceph-mon[23158]: pgmap v5071: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:13 smithi016 conmon[32727]: 2022-03-21T20:01:13.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:14 smithi016 ceph-mon[22010]: pgmap v5072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:14 smithi016 conmon[32727]: 2022-03-21T20:01:14.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:14 smithi079 ceph-mon[23158]: pgmap v5072: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:15 smithi016 conmon[32727]: 2022-03-21T20:01:15.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9515 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:16 smithi016 ceph-mon[22010]: pgmap v5073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:16 smithi016 conmon[32727]: 2022-03-21T20:01:16.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:16 smithi079 ceph-mon[23158]: pgmap v5073: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:17 smithi016 conmon[32727]: 2022-03-21T20:01:17.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:18 smithi016 ceph-mon[22010]: pgmap v5074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:18 smithi016 conmon[32727]: 2022-03-21T20:01:18.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:18 smithi079 ceph-mon[23158]: pgmap v5074: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:19 smithi016 conmon[32727]: 2022-03-21T20:01:19.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:20 smithi016 conmon[32727]: 2022-03-21T20:01:20.584+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:20 smithi016 ceph-mon[22010]: pgmap v5075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:20 smithi079 ceph-mon[23158]: pgmap v5075: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9525 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:21 smithi016 conmon[32727]: 2022-03-21T20:01:21.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:22 smithi016 conmon[32727]: 2022-03-21T20:01:22.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:22 smithi016 ceph-mon[22010]: pgmap v5076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:22 smithi079 ceph-mon[23158]: pgmap v5076: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:23 smithi016 conmon[32727]: 2022-03-21T20:01:23.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:24 smithi016 ceph-mon[22010]: pgmap v5077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:24 smithi016 conmon[32727]: 2022-03-21T20:01:24.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:24 smithi079 ceph-mon[23158]: pgmap v5077: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:25 smithi016 conmon[32727]: 2022-03-21T20:01:25.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:26 smithi016 conmon[32727]: 2022-03-21T20:01:26.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:26 smithi016 ceph-mon[22010]: pgmap v5078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:26 smithi079 ceph-mon[23158]: pgmap v5078: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9530 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:27 smithi016 conmon[32727]: 2022-03-21T20:01:27.518+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:28 smithi016 ceph-mon[22010]: pgmap v5079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:28 smithi016 conmon[32727]: 2022-03-21T20:01:28.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:28 smithi079 ceph-mon[23158]: pgmap v5079: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:29 smithi016 conmon[32727]: 2022-03-21T20:01:29.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:30 smithi016 ceph-mon[22010]: pgmap v5080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:30 smithi016 conmon[32727]: 2022-03-21T20:01:30.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:30 smithi079 ceph-mon[23158]: pgmap v5080: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:31.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:31 smithi016 conmon[32727]: 2022-03-21T20:01:31.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:32 smithi016 ceph-mon[22010]: pgmap v5081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:32 smithi016 conmon[32727]: 2022-03-21T20:01:32.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:32 smithi079 ceph-mon[23158]: pgmap v5081: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:33 smithi016 conmon[32727]: 2022-03-21T20:01:33.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:34 smithi016 ceph-mon[22010]: pgmap v5082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:34 smithi016 conmon[32727]: 2022-03-21T20:01:34.506+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:34.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:34 smithi079 ceph-mon[23158]: pgmap v5082: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:35 smithi016 conmon[32727]: 2022-03-21T20:01:35.505+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9535 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:36 smithi016 ceph-mon[22010]: pgmap v5083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:36.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:36 smithi016 conmon[32727]: 2022-03-21T20:01:36.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:36 smithi079 ceph-mon[23158]: pgmap v5083: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:37 smithi016 conmon[32727]: 2022-03-21T20:01:37.510+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:38 smithi016 ceph-mon[22010]: pgmap v5084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:38 smithi016 conmon[32727]: 2022-03-21T20:01:38.506+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:38 smithi079 ceph-mon[23158]: pgmap v5084: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:39 smithi016 conmon[32727]: 2022-03-21T20:01:39.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:40 smithi016 ceph-mon[22010]: pgmap v5085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:40 smithi016 conmon[32727]: 2022-03-21T20:01:40.464+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:40.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:40 smithi079 ceph-mon[23158]: pgmap v5085: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9545 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:41 smithi016 conmon[32727]: 2022-03-21T20:01:41.429+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:42 smithi016 ceph-mon[22010]: pgmap v5086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:42 smithi016 conmon[32727]: 2022-03-21T20:01:42.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:42 smithi079 ceph-mon[23158]: pgmap v5086: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:43 smithi016 conmon[32727]: 2022-03-21T20:01:43.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:44 smithi016 ceph-mon[22010]: pgmap v5087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:44.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:44.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:44.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:44 smithi016 conmon[32727]: 2022-03-21T20:01:44.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:44 smithi079 ceph-mon[23158]: pgmap v5087: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:45 smithi016 conmon[32727]: 2022-03-21T20:01:45.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:46 smithi016 ceph-mon[22010]: pgmap v5088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:46 smithi079 ceph-mon[23158]: pgmap v5088: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:46.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9550 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:46.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:46 smithi016 conmon[32727]: 2022-03-21T20:01:46.547+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:47.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:47.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:47 smithi016 conmon[32727]: 2022-03-21T20:01:47.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:48 smithi016 ceph-mon[22010]: pgmap v5089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:48.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:48 smithi079 ceph-mon[23158]: pgmap v5089: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:48.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:48 smithi016 conmon[32727]: 2022-03-21T20:01:48.514+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:49.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:49.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:49 smithi016 conmon[32727]: 2022-03-21T20:01:49.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:50 smithi016 ceph-mon[22010]: pgmap v5090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:50 smithi079 ceph-mon[23158]: pgmap v5090: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:50.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:50 smithi016 conmon[32727]: 2022-03-21T20:01:50.476+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:51.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:51 smithi016 conmon[32727]: 2022-03-21T20:01:51.500+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:52.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:52 smithi016 ceph-mon[22010]: pgmap v5091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:52.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:52 smithi079 ceph-mon[23158]: pgmap v5091: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:52.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:52 smithi016 conmon[32727]: 2022-03-21T20:01:52.488+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:53.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:53.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:53 smithi016 conmon[32727]: 2022-03-21T20:01:53.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:54 smithi016 ceph-mon[22010]: pgmap v5092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:54.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:54 smithi079 ceph-mon[23158]: pgmap v5092: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:54.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:54 smithi016 conmon[32727]: 2022-03-21T20:01:54.445+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:55.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:55.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9555 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:01:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:55 smithi016 conmon[32727]: 2022-03-21T20:01:55.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:56 smithi016 ceph-mon[22010]: pgmap v5093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:56 smithi079 ceph-mon[23158]: pgmap v5093: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:01:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:56 smithi016 conmon[32727]: 2022-03-21T20:01:56.413+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:57.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:57.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:57 smithi016 conmon[32727]: 2022-03-21T20:01:57.419+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:58 smithi079 ceph-mon[23158]: pgmap v5094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:58.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:58 smithi016 ceph-mon[22010]: pgmap v5094: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:01:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:58.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:58 smithi016 conmon[32727]: 2022-03-21T20:01:58.377+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:01:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:01:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:01:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:01:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:01:59.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:01:59 smithi016 conmon[32727]: 2022-03-21T20:01:59.369+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:01:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:01:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:01:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:01:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:01:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:01:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:02:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:00 smithi079 ceph-mon[23158]: pgmap v5095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:00.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:00.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:00 smithi016 ceph-mon[22010]: pgmap v5095: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9565 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:00.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:00 smithi016 conmon[32727]: 2022-03-21T20:02:00.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:01.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:01.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:01 smithi016 conmon[32727]: 2022-03-21T20:02:01.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:02 smithi079 ceph-mon[23158]: pgmap v5096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:02.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:02 smithi016 ceph-mon[22010]: pgmap v5096: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:02 smithi016 conmon[32727]: 2022-03-21T20:02:02.360+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:03.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:03.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:03 smithi016 conmon[32727]: 2022-03-21T20:02:03.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:04.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:04 smithi079 ceph-mon[23158]: pgmap v5097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:04.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:04.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:04 smithi016 conmon[32727]: 2022-03-21T20:02:04.406+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:04.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:04 smithi016 ceph-mon[22010]: pgmap v5097: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:05.400 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:05.400 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:02:05.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:02:05.401 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:05 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:02:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:05.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:02:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:02:05.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:05 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:02:05.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:05 smithi016 conmon[32727]: 2022-03-21T20:02:05.398+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:06 smithi079 ceph-mon[23158]: pgmap v5098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:06.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:06 smithi016 conmon[32727]: 2022-03-21T20:02:06.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:06 smithi016 ceph-mon[22010]: pgmap v5098: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9570 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:06.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[32727]: 2022-03-21T20:02:07.421+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:08.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: 2022-03-21T20:02:07.693+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:02:08.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:08.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: yield 2022-03-21T20:02:08.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:08.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:02:08.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:08.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:02:08.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:08.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:02:08.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:02:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:08.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:02:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:08.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:07 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:08 smithi079 ceph-mon[23158]: pgmap v5099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:08 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:02:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:02:08.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:08.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:02:08.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:08.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:02:08.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:08.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:02:08.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:08.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:08.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:08.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:02:08.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:08.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:08.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.481 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:08 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:02:08.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:08 smithi016 conmon[32727]: 2022-03-21T20:02:08.387+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:08 smithi016 ceph-mon[22010]: pgmap v5099: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:08 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:02:08.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:02:08.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:08.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:02:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:02:08.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:02:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:08.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:02:08.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:08.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:08 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:02:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:09 smithi016 conmon[32727]: 2022-03-21T20:02:09.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:10 smithi079 ceph-mon[23158]: pgmap v5100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:10 smithi016 ceph-mon[22010]: pgmap v5100: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:10.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[32727]: 2022-03-21T20:02:10.373+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:11.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 2022-03-21T20:02:10.765+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:02:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:11.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: yield 2022-03-21T20:02:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:11.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:02:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:11.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:02:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:11.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:02:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:02:11.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:02:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:11.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 2022-03-21T20:02:10.766+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:02:11.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:02:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: yield 2022-03-21T20:02:11.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:11.174 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:02:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:02:11.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:11.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:02:11.176 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:02:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:11.177 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:02:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:11.178 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:11.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 2022-03-21T20:02:11.179 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:02:11.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: 2022-03-21T20:02:11.180 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:02:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:02:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:02:11.181 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:02:11.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:02:11.182 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:02:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:02:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:02:11.183 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:02:11.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:02:11.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:02:11.184 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:02:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:02:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:02:11.185 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:02:11.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:02:11.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:02:11.186 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:02:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:02:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:02:11.187 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:02:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:02:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise 2022-03-21T20:02:11.188 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:02:11.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:02:11.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:02:11.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:02:11.190 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:02:10 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:11.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:11.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:11.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:11 smithi016 conmon[32727]: 2022-03-21T20:02:11.390+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:02:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:12.475 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:02:12.476 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:02:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:12.477 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:02:12.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:12.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:12.478 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:12.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:02:12.479 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.480 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:12 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:02:12.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:12.481 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:02:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:12.482 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:12.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:02:12.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:12.483 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:02:12.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:12.484 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:02:12.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:12.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:12.485 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:12.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:02:12.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:12.486 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:02:12.487 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:02:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:02:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.488 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:02:12.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:02:12.489 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:02:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:02:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:02:12.490 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:02:12.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:02:12.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:02:12.491 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:02:12.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:02:12.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:02:12.492 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:02:12.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:02:12.493 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:02:12.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:02:12.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:02:12.494 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:02:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:02:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:02:12.495 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:02:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:02:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:02:12.496 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:02:12.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:02:12.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:02:12.497 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:02:12.498 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.498 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:12 smithi079 ceph-mon[23158]: pgmap v5101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:12.499 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:12 smithi016 conmon[32727]: 2022-03-21T20:02:12.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:02:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:02:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:02:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:02:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:02:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:12 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:02:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:02:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:02:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:02:12.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:02:12.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:02:12.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:02:12.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:02:12.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:02:12.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:02:12.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:02:12.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:02:12.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:02:12.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:02:12.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:02:12.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:02:12.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:02:12.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:02:12.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:02:12.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:02:12.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:02:12.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:02:12.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:02:12.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:02:12.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:02:12.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:02:12.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:02:12.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:02:12.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:02:12.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:02:12.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:02:12.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:02:12.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:02:12.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:02:12.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:02:12.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:02:12.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:02:12.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:02:12.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:02:12.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:02:12.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:02:12.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:02:12.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:02:12.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:02:12.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:02:12.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:02:12.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:02:12.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:02:12.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:12 smithi016 ceph-mon[22010]: pgmap v5101: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:12.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:13.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:13 smithi016 conmon[32727]: 2022-03-21T20:02:13.408+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:14 smithi079 ceph-mon[23158]: pgmap v5102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:14.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:14 smithi016 ceph-mon[22010]: pgmap v5102: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:14.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:14 smithi016 conmon[32727]: 2022-03-21T20:02:14.418+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:15.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9575 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:15.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:15 smithi016 conmon[32727]: 2022-03-21T20:02:15.371+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:16 smithi079 ceph-mon[23158]: pgmap v5103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:16.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:16 smithi016 ceph-mon[22010]: pgmap v5103: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:16.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:16.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:16 smithi016 conmon[32727]: 2022-03-21T20:02:16.396+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:17.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:17 smithi016 conmon[32727]: 2022-03-21T20:02:17.402+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:18 smithi016 ceph-mon[22010]: pgmap v5104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:18.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:18 smithi016 conmon[32727]: 2022-03-21T20:02:18.448+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:18 smithi079 ceph-mon[23158]: pgmap v5104: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:19.492 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:19 smithi016 conmon[32727]: 2022-03-21T20:02:19.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:20.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:20 smithi016 ceph-mon[22010]: pgmap v5105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:20.533 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:20.534 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:20 smithi079 ceph-mon[23158]: pgmap v5105: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9585 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:20 smithi016 conmon[32727]: 2022-03-21T20:02:20.532+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:21.547 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:21 smithi016 conmon[32727]: 2022-03-21T20:02:21.546+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:22.580 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:22 smithi016 ceph-mon[22010]: pgmap v5106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:22.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:22 smithi079 ceph-mon[23158]: pgmap v5106: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:22 smithi016 conmon[32727]: 2022-03-21T20:02:22.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:23.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:23 smithi016 conmon[32727]: 2022-03-21T20:02:23.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:24.499 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:24.499 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:24 smithi016 ceph-mon[22010]: pgmap v5107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:24.500 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:24 smithi079 ceph-mon[23158]: pgmap v5107: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:24 smithi016 conmon[32727]: 2022-03-21T20:02:24.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:25.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:25 smithi016 conmon[32727]: 2022-03-21T20:02:25.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:26.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:26 smithi016 ceph-mon[22010]: pgmap v5108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:26.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:26.568 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:26 smithi079 ceph-mon[23158]: pgmap v5108: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9590 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:26 smithi016 conmon[32727]: 2022-03-21T20:02:26.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:27.592 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:27.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:27.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:27 smithi016 conmon[32727]: 2022-03-21T20:02:27.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:28.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:28 smithi016 ceph-mon[22010]: pgmap v5109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:28 smithi079 ceph-mon[23158]: pgmap v5109: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:28 smithi016 conmon[32727]: 2022-03-21T20:02:28.626+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:29.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:29.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:29 smithi016 conmon[32727]: 2022-03-21T20:02:29.595+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:30.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:30.601 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:30 smithi016 ceph-mon[22010]: pgmap v5110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:30 smithi079 ceph-mon[23158]: pgmap v5110: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:30 smithi016 conmon[32727]: 2022-03-21T20:02:30.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:31.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:31.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:31 smithi016 conmon[32727]: 2022-03-21T20:02:31.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:32.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:32.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:32 smithi016 ceph-mon[22010]: pgmap v5111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:32 smithi079 ceph-mon[23158]: pgmap v5111: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:32 smithi016 conmon[32727]: 2022-03-21T20:02:32.607+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:33.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:33 smithi016 conmon[32727]: 2022-03-21T20:02:33.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:34.575 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:34.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:34 smithi016 ceph-mon[22010]: pgmap v5112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:34.576 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:34 smithi079 ceph-mon[23158]: pgmap v5112: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:34.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:34 smithi016 conmon[32727]: 2022-03-21T20:02:34.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:35.570 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:35.571 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:35.571 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:35 smithi016 conmon[32727]: 2022-03-21T20:02:35.568+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9595 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:36 smithi016 ceph-mon[22010]: pgmap v5113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:36.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:36 smithi016 conmon[32727]: 2022-03-21T20:02:36.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:36 smithi079 ceph-mon[23158]: pgmap v5113: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:36.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:37 smithi016 conmon[32727]: 2022-03-21T20:02:37.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:38 smithi016 ceph-mon[22010]: pgmap v5114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:38.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:38 smithi016 conmon[32727]: 2022-03-21T20:02:38.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:38 smithi079 ceph-mon[23158]: pgmap v5114: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:39.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:39 smithi016 conmon[32727]: 2022-03-21T20:02:39.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:40.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:40 smithi016 conmon[32727]: 2022-03-21T20:02:40.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:40 smithi016 ceph-mon[22010]: pgmap v5115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:40 smithi079 ceph-mon[23158]: pgmap v5115: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9605 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:41 smithi016 conmon[32727]: 2022-03-21T20:02:41.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:41.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:42 smithi016 ceph-mon[22010]: pgmap v5116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:42.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:42 smithi016 conmon[32727]: 2022-03-21T20:02:42.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:42 smithi079 ceph-mon[23158]: pgmap v5116: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:43.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:43 smithi016 conmon[32727]: 2022-03-21T20:02:43.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:44 smithi016 ceph-mon[22010]: pgmap v5117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:44 smithi016 conmon[32727]: 2022-03-21T20:02:44.505+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:44 smithi079 ceph-mon[23158]: pgmap v5117: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:45 smithi016 conmon[32727]: 2022-03-21T20:02:45.533+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:46.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:46 smithi016 conmon[32727]: 2022-03-21T20:02:46.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:46 smithi016 ceph-mon[22010]: pgmap v5118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:46 smithi079 ceph-mon[23158]: pgmap v5118: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9610 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:47.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:47 smithi016 conmon[32727]: 2022-03-21T20:02:47.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:48 smithi016 ceph-mon[22010]: pgmap v5119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:48 smithi016 conmon[32727]: 2022-03-21T20:02:48.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:48 smithi079 ceph-mon[23158]: pgmap v5119: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:49 smithi016 conmon[32727]: 2022-03-21T20:02:49.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:50 smithi016 ceph-mon[22010]: pgmap v5120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:50.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:50 smithi016 conmon[32727]: 2022-03-21T20:02:50.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:50 smithi079 ceph-mon[23158]: pgmap v5120: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:51 smithi016 conmon[32727]: 2022-03-21T20:02:51.529+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:52 smithi016 ceph-mon[22010]: pgmap v5121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:52 smithi016 conmon[32727]: 2022-03-21T20:02:52.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:52 smithi079 ceph-mon[23158]: pgmap v5121: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:53 smithi016 conmon[32727]: 2022-03-21T20:02:53.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:54 smithi016 ceph-mon[22010]: pgmap v5122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:54 smithi016 conmon[32727]: 2022-03-21T20:02:54.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:54 smithi079 ceph-mon[23158]: pgmap v5122: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:55 smithi016 conmon[32727]: 2022-03-21T20:02:55.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9615 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:02:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:56 smithi016 ceph-mon[22010]: pgmap v5123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:56.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:56.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:56 smithi016 conmon[32727]: 2022-03-21T20:02:56.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:56 smithi079 ceph-mon[23158]: pgmap v5123: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:02:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:57 smithi016 conmon[32727]: 2022-03-21T20:02:57.582+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:58.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:58 smithi016 ceph-mon[22010]: pgmap v5124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:58.613 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:58 smithi079 ceph-mon[23158]: pgmap v5124: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:02:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:02:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:58 smithi016 conmon[32727]: 2022-03-21T20:02:58.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:02:59.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:02:59.657 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:02:59.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:02:59.658 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:02:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:02:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:02:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:02:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:02:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:02:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:02:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:02:59 smithi016 conmon[32727]: 2022-03-21T20:02:59.655+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:00.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:00.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:00 smithi016 ceph-mon[22010]: pgmap v5125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:00.641 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:00 smithi079 ceph-mon[23158]: pgmap v5125: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9625 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:00 smithi016 conmon[32727]: 2022-03-21T20:03:00.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:01.659 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:01 smithi016 conmon[32727]: 2022-03-21T20:03:01.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:02 smithi016 ceph-mon[22010]: pgmap v5126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:02 smithi079 ceph-mon[23158]: pgmap v5126: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:02 smithi016 conmon[32727]: 2022-03-21T20:03:02.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:03.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:03.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:03 smithi016 conmon[32727]: 2022-03-21T20:03:03.662+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:04 smithi016 ceph-mon[22010]: pgmap v5127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:04 smithi079 ceph-mon[23158]: pgmap v5127: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:04 smithi016 conmon[32727]: 2022-03-21T20:03:04.664+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:05 smithi016 conmon[32727]: 2022-03-21T20:03:05.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:06 smithi016 conmon[32727]: 2022-03-21T20:03:06.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:06 smithi016 ceph-mon[22010]: pgmap v5128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:06 smithi079 ceph-mon[23158]: pgmap v5128: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9630 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:07 smithi016 conmon[32727]: 2022-03-21T20:03:07.590+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:08 smithi016 ceph-mon[22010]: pgmap v5129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:08 smithi016 conmon[32727]: 2022-03-21T20:03:08.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:08 smithi079 ceph-mon[23158]: pgmap v5129: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:09 smithi016 conmon[32727]: 2022-03-21T20:03:09.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:10.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:10.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:10 smithi016 ceph-mon[22010]: pgmap v5130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:10 smithi079 ceph-mon[23158]: pgmap v5130: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:10.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:10 smithi016 conmon[32727]: 2022-03-21T20:03:10.661+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:11.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:11.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:03:11.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:03:11.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:11 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:03:11.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: 2022-03-21T20:03:11.373+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:11.627 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:03:11.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:11.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: yield 2022-03-21T20:03:11.628 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:11.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:11.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:11.629 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:03:11.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:11.630 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:03:11.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:11.631 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:03:11.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:11.632 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:03:11.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:11.633 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:03:11.634 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:11.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:11.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:03:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:03:11.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:11 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:03:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:11 smithi016 conmon[32727]: 2022-03-21T20:03:11.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:12.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:12 smithi016 conmon[32727]: 2022-03-21T20:03:12.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:12 smithi016 ceph-mon[22010]: pgmap v5131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:12 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:12.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:03:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:03:12.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:12.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:03:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:12.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:03:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:03:12.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:12.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:03:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:12.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:12.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:12 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:03:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:12 smithi079 ceph-mon[23158]: pgmap v5131: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:12 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:03:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:03:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:12.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:03:12.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:03:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:12.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:03:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:12.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:03:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:12.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:12.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:12 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:03:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:13.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:13 smithi016 conmon[32727]: 2022-03-21T20:03:13.643+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:14.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:14 smithi016 ceph-mon[22010]: pgmap v5132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:14.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[32727]: 2022-03-21T20:03:14.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:14.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 2022-03-21T20:03:14.445+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:14.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:03:14.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:14.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: yield 2022-03-21T20:03:14.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:14.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:14.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:14.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:03:14.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:14.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:03:14.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:14.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:03:14.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:14.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:03:14.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:14.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:03:14.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:14.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:14.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:14.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 2022-03-21T20:03:14.446+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:03:14.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:03:14.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:14.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: yield 2022-03-21T20:03:14.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:14.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:14.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:14.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:03:14.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:14.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:03:14.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:14.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:03:14.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:14.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:03:14.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:14.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:03:14.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:14.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:14.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:14.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 2022-03-21T20:03:14.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:03:14.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: 2022-03-21T20:03:14.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:03:14.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:03:14.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:03:14.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:03:14.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:03:14.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:03:14.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:03:14.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:03:14.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:03:14.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:03:14.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:03:14.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:03:14.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:03:14.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:03:14.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:03:14.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:03:14.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:03:14.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:03:14.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:03:14.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:03:14.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:03:14.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:03:14.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise 2022-03-21T20:03:14.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:03:14.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:03:14.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:03:14.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:03:14.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:03:14 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:14 smithi079 ceph-mon[23158]: pgmap v5132: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:15 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:03:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:15.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:15.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:15.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:03:15.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:15.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:03:15.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:03:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:15.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:03:15.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:15 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:03:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:15.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:03:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:15.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:03:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:15.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:03:15.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:15.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:03:15.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:15.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:15.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:03:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:15.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:03:15.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:03:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:03:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:03:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:03:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:03:15.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:03:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:03:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:03:15.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:03:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:03:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:03:15.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:03:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:03:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:03:15.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:03:15.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:03:15.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:03:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:03:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:03:15.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:03:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:03:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:03:15.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:03:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:03:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:03:15.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:03:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:03:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:03:15.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:15 smithi016 conmon[32727]: 2022-03-21T20:03:15.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:15 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:03:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:15.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:03:15.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:03:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:15.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:03:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:15.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:03:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:15.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:15.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:15 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:03:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:03:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:03:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:03:15.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:03:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:03:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:03:15.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:03:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:03:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:03:15.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:03:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:03:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:03:15.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:03:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:03:15.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:03:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:03:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:03:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:03:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:03:15.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:03:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:03:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:03:15.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:03:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:03:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:03:15.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:03:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:03:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:03:15.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:03:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:03:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:03:15.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:03:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:03:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:03:15.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:03:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:03:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:03:15.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:03:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:03:15.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:03:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:03:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:03:15.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:03:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:03:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:03:15.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:03:15.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:03:15.940 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9635 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:16 smithi079 ceph-mon[23158]: pgmap v5133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:16 smithi016 ceph-mon[22010]: pgmap v5133: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:16 smithi016 conmon[32727]: 2022-03-21T20:03:16.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:17.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:17 smithi016 conmon[32727]: 2022-03-21T20:03:17.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:18 smithi079 ceph-mon[23158]: pgmap v5134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:18.724 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:18.724 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:18 smithi016 ceph-mon[22010]: pgmap v5134: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:18 smithi016 conmon[32727]: 2022-03-21T20:03:18.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:19 smithi016 conmon[32727]: 2022-03-21T20:03:19.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:20 smithi079 ceph-mon[23158]: pgmap v5135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:20 smithi016 conmon[32727]: 2022-03-21T20:03:20.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:20 smithi016 ceph-mon[22010]: pgmap v5135: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9645 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:21 smithi016 conmon[32727]: 2022-03-21T20:03:21.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:22 smithi079 ceph-mon[23158]: pgmap v5136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:22 smithi016 ceph-mon[22010]: pgmap v5136: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:22 smithi016 conmon[32727]: 2022-03-21T20:03:22.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:23 smithi016 conmon[32727]: 2022-03-21T20:03:23.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:24 smithi079 ceph-mon[23158]: pgmap v5137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:24 smithi016 conmon[32727]: 2022-03-21T20:03:24.687+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:24 smithi016 ceph-mon[22010]: pgmap v5137: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:25 smithi016 conmon[32727]: 2022-03-21T20:03:25.693+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:26.710 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:26.711 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:26 smithi016 ceph-mon[22010]: pgmap v5138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:26.711 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:26 smithi079 ceph-mon[23158]: pgmap v5138: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9650 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:26 smithi016 conmon[32727]: 2022-03-21T20:03:26.709+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:27.729 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:28.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:27 smithi016 conmon[32727]: 2022-03-21T20:03:27.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:28 smithi079 ceph-mon[23158]: pgmap v5139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:28.729 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:28.729 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:28 smithi016 ceph-mon[22010]: pgmap v5139: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:29.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:28 smithi016 conmon[32727]: 2022-03-21T20:03:28.728+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:29.730 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:29 smithi016 conmon[32727]: 2022-03-21T20:03:29.729+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:30 smithi079 ceph-mon[23158]: pgmap v5140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:30.772 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:30.773 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:30 smithi016 ceph-mon[22010]: pgmap v5140: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:31.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:30 smithi016 conmon[32727]: 2022-03-21T20:03:30.772+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:31.768 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:31 smithi016 conmon[32727]: 2022-03-21T20:03:31.766+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:32.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:32 smithi079 ceph-mon[23158]: pgmap v5141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:32.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:32.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:32 smithi016 ceph-mon[22010]: pgmap v5141: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:33.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:32 smithi016 conmon[32727]: 2022-03-21T20:03:32.816+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:33.791 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:33 smithi016 conmon[32727]: 2022-03-21T20:03:33.791+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:34.479 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:34.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:34 smithi016 ceph-mon[22010]: pgmap v5142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:34 smithi079 ceph-mon[23158]: pgmap v5142: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:35.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:34 smithi016 conmon[32727]: 2022-03-21T20:03:34.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:35.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:35.877 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9655 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:36.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:35 smithi016 conmon[32727]: 2022-03-21T20:03:35.875+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:36.861 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:36.862 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:36 smithi016 ceph-mon[22010]: pgmap v5143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:36 smithi079 ceph-mon[23158]: pgmap v5143: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:37.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:36 smithi016 conmon[32727]: 2022-03-21T20:03:36.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:37.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:38.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:37 smithi016 conmon[32727]: 2022-03-21T20:03:37.821+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:38.787 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:38.787 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:38 smithi016 ceph-mon[22010]: pgmap v5144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:38 smithi079 ceph-mon[23158]: pgmap v5144: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:38 smithi016 conmon[32727]: 2022-03-21T20:03:38.785+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:39.760 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:39 smithi016 conmon[32727]: 2022-03-21T20:03:39.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:40.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:40.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:40 smithi016 ceph-mon[22010]: pgmap v5145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:40.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:40.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:40 smithi079 ceph-mon[23158]: pgmap v5145: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9665 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:40 smithi016 conmon[32727]: 2022-03-21T20:03:40.803+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:41.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:41 smithi016 conmon[32727]: 2022-03-21T20:03:41.823+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:42.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:42.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:42 smithi016 ceph-mon[22010]: pgmap v5146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:42 smithi079 ceph-mon[23158]: pgmap v5146: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:43.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:42 smithi016 conmon[32727]: 2022-03-21T20:03:42.841+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:43.868 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:43 smithi016 conmon[32727]: 2022-03-21T20:03:43.865+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:44.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:44.881 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:44 smithi016 ceph-mon[22010]: pgmap v5147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:44 smithi079 ceph-mon[23158]: pgmap v5147: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:44 smithi016 conmon[32727]: 2022-03-21T20:03:44.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:45.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:45 smithi016 conmon[32727]: 2022-03-21T20:03:45.884+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:46.883 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:46.883 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:46 smithi016 ceph-mon[22010]: pgmap v5148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:46.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9669 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:46 smithi079 ceph-mon[23158]: pgmap v5148: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9669 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:46 smithi016 conmon[32727]: 2022-03-21T20:03:46.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:47.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:47 smithi016 conmon[32727]: 2022-03-21T20:03:47.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:48.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:48.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:48 smithi016 ceph-mon[22010]: pgmap v5149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:48.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:48 smithi079 ceph-mon[23158]: pgmap v5149: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:48 smithi016 conmon[32727]: 2022-03-21T20:03:48.848+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:49.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:49 smithi016 conmon[32727]: 2022-03-21T20:03:49.820+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:50.802 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:50.803 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:50 smithi016 ceph-mon[22010]: pgmap v5150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:50 smithi079 ceph-mon[23158]: pgmap v5150: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:50 smithi016 conmon[32727]: 2022-03-21T20:03:50.801+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:51 smithi016 conmon[32727]: 2022-03-21T20:03:51.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:52.787 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:52.788 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:52 smithi016 ceph-mon[22010]: pgmap v5151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:52 smithi079 ceph-mon[23158]: pgmap v5151: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:52 smithi016 conmon[32727]: 2022-03-21T20:03:52.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:53 smithi016 conmon[32727]: 2022-03-21T20:03:53.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:54.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:54.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:54 smithi016 ceph-mon[22010]: pgmap v5152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:54 smithi079 ceph-mon[23158]: pgmap v5152: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:54 smithi016 conmon[32727]: 2022-03-21T20:03:54.806+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:55.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:55.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:55.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:55.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9675 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:03:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:55 smithi016 conmon[32727]: 2022-03-21T20:03:55.822+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:56.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:56.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:56 smithi016 ceph-mon[22010]: pgmap v5153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:56 smithi079 ceph-mon[23158]: pgmap v5153: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:03:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:56 smithi016 conmon[32727]: 2022-03-21T20:03:56.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:57.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:57 smithi016 conmon[32727]: 2022-03-21T20:03:57.838+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:58.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:58.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:58 smithi016 ceph-mon[22010]: pgmap v5154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:58 smithi079 ceph-mon[23158]: pgmap v5154: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:03:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:58 smithi016 conmon[32727]: 2022-03-21T20:03:58.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:03:59.865 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:59.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:03:59.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:03:59.866 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:03:59.867 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:03:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:03:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:03:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:03:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:03:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:03:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:03:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:04:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:03:59 smithi016 conmon[32727]: 2022-03-21T20:03:59.864+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:00.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:00.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:00 smithi016 ceph-mon[22010]: pgmap v5155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:00.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9684 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:00 smithi079 ceph-mon[23158]: pgmap v5155: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9684 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:00 smithi016 conmon[32727]: 2022-03-21T20:04:00.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:01.893 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:01 smithi016 conmon[32727]: 2022-03-21T20:04:01.892+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:02.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:02.902 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:02 smithi016 ceph-mon[22010]: pgmap v5156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:02 smithi079 ceph-mon[23158]: pgmap v5156: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:02 smithi016 conmon[32727]: 2022-03-21T20:04:02.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:03.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:03 smithi016 conmon[32727]: 2022-03-21T20:04:03.894+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:04.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:04.851 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:04 smithi016 ceph-mon[22010]: pgmap v5157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:04 smithi079 ceph-mon[23158]: pgmap v5157: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:04 smithi016 conmon[32727]: 2022-03-21T20:04:04.850+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:05.896 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:05 smithi016 conmon[32727]: 2022-03-21T20:04:05.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:06.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:06.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:06 smithi016 ceph-mon[22010]: pgmap v5158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:06.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:06 smithi079 ceph-mon[23158]: pgmap v5158: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9690 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:07.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:06 smithi016 conmon[32727]: 2022-03-21T20:04:06.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:07.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:07 smithi016 conmon[32727]: 2022-03-21T20:04:07.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:08.847 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:08.848 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:08 smithi016 ceph-mon[22010]: pgmap v5159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:08 smithi079 ceph-mon[23158]: pgmap v5159: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:08 smithi016 conmon[32727]: 2022-03-21T20:04:08.846+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:09.892 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:09 smithi016 conmon[32727]: 2022-03-21T20:04:09.891+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:10 smithi016 ceph-mon[22010]: pgmap v5160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:10 smithi079 ceph-mon[23158]: pgmap v5160: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:10 smithi016 conmon[32727]: 2022-03-21T20:04:10.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:11 smithi016 conmon[32727]: 2022-03-21T20:04:11.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:12 smithi016 ceph-mon[22010]: pgmap v5161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:12 smithi079 ceph-mon[23158]: pgmap v5161: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:12.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:12 smithi016 conmon[32727]: 2022-03-21T20:04:12.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:14 smithi016 conmon[32727]: 2022-03-21T20:04:14.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:14.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:14 smithi079 ceph-mon[23158]: pgmap v5162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:14.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:14.898 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:04:14.898 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:04:14.899 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:14 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:04:14.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:14 smithi016 ceph-mon[22010]: pgmap v5162: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:14.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:14.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:04:14.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:04:14.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:14 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:04:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:14 smithi016 conmon[32727]: 2022-03-21T20:04:14.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9695 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:16 smithi016 conmon[32727]: 2022-03-21T20:04:16.005+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:16 smithi016 ceph-mon[22010]: pgmap v5163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:16 smithi079 ceph-mon[23158]: pgmap v5163: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:16.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:16 smithi016 conmon[32727]: 2022-03-21T20:04:16.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:17.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: 2022-03-21T20:04:17.549+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:17.915 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:04:17.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:17.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: yield 2022-03-21T20:04:17.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:17.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:04:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:04:17.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:04:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:17.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:04:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:04:17.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:17.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:17.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:17 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:18 smithi016 conmon[32727]: 2022-03-21T20:04:18.030+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:18 smithi016 ceph-mon[22010]: pgmap v5164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:18 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:04:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:04:18.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:18.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:04:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:18.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:04:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:04:18.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:18.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:04:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:18.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:18 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:04:18.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:18 smithi079 ceph-mon[23158]: pgmap v5164: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:18 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:04:18.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:04:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:18.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:18.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:04:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:04:18.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:18.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:04:18.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:18.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:18.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:18.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:04:18.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:18.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:18.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:18.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:18 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:04:18.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:19.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:18 smithi016 conmon[32727]: 2022-03-21T20:04:18.997+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:19.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:19 smithi016 conmon[32727]: 2022-03-21T20:04:19.970+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:20 smithi016 ceph-mon[22010]: pgmap v5165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 2022-03-21T20:04:20.621+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:20.916 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:04:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: yield 2022-03-21T20:04:20.917 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:20.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:20.918 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:04:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:20.919 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:04:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:04:20.920 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:04:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:20.921 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:04:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:20.922 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:20.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 2022-03-21T20:04:20.622+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:04:20.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:04:20.923 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:20.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: yield 2022-03-21T20:04:20.924 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:20.925 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:04:20.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:20.926 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:04:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:04:20.927 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:20.928 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:04:20.929 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:20.930 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:04:20.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:20.932 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:20.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:20.933 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 2022-03-21T20:04:20.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:04:20.934 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: 2022-03-21T20:04:20.935 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:04:20.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:04:20.936 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:04:20.937 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:04:20.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:04:20.938 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:04:20.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:04:20.939 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:04:20.940 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:04:20.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:04:20.941 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:04:20.943 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:04:20.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:04:20.944 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:04:20.945 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:04:20.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:04:20.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:04:20.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:04:20.947 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:04:20.948 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:04:20.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:04:20.949 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:04:20.950 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise 2022-03-21T20:04:20.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:04:20.951 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:04:20.952 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:04:20.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:04:20.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:20 smithi079 ceph-mon[23158]: pgmap v5165: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9705 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:20 smithi016 conmon[32727]: 2022-03-21T20:04:20.962+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:21 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:04:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:21.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:04:21.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:21.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:04:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:04:21.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:21.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:04:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:21.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:21 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:04:21.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:04:21.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:21.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:04:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:04:21.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:04:21.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:21.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:04:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:21.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:04:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:04:21.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:04:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:04:21.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:04:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:04:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:04:21.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:04:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:04:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:04:21.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:04:21.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:04:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:04:21.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:04:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:04:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:04:21.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:04:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:04:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:04:21.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:04:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:04:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:04:21.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:04:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:04:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:04:21.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:04:21.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.941 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:21 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:04:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:21.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:21.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:04:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:21.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:04:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:04:21.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:21.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:04:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:21.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:21 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:04:21.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:04:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:04:21.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:04:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:04:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:04:21.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:04:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:04:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:04:21.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:04:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:04:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:04:21.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:04:21.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:04:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:04:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:04:21.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:04:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:04:21.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:04:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:04:21.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:04:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:04:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:04:21.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:04:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:04:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:04:21.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:04:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:04:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:04:21.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:04:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:04:21.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:04:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:04:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:04:21.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:04:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:04:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:04:21.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:04:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:04:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:04:21.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:04:21.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:04:21.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:04:21.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:04:21.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:04:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:04:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:04:21.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:04:22.000 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:21 smithi016 conmon[32727]: 2022-03-21T20:04:21.952+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:22 smithi016 ceph-mon[22010]: pgmap v5166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:22 smithi079 ceph-mon[23158]: pgmap v5166: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:22.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:22 smithi016 conmon[32727]: 2022-03-21T20:04:22.928+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:23 smithi016 conmon[32727]: 2022-03-21T20:04:23.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:24.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:24 smithi016 ceph-mon[22010]: pgmap v5167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:24.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:24 smithi079 ceph-mon[23158]: pgmap v5167: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:24 smithi016 conmon[32727]: 2022-03-21T20:04:24.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:25 smithi016 conmon[32727]: 2022-03-21T20:04:25.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:26.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:26 smithi079 ceph-mon[23158]: pgmap v5168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:26.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:26 smithi016 ceph-mon[22010]: pgmap v5168: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9710 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:27.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:27.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:26 smithi016 conmon[32727]: 2022-03-21T20:04:26.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:27.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:28.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:28.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:28 smithi016 ceph-mon[22010]: pgmap v5169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:28.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:27 smithi016 conmon[32727]: 2022-03-21T20:04:27.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:28.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:28 smithi079 ceph-mon[23158]: pgmap v5169: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:29.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:28 smithi016 conmon[32727]: 2022-03-21T20:04:28.881+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:29.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:29 smithi016 conmon[32727]: 2022-03-21T20:04:29.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:30.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:30 smithi079 ceph-mon[23158]: pgmap v5170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:30 smithi016 ceph-mon[22010]: pgmap v5170: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:30 smithi016 conmon[32727]: 2022-03-21T20:04:30.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:31.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:31 smithi016 conmon[32727]: 2022-03-21T20:04:31.836+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:32.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:32 smithi079 ceph-mon[23158]: pgmap v5171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:32 smithi016 ceph-mon[22010]: pgmap v5171: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:33.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:32 smithi016 conmon[32727]: 2022-03-21T20:04:32.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:33.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:34.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:33 smithi016 conmon[32727]: 2022-03-21T20:04:33.798+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:34.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:34.476 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:34 smithi079 ceph-mon[23158]: pgmap v5172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:34 smithi016 ceph-mon[22010]: pgmap v5172: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:35.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:34 smithi016 conmon[32727]: 2022-03-21T20:04:34.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:35.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:35.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9715 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:36.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:35 smithi016 conmon[32727]: 2022-03-21T20:04:35.768+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:36.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:36 smithi079 ceph-mon[23158]: pgmap v5173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:36 smithi016 ceph-mon[22010]: pgmap v5173: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:37.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:36 smithi016 conmon[32727]: 2022-03-21T20:04:36.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:37.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:38.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:37 smithi016 conmon[32727]: 2022-03-21T20:04:37.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:38.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:38 smithi079 ceph-mon[23158]: pgmap v5174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:38 smithi016 ceph-mon[22010]: pgmap v5174: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:38 smithi016 conmon[32727]: 2022-03-21T20:04:38.769+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:39.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:39 smithi016 conmon[32727]: 2022-03-21T20:04:39.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:40.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:40 smithi079 ceph-mon[23158]: pgmap v5175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:40.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:40 smithi016 ceph-mon[22010]: pgmap v5175: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9725 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:40 smithi016 conmon[32727]: 2022-03-21T20:04:40.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:41.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:41 smithi016 conmon[32727]: 2022-03-21T20:04:41.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:42.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:42 smithi079 ceph-mon[23158]: pgmap v5176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:42 smithi016 ceph-mon[22010]: pgmap v5176: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:43.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:42 smithi016 conmon[32727]: 2022-03-21T20:04:42.718+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:43.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:43 smithi016 conmon[32727]: 2022-03-21T20:04:43.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:44 smithi016 ceph-mon[22010]: pgmap v5177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:44.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:44 smithi079 ceph-mon[23158]: pgmap v5177: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:44 smithi016 conmon[32727]: 2022-03-21T20:04:44.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:45 smithi016 conmon[32727]: 2022-03-21T20:04:45.750+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:46 smithi016 ceph-mon[22010]: pgmap v5178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:46.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:46 smithi079 ceph-mon[23158]: pgmap v5178: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9730 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:46 smithi016 conmon[32727]: 2022-03-21T20:04:46.794+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:47.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:47 smithi016 conmon[32727]: 2022-03-21T20:04:47.781+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:48 smithi016 ceph-mon[22010]: pgmap v5179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:48 smithi079 ceph-mon[23158]: pgmap v5179: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:48 smithi016 conmon[32727]: 2022-03-21T20:04:48.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:49.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:49 smithi016 conmon[32727]: 2022-03-21T20:04:49.738+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:50 smithi016 ceph-mon[22010]: pgmap v5180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:50 smithi079 ceph-mon[23158]: pgmap v5180: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:50 smithi016 conmon[32727]: 2022-03-21T20:04:50.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:51 smithi016 conmon[32727]: 2022-03-21T20:04:51.707+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:52 smithi016 ceph-mon[22010]: pgmap v5181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:52 smithi079 ceph-mon[23158]: pgmap v5181: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:52 smithi016 conmon[32727]: 2022-03-21T20:04:52.740+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:53 smithi016 conmon[32727]: 2022-03-21T20:04:53.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:54 smithi016 ceph-mon[22010]: pgmap v5182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:54.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:54 smithi079 ceph-mon[23158]: pgmap v5182: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:54 smithi016 conmon[32727]: 2022-03-21T20:04:54.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9735 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:04:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:55 smithi016 conmon[32727]: 2022-03-21T20:04:55.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:56 smithi016 ceph-mon[22010]: pgmap v5183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:56 smithi079 ceph-mon[23158]: pgmap v5183: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:04:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:56 smithi016 conmon[32727]: 2022-03-21T20:04:56.727+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:57 smithi016 conmon[32727]: 2022-03-21T20:04:57.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:58 smithi016 ceph-mon[22010]: pgmap v5184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:58 smithi079 ceph-mon[23158]: pgmap v5184: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:04:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:58 smithi016 conmon[32727]: 2022-03-21T20:04:58.722+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:04:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:04:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:04:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:04:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:04:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:04:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:04:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:04:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:04:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:04:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:04:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:05:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:04:59 smithi016 conmon[32727]: 2022-03-21T20:04:59.759+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:00 smithi016 ceph-mon[22010]: pgmap v5185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:00 smithi079 ceph-mon[23158]: pgmap v5185: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9745 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:00 smithi016 conmon[32727]: 2022-03-21T20:05:00.730+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:01 smithi016 conmon[32727]: 2022-03-21T20:05:01.772+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:02 smithi016 ceph-mon[22010]: pgmap v5186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:02 smithi079 ceph-mon[23158]: pgmap v5186: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:02 smithi016 conmon[32727]: 2022-03-21T20:05:02.731+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:03 smithi016 conmon[32727]: 2022-03-21T20:05:03.720+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:04 smithi016 ceph-mon[22010]: pgmap v5187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:04 smithi079 ceph-mon[23158]: pgmap v5187: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:04 smithi016 conmon[32727]: 2022-03-21T20:05:04.744+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:05 smithi016 conmon[32727]: 2022-03-21T20:05:05.714+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:06 smithi016 ceph-mon[22010]: pgmap v5188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:06 smithi079 ceph-mon[23158]: pgmap v5188: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9750 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:06 smithi016 conmon[32727]: 2022-03-21T20:05:06.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:07.656 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:07.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:07 smithi016 conmon[32727]: 2022-03-21T20:05:07.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:08.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:08 smithi016 ceph-mon[22010]: pgmap v5189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:08 smithi079 ceph-mon[23158]: pgmap v5189: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:08 smithi016 conmon[32727]: 2022-03-21T20:05:08.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:09 smithi016 conmon[32727]: 2022-03-21T20:05:09.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:10 smithi016 ceph-mon[22010]: pgmap v5190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:10 smithi079 ceph-mon[23158]: pgmap v5190: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:10 smithi016 conmon[32727]: 2022-03-21T20:05:10.671+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:11.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:11 smithi016 conmon[32727]: 2022-03-21T20:05:11.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:12.614 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:12.615 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:12 smithi016 ceph-mon[22010]: pgmap v5191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:12 smithi079 ceph-mon[23158]: pgmap v5191: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:12 smithi016 conmon[32727]: 2022-03-21T20:05:12.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:13.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:13 smithi016 conmon[32727]: 2022-03-21T20:05:13.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:14.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:14.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:14 smithi016 ceph-mon[22010]: pgmap v5192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:14.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:14 smithi079 ceph-mon[23158]: pgmap v5192: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:14 smithi016 conmon[32727]: 2022-03-21T20:05:14.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:15.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:15.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9755 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:15 smithi016 conmon[32727]: 2022-03-21T20:05:15.703+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:16.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:16 smithi016 ceph-mon[22010]: pgmap v5193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:16 smithi079 ceph-mon[23158]: pgmap v5193: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:16 smithi016 conmon[32727]: 2022-03-21T20:05:16.697+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:17.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:17 smithi016 conmon[32727]: 2022-03-21T20:05:17.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:18.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:18 smithi016 ceph-mon[22010]: pgmap v5194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:18 smithi079 ceph-mon[23158]: pgmap v5194: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:18 smithi016 conmon[32727]: 2022-03-21T20:05:18.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:19 smithi016 conmon[32727]: 2022-03-21T20:05:19.666+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:20.636 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:20.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:20 smithi016 ceph-mon[22010]: pgmap v5195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:20.637 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:20.669 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:20.669 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:20 smithi079 ceph-mon[23158]: pgmap v5195: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:20.670 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9765 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:20 smithi016 conmon[32727]: 2022-03-21T20:05:20.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:21.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: 2022-03-21T20:05:21.101+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:21.347 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:05:21.347 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:21.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: yield 2022-03-21T20:05:21.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:21.349 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:21.349 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:21.350 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:05:21.351 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:21.351 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:05:21.352 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:21.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:05:21.353 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:21.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:05:21.354 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:21.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:05:21.355 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:21.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:21.356 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:21.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:21.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:05:21.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:05:21.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:21 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:05:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:05:21.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:21 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:05:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:21 smithi016 conmon[32727]: 2022-03-21T20:05:21.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:22.625 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:22.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:22 smithi016 ceph-mon[22010]: pgmap v5196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:22.626 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:22 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:22.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:05:22.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:22.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:05:22.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:22.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:22.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:22.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:05:22.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:22.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:05:22.630 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:22.630 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:05:22.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:22.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:22.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:22.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:05:22.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:22.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:22.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:22.633 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:22 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:05:22.634 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:22 smithi079 ceph-mon[23158]: pgmap v5196: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:22 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:05:22.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:05:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:22.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:05:22.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:22.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:05:22.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:05:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:22.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:05:22.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:22.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:22.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:22.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:22 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:05:22.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:22 smithi016 conmon[32727]: 2022-03-21T20:05:22.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:23 smithi016 conmon[32727]: 2022-03-21T20:05:23.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:24.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:24.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:24 smithi016 ceph-mon[22010]: pgmap v5197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:24.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 2022-03-21T20:05:24.173+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:24.647 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:05:24.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:24.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: yield 2022-03-21T20:05:24.648 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:24.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:24.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:24.649 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:05:24.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:24.650 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:05:24.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:24.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:05:24.651 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:24.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:05:24.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:24.652 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:05:24.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:24.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:24.653 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:24.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 2022-03-21T20:05:24.174+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:05:24.654 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:05:24.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:24.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: yield 2022-03-21T20:05:24.655 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:24.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:24.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:24.656 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:05:24.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:24.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:05:24.657 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:24.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:05:24.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:24.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:05:24.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:24.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:05:24.659 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:24.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:24.660 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:24.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 2022-03-21T20:05:24.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:05:24.661 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: 2022-03-21T20:05:24.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:05:24.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:05:24.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:05:24.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:05:24.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:05:24.663 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:05:24.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:05:24.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:05:24.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:05:24.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:05:24.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:05:24.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:05:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:05:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:05:24.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:05:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:05:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:05:24.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:05:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:05:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:05:24.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:05:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:05:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise 2022-03-21T20:05:24.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:05:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:05:24.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:05:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:05:24.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:24 smithi079 ceph-mon[23158]: pgmap v5197: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:24 smithi016 conmon[32727]: 2022-03-21T20:05:24.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:25.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:25.610 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:25 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.611 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.611 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:25.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:05:25.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:25.612 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:25.613 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:25.613 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:05:25.613 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:25.614 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:05:25.614 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:25.614 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:05:25.615 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:25.615 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:25.615 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:25.616 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:05:25.616 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:25.617 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:25.617 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.617 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:25 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:05:25.618 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.619 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:25.619 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:05:25.619 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:25.620 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:25.620 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:25.620 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:05:25.621 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:25.621 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:05:25.621 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:25.622 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:05:25.622 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:25.622 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:25.623 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:25.623 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:05:25.623 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:25.624 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:25.624 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.625 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:05:25.625 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:05:25.625 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:05:25.626 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.626 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:05:25.626 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:05:25.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:05:25.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:05:25.627 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:05:25.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:05:25.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:05:25.628 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:05:25.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:05:25.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:05:25.629 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:05:25.630 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:05:25.630 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:05:25.630 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:05:25.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:05:25.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:05:25.631 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:05:25.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:05:25.632 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:05:25.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:05:25.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:05:25.633 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:05:25.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:05:25.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:05:25.634 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:05:25.635 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:05:25.635 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:25 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:05:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:25.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:25.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:05:25.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:05:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:25.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:05:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:25.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:25.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:05:25.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:25.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:25.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.733 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:25 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:05:25.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:05:25.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:05:25.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:05:25.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:05:25.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:05:25.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:05:25.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:05:25.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:05:25.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:05:25.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:05:25.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:05:25.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:05:25.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:05:25.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:05:25.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:05:25.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:05:25.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:05:25.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:05:25.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:05:25.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:05:25.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:05:25.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:05:25.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:05:25.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:05:25.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:05:25.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:05:25.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:05:25.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:05:25.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:05:25.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:05:25.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:05:25.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:05:25.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:05:25.751 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:05:25.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:05:25.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:05:25.752 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:05:25.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:05:25.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:05:25.754 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:05:25.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:05:25.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:05:25.755 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:05:25.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:05:25.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:05:25.756 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:05:25.757 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:05:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:25 smithi016 conmon[32727]: 2022-03-21T20:05:25.605+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:26.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:26 smithi016 conmon[32727]: 2022-03-21T20:05:26.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:26 smithi016 ceph-mon[22010]: pgmap v5198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:26.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:26 smithi079 ceph-mon[23158]: pgmap v5198: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9770 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:27 smithi016 conmon[32727]: 2022-03-21T20:05:27.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:27.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:28 smithi016 ceph-mon[22010]: pgmap v5199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:28 smithi016 conmon[32727]: 2022-03-21T20:05:28.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:28 smithi079 ceph-mon[23158]: pgmap v5199: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:29 smithi016 conmon[32727]: 2022-03-21T20:05:29.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:30 smithi016 ceph-mon[22010]: pgmap v5200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:30 smithi016 conmon[32727]: 2022-03-21T20:05:30.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:30 smithi079 ceph-mon[23158]: pgmap v5200: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:31 smithi016 conmon[32727]: 2022-03-21T20:05:31.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:32 smithi016 conmon[32727]: 2022-03-21T20:05:32.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:32 smithi016 ceph-mon[22010]: pgmap v5201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:32.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:32 smithi079 ceph-mon[23158]: pgmap v5201: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:33.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:33 smithi016 conmon[32727]: 2022-03-21T20:05:33.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:33.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:34.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:34 smithi016 conmon[32727]: 2022-03-21T20:05:34.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:34 smithi016 ceph-mon[22010]: pgmap v5202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:34 smithi079 ceph-mon[23158]: pgmap v5202: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:35 smithi016 conmon[32727]: 2022-03-21T20:05:35.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9775 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:36.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:36 smithi016 conmon[32727]: 2022-03-21T20:05:36.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:36 smithi016 ceph-mon[22010]: pgmap v5203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:36 smithi079 ceph-mon[23158]: pgmap v5203: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:37.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:37 smithi016 conmon[32727]: 2022-03-21T20:05:37.604+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:38.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:38 smithi016 conmon[32727]: 2022-03-21T20:05:38.585+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:38 smithi016 ceph-mon[22010]: pgmap v5204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:38 smithi079 ceph-mon[23158]: pgmap v5204: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:39 smithi016 conmon[32727]: 2022-03-21T20:05:39.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:39.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:40.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:40 smithi016 ceph-mon[22010]: pgmap v5205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:40.663 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:40 smithi079 ceph-mon[23158]: pgmap v5205: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9785 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:40 smithi016 conmon[32727]: 2022-03-21T20:05:40.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:41.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:41 smithi016 conmon[32727]: 2022-03-21T20:05:41.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:42 smithi016 ceph-mon[22010]: pgmap v5206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:42 smithi079 ceph-mon[23158]: pgmap v5206: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:42 smithi016 conmon[32727]: 2022-03-21T20:05:42.665+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:43.704 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:43 smithi016 conmon[32727]: 2022-03-21T20:05:43.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:44.704 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:44.705 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:44 smithi016 ceph-mon[22010]: pgmap v5207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:44 smithi079 ceph-mon[23158]: pgmap v5207: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:44 smithi016 conmon[32727]: 2022-03-21T20:05:44.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:45.712 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:45 smithi016 conmon[32727]: 2022-03-21T20:05:45.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:46 smithi079 ceph-mon[23158]: pgmap v5208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:46.735 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:46.735 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:46 smithi016 ceph-mon[22010]: pgmap v5208: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:46.735 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9790 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:46 smithi016 conmon[32727]: 2022-03-21T20:05:46.732+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:47.715 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:47 smithi016 conmon[32727]: 2022-03-21T20:05:47.713+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:48.699 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:48.699 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:48 smithi016 ceph-mon[22010]: pgmap v5209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:48 smithi079 ceph-mon[23158]: pgmap v5209: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:48 smithi016 conmon[32727]: 2022-03-21T20:05:48.697+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:49.724 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:49 smithi016 conmon[32727]: 2022-03-21T20:05:49.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:50 smithi079 ceph-mon[23158]: pgmap v5210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:50.754 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:50.755 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:50 smithi016 ceph-mon[22010]: pgmap v5210: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:50 smithi016 conmon[32727]: 2022-03-21T20:05:50.752+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:51.782 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:51 smithi016 conmon[32727]: 2022-03-21T20:05:51.780+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:52 smithi079 ceph-mon[23158]: pgmap v5211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:52.788 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:52.788 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:52 smithi016 ceph-mon[22010]: pgmap v5211: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:52 smithi016 conmon[32727]: 2022-03-21T20:05:52.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:53.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:53 smithi016 conmon[32727]: 2022-03-21T20:05:53.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:54 smithi079 ceph-mon[23158]: pgmap v5212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:54.776 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:54.777 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:54 smithi016 ceph-mon[22010]: pgmap v5212: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:54 smithi016 conmon[32727]: 2022-03-21T20:05:54.775+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:55.797 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:55.797 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9795 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:05:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:55 smithi016 conmon[32727]: 2022-03-21T20:05:55.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:56 smithi079 ceph-mon[23158]: pgmap v5213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:56.765 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:56.766 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:56 smithi016 ceph-mon[22010]: pgmap v5213: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:05:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:56 smithi016 conmon[32727]: 2022-03-21T20:05:56.764+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:57.784 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:57 smithi016 conmon[32727]: 2022-03-21T20:05:57.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:58 smithi079 ceph-mon[23158]: pgmap v5214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:58.777 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:58.777 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:58 smithi016 ceph-mon[22010]: pgmap v5214: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:05:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:58 smithi016 conmon[32727]: 2022-03-21T20:05:58.776+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:05:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:05:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:05:59.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:05:59.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:05:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:05:59.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:05:59.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:05:59.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:05:59.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:05:59.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:05:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:06:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:05:59 smithi016 conmon[32727]: 2022-03-21T20:05:59.805+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:00.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:00.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:00 smithi016 ceph-mon[22010]: pgmap v5215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:00.808 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:00 smithi079 ceph-mon[23158]: pgmap v5215: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:00.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9805 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:00 smithi016 conmon[32727]: 2022-03-21T20:06:00.805+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:01.844 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:01 smithi016 conmon[32727]: 2022-03-21T20:06:01.842+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:02.878 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:02.879 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:02 smithi016 ceph-mon[22010]: pgmap v5216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:02 smithi079 ceph-mon[23158]: pgmap v5216: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:02 smithi016 conmon[32727]: 2022-03-21T20:06:02.877+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:04.096 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:04.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:03 smithi016 conmon[32727]: 2022-03-21T20:06:03.922+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:04.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:04.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:04 smithi016 ceph-mon[22010]: pgmap v5217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:04 smithi079 ceph-mon[23158]: pgmap v5217: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:04 smithi016 conmon[32727]: 2022-03-21T20:06:04.899+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:05 smithi016 conmon[32727]: 2022-03-21T20:06:05.927+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:06.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:06.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:06 smithi016 ceph-mon[22010]: pgmap v5218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:06.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:06 smithi079 ceph-mon[23158]: pgmap v5218: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9810 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:07.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:06 smithi016 conmon[32727]: 2022-03-21T20:06:06.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:07.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:08.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:07 smithi016 conmon[32727]: 2022-03-21T20:06:07.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:08.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:08.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:08 smithi016 ceph-mon[22010]: pgmap v5219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:08 smithi079 ceph-mon[23158]: pgmap v5219: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:09.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:08 smithi016 conmon[32727]: 2022-03-21T20:06:08.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:09.880 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:09 smithi016 conmon[32727]: 2022-03-21T20:06:09.879+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:10.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:10.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:10 smithi016 ceph-mon[22010]: pgmap v5220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:10.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:10 smithi079 ceph-mon[23158]: pgmap v5220: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:10 smithi016 conmon[32727]: 2022-03-21T20:06:10.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:11.881 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:12.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:11 smithi016 conmon[32727]: 2022-03-21T20:06:11.880+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:12.863 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:12.864 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:12 smithi016 ceph-mon[22010]: pgmap v5221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:12 smithi079 ceph-mon[23158]: pgmap v5221: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:12 smithi016 conmon[32727]: 2022-03-21T20:06:12.862+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:13.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:13 smithi016 conmon[32727]: 2022-03-21T20:06:13.839+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:14.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:14.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:14 smithi016 ceph-mon[22010]: pgmap v5222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:14 smithi079 ceph-mon[23158]: pgmap v5222: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:15.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:14 smithi016 conmon[32727]: 2022-03-21T20:06:14.840+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:15.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:15.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9815 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:16.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:15 smithi016 conmon[32727]: 2022-03-21T20:06:15.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:16.883 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:16.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:16 smithi016 ceph-mon[22010]: pgmap v5223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:16 smithi079 ceph-mon[23158]: pgmap v5223: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:17.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:16 smithi016 conmon[32727]: 2022-03-21T20:06:16.882+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:17.853 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:18.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:17 smithi016 conmon[32727]: 2022-03-21T20:06:17.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:18.869 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:18.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:18 smithi016 ceph-mon[22010]: pgmap v5224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:18 smithi079 ceph-mon[23158]: pgmap v5224: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:18 smithi016 conmon[32727]: 2022-03-21T20:06:18.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:19.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:19 smithi016 conmon[32727]: 2022-03-21T20:06:19.853+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:20.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:20.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:20 smithi016 ceph-mon[22010]: pgmap v5225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:20.904 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:20 smithi079 ceph-mon[23158]: pgmap v5225: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9825 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:20 smithi016 conmon[32727]: 2022-03-21T20:06:20.902+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:21 smithi016 conmon[32727]: 2022-03-21T20:06:21.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:22.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:22 smithi016 ceph-mon[22010]: pgmap v5226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:22 smithi079 ceph-mon[23158]: pgmap v5226: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:23.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:22 smithi016 conmon[32727]: 2022-03-21T20:06:22.917+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:23.908 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:23 smithi016 conmon[32727]: 2022-03-21T20:06:23.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:24.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:24.871 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:24 smithi016 ceph-mon[22010]: pgmap v5227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:24.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:06:24.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:06:24.872 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:24 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:06:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:24 smithi079 ceph-mon[23158]: pgmap v5227: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:06:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:06:24.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:24 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:06:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:24 smithi016 conmon[32727]: 2022-03-21T20:06:24.870+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:25.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:25 smithi016 conmon[32727]: 2022-03-21T20:06:25.884+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:26.874 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:26.874 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:26 smithi016 ceph-mon[22010]: pgmap v5228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:26.875 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:26 smithi079 ceph-mon[23158]: pgmap v5228: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9830 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:26 smithi016 conmon[32727]: 2022-03-21T20:06:26.873+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:27.557 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: 2022-03-21T20:06:27.277+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:27.557 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:06:27.557 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:27.558 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: yield 2022-03-21T20:06:27.558 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:27.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:27.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:27.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:06:27.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:27.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:06:27.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:27.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:06:27.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:27.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:06:27.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:27.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:06:27.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:27.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:27.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:27.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:27.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:06:27.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:06:27.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.910 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.911 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.911 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.912 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:27 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:27.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:06:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:06:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:27.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:27 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-03-21T20:06:28.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:27 smithi016 conmon[32727]: 2022-03-21T20:06:27.908+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:28.894 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:28.894 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: pgmap v5229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:28.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T20:06:28.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T20:06:28.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Adjusting osd_memory_target on smithi016 to 160.2M 2022-03-21T20:06:28.896 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Unable to set osd_memory_target on smithi016 to 168027431: error parsing value: Value '168027431' is below minimum 939524096 2022-03-21T20:06:28.896 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:28.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:06:28.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:28.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:06:28.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:28.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:28.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:28.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:06:28.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:28.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:06:28.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:28.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:06:28.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:28.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:28.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:28.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:06:28.902 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:28.902 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:28.903 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:28.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:28 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:06:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: pgmap v5229: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi079 to 197.0M 2022-03-21T20:06:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi079 to 206644838: error parsing value: Value '206644838' is below minimum 939524096 2022-03-21T20:06:28.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Adjusting osd_memory_target on smithi016 to 160.2M 2022-03-21T20:06:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Unable to set osd_memory_target on smithi016 to 168027431: error parsing value: Value '168027431' is below minimum 939524096 2022-03-21T20:06:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:28.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:06:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:06:28.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:28.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:28.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:06:28.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:06:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:28.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:06:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:28.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:28.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:06:28.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:28.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:28.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:28.982 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:28 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:06:29.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:28 smithi016 conmon[32727]: 2022-03-21T20:06:28.893+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:29.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:29 smithi016 conmon[32727]: 2022-03-21T20:06:29.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:30 smithi016 ceph-mon[22010]: pgmap v5230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:30.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 2022-03-21T20:06:30.349+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:30.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:06:30.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:30.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: yield 2022-03-21T20:06:30.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:30.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:30.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:30.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:06:30.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:30.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:06:30.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:30.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:06:30.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:30.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:06:30.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:30.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:06:30.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:30.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:30.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:30.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 2022-03-21T20:06:30.350+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:06:30.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:06:30.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:30.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: yield 2022-03-21T20:06:30.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:30.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:30.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:30.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:06:30.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:30.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:06:30.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:30.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:06:30.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:30.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:06:30.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:30.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:06:30.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:30.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:30.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:30.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 2022-03-21T20:06:30.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:06:30.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: 2022-03-21T20:06:30.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:06:30.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:06:30.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:06:30.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:06:30.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:06:30.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:06:30.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:06:30.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:06:30.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:06:30.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:06:30.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:06:30.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:06:30.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:06:30.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:06:30.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:06:30.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:06:30.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:06:30.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:06:30.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:06:30.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:06:30.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:06:30.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:06:30.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise 2022-03-21T20:06:30.693 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:06:30.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:06:30.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:06:30.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:06:30.695 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:30 smithi079 ceph-mon[23158]: pgmap v5230: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:30 smithi016 conmon[32727]: 2022-03-21T20:06:30.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:31.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:31.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:31 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.828 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.828 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:31.829 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:06:31.829 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:31.829 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:31.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:31.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:06:31.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:31.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:06:31.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:31.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:06:31.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:31.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:31.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:31.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:06:31.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:31.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:31.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:31 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:06:31.835 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.835 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:31.836 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:06:31.836 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:31.836 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:31.837 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:31.837 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:06:31.837 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:31.838 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:06:31.838 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:31.839 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:06:31.839 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:31.839 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:31.840 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:31.840 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:06:31.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:31.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:31.841 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:06:31.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:06:31.842 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:06:31.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:06:31.843 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:06:31.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:06:31.844 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:06:31.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:06:31.845 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:06:31.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:06:31.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:06:31.846 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:06:31.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:06:31.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:06:31.847 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:06:31.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:06:31.848 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:06:31.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:06:31.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:06:31.849 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:06:31.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:06:31.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:06:31.850 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:06:31.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:06:31.851 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:06:31.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:06:31.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:06:31.852 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:06:31.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:06:31.853 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:31 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:06:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:31.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:06:31.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:06:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:31.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:06:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:31.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:06:31.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:31.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:31.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:31 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:06:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:06:31.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:06:31.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:06:31.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:06:31.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:06:31.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:06:31.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:06:31.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:06:31.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:06:31.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:06:31.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:06:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:06:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:06:31.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:06:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:06:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:06:31.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:31.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:06:31.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:06:31.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:06:31.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:06:31.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:06:31.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:06:31.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:06:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:06:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:06:31.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:06:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:06:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:06:31.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:06:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:06:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:06:31.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:06:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:06:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:06:31.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:06:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:06:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:06:31.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:06:31.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:06:31.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:06:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:06:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:06:31.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:06:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:06:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:06:31.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:06:31.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:06:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:31 smithi016 conmon[32727]: 2022-03-21T20:06:31.826+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:32 smithi016 ceph-mon[22010]: pgmap v5231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:32 smithi016 conmon[32727]: 2022-03-21T20:06:32.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:32 smithi079 ceph-mon[23158]: pgmap v5231: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:33 smithi016 conmon[32727]: 2022-03-21T20:06:33.819+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:33.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:34 smithi016 ceph-mon[22010]: pgmap v5232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:34 smithi016 conmon[32727]: 2022-03-21T20:06:34.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:34 smithi079 ceph-mon[23158]: pgmap v5232: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:35 smithi016 conmon[32727]: 2022-03-21T20:06:35.813+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9835 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:36.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:36.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:36 smithi016 ceph-mon[22010]: pgmap v5233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:36.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:36.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:36 smithi079 ceph-mon[23158]: pgmap v5233: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:37.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:36 smithi016 conmon[32727]: 2022-03-21T20:06:36.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:37.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:38.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:37 smithi016 conmon[32727]: 2022-03-21T20:06:37.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:38.897 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:38.897 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:38 smithi016 ceph-mon[22010]: pgmap v5234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:38 smithi079 ceph-mon[23158]: pgmap v5234: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:39.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:38 smithi016 conmon[32727]: 2022-03-21T20:06:38.896+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:39.890 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:40.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:39 smithi016 conmon[32727]: 2022-03-21T20:06:39.888+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:40.888 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:40.888 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:40 smithi016 ceph-mon[22010]: pgmap v5235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:40.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:40 smithi079 ceph-mon[23158]: pgmap v5235: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:40.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9845 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:40 smithi016 conmon[32727]: 2022-03-21T20:06:40.886+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:41.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:42.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:41 smithi016 conmon[32727]: 2022-03-21T20:06:41.918+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:42.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:42.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:42 smithi016 ceph-mon[22010]: pgmap v5236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:42 smithi079 ceph-mon[23158]: pgmap v5236: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:43.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:42 smithi016 conmon[32727]: 2022-03-21T20:06:42.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:44.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:43 smithi016 conmon[32727]: 2022-03-21T20:06:43.915+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:44.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:44.900 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:44 smithi016 ceph-mon[22010]: pgmap v5237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:44 smithi079 ceph-mon[23158]: pgmap v5237: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:44 smithi016 conmon[32727]: 2022-03-21T20:06:44.898+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:45.876 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:45 smithi016 conmon[32727]: 2022-03-21T20:06:45.874+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:46.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:46 smithi016 conmon[32727]: 2022-03-21T20:06:46.857+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:46 smithi016 ceph-mon[22010]: pgmap v5238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:46 smithi079 ceph-mon[23158]: pgmap v5238: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9850 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:47.896 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:47 smithi016 conmon[32727]: 2022-03-21T20:06:47.895+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:48.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:48.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:48 smithi016 ceph-mon[22010]: pgmap v5239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:48 smithi079 ceph-mon[23158]: pgmap v5239: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:48 smithi016 conmon[32727]: 2022-03-21T20:06:48.897+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:49 smithi016 conmon[32727]: 2022-03-21T20:06:49.851+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:49.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:50 smithi016 conmon[32727]: 2022-03-21T20:06:50.855+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:50 smithi016 ceph-mon[22010]: pgmap v5240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:50 smithi079 ceph-mon[23158]: pgmap v5240: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:51 smithi016 conmon[32727]: 2022-03-21T20:06:51.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:52 smithi016 ceph-mon[22010]: pgmap v5241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:52 smithi016 conmon[32727]: 2022-03-21T20:06:52.843+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:52 smithi079 ceph-mon[23158]: pgmap v5241: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:53 smithi016 conmon[32727]: 2022-03-21T20:06:53.838+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:54 smithi016 conmon[32727]: 2022-03-21T20:06:54.878+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:54 smithi016 ceph-mon[22010]: pgmap v5242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:54 smithi079 ceph-mon[23158]: pgmap v5242: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:55.906 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:55.906 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9855 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:06:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:55 smithi016 conmon[32727]: 2022-03-21T20:06:55.904+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:56 smithi016 ceph-mon[22010]: pgmap v5243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:56 smithi079 ceph-mon[23158]: pgmap v5243: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:06:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:57.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:56 smithi016 conmon[32727]: 2022-03-21T20:06:56.934+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:57 smithi016 conmon[32727]: 2022-03-21T20:06:57.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:58 smithi016 ceph-mon[22010]: pgmap v5244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:58 smithi079 ceph-mon[23158]: pgmap v5244: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:06:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:58 smithi016 conmon[32727]: 2022-03-21T20:06:58.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:06:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:06:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:06:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:06:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:06:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:06:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:06:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:06:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:06:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:06:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:06:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:06:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:06:59 smithi016 conmon[32727]: 2022-03-21T20:06:59.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:00 smithi016 ceph-mon[22010]: pgmap v5245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:00 smithi016 conmon[32727]: 2022-03-21T20:07:00.900+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:00 smithi079 ceph-mon[23158]: pgmap v5245: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9865 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:01.933 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:01 smithi016 conmon[32727]: 2022-03-21T20:07:01.932+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:02.957 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:02 smithi016 ceph-mon[22010]: pgmap v5246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:02.958 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:02.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:02 smithi079 ceph-mon[23158]: pgmap v5246: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:02 smithi016 conmon[32727]: 2022-03-21T20:07:02.956+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:03.956 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:03 smithi016 conmon[32727]: 2022-03-21T20:07:03.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:04.955 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:04 smithi016 ceph-mon[22010]: pgmap v5247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:04.955 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:04 smithi079 ceph-mon[23158]: pgmap v5247: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:04 smithi016 conmon[32727]: 2022-03-21T20:07:04.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:06.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:05 smithi016 conmon[32727]: 2022-03-21T20:07:05.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:06.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:06 smithi016 ceph-mon[22010]: pgmap v5248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:06.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:06 smithi079 ceph-mon[23158]: pgmap v5248: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:06.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9870 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:07.206 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:06 smithi016 conmon[32727]: 2022-03-21T20:07:06.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:07.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:08.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:07 smithi016 conmon[32727]: 2022-03-21T20:07:07.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:08.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:08 smithi016 ceph-mon[22010]: pgmap v5249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:08.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:08.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:08 smithi079 ceph-mon[23158]: pgmap v5249: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:09.212 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:08 smithi016 conmon[32727]: 2022-03-21T20:07:08.932+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:09.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:09 smithi016 conmon[32727]: 2022-03-21T20:07:09.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:10.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:10.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:10 smithi016 ceph-mon[22010]: pgmap v5250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:10.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:10 smithi079 ceph-mon[23158]: pgmap v5250: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:11.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:11.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:10 smithi016 conmon[32727]: 2022-03-21T20:07:10.985+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:11.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:12.220 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:11 smithi016 conmon[32727]: 2022-03-21T20:07:11.939+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:12.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:12.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:12 smithi079 ceph-mon[23158]: pgmap v5251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:12.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:12.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:12 smithi016 ceph-mon[22010]: pgmap v5251: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:13.225 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:12 smithi016 conmon[32727]: 2022-03-21T20:07:12.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:14.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:14.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:14 smithi016 ceph-mon[22010]: pgmap v5252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:14.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:13 smithi016 conmon[32727]: 2022-03-21T20:07:13.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:14 smithi079 ceph-mon[23158]: pgmap v5252: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:15.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:15.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:15.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:15 smithi016 conmon[32727]: 2022-03-21T20:07:15.013+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9875 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:16.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:16 smithi016 ceph-mon[22010]: pgmap v5253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:16.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:16 smithi016 conmon[32727]: 2022-03-21T20:07:16.033+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:16 smithi079 ceph-mon[23158]: pgmap v5253: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:17.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:17.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:17 smithi016 conmon[32727]: 2022-03-21T20:07:17.064+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:18.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:18 smithi016 ceph-mon[22010]: pgmap v5254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:18.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:18.416 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:18 smithi016 conmon[32727]: 2022-03-21T20:07:18.035+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:18 smithi079 ceph-mon[23158]: pgmap v5254: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:18.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:19.244 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:18 smithi016 conmon[32727]: 2022-03-21T20:07:18.989+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:20 smithi016 conmon[32727]: 2022-03-21T20:07:20.007+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:20 smithi016 ceph-mon[22010]: pgmap v5255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:20.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:20 smithi079 ceph-mon[23158]: pgmap v5255: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9885 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:21.251 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:20 smithi016 conmon[32727]: 2022-03-21T20:07:20.987+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:22.252 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:21 smithi016 conmon[32727]: 2022-03-21T20:07:21.959+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:22 smithi016 ceph-mon[22010]: pgmap v5256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:22 smithi079 ceph-mon[23158]: pgmap v5256: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:23.257 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:22 smithi016 conmon[32727]: 2022-03-21T20:07:22.930+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:24.259 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:23 smithi016 conmon[32727]: 2022-03-21T20:07:23.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:24 smithi016 ceph-mon[22010]: pgmap v5257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:24 smithi079 ceph-mon[23158]: pgmap v5257: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:25.263 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:24 smithi016 conmon[32727]: 2022-03-21T20:07:24.924+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:26.264 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:25 smithi016 conmon[32727]: 2022-03-21T20:07:25.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:26 smithi016 ceph-mon[22010]: pgmap v5258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:26 smithi079 ceph-mon[23158]: pgmap v5258: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9890 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:27.269 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:26 smithi016 conmon[32727]: 2022-03-21T20:07:26.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:28.271 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:27 smithi016 conmon[32727]: 2022-03-21T20:07:27.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:28 smithi016 ceph-mon[22010]: pgmap v5259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:28 smithi079 ceph-mon[23158]: pgmap v5259: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:29.276 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:29 smithi016 conmon[32727]: 2022-03-21T20:07:29.008+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:30.278 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:30 smithi016 conmon[32727]: 2022-03-21T20:07:30.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:30 smithi016 ceph-mon[22010]: pgmap v5260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:30 smithi079 ceph-mon[23158]: pgmap v5260: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:31 smithi016 conmon[32727]: 2022-03-21T20:07:31.002+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:07:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:07:31.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:31 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:07:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:07:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:07:31.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:31 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:07:32.283 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:31 smithi016 conmon[32727]: 2022-03-21T20:07:31.960+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:32 smithi016 ceph-mon[22010]: pgmap v5261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:32 smithi079 ceph-mon[23158]: pgmap v5261: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:33.288 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:32 smithi016 conmon[32727]: 2022-03-21T20:07:32.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:33.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: 2022-03-21T20:07:33.453+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:33.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:07:33.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:33.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: yield 2022-03-21T20:07:33.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:33.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:33.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:33.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:07:33.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:33.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:07:33.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:33.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:07:33.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:33.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:07:33.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:33.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:07:33.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:33.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:33.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:34.290 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:33 smithi016 conmon[32727]: 2022-03-21T20:07:33.935+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:34 smithi016 ceph-mon[22010]: pgmap v5262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:34 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:07:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:34.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:07:34.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:34.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:07:34.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:07:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:34.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:07:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:34.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:07:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:34.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:34.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:34.673 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:34 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:07:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:34 smithi079 ceph-mon[23158]: pgmap v5262: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:34 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:07:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:34.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:07:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:34.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:07:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:34.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:07:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:07:34.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:34.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:07:34.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:34.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:34 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:07:35.294 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:34 smithi016 conmon[32727]: 2022-03-21T20:07:34.937+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:35.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9895 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:36.296 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:35 smithi016 conmon[32727]: 2022-03-21T20:07:35.929+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:36 smithi016 ceph-mon[22010]: pgmap v5263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:36.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 2022-03-21T20:07:36.525+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:36.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:07:36.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:36.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: yield 2022-03-21T20:07:36.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:36.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:36.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:36.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:07:36.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:36.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:07:36.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:36.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:07:36.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:36.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:07:36.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:36.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:07:36.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:36.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:36.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:36.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 2022-03-21T20:07:36.526+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:07:36.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:07:36.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:36.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: yield 2022-03-21T20:07:36.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:36.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:36.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:36.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:07:36.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:36.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:07:36.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:36.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:07:36.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:36.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:07:36.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:36.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:07:36.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:36.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:36.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:36.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 2022-03-21T20:07:36.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:07:36.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: 2022-03-21T20:07:36.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:07:36.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:07:36.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:07:36.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:07:36.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:07:36.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:07:36.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:07:36.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:07:36.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:07:36.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:07:36.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:07:36.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:07:36.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:07:36.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:07:36.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:07:36.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:07:36.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:07:36.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:07:36.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:07:36.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:07:36.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:07:36.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:07:36.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise 2022-03-21T20:07:36.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:07:36.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:07:36.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:07:36.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:07:36.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:36 smithi079 ceph-mon[23158]: pgmap v5263: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:37.300 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:36 smithi016 conmon[32727]: 2022-03-21T20:07:36.975+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:37 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:37.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:07:37.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:37.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:37.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:37.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:07:37.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:37.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:07:37.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:37.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:07:37.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:37.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:37.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:37.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:07:37.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:37.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:37.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:37 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:07:37.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:37.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:07:37.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:37.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:37.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:37.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:07:37.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:37.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:07:37.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:37.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:07:37.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:37.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:37.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:37.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:07:37.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:37.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:37.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:07:37.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:07:37.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:07:37.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:07:37.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:07:37.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:07:37.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:07:37.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:07:37.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:07:37.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:07:37.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:07:37.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:07:37.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:07:37.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:07:37.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:07:37.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:07:37.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:07:37.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:07:37.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:07:37.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:07:37.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:07:37.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:07:37.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:07:37.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:07:37.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:07:37.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:07:37.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:07:37.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:07:37.690 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:07:37.691 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:37 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:07:37.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:37.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:07:37.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:37.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:07:37.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:37.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:07:37.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:37.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:37.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:37.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:07:37.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:37 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:07:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:07:37.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:07:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:07:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:07:37.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:07:37.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:07:37.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:07:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:07:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:07:37.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:07:37.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:07:37.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:07:37.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:07:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:07:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:07:37.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:07:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:07:37.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:07:37.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:07:37.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:07:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:07:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:07:37.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:07:37.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:07:37.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:07:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:07:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:07:37.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:07:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:07:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:07:37.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:07:37.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:07:37.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:07:37.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:07:37.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:07:37.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:07:37.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:07:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:07:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:07:37.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:07:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:07:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:07:37.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:07:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:07:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:07:37.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:07:37.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:07:38.301 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:37 smithi016 conmon[32727]: 2022-03-21T20:07:37.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:38.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:38 smithi016 ceph-mon[22010]: pgmap v5264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:38 smithi079 ceph-mon[23158]: pgmap v5264: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:39.303 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:38 smithi016 conmon[32727]: 2022-03-21T20:07:38.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:40.309 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:39 smithi016 conmon[32727]: 2022-03-21T20:07:39.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:40 smithi016 ceph-mon[22010]: pgmap v5265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:40.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:40 smithi079 ceph-mon[23158]: pgmap v5265: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9905 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:41.313 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:41 smithi016 conmon[32727]: 2022-03-21T20:07:41.026+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:42.315 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:42 smithi016 conmon[32727]: 2022-03-21T20:07:42.045+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:42.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:42 smithi016 ceph-mon[22010]: pgmap v5266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:42.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:42 smithi079 ceph-mon[23158]: pgmap v5266: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:43.320 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:43 smithi016 conmon[32727]: 2022-03-21T20:07:43.029+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:43.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:43.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:44.322 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:44 smithi016 conmon[32727]: 2022-03-21T20:07:44.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:44.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:44.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:44 smithi016 ceph-mon[22010]: pgmap v5267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:44 smithi079 ceph-mon[23158]: pgmap v5267: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:45.326 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:45 smithi016 conmon[32727]: 2022-03-21T20:07:45.067+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:46 smithi016 ceph-mon[22010]: pgmap v5268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:46.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:46.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:46.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:46 smithi016 conmon[32727]: 2022-03-21T20:07:46.117+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:46 smithi079 ceph-mon[23158]: pgmap v5268: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9910 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:47.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:47.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:47 smithi016 conmon[32727]: 2022-03-21T20:07:47.109+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:48 smithi016 ceph-mon[22010]: pgmap v5269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:48.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:48.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:48 smithi016 conmon[32727]: 2022-03-21T20:07:48.100+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:48 smithi079 ceph-mon[23158]: pgmap v5269: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:49.339 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:49 smithi016 conmon[32727]: 2022-03-21T20:07:49.052+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:50 smithi016 ceph-mon[22010]: pgmap v5270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:50.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:50.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:50 smithi016 conmon[32727]: 2022-03-21T20:07:50.101+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:50 smithi079 ceph-mon[23158]: pgmap v5270: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:51.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:51.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:51 smithi016 conmon[32727]: 2022-03-21T20:07:51.106+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:52.348 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:52 smithi016 conmon[32727]: 2022-03-21T20:07:52.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:52 smithi016 ceph-mon[22010]: pgmap v5271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:52 smithi079 ceph-mon[23158]: pgmap v5271: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:53.359 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:53 smithi016 conmon[32727]: 2022-03-21T20:07:53.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:54.360 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:54 smithi016 conmon[32727]: 2022-03-21T20:07:54.093+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:54 smithi016 ceph-mon[22010]: pgmap v5272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:54.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:54 smithi079 ceph-mon[23158]: pgmap v5272: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:55.364 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:55 smithi016 conmon[32727]: 2022-03-21T20:07:55.049+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9915 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:07:56.366 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:56 smithi016 conmon[32727]: 2022-03-21T20:07:56.078+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:56 smithi016 ceph-mon[22010]: pgmap v5273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:56 smithi079 ceph-mon[23158]: pgmap v5273: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:07:56.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:57.372 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:57 smithi016 conmon[32727]: 2022-03-21T20:07:57.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:57.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:58.373 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:58 smithi016 conmon[32727]: 2022-03-21T20:07:58.094+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:58 smithi016 ceph-mon[22010]: pgmap v5274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:58 smithi079 ceph-mon[23158]: pgmap v5274: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:07:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:59.378 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:07:59 smithi016 conmon[32727]: 2022-03-21T20:07:59.075+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:07:59.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:07:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:07:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:07:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:07:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:07:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:07:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:07:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:07:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:07:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:07:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:08:00.380 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:00 smithi016 conmon[32727]: 2022-03-21T20:08:00.077+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:00 smithi016 ceph-mon[22010]: pgmap v5275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:00 smithi079 ceph-mon[23158]: pgmap v5275: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9925 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:01.384 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:01 smithi016 conmon[32727]: 2022-03-21T20:08:01.110+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:02.386 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:02 smithi016 conmon[32727]: 2022-03-21T20:08:02.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:02 smithi016 ceph-mon[22010]: pgmap v5276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:02 smithi079 ceph-mon[23158]: pgmap v5276: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:03.390 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:03 smithi016 conmon[32727]: 2022-03-21T20:08:03.082+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:04.392 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:04 smithi016 conmon[32727]: 2022-03-21T20:08:04.119+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:04 smithi016 ceph-mon[22010]: pgmap v5277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:04 smithi079 ceph-mon[23158]: pgmap v5277: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:05.396 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:05 smithi016 conmon[32727]: 2022-03-21T20:08:05.088+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:06.398 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:06 smithi016 conmon[32727]: 2022-03-21T20:08:06.066+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:06 smithi016 ceph-mon[22010]: pgmap v5278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:06.769 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:06.770 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:06.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:06 smithi079 ceph-mon[23158]: pgmap v5278: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:06.771 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:06.772 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9930 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:07.403 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:07 smithi016 conmon[32727]: 2022-03-21T20:08:07.108+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:08.405 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:08 smithi016 conmon[32727]: 2022-03-21T20:08:08.071+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:08 smithi016 ceph-mon[22010]: pgmap v5279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:08 smithi079 ceph-mon[23158]: pgmap v5279: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:08.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:09.410 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:09 smithi016 conmon[32727]: 2022-03-21T20:08:09.021+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:10.420 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:09 smithi016 conmon[32727]: 2022-03-21T20:08:09.980+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:10 smithi016 ceph-mon[22010]: pgmap v5280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:10 smithi079 ceph-mon[23158]: pgmap v5280: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:10 smithi016 conmon[32727]: 2022-03-21T20:08:10.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:11.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:11 smithi016 conmon[32727]: 2022-03-21T20:08:11.971+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:12 smithi079 ceph-mon[23158]: pgmap v5281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:12 smithi016 ceph-mon[22010]: pgmap v5281: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:12 smithi016 conmon[32727]: 2022-03-21T20:08:12.941+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:13 smithi016 conmon[32727]: 2022-03-21T20:08:13.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:14 smithi079 ceph-mon[23158]: pgmap v5282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:14 smithi016 ceph-mon[22010]: pgmap v5282: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:14 smithi016 conmon[32727]: 2022-03-21T20:08:14.921+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9935 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:15 smithi016 conmon[32727]: 2022-03-21T20:08:15.943+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:16 smithi079 ceph-mon[23158]: pgmap v5283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:16 smithi016 ceph-mon[22010]: pgmap v5283: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:16 smithi016 conmon[32727]: 2022-03-21T20:08:16.988+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:17 smithi016 conmon[32727]: 2022-03-21T20:08:17.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:18 smithi079 ceph-mon[23158]: pgmap v5284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:18.905 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:18.906 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:18 smithi016 ceph-mon[22010]: pgmap v5284: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:18 smithi016 conmon[32727]: 2022-03-21T20:08:18.904+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:19 smithi016 conmon[32727]: 2022-03-21T20:08:19.945+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:20 smithi079 ceph-mon[23158]: pgmap v5285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:20 smithi016 ceph-mon[22010]: pgmap v5285: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9945 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:20 smithi016 conmon[32727]: 2022-03-21T20:08:20.955+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:21 smithi016 conmon[32727]: 2022-03-21T20:08:21.998+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:22 smithi079 ceph-mon[23158]: pgmap v5286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:22 smithi016 ceph-mon[22010]: pgmap v5286: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:22 smithi016 conmon[32727]: 2022-03-21T20:08:22.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:23.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:24.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:23 smithi016 conmon[32727]: 2022-03-21T20:08:23.993+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:24 smithi079 ceph-mon[23158]: pgmap v5287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:24 smithi016 ceph-mon[22010]: pgmap v5287: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:25.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:24 smithi016 conmon[32727]: 2022-03-21T20:08:24.949+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:26.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:25 smithi016 conmon[32727]: 2022-03-21T20:08:25.961+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:26 smithi079 ceph-mon[23158]: pgmap v5288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:26 smithi016 ceph-mon[22010]: pgmap v5288: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9950 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:27.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:26 smithi016 conmon[32727]: 2022-03-21T20:08:26.994+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:28.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:28 smithi016 conmon[32727]: 2022-03-21T20:08:28.031+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:28 smithi079 ceph-mon[23158]: pgmap v5289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:28 smithi016 ceph-mon[22010]: pgmap v5289: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:29.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:29 smithi016 conmon[32727]: 2022-03-21T20:08:29.061+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:30.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:30 smithi016 conmon[32727]: 2022-03-21T20:08:30.046+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:30 smithi016 ceph-mon[22010]: pgmap v5290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:30 smithi079 ceph-mon[23158]: pgmap v5290: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:31.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:31 smithi016 conmon[32727]: 2022-03-21T20:08:31.017+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:32.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:31 smithi016 conmon[32727]: 2022-03-21T20:08:31.968+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:32.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:32 smithi016 ceph-mon[22010]: pgmap v5291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:32 smithi079 ceph-mon[23158]: pgmap v5291: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:33.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:32 smithi016 conmon[32727]: 2022-03-21T20:08:32.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:34.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:33 smithi016 conmon[32727]: 2022-03-21T20:08:33.954+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:34 smithi016 ceph-mon[22010]: pgmap v5292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:34.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:34 smithi079 ceph-mon[23158]: pgmap v5292: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:35.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:34 smithi016 conmon[32727]: 2022-03-21T20:08:34.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9955 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:36.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:35 smithi016 conmon[32727]: 2022-03-21T20:08:35.964+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:36 smithi016 ceph-mon[22010]: pgmap v5293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:36.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:36 smithi079 ceph-mon[23158]: pgmap v5293: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:37.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:36 smithi016 conmon[32727]: 2022-03-21T20:08:36.965+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:37.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: 2022-03-21T20:08:37.133+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:37.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:08:37.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:37.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: yield 2022-03-21T20:08:37.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:37.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:37.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:37.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:08:37.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:37.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:08:37.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:37.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:08:37.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:37.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:08:37.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:37.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:08:37.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:37.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:37.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:08:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:08:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:37 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:08:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:08:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:08:37.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:37 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:08:38.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:37 smithi016 conmon[32727]: 2022-03-21T20:08:37.972+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:38 smithi016 ceph-mon[22010]: pgmap v5294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:38 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:08:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:08:38.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:38.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:08:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:38.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:08:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:08:38.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:38.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:08:38.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:38.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:38.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:38 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:08:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:38 smithi079 ceph-mon[23158]: pgmap v5294: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:38 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:08:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:08:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:38.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:08:38.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:08:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:38.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:08:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:38.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:38.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:08:38.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:38.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:38.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:38.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:38 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:08:39.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:38 smithi016 conmon[32727]: 2022-03-21T20:08:38.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:40.206 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:39 smithi016 conmon[32727]: 2022-03-21T20:08:39.926+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:40.508 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 2022-03-21T20:08:40.205+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:40.508 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:08:40.509 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:40.509 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: yield 2022-03-21T20:08:40.510 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:40.510 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:40.511 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:40.511 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:08:40.511 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:40.512 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:08:40.512 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:40.512 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:08:40.513 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:40.513 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:08:40.513 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:40.514 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:08:40.514 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:40.514 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:40.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:40.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 2022-03-21T20:08:40.206+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:08:40.515 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:08:40.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:40.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: yield 2022-03-21T20:08:40.516 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:40.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:40.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:40.517 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:08:40.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:40.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:08:40.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:40.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:08:40.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:40.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:08:40.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:40.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:08:40.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:40.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:40.521 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:40.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 2022-03-21T20:08:40.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:08:40.522 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: 2022-03-21T20:08:40.523 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:08:40.523 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:08:40.523 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:08:40.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:08:40.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:08:40.524 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:08:40.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:08:40.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:08:40.525 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:08:40.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:08:40.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:08:40.526 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:08:40.527 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:08:40.527 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:08:40.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:08:40.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:08:40.528 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:08:40.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:08:40.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:08:40.529 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:08:40.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:08:40.530 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:08:40.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise 2022-03-21T20:08:40.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:08:40.531 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:08:40.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:08:40.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:08:40.532 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:40.888 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:40.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:40 smithi016 ceph-mon[22010]: pgmap v5295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:40.889 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:40 smithi079 ceph-mon[23158]: pgmap v5295: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9965 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:41.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:40 smithi016 conmon[32727]: 2022-03-21T20:08:40.887+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:41.884 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:41.885 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:41 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.885 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.886 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:41.886 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:08:41.886 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:41.887 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:41.887 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:41.887 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:08:41.888 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:41.888 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:08:41.889 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:41.889 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:08:41.889 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:41.890 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:41.890 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:41.890 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:08:41.891 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:41.891 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:41.891 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.892 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:41 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:08:41.892 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.892 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:41.893 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:08:41.893 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:41.893 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:41.894 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:41.894 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:08:41.894 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:41.895 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:08:41.895 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:41.896 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:08:41.896 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:41.896 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:41.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:41.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:08:41.897 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:41.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:41.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.898 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:08:41.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:08:41.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:08:41.899 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:08:41.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:08:41.900 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:08:41.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:08:41.901 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:08:41.902 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:08:41.903 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:08:41.903 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:08:41.904 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:08:41.904 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:08:41.904 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:08:41.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:08:41.905 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:08:41.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:08:41.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:08:41.906 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:08:41.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:08:41.907 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:08:41.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:08:41.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:08:41.908 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:08:41.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:08:41.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:08:41.909 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:08:41.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:08:41.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:08:41.910 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:41 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:08:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:41.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:08:41.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:41.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:08:41.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:08:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:41.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:08:41.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:41 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:08:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:08:41.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:08:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:08:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:08:41.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:08:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:08:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:08:41.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:08:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:08:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:08:41.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:08:41.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:08:41.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:08:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:08:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:08:41.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:08:41.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:41.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:08:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:08:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:08:41.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:08:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:08:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:08:41.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:08:41.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:08:41.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:08:41.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:08:41.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:08:41.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:08:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:08:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:08:41.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:08:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:08:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:08:41.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:08:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:08:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:08:41.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:08:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:08:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:08:41.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:08:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:08:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:08:41.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:08:41.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:08:41.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:08:41.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:08:41.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:08:42.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:41 smithi016 conmon[32727]: 2022-03-21T20:08:41.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:42.886 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:42.887 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:42 smithi016 ceph-mon[22010]: pgmap v5296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:42 smithi079 ceph-mon[23158]: pgmap v5296: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:43.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:42 smithi016 conmon[32727]: 2022-03-21T20:08:42.884+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:43.870 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:43 smithi016 conmon[32727]: 2022-03-21T20:08:43.868+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:44.913 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:44 smithi016 ceph-mon[22010]: pgmap v5297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:44 smithi079 ceph-mon[23158]: pgmap v5297: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:45.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:44 smithi016 conmon[32727]: 2022-03-21T20:08:44.911+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:45.887 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:45 smithi016 conmon[32727]: 2022-03-21T20:08:45.885+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:46.847 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:46.848 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:46 smithi016 ceph-mon[22010]: pgmap v5298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:46.848 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:46 smithi079 ceph-mon[23158]: pgmap v5298: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9970 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:46 smithi016 conmon[32727]: 2022-03-21T20:08:46.846+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:47.895 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:47.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:47 smithi016 conmon[32727]: 2022-03-21T20:08:47.893+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:48 smithi016 ceph-mon[22010]: pgmap v5299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:48 smithi079 ceph-mon[23158]: pgmap v5299: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:49.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:48 smithi016 conmon[32727]: 2022-03-21T20:08:48.923+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:50.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:49 smithi016 conmon[32727]: 2022-03-21T20:08:49.914+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:50 smithi016 ceph-mon[22010]: pgmap v5300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:50 smithi079 ceph-mon[23158]: pgmap v5300: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:51.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:50 smithi016 conmon[32727]: 2022-03-21T20:08:50.942+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:52.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:51 smithi016 conmon[32727]: 2022-03-21T20:08:51.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:52 smithi016 ceph-mon[22010]: pgmap v5301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:52 smithi079 ceph-mon[23158]: pgmap v5301: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:53.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:52 smithi016 conmon[32727]: 2022-03-21T20:08:52.933+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:53.903 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:53 smithi016 conmon[32727]: 2022-03-21T20:08:53.901+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:54.854 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:54.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:54 smithi016 ceph-mon[22010]: pgmap v5302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:54 smithi079 ceph-mon[23158]: pgmap v5302: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-03-21T20:08:55.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:54 smithi016 conmon[32727]: 2022-03-21T20:08:54.852+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:55.901 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:55.902 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:55.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9975 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:08:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:55 smithi016 conmon[32727]: 2022-03-21T20:08:55.900+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:56.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:56.909 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:56 smithi016 ceph-mon[22010]: pgmap v5303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:56 smithi079 ceph-mon[23158]: pgmap v5303: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:08:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:56 smithi016 conmon[32727]: 2022-03-21T20:08:56.907+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:58.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:57 smithi016 conmon[32727]: 2022-03-21T20:08:57.919+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:58 smithi016 ceph-mon[22010]: pgmap v5304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:58 smithi079 ceph-mon[23158]: pgmap v5304: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:08:59.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:08:58 smithi016 conmon[32727]: 2022-03-21T20:08:58.966+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:08:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:08:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:08:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:08:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:08:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:08:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:08:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:08:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:08:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:08:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:08:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:09:00.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:00 smithi016 conmon[32727]: 2022-03-21T20:09:00.004+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:00 smithi016 ceph-mon[22010]: pgmap v5305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:00 smithi079 ceph-mon[23158]: pgmap v5305: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9985 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:01.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:01 smithi016 conmon[32727]: 2022-03-21T20:09:00.999+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:02.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:01 smithi016 conmon[32727]: 2022-03-21T20:09:01.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:02 smithi016 ceph-mon[22010]: pgmap v5306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:02 smithi079 ceph-mon[23158]: pgmap v5306: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:03.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:03 smithi016 conmon[32727]: 2022-03-21T20:09:03.019+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:04.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:03 smithi016 conmon[32727]: 2022-03-21T20:09:03.976+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:04 smithi016 ceph-mon[22010]: pgmap v5307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:04 smithi079 ceph-mon[23158]: pgmap v5307: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:05.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:04 smithi016 conmon[32727]: 2022-03-21T20:09:04.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:06.415 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:05 smithi016 conmon[32727]: 2022-03-21T20:09:05.991+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:06 smithi016 ceph-mon[22010]: pgmap v5308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:06.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:06 smithi079 ceph-mon[23158]: pgmap v5308: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9990 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:07.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:06 smithi016 conmon[32727]: 2022-03-21T20:09:06.946+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:08.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:07 smithi016 conmon[32727]: 2022-03-21T20:09:07.995+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:08 smithi016 ceph-mon[22010]: pgmap v5309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:08.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:08 smithi079 ceph-mon[23158]: pgmap v5309: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:09.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:08 smithi016 conmon[32727]: 2022-03-21T20:09:08.967+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:10.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:09 smithi016 conmon[32727]: 2022-03-21T20:09:09.953+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:10 smithi016 ceph-mon[22010]: pgmap v5310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:10 smithi079 ceph-mon[23158]: pgmap v5310: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:11.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:10 smithi016 conmon[32727]: 2022-03-21T20:09:10.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:11.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:12.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:11 smithi016 conmon[32727]: 2022-03-21T20:09:11.958+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:12 smithi016 ceph-mon[22010]: pgmap v5311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:12 smithi079 ceph-mon[23158]: pgmap v5311: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:13.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:12 smithi016 conmon[32727]: 2022-03-21T20:09:12.984+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:14.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:13 smithi016 conmon[32727]: 2022-03-21T20:09:13.982+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:14 smithi016 ceph-mon[22010]: pgmap v5312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:14 smithi079 ceph-mon[23158]: pgmap v5312: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:15.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:14 smithi016 conmon[32727]: 2022-03-21T20:09:14.998+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 9995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 9995 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:16.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:15 smithi016 conmon[32727]: 2022-03-21T20:09:15.969+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:16 smithi016 ceph-mon[22010]: pgmap v5313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:16 smithi079 ceph-mon[23158]: pgmap v5313: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:17.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:16 smithi016 conmon[32727]: 2022-03-21T20:09:16.979+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:18.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:17 smithi016 conmon[32727]: 2022-03-21T20:09:17.978+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:18 smithi016 ceph-mon[22010]: pgmap v5314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:18 smithi079 ceph-mon[23158]: pgmap v5314: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:19.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:18 smithi016 conmon[32727]: 2022-03-21T20:09:18.986+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:20.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:19 smithi016 conmon[32727]: 2022-03-21T20:09:19.938+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:20 smithi016 ceph-mon[22010]: pgmap v5315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:20 smithi079 ceph-mon[23158]: pgmap v5315: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10005 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:21.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:20 smithi016 conmon[32727]: 2022-03-21T20:09:20.947+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:22.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:21 smithi016 conmon[32727]: 2022-03-21T20:09:21.936+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:22 smithi016 ceph-mon[22010]: pgmap v5316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:22 smithi079 ceph-mon[23158]: pgmap v5316: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:23.414 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:22 smithi016 conmon[32727]: 2022-03-21T20:09:22.917+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:23 smithi016 conmon[32727]: 2022-03-21T20:09:23.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:23.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:24 smithi016 ceph-mon[22010]: pgmap v5317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:24 smithi016 conmon[32727]: 2022-03-21T20:09:24.849+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:24 smithi079 ceph-mon[23158]: pgmap v5317: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:25 smithi016 conmon[32727]: 2022-03-21T20:09:25.863+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:25.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:26 smithi016 ceph-mon[22010]: pgmap v5318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:26 smithi016 conmon[32727]: 2022-03-21T20:09:26.889+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:26 smithi079 ceph-mon[23158]: pgmap v5318: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10010 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:27 smithi016 conmon[32727]: 2022-03-21T20:09:27.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:27.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:28 smithi016 ceph-mon[22010]: pgmap v5319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:28 smithi016 conmon[32727]: 2022-03-21T20:09:28.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:28 smithi079 ceph-mon[23158]: pgmap v5319: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:30.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:29 smithi016 conmon[32727]: 2022-03-21T20:09:29.883+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:30 smithi079 ceph-mon[23158]: pgmap v5320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:30 smithi016 ceph-mon[22010]: pgmap v5320: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:31.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:31.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:30 smithi016 conmon[32727]: 2022-03-21T20:09:30.913+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:32.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:32.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:31 smithi016 conmon[32727]: 2022-03-21T20:09:31.871+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:32 smithi079 ceph-mon[23158]: pgmap v5321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:32 smithi016 ceph-mon[22010]: pgmap v5321: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:33.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:32 smithi016 conmon[32727]: 2022-03-21T20:09:32.866+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:34.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:33 smithi016 conmon[32727]: 2022-03-21T20:09:33.861+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:34 smithi079 ceph-mon[23158]: pgmap v5322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:34 smithi016 ceph-mon[22010]: pgmap v5322: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:35.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:35.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:34 smithi016 conmon[32727]: 2022-03-21T20:09:34.860+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:36.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10015 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:36.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:35 smithi016 conmon[32727]: 2022-03-21T20:09:35.815+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:36 smithi079 ceph-mon[23158]: pgmap v5323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:37.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:36 smithi016 ceph-mon[22010]: pgmap v5323: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:37.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:37.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:36 smithi016 conmon[32727]: 2022-03-21T20:09:36.792+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:38.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:38.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:37 smithi016 conmon[32727]: 2022-03-21T20:09:37.833+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:38 smithi079 ceph-mon[23158]: pgmap v5324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:38.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:39.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:38 smithi016 ceph-mon[22010]: pgmap v5324: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:39.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:39.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:38 smithi016 conmon[32727]: 2022-03-21T20:09:38.802+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:39.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:40.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:40.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:39 smithi016 conmon[32727]: 2022-03-21T20:09:39.811+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:40 smithi016 conmon[32727]: 2022-03-21T20:09:40.825+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: pgmap v5325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:09:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:09:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:09:40.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:09:40.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:40 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:09:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: pgmap v5325: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:40.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:40.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:09:40.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:09:40.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:09:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10025 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:09:40.977 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:40 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:09:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:42.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:42.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:41 smithi016 conmon[32727]: 2022-03-21T20:09:41.854+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:42 smithi079 ceph-mon[23158]: pgmap v5326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:42 smithi016 ceph-mon[22010]: pgmap v5326: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:43.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:43.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:42 smithi016 conmon[32727]: 2022-03-21T20:09:42.859+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:43.664 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: 2022-03-21T20:09:43.309+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:43.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:09:43.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:43.665 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: yield 2022-03-21T20:09:43.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:43.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:43.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:43.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:09:43.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:43.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:09:43.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:43.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:09:43.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:43.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:09:43.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:43.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:09:43.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:43.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:43.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:44.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:43 smithi016 conmon[32727]: 2022-03-21T20:09:43.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:44.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:44 smithi079 ceph-mon[23158]: pgmap v5327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:44 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:09:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:09:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:44.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:09:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:44.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:09:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:09:44.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:44.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:09:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:44.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:44.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:44 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:09:44.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:45.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:44 smithi016 ceph-mon[22010]: pgmap v5327: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:44 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:45.165 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:09:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:45.166 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:09:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:45.167 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:45.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:09:45.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:45.168 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:09:45.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:45.169 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:09:45.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:45.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:45.170 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:45.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:09:45.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:45.171 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:45.172 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:45.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:44 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:09:45.172 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:45.173 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:44 smithi016 conmon[32727]: 2022-03-21T20:09:44.785+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:46.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:45 smithi016 conmon[32727]: 2022-03-21T20:09:45.763+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:46 smithi016 ceph-mon[22010]: pgmap v5328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:46.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 2022-03-21T20:09:46.380+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:46.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:09:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: yield 2022-03-21T20:09:46.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:46.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:09:46.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:46.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:09:46.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:09:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:46.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:09:46.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:46.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:09:46.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:46.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:46.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:46.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 2022-03-21T20:09:46.381+0000 7f92716dd700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:09:46.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:09:46.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:46.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: yield 2022-03-21T20:09:46.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:46.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:46.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:46.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:09:46.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:46.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:09:46.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:46.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:09:46.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:46.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:09:46.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:46.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:09:46.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:46.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:46.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:46.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 2022-03-21T20:09:46.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:09:46.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: 2022-03-21T20:09:46.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:09:46.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:09:46.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:09:46.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:09:46.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:09:46.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:09:46.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:09:46.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:09:46.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:09:46.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:09:46.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:09:46.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:09:46.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:09:46.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:09:46.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:09:46.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:09:46.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:09:46.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:09:46.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:09:46.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:09:46.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:09:46.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:09:46.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise 2022-03-21T20:09:46.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:09:46.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:09:46.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:09:46.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:09:46.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:46.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:46 smithi079 ceph-mon[23158]: pgmap v5328: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10030 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:47.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:46 smithi016 conmon[32727]: 2022-03-21T20:09:46.761+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:47 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:09:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:47.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:47.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:47.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:09:47.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:47.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:09:47.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:47.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:09:47.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:47.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:47.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:47.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:09:47.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:47.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:47.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:47 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:09:47.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:47.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:09:47.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:47.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:47.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:47.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:09:47.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:47.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:09:47.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:47.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:09:47.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:47.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:47.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:47.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:09:47.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:47.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:47.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:09:47.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:09:47.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:09:47.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:09:47.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:09:47.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:09:47.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:09:47.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:09:47.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:09:47.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:09:47.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:09:47.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:09:47.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:09:47.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:09:47.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:09:47.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:09:47.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:09:47.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:09:47.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:09:47.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:09:47.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:09:47.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:09:47.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:09:47.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:09:47.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:09:47.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:09:47.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:09:47.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:09:47.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:09:47.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:47 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:47.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:09:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:47.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:47.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:09:47.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:47.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:09:47.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:47.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:09:47.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:47.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:47.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:47.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:09:47.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:47.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:47.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:47 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:09:47.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:09:47.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:09:47.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:09:47.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:09:47.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:09:47.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:09:47.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:09:47.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:09:47.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:09:47.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:09:47.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:09:47.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:09:47.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:09:47.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:09:47.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:09:47.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:09:47.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:47.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:09:47.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:09:47.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:09:47.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:09:47.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:09:47.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:09:47.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:09:47.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:09:47.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:09:47.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:09:47.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:09:47.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:09:47.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:09:47.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:09:47.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:09:47.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:09:47.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:09:47.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:09:47.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:09:47.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:09:47.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:09:47.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:09:47.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:09:47.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:09:47.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:09:47.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:09:47.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:09:47.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:09:47.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:09:47.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:09:47.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:09:48.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:47 smithi016 conmon[32727]: 2022-03-21T20:09:47.770+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:48.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:48 smithi016 ceph-mon[22010]: pgmap v5329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:48 smithi079 ceph-mon[23158]: pgmap v5329: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:49.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:48 smithi016 conmon[32727]: 2022-03-21T20:09:48.762+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:49.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:50.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:49 smithi016 conmon[32727]: 2022-03-21T20:09:49.795+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:50 smithi016 ceph-mon[22010]: pgmap v5330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:50.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:50 smithi079 ceph-mon[23158]: pgmap v5330: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:51.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:50 smithi016 conmon[32727]: 2022-03-21T20:09:50.834+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:52.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:51 smithi016 conmon[32727]: 2022-03-21T20:09:51.858+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:52 smithi016 ceph-mon[22010]: pgmap v5331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:52 smithi079 ceph-mon[23158]: pgmap v5331: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:53.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:52 smithi016 conmon[32727]: 2022-03-21T20:09:52.818+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:54.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:53 smithi016 conmon[32727]: 2022-03-21T20:09:53.784+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:54 smithi016 ceph-mon[22010]: pgmap v5332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:54 smithi079 ceph-mon[23158]: pgmap v5332: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:55.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:54 smithi016 conmon[32727]: 2022-03-21T20:09:54.817+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10035 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:09:56.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:55 smithi016 conmon[32727]: 2022-03-21T20:09:55.787+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:56.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:56 smithi016 ceph-mon[22010]: pgmap v5333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:56.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:56 smithi079 ceph-mon[23158]: pgmap v5333: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:09:57.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:56 smithi016 conmon[32727]: 2022-03-21T20:09:56.807+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:57.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:57 smithi016 conmon[32727]: 2022-03-21T20:09:57.758+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:58.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:58 smithi016 ceph-mon[22010]: pgmap v5334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:58.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:58 smithi079 ceph-mon[23158]: pgmap v5334: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:09:59.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:58 smithi016 conmon[32727]: 2022-03-21T20:09:58.785+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:09:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:59.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:09:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:09:59.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:09:59.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:09:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:09:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:09:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:09:59.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:09:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:09:59.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:09:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:10:00.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:09:59 smithi016 conmon[32727]: 2022-03-21T20:09:59.782+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:00.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: pgmap v5335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops 2022-03-21T20:10:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T20:10:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T20:10:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T20:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: pg 2.1 is stuck inactive for 6h, current state unknown, last acting [] 2022-03-21T20:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: pg 3.6 is stuck inactive for 6h, current state unknown, last acting [] 2022-03-21T20:10:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops 2022-03-21T20:10:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: pgmap v5335: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 2 pgs inactive; 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops 2022-03-21T20:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-03-21T20:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-03-21T20:10:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive 2022-03-21T20:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: pg 2.1 is stuck inactive for 6h, current state unknown, last acting [] 2022-03-21T20:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: pg 3.6 is stuck inactive for 6h, current state unknown, last acting [] 2022-03-21T20:10:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops 2022-03-21T20:10:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10045 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:01.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:00 smithi016 conmon[32727]: 2022-03-21T20:10:00.743+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:02.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:01 smithi016 conmon[32727]: 2022-03-21T20:10:01.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:02 smithi016 ceph-mon[22010]: pgmap v5336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:02 smithi079 ceph-mon[23158]: pgmap v5336: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:03.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:02 smithi016 conmon[32727]: 2022-03-21T20:10:02.753+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:03.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:04.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:03 smithi016 conmon[32727]: 2022-03-21T20:10:03.719+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:04 smithi016 ceph-mon[22010]: pgmap v5337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:04.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:04 smithi079 ceph-mon[23158]: pgmap v5337: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:05.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:04 smithi016 conmon[32727]: 2022-03-21T20:10:04.690+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:05.643 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:05 smithi016 conmon[32727]: 2022-03-21T20:10:05.641+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:06.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:06.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:06 smithi016 ceph-mon[22010]: pgmap v5338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:06.608 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:06.609 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:06 smithi079 ceph-mon[23158]: pgmap v5338: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10050 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:06 smithi016 conmon[32727]: 2022-03-21T20:10:06.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:07.587 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:07 smithi016 conmon[32727]: 2022-03-21T20:10:07.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:08.594 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:08 smithi016 ceph-mon[22010]: pgmap v5339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:08.595 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:08 smithi079 ceph-mon[23158]: pgmap v5339: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:08.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:08 smithi016 conmon[32727]: 2022-03-21T20:10:08.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:09.594 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:09.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:09 smithi016 conmon[32727]: 2022-03-21T20:10:09.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:10.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:10 smithi016 ceph-mon[22010]: pgmap v5340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:10.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:10 smithi016 conmon[32727]: 2022-03-21T20:10:10.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:10 smithi079 ceph-mon[23158]: pgmap v5340: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:11.581 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:11 smithi016 conmon[32727]: 2022-03-21T20:10:11.580+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:12.620 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:12 smithi016 ceph-mon[22010]: pgmap v5341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:12.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:12 smithi079 ceph-mon[23158]: pgmap v5341: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:12 smithi016 conmon[32727]: 2022-03-21T20:10:12.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:13.635 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:13.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:13 smithi016 conmon[32727]: 2022-03-21T20:10:13.634+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:14.620 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:14 smithi016 ceph-mon[22010]: pgmap v5342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:14.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:14 smithi079 ceph-mon[23158]: pgmap v5342: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:14 smithi016 conmon[32727]: 2022-03-21T20:10:14.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:15.631 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:15.632 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10055 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:15 smithi016 conmon[32727]: 2022-03-21T20:10:15.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:16.650 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:16 smithi016 ceph-mon[22010]: pgmap v5343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:16.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:16 smithi079 ceph-mon[23158]: pgmap v5343: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:16 smithi016 conmon[32727]: 2022-03-21T20:10:16.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:17.621 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:17 smithi016 conmon[32727]: 2022-03-21T20:10:17.620+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:18.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:18 smithi016 ceph-mon[22010]: pgmap v5344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:18.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:18 smithi079 ceph-mon[23158]: pgmap v5344: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:18.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:18 smithi016 conmon[32727]: 2022-03-21T20:10:18.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:19.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:19 smithi016 conmon[32727]: 2022-03-21T20:10:19.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:19.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:20.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:20 smithi016 conmon[32727]: 2022-03-21T20:10:20.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:20 smithi016 ceph-mon[22010]: pgmap v5345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:20 smithi079 ceph-mon[23158]: pgmap v5345: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10065 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:21.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:21.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:21 smithi016 conmon[32727]: 2022-03-21T20:10:21.661+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:22 smithi016 ceph-mon[22010]: pgmap v5346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:22.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:22 smithi079 ceph-mon[23158]: pgmap v5346: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:23.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:22 smithi016 conmon[32727]: 2022-03-21T20:10:22.702+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:23 smithi016 conmon[32727]: 2022-03-21T20:10:23.747+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:24.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:24 smithi016 ceph-mon[22010]: pgmap v5347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:24.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:24 smithi079 ceph-mon[23158]: pgmap v5347: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:25.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:24 smithi016 conmon[32727]: 2022-03-21T20:10:24.710+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:25.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:25 smithi016 conmon[32727]: 2022-03-21T20:10:25.665+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:26.638 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:26.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:26 smithi016 ceph-mon[22010]: pgmap v5348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:26.639 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:26.640 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:26 smithi079 ceph-mon[23158]: pgmap v5348: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10070 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:26 smithi016 conmon[32727]: 2022-03-21T20:10:26.637+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:27.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:27.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:27 smithi016 conmon[32727]: 2022-03-21T20:10:27.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:28 smithi016 ceph-mon[22010]: pgmap v5349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:28.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:28 smithi016 conmon[32727]: 2022-03-21T20:10:28.578+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:28 smithi079 ceph-mon[23158]: pgmap v5349: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:29.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:29 smithi016 conmon[32727]: 2022-03-21T20:10:29.589+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:30 smithi016 ceph-mon[22010]: pgmap v5350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:30.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:30.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:30 smithi016 conmon[32727]: 2022-03-21T20:10:30.615+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:30 smithi079 ceph-mon[23158]: pgmap v5350: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:31 smithi016 conmon[32727]: 2022-03-21T20:10:31.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:32 smithi016 ceph-mon[22010]: pgmap v5351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:32.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:32.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:32 smithi016 conmon[32727]: 2022-03-21T20:10:32.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:32 smithi079 ceph-mon[23158]: pgmap v5351: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:33.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:33 smithi016 conmon[32727]: 2022-03-21T20:10:33.646+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:34 smithi016 ceph-mon[22010]: pgmap v5352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:34.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:34 smithi016 conmon[32727]: 2022-03-21T20:10:34.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:34 smithi079 ceph-mon[23158]: pgmap v5352: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:35.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:35 smithi016 conmon[32727]: 2022-03-21T20:10:35.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10075 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:36 smithi079 ceph-mon[23158]: pgmap v5353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:36 smithi016 conmon[32727]: 2022-03-21T20:10:36.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:36 smithi016 ceph-mon[22010]: pgmap v5353: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:37.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:37 smithi016 conmon[32727]: 2022-03-21T20:10:37.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:37.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:38 smithi079 ceph-mon[23158]: pgmap v5354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:38 smithi016 conmon[32727]: 2022-03-21T20:10:38.624+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:38 smithi016 ceph-mon[22010]: pgmap v5354: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:39 smithi016 conmon[32727]: 2022-03-21T20:10:39.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:40 smithi079 ceph-mon[23158]: pgmap v5355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:40.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:40 smithi016 conmon[32727]: 2022-03-21T20:10:40.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:40 smithi016 ceph-mon[22010]: pgmap v5355: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:40.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10085 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:41.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:41 smithi016 conmon[32727]: 2022-03-21T20:10:41.548+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:42 smithi079 ceph-mon[23158]: pgmap v5356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:42.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:42 smithi016 conmon[32727]: 2022-03-21T20:10:42.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:42 smithi016 ceph-mon[22010]: pgmap v5356: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:43 smithi016 conmon[32727]: 2022-03-21T20:10:43.610+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:43.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:44.704 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:44 smithi016 ceph-mon[22010]: pgmap v5357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:44.704 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:44.705 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:44 smithi016 conmon[32727]: 2022-03-21T20:10:44.614+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:44 smithi079 ceph-mon[23158]: pgmap v5357: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:45 smithi016 conmon[32727]: 2022-03-21T20:10:45.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: pgmap v5358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:10:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:10:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:46 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:10:46.837 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[32727]: 2022-03-21T20:10:46.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:46.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: pgmap v5358: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:46.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10090 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:46.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:46.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:10:46.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:10:46.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:46 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:10:47.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: 2022-03-21T20:10:46.860+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:47.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:10:47.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:47.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: yield 2022-03-21T20:10:47.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:47.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:47.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:47.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:10:47.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:47.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:10:47.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:10:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:47.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:10:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:10:47.172 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:47.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:47.173 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:46 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:47 smithi016 conmon[32727]: 2022-03-21T20:10:47.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:48 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:10:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:10:48.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:48.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:10:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:48.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:10:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:48.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:10:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:48.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:10:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:48.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:48.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:48 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:10:48.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:48 smithi079 ceph-mon[23158]: pgmap v5359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:48.732 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:48.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:48 smithi016 conmon[32727]: 2022-03-21T20:10:48.684+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:48 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:10:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:48.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:10:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:48.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:48.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:10:48.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:48.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:10:48.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:48.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:10:48.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:48.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:48.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:48.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:10:48.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:48.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:48.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:48.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:48 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:10:48.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:48 smithi016 ceph-mon[22010]: pgmap v5359: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:48.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:49.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[32727]: 2022-03-21T20:10:49.638+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:50.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 2022-03-21T20:10:49.933+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:50.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:10:50.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:50.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: yield 2022-03-21T20:10:50.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:50.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:50.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:50.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:10:50.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:50.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:10:50.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:50.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:10:50.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:50.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:10:50.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:50.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:10:50.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:50.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:50.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:50.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 2022-03-21T20:10:49.934+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:10:50.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:10:50.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:50.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: yield 2022-03-21T20:10:50.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:50.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:50.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:50.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:10:50.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:50.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:10:50.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:50.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:10:50.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:50.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:10:50.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:50.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:10:50.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:50.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:50.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:50.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 2022-03-21T20:10:50.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:10:50.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: 2022-03-21T20:10:50.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:10:50.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:10:50.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:10:50.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:10:50.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:10:50.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:10:50.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:10:50.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:10:50.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:10:50.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:10:50.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:10:50.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:10:50.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:10:50.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:10:50.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:10:50.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:10:50.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:10:50.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:10:50.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:10:50.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:10:50.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:10:50.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:10:50.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise 2022-03-21T20:10:50.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:10:50.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:10:50.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:10:50.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:10:50.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:10:49 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:50 smithi079 ceph-mon[23158]: pgmap v5360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:50.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:50 smithi016 ceph-mon[22010]: pgmap v5360: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:50.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:50 smithi016 conmon[32727]: 2022-03-21T20:10:50.674+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:51 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:10:51.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:51.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:10:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:10:51.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:10:51.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:51.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:10:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:51.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:51 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:10:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:10:51.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:51.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:51.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:51.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:10:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:10:51.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:10:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:51.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:51.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:51.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:10:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:51.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:10:51.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:10:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:10:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:10:51.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:10:51.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:10:51.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:10:51.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:10:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:10:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:10:51.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:10:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:10:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:10:51.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:10:51.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:10:51.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:10:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:10:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:10:51.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:10:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:10:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:10:51.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:10:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:10:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:10:51.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:10:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:10:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:10:51.749 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:10:51.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:10:51.750 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:51 smithi016 conmon[32727]: 2022-03-21T20:10:51.684+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:51 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:51.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:10:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:51.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:10:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:51.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:10:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:10:51.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:51.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:10:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:51.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:51 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:10:51.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:10:51.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:10:51.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:10:51.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:10:51.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:10:51.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:10:51.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:10:51.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:10:51.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:10:51.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:10:51.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:10:51.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:10:51.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:10:51.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:10:51.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:10:51.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:10:51.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:10:51.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:10:51.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:10:51.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:10:51.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:10:51.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:10:51.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:10:51.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:10:51.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:10:51.934 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:10:51.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:10:51.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:10:51.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:10:51.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:10:51.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:10:51.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:10:51.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:10:51.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:10:51.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:10:51.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:10:51.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:10:51.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:10:51.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:10:51.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:10:51.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:10:51.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:10:51.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:10:51.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:10:51.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:10:51.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:10:51.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:10:51.944 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:52 smithi079 ceph-mon[23158]: pgmap v5361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:52 smithi016 ceph-mon[22010]: pgmap v5361: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:52 smithi016 conmon[32727]: 2022-03-21T20:10:52.657+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:53 smithi016 conmon[32727]: 2022-03-21T20:10:53.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:54 smithi079 ceph-mon[23158]: pgmap v5362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:54 smithi016 ceph-mon[22010]: pgmap v5362: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:54 smithi016 conmon[32727]: 2022-03-21T20:10:54.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:55.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:55 smithi016 conmon[32727]: 2022-03-21T20:10:55.677+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10095 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:10:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:56 smithi016 ceph-mon[22010]: pgmap v5363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:56.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:56 smithi016 conmon[32727]: 2022-03-21T20:10:56.684+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:56 smithi079 ceph-mon[23158]: pgmap v5363: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:10:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:58.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:57 smithi016 conmon[32727]: 2022-03-21T20:10:57.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:58.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:58 smithi016 conmon[32727]: 2022-03-21T20:10:58.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:58 smithi016 ceph-mon[22010]: pgmap v5364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:58 smithi079 ceph-mon[23158]: pgmap v5364: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:10:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:10:59 smithi016 conmon[32727]: 2022-03-21T20:10:59.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:10:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:10:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:10:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:10:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:10:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:10:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:10:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:10:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:10:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:10:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:10:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:11:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:00 smithi016 ceph-mon[22010]: pgmap v5365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:00 smithi016 conmon[32727]: 2022-03-21T20:11:00.653+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:00.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:00 smithi079 ceph-mon[23158]: pgmap v5365: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:00.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10105 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:01.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:01 smithi016 conmon[32727]: 2022-03-21T20:11:01.635+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:02 smithi016 ceph-mon[22010]: pgmap v5366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:02.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:02.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:02 smithi016 conmon[32727]: 2022-03-21T20:11:02.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:02 smithi079 ceph-mon[23158]: pgmap v5366: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:03 smithi016 conmon[32727]: 2022-03-21T20:11:03.675+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:03.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:04 smithi016 ceph-mon[22010]: pgmap v5367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:04.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:04.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:04 smithi016 conmon[32727]: 2022-03-21T20:11:04.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:04 smithi079 ceph-mon[23158]: pgmap v5367: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:05.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:05 smithi016 conmon[32727]: 2022-03-21T20:11:05.593+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:06 smithi016 ceph-mon[22010]: pgmap v5368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:06.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:06.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:06 smithi016 conmon[32727]: 2022-03-21T20:11:06.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:06 smithi079 ceph-mon[23158]: pgmap v5368: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10110 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:07.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:07.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:07 smithi016 conmon[32727]: 2022-03-21T20:11:07.629+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:08.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:08 smithi016 ceph-mon[22010]: pgmap v5369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:08.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:08.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:08 smithi016 conmon[32727]: 2022-03-21T20:11:08.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:08 smithi079 ceph-mon[23158]: pgmap v5369: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:08.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:09.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:09.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:09 smithi016 conmon[32727]: 2022-03-21T20:11:09.651+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:10 smithi016 ceph-mon[22010]: pgmap v5370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:10.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:10.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:10 smithi016 conmon[32727]: 2022-03-21T20:11:10.659+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:10 smithi079 ceph-mon[23158]: pgmap v5370: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:11.076 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:11 smithi016 conmon[32727]: 2022-03-21T20:11:11.683+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:11.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:12 smithi016 ceph-mon[22010]: pgmap v5371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:12 smithi016 conmon[32727]: 2022-03-21T20:11:12.636+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:12 smithi079 ceph-mon[23158]: pgmap v5371: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:13 smithi016 conmon[32727]: 2022-03-21T20:11:13.630+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:13.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:14 smithi016 ceph-mon[22010]: pgmap v5372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:14.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:14 smithi016 conmon[32727]: 2022-03-21T20:11:14.622+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:14 smithi079 ceph-mon[23158]: pgmap v5372: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:15 smithi016 conmon[32727]: 2022-03-21T20:11:15.573+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:15.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10115 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:16 smithi016 ceph-mon[22010]: pgmap v5373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:16 smithi016 conmon[32727]: 2022-03-21T20:11:16.623+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:16 smithi079 ceph-mon[23158]: pgmap v5373: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:17.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:17 smithi016 conmon[32727]: 2022-03-21T20:11:17.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:18 smithi016 ceph-mon[22010]: pgmap v5374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:18 smithi016 conmon[32727]: 2022-03-21T20:11:18.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:18 smithi079 ceph-mon[23158]: pgmap v5374: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:19.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:19 smithi016 conmon[32727]: 2022-03-21T20:11:19.576+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:20 smithi016 ceph-mon[22010]: pgmap v5375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:20.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:20 smithi016 conmon[32727]: 2022-03-21T20:11:20.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:20.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:20 smithi079 ceph-mon[23158]: pgmap v5375: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10125 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:21 smithi016 conmon[32727]: 2022-03-21T20:11:21.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:21.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:22 smithi016 ceph-mon[22010]: pgmap v5376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:22 smithi016 conmon[32727]: 2022-03-21T20:11:22.596+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:22.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:22 smithi079 ceph-mon[23158]: pgmap v5376: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:23 smithi016 conmon[32727]: 2022-03-21T20:11:23.645+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:24 smithi016 ceph-mon[22010]: pgmap v5377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:24 smithi016 conmon[32727]: 2022-03-21T20:11:24.617+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:24 smithi079 ceph-mon[23158]: pgmap v5377: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:25 smithi016 conmon[32727]: 2022-03-21T20:11:25.625+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:26 smithi016 conmon[32727]: 2022-03-21T20:11:26.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:26 smithi016 ceph-mon[22010]: pgmap v5378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:26.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:26 smithi079 ceph-mon[23158]: pgmap v5378: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10130 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:27 smithi016 conmon[32727]: 2022-03-21T20:11:27.692+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:28 smithi016 ceph-mon[22010]: pgmap v5379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:28 smithi016 conmon[32727]: 2022-03-21T20:11:28.717+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:28 smithi079 ceph-mon[23158]: pgmap v5379: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:29 smithi016 conmon[32727]: 2022-03-21T20:11:29.703+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:30 smithi016 ceph-mon[22010]: pgmap v5380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:30 smithi016 conmon[32727]: 2022-03-21T20:11:30.721+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:30 smithi079 ceph-mon[23158]: pgmap v5380: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:31 smithi016 conmon[32727]: 2022-03-21T20:11:31.763+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:32 smithi016 ceph-mon[22010]: pgmap v5381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:32 smithi016 conmon[32727]: 2022-03-21T20:11:32.755+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:32.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:32 smithi079 ceph-mon[23158]: pgmap v5381: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:33 smithi016 conmon[32727]: 2022-03-21T20:11:33.736+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:33.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:34.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:34 smithi016 ceph-mon[22010]: pgmap v5382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:34 smithi016 conmon[32727]: 2022-03-21T20:11:34.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:34.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:34 smithi079 ceph-mon[23158]: pgmap v5382: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:35 smithi016 conmon[32727]: 2022-03-21T20:11:35.701+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10135 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:36.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:36 smithi016 conmon[32727]: 2022-03-21T20:11:36.661+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:36 smithi016 ceph-mon[22010]: pgmap v5383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:36.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:36 smithi079 ceph-mon[23158]: pgmap v5383: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:37 smithi016 conmon[32727]: 2022-03-21T20:11:37.656+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:38 smithi016 ceph-mon[22010]: pgmap v5384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:38 smithi016 conmon[32727]: 2022-03-21T20:11:38.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:38 smithi079 ceph-mon[23158]: pgmap v5384: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:39 smithi016 conmon[32727]: 2022-03-21T20:11:39.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:40 smithi016 ceph-mon[22010]: pgmap v5385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:40 smithi016 conmon[32727]: 2022-03-21T20:11:40.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:40 smithi079 ceph-mon[23158]: pgmap v5385: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10145 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:41 smithi016 conmon[32727]: 2022-03-21T20:11:41.619+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:41.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:42 smithi016 ceph-mon[22010]: pgmap v5386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:42 smithi016 conmon[32727]: 2022-03-21T20:11:42.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:42 smithi079 ceph-mon[23158]: pgmap v5386: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:43 smithi016 conmon[32727]: 2022-03-21T20:11:43.680+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:43.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:44 smithi016 ceph-mon[22010]: pgmap v5387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:44 smithi016 conmon[32727]: 2022-03-21T20:11:44.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:44 smithi079 ceph-mon[23158]: pgmap v5387: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:45 smithi016 conmon[32727]: 2022-03-21T20:11:45.691+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:46 smithi016 ceph-mon[22010]: pgmap v5388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:46.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:46.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:46 smithi016 conmon[32727]: 2022-03-21T20:11:46.723+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:46.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:46 smithi079 ceph-mon[23158]: pgmap v5388: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10150 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:48.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:47 smithi016 conmon[32727]: 2022-03-21T20:11:47.706+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:48.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:48 smithi016 ceph-mon[22010]: pgmap v5389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:48.918 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:48 smithi016 conmon[32727]: 2022-03-21T20:11:48.679+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:48.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:48 smithi079 ceph-mon[23158]: pgmap v5389: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:49 smithi016 conmon[32727]: 2022-03-21T20:11:49.699+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:49.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:49.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:50 smithi016 ceph-mon[22010]: pgmap v5390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:11:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:11:50.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:50 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:11:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:50 smithi079 ceph-mon[23158]: pgmap v5390: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:11:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:11:50.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:50 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:11:51.095 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:50 smithi016 conmon[32727]: 2022-03-21T20:11:50.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:51.812 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:51 smithi016 conmon[32727]: 2022-03-21T20:11:51.632+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:51.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:51.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:51.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:52 smithi016 ceph-mon[22010]: pgmap v5391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:52 smithi016 conmon[32727]: 2022-03-21T20:11:52.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:52 smithi079 ceph-mon[23158]: pgmap v5391: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:52.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:53.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: 2022-03-21T20:11:53.036+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:53.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:11:53.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:53.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: yield 2022-03-21T20:11:53.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:53.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:53.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:53.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:11:53.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:53.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:11:53.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:53.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:11:53.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:53.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:11:53.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:53.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:11:53.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:53.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:53.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:53.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:53 smithi016 conmon[32727]: 2022-03-21T20:11:53.621+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:53.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:53 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:11:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:53.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:53 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:11:54.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:54 smithi016 conmon[32727]: 2022-03-21T20:11:54.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:54 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi016 2022-03-21T20:11:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:54 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:54.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:11:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:11:54.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:54.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:11:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:54.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:11:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:11:54.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:54.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:11:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:54.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:54 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:11:54.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:54 smithi016 ceph-mon[22010]: pgmap v5392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:54.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:54 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi016 2022-03-21T20:11:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:54 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:11:54.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:11:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:54.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:11:54.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:11:54.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:11:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:54.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:11:54.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:54.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:54 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:11:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:54 smithi079 ceph-mon[23158]: pgmap v5392: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:54.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:55 smithi016 conmon[32727]: 2022-03-21T20:11:55.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:55.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:55.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10155 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:11:56.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 2022-03-21T20:11:56.108+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:56.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:11:56.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:56.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: yield 2022-03-21T20:11:56.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:56.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:56.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:56.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:11:56.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:56.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:11:56.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:56.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:11:56.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:56.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:11:56.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:56.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:11:56.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:56.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:56.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:56.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 2022-03-21T20:11:56.109+0000 7f9273ee2700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:11:56.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:11:56.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:56.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: yield 2022-03-21T20:11:56.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:56.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:56.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:56.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:11:56.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:56.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:11:56.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:56.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:11:56.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:56.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:11:56.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:56.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:11:56.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:56.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:56.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:56.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 2022-03-21T20:11:56.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:11:56.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: 2022-03-21T20:11:56.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:11:56.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:11:56.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:11:56.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:11:56.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:11:56.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:11:56.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:11:56.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:11:56.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:11:56.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:11:56.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:11:56.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:11:56.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:11:56.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:11:56.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:11:56.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:11:56.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:11:56.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:11:56.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:11:56.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:11:56.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:11:56.442 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:11:56.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise 2022-03-21T20:11:56.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:11:56.444 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:11:56.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:11:56.445 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:11:56.446 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:56 smithi016 ceph-mon[22010]: pgmap v5393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:56 smithi016 conmon[32727]: 2022-03-21T20:11:56.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:56.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:56 smithi079 ceph-mon[23158]: pgmap v5393: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:11:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:57 smithi016 conmon[32727]: 2022-03-21T20:11:57.577+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:57 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:57.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:11:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:57.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:11:57.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:11:57.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:57.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:11:57.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:57.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:11:57.922 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:57.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:57.923 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.924 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:57 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:11:57.924 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:57.925 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:11:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:57.926 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:57.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:11:57.927 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:57.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:11:57.928 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:57.929 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:11:57.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:57.930 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:57.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:57.931 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:11:57.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:57.932 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:57.933 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.935 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:11:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:11:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:11:57.936 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:11:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:11:57.937 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:11:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:11:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:11:57.938 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:11:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:11:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:11:57.939 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:11:57.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:11:57.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:11:57.940 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:11:57.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:11:57.941 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:11:57.942 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:11:57.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:11:57.943 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:11:57.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:11:57.944 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:11:57.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:11:57.945 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:11:57.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:11:57.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:11:57.946 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:11:57.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:11:57.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:11:57.947 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.948 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:57 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:11:57.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:57.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:11:57.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:57.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:11:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:11:57.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:57.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:57.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:11:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:57.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:57.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:57 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:11:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:11:57.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:11:57.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:11:57.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:11:57.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:11:57.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:11:57.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:11:57.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:11:57.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:11:57.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:11:57.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:11:57.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:11:57.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:11:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:11:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:11:57.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:11:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:11:57.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:11:57.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:11:57.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:11:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:11:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:11:57.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:11:57.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:11:57.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:11:57.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:11:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:11:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:11:57.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:11:57.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:11:57.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:11:57.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:11:57.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:11:57.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:11:57.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:11:57.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:11:57.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:11:57.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:11:57.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:11:57.999 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:11:58.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:11:58.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:11:58.000 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:11:58.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:11:58.001 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:11:58.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:11:58.002 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:11:58.003 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:58 smithi016 ceph-mon[22010]: pgmap v5394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:58.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:58 smithi016 conmon[32727]: 2022-03-21T20:11:58.588+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:58 smithi079 ceph-mon[23158]: pgmap v5394: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:11:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:11:59 smithi016 conmon[32727]: 2022-03-21T20:11:59.613+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:11:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:11:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:11:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:11:59.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:11:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:11:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:11:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:11:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:11:59.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:11:59.976 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:11:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:12:00.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:00 smithi016 conmon[32727]: 2022-03-21T20:12:00.640+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:00 smithi016 ceph-mon[22010]: pgmap v5395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:00.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:00 smithi079 ceph-mon[23158]: pgmap v5395: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10165 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:01.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:01 smithi016 conmon[32727]: 2022-03-21T20:12:01.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:01.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:02.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:02 smithi079 ceph-mon[23158]: pgmap v5396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:02.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:02 smithi016 ceph-mon[22010]: pgmap v5396: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:03.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:02 smithi016 conmon[32727]: 2022-03-21T20:12:02.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:04.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:04.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:03 smithi016 conmon[32727]: 2022-03-21T20:12:03.698+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:04 smithi079 ceph-mon[23158]: pgmap v5397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:04 smithi016 ceph-mon[22010]: pgmap v5397: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:05.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:04 smithi016 conmon[32727]: 2022-03-21T20:12:04.691+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:05.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:06.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:05 smithi016 conmon[32727]: 2022-03-21T20:12:05.741+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:06.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:06.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:06 smithi079 ceph-mon[23158]: pgmap v5398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:06 smithi016 ceph-mon[22010]: pgmap v5398: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:07.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10170 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:07.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:06 smithi016 conmon[32727]: 2022-03-21T20:12:06.746+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:07.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:08.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:08.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:07 smithi016 conmon[32727]: 2022-03-21T20:12:07.763+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:08 smithi079 ceph-mon[23158]: pgmap v5399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:09.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:09.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:08 smithi016 ceph-mon[22010]: pgmap v5399: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:09.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:08 smithi016 conmon[32727]: 2022-03-21T20:12:08.734+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:10.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:10.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:09 smithi016 conmon[32727]: 2022-03-21T20:12:09.757+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:10 smithi079 ceph-mon[23158]: pgmap v5400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:11.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:10 smithi016 conmon[32727]: 2022-03-21T20:12:10.762+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:11.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:10 smithi016 ceph-mon[22010]: pgmap v5400: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:11.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:12.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:12.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:11 smithi016 conmon[32727]: 2022-03-21T20:12:11.778+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:12.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:12 smithi079 ceph-mon[23158]: pgmap v5401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:12 smithi016 ceph-mon[22010]: pgmap v5401: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:13.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:13.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:12 smithi016 conmon[32727]: 2022-03-21T20:12:12.745+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:13.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:14.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:14.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:13 smithi016 conmon[32727]: 2022-03-21T20:12:13.786+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:14 smithi079 ceph-mon[23158]: pgmap v5402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:14.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:14 smithi016 ceph-mon[22010]: pgmap v5402: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:15.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:15.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:14 smithi016 conmon[32727]: 2022-03-21T20:12:14.756+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:15.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:16.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:16.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10175 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:16.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:15 smithi016 conmon[32727]: 2022-03-21T20:12:15.716+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:16 smithi016 ceph-mon[22010]: pgmap v5403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:17.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:17.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:16 smithi016 conmon[32727]: 2022-03-21T20:12:16.704+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:16 smithi079 ceph-mon[23158]: pgmap v5403: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:17.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:17 smithi016 conmon[32727]: 2022-03-21T20:12:17.658+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:17.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:18.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:18.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:18 smithi016 ceph-mon[22010]: pgmap v5404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:18.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:18 smithi016 conmon[32727]: 2022-03-21T20:12:18.661+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:19.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:18 smithi079 ceph-mon[23158]: pgmap v5404: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:19.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:20.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:20.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:19 smithi016 conmon[32727]: 2022-03-21T20:12:19.697+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:20.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:20.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:20 smithi016 conmon[32727]: 2022-03-21T20:12:20.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:20 smithi016 ceph-mon[22010]: pgmap v5405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:20.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:20.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:20 smithi079 ceph-mon[23158]: pgmap v5405: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10185 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:21.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:22.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:22.166 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:21 smithi016 conmon[32727]: 2022-03-21T20:12:21.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:22.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:22 smithi016 ceph-mon[22010]: pgmap v5406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:22 smithi016 conmon[32727]: 2022-03-21T20:12:22.628+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:22 smithi079 ceph-mon[23158]: pgmap v5406: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:23.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:24.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:24.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:23 smithi016 conmon[32727]: 2022-03-21T20:12:23.676+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:24.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:25.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:24 smithi016 ceph-mon[22010]: pgmap v5407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:25.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:25.165 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:24 smithi016 conmon[32727]: 2022-03-21T20:12:24.668+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:24 smithi079 ceph-mon[23158]: pgmap v5407: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:25.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:26.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:25 smithi016 conmon[32727]: 2022-03-21T20:12:25.678+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:26 smithi016 ceph-mon[22010]: pgmap v5408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:26.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:26.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:26 smithi079 ceph-mon[23158]: pgmap v5408: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10190 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:27.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:26 smithi016 conmon[32727]: 2022-03-21T20:12:26.711+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:28.167 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:27 smithi016 conmon[32727]: 2022-03-21T20:12:27.672+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:28.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:28.645 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:28 smithi016 ceph-mon[22010]: pgmap v5409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:28.646 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:28 smithi079 ceph-mon[23158]: pgmap v5409: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:28.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:28.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:28 smithi016 conmon[32727]: 2022-03-21T20:12:28.644+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:29.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:29.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:29 smithi016 conmon[32727]: 2022-03-21T20:12:29.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:30.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:30 smithi016 ceph-mon[22010]: pgmap v5410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:30.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:30.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:30 smithi079 ceph-mon[23158]: pgmap v5410: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:30.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:30 smithi016 conmon[32727]: 2022-03-21T20:12:30.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:31.617 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:31 smithi016 conmon[32727]: 2022-03-21T20:12:31.616+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:32.619 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:32 smithi016 ceph-mon[22010]: pgmap v5411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:32.619 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:32.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:32 smithi079 ceph-mon[23158]: pgmap v5411: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:32.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:32 smithi016 conmon[32727]: 2022-03-21T20:12:32.618+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:33.591 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:33.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:33 smithi016 conmon[32727]: 2022-03-21T20:12:33.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:34 smithi016 ceph-mon[22010]: pgmap v5412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:34.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:34.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:34 smithi016 conmon[32727]: 2022-03-21T20:12:34.542+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:34.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:34 smithi079 ceph-mon[23158]: pgmap v5412: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:35.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:35 smithi016 conmon[32727]: 2022-03-21T20:12:35.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:35.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:35.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:35.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10195 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:36.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:36 smithi016 ceph-mon[22010]: pgmap v5413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:36.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:36.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:36 smithi016 conmon[32727]: 2022-03-21T20:12:36.515+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:36 smithi079 ceph-mon[23158]: pgmap v5413: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:37.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:37.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:37 smithi016 conmon[32727]: 2022-03-21T20:12:37.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:38.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:38 smithi016 ceph-mon[22010]: pgmap v5414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:38.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:38.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:38 smithi016 conmon[32727]: 2022-03-21T20:12:38.509+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:38 smithi079 ceph-mon[23158]: pgmap v5414: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:38.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:39.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:39.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:39 smithi016 conmon[32727]: 2022-03-21T20:12:39.474+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:39.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:40 smithi016 ceph-mon[22010]: pgmap v5415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:40.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:40.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:40.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:40 smithi016 conmon[32727]: 2022-03-21T20:12:40.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:40 smithi079 ceph-mon[23158]: pgmap v5415: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10205 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:41.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:41.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:41 smithi016 conmon[32727]: 2022-03-21T20:12:41.505+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:42.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:42 smithi016 conmon[32727]: 2022-03-21T20:12:42.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:42 smithi016 ceph-mon[22010]: pgmap v5416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:42.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:42.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:42 smithi079 ceph-mon[23158]: pgmap v5416: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:43.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:43.819 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:43 smithi016 conmon[32727]: 2022-03-21T20:12:43.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:43.820 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:44.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:44 smithi016 ceph-mon[22010]: pgmap v5417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:44.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:44.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:44 smithi016 conmon[32727]: 2022-03-21T20:12:44.522+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:44.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:44 smithi079 ceph-mon[23158]: pgmap v5417: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:45.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:45.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:45 smithi016 conmon[32727]: 2022-03-21T20:12:45.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:45.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:46.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:46 smithi016 ceph-mon[22010]: pgmap v5418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:46.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:46.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:46 smithi016 conmon[32727]: 2022-03-21T20:12:46.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:46 smithi079 ceph-mon[23158]: pgmap v5418: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10210 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:47.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:47.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:47 smithi016 conmon[32727]: 2022-03-21T20:12:47.529+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:47.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:48 smithi016 ceph-mon[22010]: pgmap v5419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:48.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:48.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:48 smithi016 conmon[32727]: 2022-03-21T20:12:48.572+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:48 smithi079 ceph-mon[23158]: pgmap v5419: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:48.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:49.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:49.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:49 smithi016 conmon[32727]: 2022-03-21T20:12:49.586+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:49.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:50 smithi016 ceph-mon[22010]: pgmap v5420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:50.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:50.670 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:50 smithi016 conmon[32727]: 2022-03-21T20:12:50.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:50 smithi079 ceph-mon[23158]: pgmap v5420: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:50.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:51.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:51.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:51 smithi016 conmon[32727]: 2022-03-21T20:12:51.553+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:51.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:52.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:52 smithi016 ceph-mon[22010]: pgmap v5421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:52.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:52.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:52 smithi016 conmon[32727]: 2022-03-21T20:12:52.561+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:52 smithi079 ceph-mon[23158]: pgmap v5421: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:52.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:53.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:53.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:53 smithi016 conmon[32727]: 2022-03-21T20:12:53.574+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:53.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:54 smithi016 ceph-mon[22010]: pgmap v5422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:54.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:54.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:54 smithi016 conmon[32727]: 2022-03-21T20:12:54.543+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:54.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:54 smithi079 ceph-mon[23158]: pgmap v5422: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:54.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:55.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:55.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:55.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:55 smithi016 conmon[32727]: 2022-03-21T20:12:55.557+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:55.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10215 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:12:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:56 smithi016 ceph-mon[22010]: pgmap v5423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:56.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:12:56.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:12:56.416 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:56 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:12:56.472 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:56 smithi079 ceph-mon[23158]: pgmap v5423: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:12:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:12:56.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:12:56.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:56 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:12:56.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:56 smithi016 conmon[32727]: 2022-03-21T20:12:56.564+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:57.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:57.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:57.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:57 smithi016 conmon[32727]: 2022-03-21T20:12:57.579+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:58.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:58 smithi016 conmon[32727]: 2022-03-21T20:12:58.539+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:12:58.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:58 smithi016 ceph-mon[22010]: pgmap v5424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:58.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:58.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:58 smithi079 ceph-mon[23158]: pgmap v5424: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:12:58.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:59.473 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:12:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:12:59.474 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:12:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:12:59.475 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:12:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:12:59.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:12:59.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:12:59.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:12:59.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:12:59.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:12:59.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:12:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' 2022-03-21T20:12:59.557 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: 2022-03-21T20:12:59.212+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:12:59.558 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:12:59.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:12:59.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: yield 2022-03-21T20:12:59.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:12:59.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:12:59.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:12:59.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:12:59.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:12:59.561 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:12:59.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:12:59.562 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:12:59.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:12:59.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:12:59.563 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:12:59.564 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:12:59.564 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:12:59.564 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:12:59.565 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:12:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:12:59 smithi016 conmon[32727]: 2022-03-21T20:12:59.552+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:00.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:00 smithi016 conmon[32727]: 2022-03-21T20:13:00.508+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: pgmap v5425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:00.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:13:00.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:13:00.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:00.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:13:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:13:00.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:13:00.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:00.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:00.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:13:00.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:00.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:00.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:00.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:13:00.675 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: Detected new or changed devices on smithi079 2022-03-21T20:13:00.676 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:00.676 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:00.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: pgmap v5425: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:00.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:13:00.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:00.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:13:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:00.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:00.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:00.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:13:00.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:00.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:13:00.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:00.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:13:00.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:00.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:00.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:00.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:13:00.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:00.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:00.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:00.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:13:00.738 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: Detected new or changed devices on smithi079 2022-03-21T20:13:00.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:00.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10225 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:01.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:01.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:01 smithi016 conmon[32727]: 2022-03-21T20:13:01.540+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:01.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:02.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:02 smithi016 ceph-mon[22010]: pgmap v5426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:02.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[32727]: 2022-03-21T20:13:02.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:02.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 2022-03-21T20:13:02.284+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:02.666 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:13:02.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:02.667 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: yield 2022-03-21T20:13:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:02.668 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:02.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:13:02.669 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:13:02.670 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:13:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:02.671 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:13:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:13:02.672 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:02.673 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 2022-03-21T20:13:02.285+0000 7f92726df700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:13:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:13:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:02.674 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: yield 2022-03-21T20:13:02.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:02.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:02.675 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:13:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:02.676 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:13:02.677 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:13:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:02.678 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:13:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:13:02.679 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:02.680 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 2022-03-21T20:13:02.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:13:02.681 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: 2022-03-21T20:13:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:13:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:13:02.682 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:13:02.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:13:02.683 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:13:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:13:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:13:02.684 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:13:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:13:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:13:02.685 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:13:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:13:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:13:02.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:13:02.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:13:02.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:13:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:13:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:13:02.688 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:13:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:13:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:13:02.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:13:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise 2022-03-21T20:13:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:13:02.690 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:13:02.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:13:02.691 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:13:02.692 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:13:02 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:02 smithi079 ceph-mon[23158]: pgmap v5426: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:03.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:03 smithi016 conmon[32727]: 2022-03-21T20:13:03.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:03 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:03.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:13:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:13:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:03.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:13:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:13:03.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:03.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:13:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:03.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:03 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:13:03.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:13:03.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:03.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:03.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:03.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:13:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:13:03.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:13:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:03.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:03.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:03.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:13:03.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:03.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:13:03.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:13:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:13:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:13:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:13:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:13:03.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:13:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:13:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:13:03.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:13:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:13:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:13:03.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:13:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:13:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:13:03.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:13:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:13:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:13:03.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:13:03.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:13:03.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:13:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:13:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:13:03.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:13:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:13:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:13:03.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:13:03.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:13:03.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:13:03.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:03 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:03.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:13:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:03.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:13:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:03.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:13:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:13:03.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:03.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:13:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:03.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:03 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:13:03.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:13:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:13:03.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:13:03.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:13:03.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:13:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:13:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:13:03.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:13:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:13:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:13:03.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:13:03.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:13:03.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:13:03.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:13:03.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:13:03.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:13:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:13:03.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:13:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:13:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:13:03.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:13:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:13:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:13:03.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:13:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:13:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:13:03.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:13:03.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:13:03.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:13:03.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:13:03.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:13:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:13:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:13:03.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:13:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:13:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:13:03.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:13:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:13:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:13:03.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:13:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:13:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:13:03.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:13:03.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:13:03.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:13:03.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:13:03.748 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:13:03.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:04.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:04 smithi016 ceph-mon[22010]: pgmap v5427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:04.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:04.666 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:04 smithi016 conmon[32727]: 2022-03-21T20:13:04.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:04 smithi079 ceph-mon[23158]: pgmap v5427: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:04.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:05.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:05.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:05 smithi016 conmon[32727]: 2022-03-21T20:13:05.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:05.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:06 smithi016 ceph-mon[22010]: pgmap v5428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:06.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:06 smithi016 conmon[32727]: 2022-03-21T20:13:06.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:06 smithi079 ceph-mon[23158]: pgmap v5428: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10230 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:07 smithi016 conmon[32727]: 2022-03-21T20:13:07.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:08 smithi016 ceph-mon[22010]: pgmap v5429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:08 smithi016 conmon[32727]: 2022-03-21T20:13:08.523+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:08 smithi079 ceph-mon[23158]: pgmap v5429: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:09.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:09 smithi016 conmon[32727]: 2022-03-21T20:13:09.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:10 smithi079 ceph-mon[23158]: pgmap v5430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:10 smithi016 ceph-mon[22010]: pgmap v5430: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:10.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:10.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:10 smithi016 conmon[32727]: 2022-03-21T20:13:10.451+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:11.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:11 smithi016 conmon[32727]: 2022-03-21T20:13:11.486+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:12 smithi079 ceph-mon[23158]: pgmap v5431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:12.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:12 smithi016 ceph-mon[22010]: pgmap v5431: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:12.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:12 smithi016 conmon[32727]: 2022-03-21T20:13:12.480+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:13.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:13.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:13 smithi016 conmon[32727]: 2022-03-21T20:13:13.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:14 smithi079 ceph-mon[23158]: pgmap v5432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:14.728 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:14 smithi016 ceph-mon[22010]: pgmap v5432: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:14.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:14 smithi016 conmon[32727]: 2022-03-21T20:13:14.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:15.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10235 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:15.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:15 smithi016 conmon[32727]: 2022-03-21T20:13:15.423+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:16.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:16 smithi079 ceph-mon[23158]: pgmap v5433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:16 smithi016 ceph-mon[22010]: pgmap v5433: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:16.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:16 smithi016 conmon[32727]: 2022-03-21T20:13:16.420+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:17 smithi016 conmon[32727]: 2022-03-21T20:13:17.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:18.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:18 smithi079 ceph-mon[23158]: pgmap v5434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:18.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:18 smithi016 ceph-mon[22010]: pgmap v5434: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:18.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:18 smithi016 conmon[32727]: 2022-03-21T20:13:18.464+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:19.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:19 smithi016 conmon[32727]: 2022-03-21T20:13:19.475+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:20 smithi079 ceph-mon[23158]: pgmap v5435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:20 smithi016 ceph-mon[22010]: pgmap v5435: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10245 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:20.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:20 smithi016 conmon[32727]: 2022-03-21T20:13:20.524+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:21.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:21 smithi016 conmon[32727]: 2022-03-21T20:13:21.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:22 smithi079 ceph-mon[23158]: pgmap v5436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:22 smithi016 ceph-mon[22010]: pgmap v5436: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:22.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:22 smithi016 conmon[32727]: 2022-03-21T20:13:22.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:23.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:23 smithi016 conmon[32727]: 2022-03-21T20:13:23.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:24 smithi079 ceph-mon[23158]: pgmap v5437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:24 smithi016 ceph-mon[22010]: pgmap v5437: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:24.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:24.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:24 smithi016 conmon[32727]: 2022-03-21T20:13:24.549+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:25.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:25 smithi016 conmon[32727]: 2022-03-21T20:13:25.503+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:26 smithi079 ceph-mon[23158]: pgmap v5438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:26.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:26.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:26 smithi016 conmon[32727]: 2022-03-21T20:13:26.521+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:26 smithi016 ceph-mon[22010]: pgmap v5438: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:26.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10250 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:26.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:27.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:27.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:27 smithi016 conmon[32727]: 2022-03-21T20:13:27.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:28.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:28 smithi016 ceph-mon[22010]: pgmap v5439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:28.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:28.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:28 smithi016 conmon[32727]: 2022-03-21T20:13:28.555+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:28 smithi079 ceph-mon[23158]: pgmap v5439: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:29.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:29.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:29 smithi016 conmon[32727]: 2022-03-21T20:13:29.591+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:30.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:30 smithi016 ceph-mon[22010]: pgmap v5440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:30.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:30.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:30 smithi016 conmon[32727]: 2022-03-21T20:13:30.633+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:30 smithi079 ceph-mon[23158]: pgmap v5440: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:30.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:31.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:31.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:31 smithi016 conmon[32727]: 2022-03-21T20:13:31.600+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:31.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:32 smithi016 ceph-mon[22010]: pgmap v5441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:32.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:32.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:32 smithi016 conmon[32727]: 2022-03-21T20:13:32.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:32 smithi079 ceph-mon[23158]: pgmap v5441: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:32.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:33.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:33 smithi016 conmon[32727]: 2022-03-21T20:13:33.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:34 smithi016 ceph-mon[22010]: pgmap v5442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:34 smithi016 conmon[32727]: 2022-03-21T20:13:34.592+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:34 smithi079 ceph-mon[23158]: pgmap v5442: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:35 smithi016 conmon[32727]: 2022-03-21T20:13:35.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:35.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10255 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:36 smithi016 ceph-mon[22010]: pgmap v5443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:36 smithi016 conmon[32727]: 2022-03-21T20:13:36.528+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:36 smithi079 ceph-mon[23158]: pgmap v5443: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:37 smithi016 conmon[32727]: 2022-03-21T20:13:37.479+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:38.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:38 smithi016 conmon[32727]: 2022-03-21T20:13:38.475+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:38 smithi016 ceph-mon[22010]: pgmap v5444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:38.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:38 smithi079 ceph-mon[23158]: pgmap v5444: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:39.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:39 smithi016 conmon[32727]: 2022-03-21T20:13:39.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:40 smithi016 conmon[32727]: 2022-03-21T20:13:40.553+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:40 smithi016 ceph-mon[22010]: pgmap v5445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:40 smithi079 ceph-mon[23158]: pgmap v5445: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10265 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:41 smithi016 conmon[32727]: 2022-03-21T20:13:41.563+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:42 smithi016 ceph-mon[22010]: pgmap v5446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:42 smithi016 conmon[32727]: 2022-03-21T20:13:42.556+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:42 smithi079 ceph-mon[23158]: pgmap v5446: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:43 smithi016 conmon[32727]: 2022-03-21T20:13:43.519+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:43.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:44 smithi016 ceph-mon[22010]: pgmap v5447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:44 smithi016 conmon[32727]: 2022-03-21T20:13:44.527+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:44.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:44 smithi079 ceph-mon[23158]: pgmap v5447: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:44.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:45.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:45 smithi016 conmon[32727]: 2022-03-21T20:13:45.490+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:45.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:46 smithi016 ceph-mon[22010]: pgmap v5448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:46 smithi016 conmon[32727]: 2022-03-21T20:13:46.487+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:46.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:46 smithi079 ceph-mon[23158]: pgmap v5448: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:46.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10270 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:47 smithi016 conmon[32727]: 2022-03-21T20:13:47.501+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:47.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:48 smithi016 ceph-mon[22010]: pgmap v5449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:48.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:48 smithi016 conmon[32727]: 2022-03-21T20:13:48.537+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:48 smithi079 ceph-mon[23158]: pgmap v5449: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:48.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:49 smithi016 conmon[32727]: 2022-03-21T20:13:49.504+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:49.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:50.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:50 smithi016 ceph-mon[22010]: pgmap v5450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:50.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:50 smithi016 conmon[32727]: 2022-03-21T20:13:50.477+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:50 smithi079 ceph-mon[23158]: pgmap v5450: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:50.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:51.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:51.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:51 smithi016 conmon[32727]: 2022-03-21T20:13:51.432+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:51.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:52 smithi016 ceph-mon[22010]: pgmap v5451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:52.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:52.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:52 smithi016 conmon[32727]: 2022-03-21T20:13:52.434+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:52 smithi079 ceph-mon[23158]: pgmap v5451: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:52.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:53.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:53.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:53 smithi016 conmon[32727]: 2022-03-21T20:13:53.461+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:53.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:54 smithi016 ceph-mon[22010]: pgmap v5452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:54.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:54.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:54 smithi016 conmon[32727]: 2022-03-21T20:13:54.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:54.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:54 smithi079 ceph-mon[23158]: pgmap v5452: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:54.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:55.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:55.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:55 smithi016 conmon[32727]: 2022-03-21T20:13:55.491+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:55.979 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:55.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10275 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:13:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:56 smithi016 ceph-mon[22010]: pgmap v5453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:56 smithi016 conmon[32727]: 2022-03-21T20:13:56.460+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:56 smithi079 ceph-mon[23158]: pgmap v5453: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:13:56.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:57.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:57.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:57 smithi016 conmon[32727]: 2022-03-21T20:13:57.470+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:57.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:58 smithi016 ceph-mon[22010]: pgmap v5454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:58.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:58.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:58 smithi016 conmon[32727]: 2022-03-21T20:13:58.436+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:58.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:58 smithi079 ceph-mon[23158]: pgmap v5454: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:13:58.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:59.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:13:59 smithi016 conmon[32727]: 2022-03-21T20:13:59.427+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:13:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:13:59.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:59 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:13:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:13:59.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:13:59 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:13:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:13:59.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:13:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:59 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:13:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:13:59.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:13:59 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:14:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:00 smithi016 ceph-mon[22010]: pgmap v5455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:00.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:00.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:00.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:00 smithi016 conmon[32727]: 2022-03-21T20:14:00.438+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:00.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:00 smithi079 ceph-mon[23158]: pgmap v5455: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:00.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10285 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:01.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:01.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:01 smithi016 conmon[32727]: 2022-03-21T20:14:01.485+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:01.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:02.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[32727]: 2022-03-21T20:14:02.474+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:02 smithi016 ceph-mon[22010]: pgmap v5456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:02.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:14:02.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:14:02.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:02 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:14:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:02 smithi079 ceph-mon[23158]: pgmap v5456: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:02.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:14:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:14:02.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:02 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:14:03.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: 2022-03-21T20:14:02.892+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:03.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:14:03.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:03.165 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: yield 2022-03-21T20:14:03.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:03.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:03.166 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:03.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:14:03.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:03.167 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:14:03.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:03.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:14:03.168 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:03.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:14:03.169 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:03.170 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:14:03.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:03.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:03.171 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:02 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:03.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:03.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:03 smithi016 conmon[32727]: 2022-03-21T20:14:03.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:03.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:04.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:04 smithi016 conmon[32727]: 2022-03-21T20:14:04.462+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:04 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:04.915 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:14:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:14:04.916 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:04.917 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:14:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:14:04.918 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:14:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:04.919 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:04.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:04.920 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:14:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:04.921 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:04.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:04 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:14:04.922 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:04 smithi016 ceph-mon[22010]: pgmap v5457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:04.923 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:04.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:04 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:04.973 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:14:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:14:04.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:04.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:14:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:14:04.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:14:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:04.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:04.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:04.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:14:04.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:04.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:04 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:14:04.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:04 smithi079 ceph-mon[23158]: pgmap v5457: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:04.981 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:05.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:05 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[32727]: 2022-03-21T20:14:05.457+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:05.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:05 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:06.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 2022-03-21T20:14:05.964+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:06.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:14:06.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:06.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: yield 2022-03-21T20:14:06.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:06.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:06.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:06.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:14:06.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:06.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:14:06.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:06.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:14:06.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:06.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:14:06.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:06.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:14:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:06.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:06.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 2022-03-21T20:14:05.965+0000 7f92736e1700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:14:06.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:14:06.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:06.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: yield 2022-03-21T20:14:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:06.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:14:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:06.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:14:06.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:06.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:14:06.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:06.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:14:06.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:06.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:14:06.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:06.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 2022-03-21T20:14:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:14:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: 2022-03-21T20:14:06.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:14:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:14:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:14:06.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:14:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:14:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:14:06.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:14:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:14:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:14:06.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:14:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:14:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:14:06.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:14:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:14:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:14:06.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:14:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:14:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:14:06.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:14:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:14:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:14:06.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:14:06.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise 2022-03-21T20:14:06.438 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:14:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:14:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:14:06.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:14:06.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:14:05 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:06.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:06 smithi016 conmon[32727]: 2022-03-21T20:14:06.422+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:06 smithi016 ceph-mon[22010]: pgmap v5458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:06.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:06 smithi079 ceph-mon[23158]: pgmap v5458: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:06.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:06.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10290 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:07.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:07 smithi016 conmon[32727]: 2022-03-21T20:14:07.374+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:07.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:07 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:07.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:14:07.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:07.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:07.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:07.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:14:07.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:07.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:14:07.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:07.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:14:07.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:07.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:07.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:07.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:14:07.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:07.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:07.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:07 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:14:07.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:07.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:14:07.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:07.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:07.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:07.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:14:07.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:07.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:14:07.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:07.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:14:07.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:07.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:07.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:07.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:14:07.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:07.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:07.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:14:07.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:14:07.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:14:07.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:14:07.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:14:07.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:14:07.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:14:07.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:14:07.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:14:07.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:14:07.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:14:07.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:14:07.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:14:07.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:14:07.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:14:07.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:14:07.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:14:07.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:14:07.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:14:07.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:14:07.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:14:07.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:14:07.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:14:07.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:14:07.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:14:07.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:14:07.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:14:07.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:14:07.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:14:07.689 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:07 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.973 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:14:07.974 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:07.975 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:14:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:07.976 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:14:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:14:07.977 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:07.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:07.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:07.978 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:14:07.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:07.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:07.979 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.980 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:07 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:14:07.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.980 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:14:07.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:14:07.981 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:14:07.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:14:07.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:14:07.982 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:14:07.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:14:07.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:14:07.983 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:14:07.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:14:07.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:14:07.984 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:14:07.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:14:07.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:14:07.985 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:14:07.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:14:07.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.986 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:14:07.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:14:07.987 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:14:07.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:14:07.988 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:14:07.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:14:07.989 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:14:07.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:14:07.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:14:07.990 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:14:07.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:14:07.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:14:07.991 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:14:07.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:14:07.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:14:07.992 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:14:07.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:14:07.993 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:14:07.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:14:07.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:14:07.994 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:14:07.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:14:07.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:14:07.995 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:14:07.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:14:07.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:14:07.996 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:14:07.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:14:07.997 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:14:07.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:14:07.998 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:14:07.998 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:08.602 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:08 smithi016 conmon[32727]: 2022-03-21T20:14:08.343+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:08 smithi016 ceph-mon[22010]: pgmap v5459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:08.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:08 smithi079 ceph-mon[23158]: pgmap v5459: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:08.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:09 smithi016 conmon[32727]: 2022-03-21T20:14:09.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:09.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:10 smithi016 conmon[32727]: 2022-03-21T20:14:10.395+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:10 smithi016 ceph-mon[22010]: pgmap v5460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:10.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:10 smithi079 ceph-mon[23158]: pgmap v5460: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:10.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:11.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:11.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:11 smithi016 conmon[32727]: 2022-03-21T20:14:11.437+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:11.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:12 smithi016 conmon[32727]: 2022-03-21T20:14:12.415+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:12 smithi016 ceph-mon[22010]: pgmap v5461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:12.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:12 smithi079 ceph-mon[23158]: pgmap v5461: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:12.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:13.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:13 smithi016 conmon[32727]: 2022-03-21T20:14:13.385+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:13.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:14.621 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:14 smithi016 conmon[32727]: 2022-03-21T20:14:14.339+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:14.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:14 smithi016 ceph-mon[22010]: pgmap v5462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:14.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:14 smithi079 ceph-mon[23158]: pgmap v5462: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:14.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:15.626 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:15 smithi016 conmon[32727]: 2022-03-21T20:14:15.333+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:15.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:15.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:15.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:15.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10295 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:16.627 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:16 smithi016 conmon[32727]: 2022-03-21T20:14:16.304+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:16.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:16 smithi016 ceph-mon[22010]: pgmap v5463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:16.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:16 smithi079 ceph-mon[23158]: pgmap v5463: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:16.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:17.632 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:17 smithi016 conmon[32727]: 2022-03-21T20:14:17.268+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:17.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:17.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:18.634 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:18 smithi016 conmon[32727]: 2022-03-21T20:14:18.317+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:18 smithi016 ceph-mon[22010]: pgmap v5464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:18.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:18 smithi079 ceph-mon[23158]: pgmap v5464: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:18.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:19.639 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:19 smithi016 conmon[32727]: 2022-03-21T20:14:19.364+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:19.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:19.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:20.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:20 smithi016 conmon[32727]: 2022-03-21T20:14:20.355+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:20.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:20 smithi016 ceph-mon[22010]: pgmap v5465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:20.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:20.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:20 smithi079 ceph-mon[23158]: pgmap v5465: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:20.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10305 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:21.646 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:21 smithi016 conmon[32727]: 2022-03-21T20:14:21.355+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:21.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:21.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:22.647 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:22 smithi016 conmon[32727]: 2022-03-21T20:14:22.314+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:22 smithi016 ceph-mon[22010]: pgmap v5466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:22.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:22 smithi079 ceph-mon[23158]: pgmap v5466: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:22.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:23.652 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:23 smithi016 conmon[32727]: 2022-03-21T20:14:23.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:23.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:23.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:24.654 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:24 smithi016 conmon[32727]: 2022-03-21T20:14:24.340+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:24 smithi016 ceph-mon[22010]: pgmap v5467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:24.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:24.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:24 smithi079 ceph-mon[23158]: pgmap v5467: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:24.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:25.661 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:25 smithi016 conmon[32727]: 2022-03-21T20:14:25.352+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:25.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:25.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:26.663 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:26 smithi016 conmon[32727]: 2022-03-21T20:14:26.310+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:26.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:26 smithi016 ceph-mon[22010]: pgmap v5468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:26.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:26 smithi079 ceph-mon[23158]: pgmap v5468: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:26.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:26.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10310 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:27 smithi016 conmon[32727]: 2022-03-21T20:14:27.282+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:27.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:28.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:28.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:28 smithi016 conmon[32727]: 2022-03-21T20:14:28.293+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:28.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:28 smithi079 ceph-mon[23158]: pgmap v5469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:28.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:28 smithi016 ceph-mon[22010]: pgmap v5469: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:29.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:29.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:29 smithi016 conmon[32727]: 2022-03-21T20:14:29.279+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:29.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:30.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:30.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:30 smithi016 conmon[32727]: 2022-03-21T20:14:30.302+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:30 smithi079 ceph-mon[23158]: pgmap v5470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:30.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:30 smithi016 ceph-mon[22010]: pgmap v5470: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:31.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:31.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:31 smithi016 conmon[32727]: 2022-03-21T20:14:31.351+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:31.974 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:32.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:32.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:32 smithi016 conmon[32727]: 2022-03-21T20:14:32.394+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:32 smithi079 ceph-mon[23158]: pgmap v5471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:32.975 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:32 smithi016 ceph-mon[22010]: pgmap v5471: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:33.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:33.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:33 smithi016 conmon[32727]: 2022-03-21T20:14:33.411+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:33.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:34.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:34 smithi016 ceph-mon[22010]: pgmap v5472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:34 smithi016 conmon[32727]: 2022-03-21T20:14:34.458+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:34 smithi079 ceph-mon[23158]: pgmap v5472: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:34.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:35 smithi016 conmon[32727]: 2022-03-21T20:14:35.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:35.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10315 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:36 smithi016 ceph-mon[22010]: pgmap v5473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:36 smithi016 conmon[32727]: 2022-03-21T20:14:36.475+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:36 smithi079 ceph-mon[23158]: pgmap v5473: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:36.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:37 smithi016 conmon[32727]: 2022-03-21T20:14:37.484+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:37.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:38 smithi016 ceph-mon[22010]: pgmap v5474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:38 smithi016 conmon[32727]: 2022-03-21T20:14:38.499+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:38 smithi079 ceph-mon[23158]: pgmap v5474: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:38.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:39.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:39.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:39 smithi016 conmon[32727]: 2022-03-21T20:14:39.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:39.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:40.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:40 smithi016 ceph-mon[22010]: pgmap v5475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:40.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:40 smithi016 conmon[32727]: 2022-03-21T20:14:40.565+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:40 smithi079 ceph-mon[23158]: pgmap v5475: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:40.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10325 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:41 smithi016 conmon[32727]: 2022-03-21T20:14:41.599+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:41.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:42 smithi016 ceph-mon[22010]: pgmap v5476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:42.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:42 smithi016 conmon[32727]: 2022-03-21T20:14:42.567+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:42.972 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:42 smithi079 ceph-mon[23158]: pgmap v5476: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:42.973 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:43.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:43 smithi016 conmon[32727]: 2022-03-21T20:14:43.612+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:44.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:44 smithi016 ceph-mon[22010]: pgmap v5477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:44.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:44 smithi016 conmon[32727]: 2022-03-21T20:14:44.566+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:45.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:44 smithi079 ceph-mon[23158]: pgmap v5477: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:45.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:45 smithi016 conmon[32727]: 2022-03-21T20:14:45.535+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:46.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:46 smithi016 ceph-mon[22010]: pgmap v5478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:46.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:46 smithi016 conmon[32727]: 2022-03-21T20:14:46.551+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:46 smithi079 ceph-mon[23158]: pgmap v5478: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:47.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10330 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:47 smithi016 conmon[32727]: 2022-03-21T20:14:47.536+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:48.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:48.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:48 smithi016 ceph-mon[22010]: pgmap v5479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:48.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:48 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:48.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:48 smithi016 conmon[32727]: 2022-03-21T20:14:48.562+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:49.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:48 smithi079 ceph-mon[23158]: pgmap v5479: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:49.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:48 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:49.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:49 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:49.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:49 smithi016 conmon[32727]: 2022-03-21T20:14:49.585+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:50.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:49 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:50.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:50 smithi016 conmon[32727]: 2022-03-21T20:14:50.608+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:50 smithi016 ceph-mon[22010]: pgmap v5480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:50.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:50 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:50 smithi079 ceph-mon[23158]: pgmap v5480: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:51.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:50 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:51.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:51 smithi016 conmon[32727]: 2022-03-21T20:14:51.569+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:51.916 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:51 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:52.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:51 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:52.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:52 smithi016 conmon[32727]: 2022-03-21T20:14:52.550+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:52 smithi016 ceph-mon[22010]: pgmap v5481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:52.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:52 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:52 smithi079 ceph-mon[23158]: pgmap v5481: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:53.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:52 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:53.916 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:53 smithi016 conmon[32727]: 2022-03-21T20:14:53.502+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:53.917 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:53 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:54.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:53 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:54.752 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:54 smithi016 conmon[32727]: 2022-03-21T20:14:54.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:54 smithi016 ceph-mon[22010]: pgmap v5482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:55.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:54 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:54 smithi079 ceph-mon[23158]: pgmap v5482: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:55.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:54 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:55 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:55.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:55 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:55.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:55 smithi016 conmon[32727]: 2022-03-21T20:14:55.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:56.222 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:55 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10335 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:14:56.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:55 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:56.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:56 smithi016 ceph-mon[22010]: pgmap v5483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:56.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:56 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:56.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:56 smithi016 conmon[32727]: 2022-03-21T20:14:56.516+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:56 smithi079 ceph-mon[23158]: pgmap v5483: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:14:57.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:56 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:57.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:57 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:57.917 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:57 smithi016 conmon[32727]: 2022-03-21T20:14:57.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:58.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:57 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:58.766 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:58 smithi016 conmon[32727]: 2022-03-21T20:14:58.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:14:59.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: pgmap v5484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:59.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:14:59.166 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:14:59.167 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:58 smithi016 ceph-mon[22010]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:14:59.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: pgmap v5484: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:59.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:14:59.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:14:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-03-21T20:14:59.226 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:58 smithi079 ceph-mon[23158]: from='mgr.24109 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-03-21T20:14:59.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:14:59 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:14:59.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:14:59 smithi016 conmon[32727]: 2022-03-21T20:14:59.544+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:00.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:14:59 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:00.772 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:00 smithi016 conmon[32727]: 2022-03-21T20:15:00.517+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:01.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:00 smithi016 ceph-mon[22010]: pgmap v5485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:00 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:01.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:00 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:01.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:00 smithi079 ceph-mon[23158]: pgmap v5485: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:00 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10345 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:01.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:00 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:01.777 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:01 smithi016 conmon[32727]: 2022-03-21T20:15:01.472+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:02.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:01 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:02.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:01 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:02.779 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:02 smithi016 conmon[32727]: 2022-03-21T20:15:02.435+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:03.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:02 smithi016 ceph-mon[22010]: pgmap v5486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:03.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:02 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:03.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:02 smithi079 ceph-mon[23158]: pgmap v5486: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:03.225 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:02 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:03.784 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:03 smithi016 conmon[32727]: 2022-03-21T20:15:03.467+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:04.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:03 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:04.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:03 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:04.785 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:04 smithi016 conmon[32727]: 2022-03-21T20:15:04.469+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:05.164 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:04 smithi016 ceph-mon[22010]: pgmap v5487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:05.165 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:04 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:05.223 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:04 smithi079 ceph-mon[23158]: pgmap v5487: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:05.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:04 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:05.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:05 smithi016 conmon[32727]: 2022-03-21T20:15:05.507+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:06.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:06 smithi016 conmon[32727]: 2022-03-21T20:15:06.481+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:06.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: pgmap v5488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:06.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:15:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:15:06.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:06 smithi016 ceph-mon[22010]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:15:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: pgmap v5488: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:06.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10350 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-03-21T20:15:06.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-03-21T20:15:06.725 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:06 smithi079 ceph-mon[23158]: from='mgr.24109 172.21.15.16:0/1065411397' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-03-21T20:15:07.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:07 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:07.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:07 smithi016 conmon[32727]: 2022-03-21T20:15:07.526+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:07.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:07 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:08 smithi016 ceph-mon[22010]: pgmap v5489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:08.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:08 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:08.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:08 smithi016 conmon[32727]: 2022-03-21T20:15:08.511+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:08 smithi079 ceph-mon[23158]: pgmap v5489: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:08.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:08 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:09.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: 2022-03-21T20:15:09.068+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:09.326 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:15:09.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:09.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: yield 2022-03-21T20:15:09.327 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:09.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:09.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:09.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:15:09.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:09.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:15:09.329 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:09.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:15:09.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:09.330 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:15:09.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:09.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:15:09.331 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:09.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:09.332 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:09.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:09 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:09.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:09 smithi016 conmon[32727]: 2022-03-21T20:15:09.558+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:09.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:09 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:10.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:10 smithi016 conmon[32727]: 2022-03-21T20:15:10.571+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:10 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:15:10.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:15:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:10.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:15:10.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:10.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:15:10.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:10.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:15:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:10.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:15:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:10.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:10.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:10.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:10 smithi016 ceph-mon[22010]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:15:10.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:10 smithi016 ceph-mon[22010]: pgmap v5490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:10.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:10 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:10 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:10.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:15:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:15:10.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:10.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:10.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:10.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:15:10.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:10.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:15:10.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:10.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:15:10.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:10.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:10.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:10.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:15:10.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:10.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:10.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:10.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:10 smithi079 ceph-mon[23158]: Removing smithi002:/etc/ceph/ceph.conf 2022-03-21T20:15:10.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:10 smithi079 ceph-mon[23158]: pgmap v5490: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:10.731 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:10 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:11.612 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:11 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:11.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:11 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:11.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:11 smithi016 conmon[32727]: 2022-03-21T20:15:11.611+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:12.414 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:12 smithi016 ceph-mon[22010]: pgmap v5491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:12.415 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:12 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:12.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 2022-03-21T20:15:12.141+0000 7f926bed2700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:12.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:15:12.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:12.416 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: yield 2022-03-21T20:15:12.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:12.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:12.417 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:12.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:15:12.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:12.418 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:15:12.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:12.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:15:12.419 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:12.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:15:12.420 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:12.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:15:12.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:12.421 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:12.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:12.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 2022-03-21T20:15:12.142+0000 7f9270edc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:15:12.422 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:15:12.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:12.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: yield 2022-03-21T20:15:12.423 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:12.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:12.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:12.424 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 'Opening SSH connection to') 2022-03-21T20:15:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: local_addr=local_addr) 2022-03-21T20:15:12.425 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:12.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise exceptions[0] 2022-03-21T20:15:12.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:12.426 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: yield from self.sock_connect(sock, address) 2022-03-21T20:15:12.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:12.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return (yield from fut) 2022-03-21T20:15:12.427 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:12.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:12.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:12.428 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 2022-03-21T20:15:12.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: During handling of the above exception, another exception occurred: 2022-03-21T20:15:12.429 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: 2022-03-21T20:15:12.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: Traceback (most recent call last): 2022-03-21T20:15:12.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:15:12.430 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return f(*arg) 2022-03-21T20:15:12.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:15:12.431 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: self._write_client_files(client_files, host) 2022-03-21T20:15:12.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:15:12.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:15:12.432 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:15:12.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:15:12.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:15:12.433 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return self.event_loop.get_result(coro) 2022-03-21T20:15:12.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:15:12.434 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:15:12.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:15:12.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: return self.__get_result() 2022-03-21T20:15:12.435 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:15:12.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise self._exception 2022-03-21T20:15:12.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:15:12.436 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:15:12.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:15:12.437 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: conn = await self._remote_connection(host, addr) 2022-03-21T20:15:12.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:15:12.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise 2022-03-21T20:15:12.439 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:15:12.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: self.gen.throw(type, value, traceback) 2022-03-21T20:15:12.440 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:15:12.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: raise OrchestratorError(msg) 2022-03-21T20:15:12.441 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[23230]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:12.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:12 smithi079 ceph-mon[23158]: pgmap v5491: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:12.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:12 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:12.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:12 smithi016 conmon[32727]: 2022-03-21T20:15:12.594+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:13.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:13 smithi016 conmon[32727]: 2022-03-21T20:15:13.587+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:13 smithi016 ceph-mon[22010]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.665 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:15:13.666 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:13.667 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:15:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:15:13.668 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:15:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:13.669 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:15:13.670 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:13 smithi016 ceph-mon[22010]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:15:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:13.672 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield 2022-03-21T20:15:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:13.673 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: 'Opening SSH connection to') 2022-03-21T20:15:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:13.674 INFO:journalctl@ceph.mon.b.smithi016.stdout: local_addr=local_addr) 2022-03-21T20:15:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise exceptions[0] 2022-03-21T20:15:13.675 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:13.676 INFO:journalctl@ceph.mon.b.smithi016.stdout: return (yield from fut) 2022-03-21T20:15:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:13.677 INFO:journalctl@ceph.mon.b.smithi016.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:15:13.678 INFO:journalctl@ceph.mon.b.smithi016.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:15:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2022-03-21T20:15:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.679 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:15:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: return f(*arg) 2022-03-21T20:15:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:15:13.680 INFO:journalctl@ceph.mon.b.smithi016.stdout: self._write_client_files(client_files, host) 2022-03-21T20:15:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:15:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:15:13.681 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:15:13.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:15:13.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:15:13.682 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:15:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:15:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:15:13.683 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:15:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: return self.__get_result() 2022-03-21T20:15:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:15:13.684 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise self._exception 2022-03-21T20:15:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:15:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:15:13.685 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:15:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:15:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:15:13.686 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise 2022-03-21T20:15:13.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:15:13.687 INFO:journalctl@ceph.mon.b.smithi016.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:15:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:15:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: raise OrchestratorError(msg) 2022-03-21T20:15:13.688 INFO:journalctl@ceph.mon.b.smithi016.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.689 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:13 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:13 smithi079 ceph-mon[23158]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.723 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:13.724 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:15:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:13.725 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:13.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:15:13.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:13.726 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:15:13.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:13.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:15:13.727 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:13.728 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:15:13.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:13.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:13.729 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:13 smithi079 ceph-mon[23158]: executing refresh((['smithi002', 'smithi016', 'smithi079'],)) failed. 2022-03-21T20:15:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.730 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 99, in redirect_log 2022-03-21T20:15:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield 2022-03-21T20:15:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 78, in _remote_connection 2022-03-21T20:15:13.731 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], known_hosts=None, config=[self.mgr.ssh_config_fname], preferred_auth=['publickey']) 2022-03-21T20:15:13.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-03-21T20:15:13.732 INFO:journalctl@ceph.mon.c.smithi079.stdout: 'Opening SSH connection to') 2022-03-21T20:15:13.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-03-21T20:15:13.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: local_addr=local_addr) 2022-03-21T20:15:13.733 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-03-21T20:15:13.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise exceptions[0] 2022-03-21T20:15:13.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-03-21T20:15:13.734 INFO:journalctl@ceph.mon.c.smithi079.stdout: yield from self.sock_connect(sock, address) 2022-03-21T20:15:13.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-03-21T20:15:13.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: return (yield from fut) 2022-03-21T20:15:13.735 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-03-21T20:15:13.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-03-21T20:15:13.736 INFO:journalctl@ceph.mon.c.smithi079.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:15:13.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: During handling of the above exception, another exception occurred: 2022-03-21T20:15:13.737 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2022-03-21T20:15:13.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2022-03-21T20:15:13.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work 2022-03-21T20:15:13.738 INFO:journalctl@ceph.mon.c.smithi079.stdout: return f(*arg) 2022-03-21T20:15:13.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-03-21T20:15:13.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._write_client_files(client_files, host) 2022-03-21T20:15:13.739 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1058, in _write_client_files 2022-03-21T20:15:13.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-03-21T20:15:13.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 190, in check_execute_command 2022-03-21T20:15:13.740 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-03-21T20:15:13.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 583, in wait_async 2022-03-21T20:15:13.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro) 2022-03-21T20:15:13.741 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-03-21T20:15:13.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-03-21T20:15:13.742 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-03-21T20:15:13.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2022-03-21T20:15:13.743 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-03-21T20:15:13.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2022-03-21T20:15:13.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 177, in _check_execute_command 2022-03-21T20:15:13.744 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-03-21T20:15:13.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 134, in _execute_command 2022-03-21T20:15:13.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: conn = await self._remote_connection(host, addr) 2022-03-21T20:15:13.745 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 84, in _remote_connection 2022-03-21T20:15:13.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise 2022-03-21T20:15:13.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-03-21T20:15:13.746 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.gen.throw(type, value, traceback) 2022-03-21T20:15:13.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 105, in redirect_log 2022-03-21T20:15:13.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError(msg) 2022-03-21T20:15:13.747 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-03-21T20:15:13.748 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:13 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:14.629 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:14 smithi016 ceph-mon[22010]: pgmap v5492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:14.630 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:14 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:14 smithi079 ceph-mon[23158]: pgmap v5492: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:14.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:14 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:14.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:14 smithi016 conmon[32727]: 2022-03-21T20:15:14.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:15.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:15 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:15.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:15 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:15 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:15.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:15 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10355 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:15.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:15 smithi016 conmon[32727]: 2022-03-21T20:15:15.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:16.661 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:16 smithi016 ceph-mon[22010]: pgmap v5493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:16.662 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:16 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:16 smithi079 ceph-mon[23158]: pgmap v5493: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:16.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:16 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:16.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:16 smithi016 conmon[32727]: 2022-03-21T20:15:16.660+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:17.651 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:17 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:17.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:17 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:17.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:17 smithi016 conmon[32727]: 2022-03-21T20:15:17.650+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:18 smithi016 ceph-mon[22010]: pgmap v5494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:18.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:18 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:18 smithi079 ceph-mon[23158]: pgmap v5494: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:18.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:18 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:19.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:18 smithi016 conmon[32727]: 2022-03-21T20:15:18.688+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:19.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:19 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:19.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:19 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:20.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:19 smithi016 conmon[32727]: 2022-03-21T20:15:19.722+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:20 smithi016 ceph-mon[22010]: pgmap v5495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:20.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:20 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:20.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:20 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:20 smithi079 ceph-mon[23158]: pgmap v5495: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:20.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:20 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:20.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:20 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10365 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:21.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:20 smithi016 conmon[32727]: 2022-03-21T20:15:20.726+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:21.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:21 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:21.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:21 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:22.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:21 smithi016 conmon[32727]: 2022-03-21T20:15:21.700+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:22.652 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:22 smithi016 ceph-mon[22010]: pgmap v5496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:22.653 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:22 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:22 smithi079 ceph-mon[23158]: pgmap v5496: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:22.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:22 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:22.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:22 smithi016 conmon[32727]: 2022-03-21T20:15:22.652+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:23.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:23 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:23.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:23 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:24.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:23 smithi016 conmon[32727]: 2022-03-21T20:15:23.673+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:24.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:24 smithi016 ceph-mon[22010]: pgmap v5497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:24.644 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:24 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:24 smithi079 ceph-mon[23158]: pgmap v5497: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:24.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:24 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:24.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:24 smithi016 conmon[32727]: 2022-03-21T20:15:24.642+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:25.655 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:25 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:25.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:25 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:25.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:25 smithi016 conmon[32727]: 2022-03-21T20:15:25.654+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:26.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:26 smithi016 ceph-mon[22010]: pgmap v5498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:26.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:26 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:26.648 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:26 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:26 smithi079 ceph-mon[23158]: pgmap v5498: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:26 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10370 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:26.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:26 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:26.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:26 smithi016 conmon[32727]: 2022-03-21T20:15:26.647+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:27.664 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:27 smithi016 conmon[32727]: 2022-03-21T20:15:27.639+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:27.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:27 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:27.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:27 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:28.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:28 smithi016 ceph-mon[22010]: pgmap v5499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:28.665 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:28 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:28 smithi079 ceph-mon[23158]: pgmap v5499: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:28.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:28 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:29.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:28 smithi016 conmon[32727]: 2022-03-21T20:15:28.681+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:29.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:29 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:29.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:29 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:30.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:29 smithi016 conmon[32727]: 2022-03-21T20:15:29.667+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:30 smithi016 ceph-mon[22010]: pgmap v5500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:30.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:30 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:30 smithi079 ceph-mon[23158]: pgmap v5500: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:30.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:30 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:31.164 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:30 smithi016 conmon[32727]: 2022-03-21T20:15:30.671+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:31.664 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:31 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:31.665 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:31 smithi016 conmon[32727]: 2022-03-21T20:15:31.649+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:31.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:31 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:32.666 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:32 smithi016 ceph-mon[22010]: pgmap v5501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:32.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:32 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:32.667 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:32 smithi016 conmon[32727]: 2022-03-21T20:15:32.627+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:32 smithi079 ceph-mon[23158]: pgmap v5501: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:32.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:32 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:33.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:33 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:33.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:33 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:33.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:33 smithi016 conmon[32727]: 2022-03-21T20:15:33.603+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:34 smithi079 ceph-mon[23158]: pgmap v5502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:34.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:34 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:34 smithi016 ceph-mon[22010]: pgmap v5502: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:34.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:34 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:34.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:34 smithi016 conmon[32727]: 2022-03-21T20:15:34.605+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:35 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:35.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:35 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:35.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:35 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:35.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:35 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10375 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:35.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:35 smithi016 conmon[32727]: 2022-03-21T20:15:35.559+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:36.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:36 smithi079 ceph-mon[23158]: pgmap v5503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:36.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:36 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:36.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:36 smithi016 ceph-mon[22010]: pgmap v5503: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:36.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:36 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:36.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:36 smithi016 conmon[32727]: 2022-03-21T20:15:36.606+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:37.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:37 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:37.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:37 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:37.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:37 smithi016 conmon[32727]: 2022-03-21T20:15:37.560+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:38.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:38 smithi079 ceph-mon[23158]: pgmap v5504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:38.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:38 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:38.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:38 smithi016 ceph-mon[22010]: pgmap v5504: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:38.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:38 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:38.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:38 smithi016 conmon[32727]: 2022-03-21T20:15:38.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:39.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:39 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:39.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:39 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:39.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:39 smithi016 conmon[32727]: 2022-03-21T20:15:39.495+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:40.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:40 smithi079 ceph-mon[23158]: pgmap v5505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:40 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:40.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:40 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:40.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:40 smithi016 conmon[32727]: 2022-03-21T20:15:40.498+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:40 smithi016 ceph-mon[22010]: pgmap v5505: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:40 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:40.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:40 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10385 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:41.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:41 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:41.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:41 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:41.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:41 smithi016 conmon[32727]: 2022-03-21T20:15:41.473+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:42 smithi079 ceph-mon[23158]: pgmap v5506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:42.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:42 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:42 smithi016 ceph-mon[22010]: pgmap v5506: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:42.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:42 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:42.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:42 smithi016 conmon[32727]: 2022-03-21T20:15:42.489+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:43.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:43 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:43.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:43 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:43.914 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:43 smithi016 conmon[32727]: 2022-03-21T20:15:43.512+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:44 smithi079 ceph-mon[23158]: pgmap v5507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:44.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:44 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:44 smithi016 ceph-mon[22010]: pgmap v5507: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-03-21T20:15:44.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:44 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:44.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:44 smithi016 conmon[32727]: 2022-03-21T20:15:44.531+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:45.722 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:45 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:45.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:45 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:45.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:45 smithi016 conmon[32727]: 2022-03-21T20:15:45.554+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:46 smithi079 ceph-mon[23158]: pgmap v5508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:46.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:46 smithi079 ceph-mon[23158]: Health check update: 9 slow ops, oldest one blocked for 10390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:46.724 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:46 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:46 smithi016 ceph-mon[22010]: pgmap v5508: 97 pgs: 95 active+clean, 2 unknown; 133 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-03-21T20:15:46.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:46 smithi016 ceph-mon[22010]: Health check update: 9 slow ops, oldest one blocked for 10390 sec, osd.4 has slow ops (SLOW_OPS) 2022-03-21T20:15:46.915 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:46 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:46.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:46 smithi016 conmon[32727]: 2022-03-21T20:15:46.525+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:47.723 INFO:journalctl@ceph.mon.c.smithi079.stdout:Mar 21 20:15:47 smithi079 ceph-mon[23158]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:47.914 INFO:journalctl@ceph.mon.b.smithi016.stdout:Mar 21 20:15:47 smithi016 ceph-mon[22010]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_data' : 9 ]) 2022-03-21T20:15:47.915 INFO:journalctl@ceph.osd.4.smithi016.stdout:Mar 21 20:15:47 smithi016 conmon[32727]: 2022-03-21T20:15:47.534+0000 7ff70c0ba700 -1 osd.4 113 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.14:3219 3.6 3.22e25d46 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) 2022-03-21T20:15:48.657 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-21T20:15:48.659 DEBUG:teuthology.task.console_log:Killing console logger for smithi002 2022-03-21T20:15:48.660 DEBUG:teuthology.task.console_log:Killing console logger for smithi016 2022-03-21T20:15:48.661 DEBUG:teuthology.task.console_log:Killing console logger for smithi079 2022-03-21T20:15:48.661 DEBUG:teuthology.exit:Finished running handlers